# Towards Benchmarking Explainable Artificial Intelligence Methods

Lars Holmberg<EMAIL_ADDRESS>
Department of Computer Science and Media Technology  Malmö University  Sweden

###### Abstract

The currently dominating artificial intelligence and machine learning
technology, neural networks, builds on inductive statistical learning. Neural
networks of today are information processing systems void of understanding and
reasoning capabilities, consequently, they cannot explain promoted decisions
in a humanly valid form. In this work, we revisit and use fundamental
philosophy of science theories as an analytical lens with the goal of
revealing, what can be expected, and more importantly, not expected, from
methods that aim to explain decisions promoted by a neural network. By
conducting a case study we investigate a selection of explainability method’s
performance over two mundane domains, animals and headgear. Through our study,
we lay bare that the usefulness of these methods relies on human domain
knowledge and our ability to understand, generalise and reason. The
explainability methods can be useful when the goal is to gain further insights
into a trained neural network’s strengths and weaknesses. If our aim instead
is to use these explainability methods to promote actionable decisions or
build trust in ML-models they need to be less ambiguous than they are today.
In this work, we conclude from our study, that benchmarking explainability
methods, is a central quest towards trustworthy artificial intelligence and
machine learning.

###### keywords:

Trustworthy Machine Learning; Explainable AI; Concept

##  1 Introduction

Digitalisation influences all parts of society. In its heart resides
Artificial Intelligence (AI) and Machine Learning (ML), technologies with
roots in natural science and a third-person objectivising stance Grimm (2016).
Research in the area then inherits values that are concerned with ground
truth, optimising class probability, minimising bias in training data and
mitigating consequences of data drift. This approach, useful when the target
domain is static and well-defined, is not well suited to promote
decisions111The output from an ML system in the form of classification,
recommendation, prediction, proposed decisions or action in a real-world
context Chollet (2019). These ML/AI systems are information processing systems
that due to their complexity becomes black boxes Lipton (2016) that promote
decisions without presenting reasons in a human-understandable form. Answering
a how-question, by associating inputs to an ML-system with a promoted
decision, compared to, answering a why-question, actionable in the real world,
are two very different challenges. In later years negative consequences
related to this information processing approach, void of reasoning and
understanding, has become increasingly apparent Hutchinson and Mitchell
(2019); Couldry and Mejias (2019).

The gap between reality and the fragment of reality modelled by an ML/AI
System can be expressed in terms of explanandum overlap. There is then a
significant explanandum overlap in medical image analyse, where the problem is
hermeneutic and concerned with interpreting, analysing and naming concepts in
images produced by an instrument that visualises concepts unreachable for
human senses. Areas with limited explanandum overlap, on the other hand, are,
for example, image search on the internet for abstract concepts like freedom,
sexual orientation, gender or ethnicity. The explanandum overlap is then
limited with respect to: how these socially constructed concepts are perceived
by humans, and how and in what way they are visually represented in an image
search. Explanations based on an image search can then only communicate very
limited visual queues to these concepts. Datasets, used for training and
benchmarking, are, in addition to this, contextually and culturally situated
and consequently come with inherited and embedded value systems. These
enormous and inconceivable datasets then risk being normative and part of
conserving and obscuring values if they are used downstream in ML-models
deployed in a real-world setting Yang et al. (2020); Gebru et al. (2021).

Figure 1: Why is this image classified as a “Tiger”?
* Photo by Ratanjot Singh on Unsplash

The work presented here then focus on this gap, between the fragment of
reality modelled by the ML/AI system and the real world in which a promoted
decision can be useful and actionable. We adhere to a Human In Command
approach (HIC) Holmberg (2021) with the goal of positioning the Machine In The
Loop (MITL) and thus presenting an alternative to a technology trend that
position Humans In The Loop (HITL) as a resource that risks being treated
instrumentally Couldry and Mejias (2019). Moving the agency in this direction
opens up for a type of ML/AI where humans have agency and can value certain
ends and goals as good and prosperous for humanity Vallor (2016).

This paper focus on inductive statistical learning approaches used in the
currently dominating ML technology, neural networks. This is a technology that
can learn from raw input data in the form of images, sound or text Lecun et
al. (2015). We selected this ML/AI approach for our analyse since knowledge
priors embedded in the system originate primarily from the selection of
training data and labels which, make our analyse simpler. We focus on
classification tasks for images picturing mundane visual objects as input data
and we select those images so the explanandum available to the human and the
ML-system overlap. We thereby construct a setting in which a human with domain
knowledge (henceforth domain experts) can assess reasons for a decision
presented by the ML system.

Our focus is to research if available eXplainable Artificial Intelligence
(XAI) methods are useful when the goal is to understand a trained ML-model’s
decisions. By selecting a mundane domain we build on a tradition of
benchmarks, but, instead of focusing on average-case performance for a static
dataset (ImageNet1000) Deng et al. (2009), our focus is on the usefulness of
XAI-methods given a pretrained ML-model (ResNet) He et al. (2016) and
carefully selected images. We, therefore, formulate the following research
question:

  * •

Are local XAI-methods capable of communicating reasons for a proposed decision
in a human understandable form?

To address this we study five local XAI-methods that utilise saliency maps to
accentuate parts of the image as important for a singular promoted decision-
[See Figure 1]). These explanation methods cannot give a comprehensive summary
of all reasons for a decision, instead, they aim to present evidence from the
part of the explanandum reachable for the specific XAI-method in question in a
human-understandable form.

In the study, we find that the XAI-methods delivers vague and imprecise
evidence and this leaves interpretation and judging the evidence presented to
human abilities. Notwithstanding this, we believe that the tested XAI-methods,
especially when used in an ensemble, can deliver valuable insights on
strengths and weaknesses related to the ML-model’s capabilities. We also find
that the XAI-methods has to be used strategically since they also have their
different strengths and weaknesses and therefore deliver complementing
insights in relation to the ML-model’s Internal Knowledge Representation
(IKR).

In the background section that follows, we present theories related to
scientific explanations, inductive statistical learning and concepts. This is
followed by a section on the methodological approach leading to our study
results. The article ends with a discussion section that contextualises our
conclusions. We then end the article by concluding the results.

##  2 Background

Human understanding of natural phenomena builds on the uniformity principle
which implies that instances of which we have no experience must resemble
those we had experience from Hume (1896). The uniformity principle cannot be
justified from a scientific perspective and we, therefore, need to define non-
inductive reasons to classify an explanation as scientific. To be
scientifically valid we need a theory that allows for deductive reasoning that
consequently can be used to predict future events without relying on
induction. This is then important for machine learning, based on induction,
since without the ability to falsify the system and scientifically explain
decisions there is no demarcation between these systems and pseudoscience.

In this work we use a form of scientific explanations, Deductive-Nomological
(DN) Hempel and Oppenheim (1948), as our analytical lens to better understand
what can be expected, and not expected, from XAI-methods used in conjunction
with neural networks. DN-explanations are used throughout science as a
blueprint for a scientific explanations and can then serve us when we discuss
what we can expect, epistemologically, from explanations that reflect the
inductive statistical learning process used when neural networks are trained.

A full DN-explanation uses a theory to create a model of the phenomena in
question, this model then describes kinds and entities that populate the model
Overton (2012) (We will in the following work denote this type of model, DN-
model, to distinguish it from an ML-model). Measurements in the form of data
are then, if the DN-model describes the phenomena well, aligned to predictions
produced by the DN-model. A scientific explanation can answer a why-question,
DN nomenclature for this is that a why-question establish an explanandum from
which explanans, or evidence for a decision, can be selected to form a
scientific explanation that answers the why-question. At least one of these
evidences needs to be a law deduced from the DN-model Woodward and Ross
(2021).

In algorithmic ML, a theory is used to select an algorithm that can encompass
the phenomena in question and consequently the DN-model and the ML-model
overlaps. A neural network is, instead of using a preselected algorithm,
limited by the universal approximation theorem Hornik (1991) which imply that
the resulting ML-model used for prediction is formed during training based on
the training data and labels. Consequently, there exists no model in a DN
sense, no model that builds on a theory that can be falsified. From this
follows that the neural network is incapable of producing a scientifically
valid explanation since they build on a theory and deductive reasoning.

In this work, we denote all human knowledge added to the ML-system as
knowledge priors. This can be choice of domain, algorithm selection, data
augmentation, feature engineering, training data selection, etc. For a neural
network knowledge priors are mainly added based on domain selection and input
data selection. Neural networks then build internal knowledge representations
from raw data (images, sound or text) and labels, the function used for
prediction is then created in an inductive statistical manner by exposing the
neural network to a large amount of training data Lecun et al. (2015).

The often assumed prerequisite for neural network training is that data and
labels are independent and identically distributed (i.i.d), a presumption that
is very challenging to fulfil in a real-world setting Schölkopf (2019). Data
not part of the intended target domain is in our work denoted out of
distribution data (o.o.d.). A consequence of the inductive learning process,
and the nonexisting DN-model, is that o.o.d. data cannot be identified as
external by a neural network. This since the network, cannot, without human
help, identify the domain borders. It is then, in a classification problem,
not possible to identify new classes, instead, class probability for a
promoted decision reflect some aspect of similarity with existing classes.

Concepts are central to human reasoning and essential for our ability to
generalise. We use and need them to explain and make predictions about new
objects and situations Margolis and Laurence (2021); Murphy (2004). In this
work, we aim to use XAI-methods and map the IKRs (Internal Knowledge
Representations) in a trained neural network to Human Understandable Concepts
(HUC). This to some extent maps to prototype theories and exemplar theories of
concepts Murphy (2016). Prototype theories are a summary description of
concepts (our HUC) and exemplar theories are remembered instance categories
(our IKR). Humans’ beliefs, related to concepts and their properties, can be
both false and incomplete and, additionally, contain both causal and
descriptive factors Genone and Lombrozo (2012). Concepts are then named using
a referent, for example, “pferd” or “horse” (double quotation marks), these
are two referents that points to the same concept, ‘horse’ (single quotation
marks).

An approach used to search for a precise definition of concepts is to specify
them as necessary and/or sufficient Brennan (2017). This approach is used in
machine learning both in experimental research Wang et al. (2020) and to
underpin representation learning Wang and Jordan (2021). To exemplify, the
concept of ‘elephant’ can be described using necessary sub-concepts shared
with many animals, for example, ‘four legs’, ‘eyes’ and sufficient sub-
concepts, for example, ‘elephant tusk’. Sufficient and necessary sub-concepts
are those that on their own can be used in classification, for example,
‘elephant trunk’ (‘elephant tusks’ are sufficient for classification but not
necessary since not all female Indian elephants have tusks). Spurious
correlations are relationships between the proposed decision that are prone to
change when a system is deployed in a real-world context Gulrajani and Lopez-
Paz (2020). For example, if all elephants are pictured close to ‘watering
holes’ this concept can wrongly be seen by the ML-model as a necessary
concept. In one part of our study, we investigate the usefulness of spurious
correlations and sufficient and necessary concepts since human insights here
can help to identify limitations in the training data in relation to the
deployment domain. In this work we use the notion of referents to concepts
synonymous with labels, promoted decisions and classifications.

In recent years there has been a surge in XAI-methods Gilpin et al. (2019);
Samek et al. (2021) but there are few user studies evaluating these methods in
a real-world context Tjoa and Guan (2019). In the last years, implementations
of many of the XAI-methods in accessible packages have been produced, which
simplifies user studies. One advantage is that the XAI-methods are implemented
by a group of developers and therefore are, from an implementation and code
quality perspective, comparable. In this work, we have selected Captum
(PyTorch) as a competitive implementation Kokhlikyan et al. (2020). It has
then become easier to use XAI-methods, but still, it is not clear if the XAI-
methods are mostly developed to give insights into the trained ML-models or if
they also are useful for our target users: domain experts.

For our study we have selected both XAI-methods that are ML-model independent,
like Occlusion Zeiler and Fergus (2014) and those that focus on IKR in layers
closer to the promoted decision, two Gradient-weighted Class Activation
Mapping (GradCAM) Selvaraju et al. (2017) methods and a SHapley Additive
exPlanations (SHAP) Lundberg and Lee (2017) method, as well as one method,
Integrated Gradients (IG) Sundararajan et al. (2017), that weighs activations
over all layers in a neural network. XAI-methods, in their original papers,
are often presented with best-case performance examples and their usefulness
can even be questioned Adebayo et al. (2018). We are then interested in
evaluating them using pictures from mundane domains and consequently get an
idea of their more general usefulness. The underlying reasoning is that the
XAI-methods should perform well in a domain that pictures well-known objects
if they are to be trusted in more demanding domains.

Figure 2: Examples of the diverse visual languages used in XAI-methods. The
original image and predictions are placed in the middle, flanked by different
XAI-methods.

##  3 Methodology

We set up a small but targeted case study with the goal of exploring some
alternative approaches that ML/AI systems can use to communicate reasons for
decisions with humans Seawnght and Gerring (2008). One of the two studies
investigated if the notion of sufficient concepts, necessary concepts and
spurious correlations can be useful as an analytical tool when the goal is to
categorise areas in images accentuated by the XAI-methods. In the other study,
we investigated if we can rely on a human intuitive understanding of the XAI-
methods explainability capabilities. In both cases, we selected images
picturing non-abstract concepts clearly visible in the images (animals and
headgear). The studies mixed harder to identify objects with easier and a few
images that pictured more than one object. We also selected images picturing
objects predicted with class probability around 50% that. The study
participants were not told which classes, within the domain, the system could
recognise, they only got information related to the domain they could expect,
animals and headgear. By this approach, we aimed towards a baseline study
where the results of the two studies give basic indications on the usefulness
of the XAI-methods in a mundane domain.

Since our goal is benchmarking XAI-methods we used well-known datasets and
pre-trained models, ImageNet1000 together with a pre-trained ResNet50 model
ResNet50, a model that has a reasonably low error rate (less than 5% in the
ILSVRC Deng et al. (2009) challenge). With this setup, together with a
coherent XAI-API, we aimed towards a setting with a clear focus on the
usability of the XAI-methods. Around this ML architecture, we created a
website containing the two studies. Each study consisted of web pages the
participant could navigate back and forth between. On each page one image
pictured an object and a prediction together with a possibility to choose an
XAI-method to investigate. A form for answers was placed under the images so
the participants could select the methods they found most useful, this was
accompanied by free text explanation fields. After the study, the participants
were asked to summarise their understanding of the ResNet50-models overall
capabilities related to the domain in question.

The XAI-methods selected, in their original implementations, use a diverse and
to some extent incompatible graphical language to accentuate areas important
for a decision in the image. For example, GradCAM images are often visually
appealing compared to gradient-based methods (See Figure 2). Differences lay
partly in colour schemes used and if the visual explanations are overlaid on
the original image. GradCAM uses, in the original implementation, colour
gradients from blue to red to indicate areas that increasingly influence the
prediction. Other methods use red colour to indicate areas that influence the
attribution negatively Wang et al. (2020). In the XAI-method Integrated
Gradient, internal gradients for a predicted class are projected on the input
image and coloured brownish. These diverse visual explanation languages are
then not comparable and each of them needs to be explained to be understood.
We discussed, in our research team, what a negative attribution implies and
came to the conclusion that it is not intuitive and its usefulness can be
questioned. A negative attribution will also be differently interpreted if the
promoted decision is perceived as erroneous. For example, what do the concept
‘non-cowboy hat’ imply for a erroneous prediction of a “top hat” as a “cowboy
hat”. So, instead of adding to the complexity more than needed, we decided to
focus on identifying sub-concepts that can be associated with the promoted
decision, whether it is perceived as correct or incorrect by the domain
expert.

Based on the reasoning above we decided to use a coherent graphical language
for the XAI-methods and thereby increase comparability between the methods in
a baseline test. We decided on a design with two images, the original image
with predictions and another image with the selected XAI-method overlaid on
the original image in black and white and slightly opaque. We then used a
bright green colour that contrasts to the greyish background and we let
opacity of the green colour indicate how important a part of an image is for
the promoted decision (See Figure 1 and Figure 3).

By keeping many parameters constant we aimed towards a controlled experiment
Ko et al. (2013) and semi-structured interviews Myers and Newman (2007).
Interviews and text were for the animal study broken down into codes related
to sufficient and necessary concepts and mentioning of spurious correlations.
In the study, four students from an Ethics and IT bachelor level course
participated together with four bachelor exam-students, with a focus on XAI,
and two researchers with a background in ML/AI. The age span was 20-40 years
and 40% of the participants were women. Except collecting the form data from
the website we interviewed six of the participants in half-hour sessions. From
the forms, we could collect some quantitative data on relative measurement of
the XAI-methods perceived usefulness.

##  4 Result

One of our two studies investigated whether it is useful, for a human, to
categorise visible sub-concepts in images picturing animals as necessary,
sufficient and, additionally, the usefulness of the notion of spurious
correlations concerning concepts not deemed as necessary or sufficient.
Concepts here were then sub-concepts to animal classifications, as, for
example, ‘watering hole’, ‘beak’ or ‘feather’. For the majority of the users,
this approach was not seen as a useful path to evaluate the XAI-methods. Only
one participant was cautiously positive, but generally, the usage of this
categorisation of concepts was seen as puzzling or even uncomfortable since
they are subjective and open for discussion.

The methods Layer GradCAM and Guided GradCAM were mentioned as the ones that
resemble a human approach to the same task closest and thus were deemed as
most useful. For example, a somewhat non-sharp image picturing a type of
lizard, ‘komodo dragon’, was by the ML-model erroneously classified as a type
of snake with 24% class probability (See central image in Figure 2). In this
case, the XAI-methods were useful since they drew attention to the form of the
tail as a possible reason. Another example: since ‘horse’ is not one of the
classes in ImageNet1000 it is not possible to classify for the ML-model. This
was disclosed by two users that used the XAI-methods to point towards reasons
why the ‘horse’ was erroneously classified as the dog bread “great dane” and
thus drew the conclusion that ‘horse’ must be an o.o.d class.

The study participants judged that the ML-model predicted correctly for 60% of
the images. In total 64 images were assessed by the participants. It was only
in one case a participant used the possibility to refrain from selecting any
method. The perceived usefulness for the XAI-methods for the animal study is
presented in Table 1. The fact that the study participants didn’t know which
classes the system was trained on led to some confusion. For example,
classifying a pond with three ducks as “drake” is from a human perspective a
questionable focus. The use of necessary concepts, sufficient concepts and
spurious correlations can probably be useful but our results indicate that the
study participants need a theoretical base to make use of these notions. A
peculiarity, when the methods are compared, is that when classifications are
judged as erroneous Guided GradCAM is perceived as more useful (42%) than
Layer GradCAM (27%). The relation is switched for decisions judged as correct,
where Layer GradCAM (45%) is perceived as more useful than Guided GradCAM
(31%).

Table 1: The participant’s subjective and relative assessment of the usefulness of XAI-methods. TP (True Positives) denotes usefulness of XAI methods for predictions perceived as correct. FP (False Positives) denotes the usefulness of methods for predictions perceived as erroneous (The participant could choose multiple XAI methods for each proposed prediction). | Animal study | Headgear study
---|---|---
XAI-method | All | TP | FP | All | TP | FP
Occlusion | 19% | 18% | 19% | 29% | 29% | 31%
Guided GradCAM | 35% | 31% | 42% | 27% | 27% | 27%
Integrated Gradients | 7% | 5% | 12% | 5% | 5% | 6%
Layer GradCAM | 38% | 45% | 27% | 36% | 38% | 32%
Gradient SHAP | n/a | n/a | n/a | 3% | 2% | 4%

The other part of the study focused on headgear and the six classes in
ImageNet1000: ‘sombrero’, ‘cowboy hat’, ‘bathing cap’, ‘crash helmet’,
‘bonnet’, ‘shower cap’. We used the same XAI-methods as in the animal study
but added Gradient SHAP, the reason being that SHAP methods are referenced in
many XAI-papers and, based on experiences from the animal study, adding one
more method would not be confusing.

The study participants judged that the ML-model predicted correctly in a
little bit more than half of the cases (55%). We found only a slight
difference between the perceived usefulness of XAI-methods for the
classifications that were judged as correct and those judged as incorrect. The
perceived usefulness for the XAI-methods for the headgear study is presented
in Table 1. In total 115 images were assessed by the participants. The
participant could select zero to five methods they found useful. All
participants selected at least one XAI-method for each image. Examples of the
perceived usefulness are, for example, related to an image picturing a
‘stormtrooper helmet’ from the Star Wars movies that were classified as a
“crash helmet”. This was for some users understood as correct since it
resembles a ‘crash helmet’ by the areas accentuated especially by Guided
GradCAM. The similarity in functionality was discussed by one user in that a
‘stormtrooper helmet’ most probable has a protective function but that
particular participant also concluded that the superclass ‘helmet’ would fit
better. In this test, the participant became aware of missing classes, for
example, that a ‘top hat’ was classified as a “cowboy hat” (51% class
probability), opened up for these types of conclusions. Other more general
opinions were that the ML-model seemed to be good at fabric for example ‘wool’
but also biased towards water-related headgear (‘bathing cap’, ‘bonnet’,
‘shower cap’) especially when these were worn by people (spurious
correlation).

Figure 3: Part of the headgear study, visualising areas in images accentuated
as important for a promoted decision. The original image and predictions are
placed in the middle, flanked by different XAI-methods.

###  4.1 Concluding remarks on the studies

It is worth emphasising that the XAI-methods usefulness heavily depends on the
human ability to compare the evidence exposed by the XAI-methods with the
reality as perceived subjectively by humans. In this setting there is not much
to learn about the reality for humans the possible usefulness of the XAI-
methods resides in a deepened understanding of the ML-model’s inner workings.
These insights can then be used to improve the ML-model or delimit its usage
domain. It is also worth noting that we only measure perceived usefulness in
relation to the other XAI-methods in the study and if all methods were
perceived as equally useful, all would end up at 25% in the animal study and
20% in the headgear study. The only conclusion we can draw is that, for these
mundane images, some methods are perceived as better than others. A deepened
study on the mapping between IKR and HUC is needed to evaluate the methods and
thus be able to generalise conclusions on their respective usefulness. The
lack of a taxonomy that relates concepts to each other also becomes evident in
the study, for example classifying an image of a pond with ducks as a “drake”
without information on that ‘drake’ is a sub-concept to ‘duck’. Here an
integration with WordNet Miller (1998) or a similar service could add further
insights for a human.

##  5 Discussion

In this work, we view neural networks as an information processing system that
through training builds internal knowledge representations (IKR) inductively
based on labels and data in the form of images. As discussed in the background
section the trained ML-model is consequently not based on any theory or law in
a DN-sense and can therefore not be expected to underpin explanations that
hold scientifically. We will in the following discuss what this implies when
the intended purpose of XAI-methods is to communicate reasons for a decision
in a human understandable fashion.

We are aware that ImageNet1000 and the ResNet50 model we used are created as a
benchmark, primarily focusing on class probability, but we still believe that
XAI-methods used on ResNet50 can open up for a discussion on central issues
for XAI in general. In the following discussion, we do not include abstract
concepts and abstract labels, instead, the focus is on concepts in the form of
physical objects clearly visible in the images. The reasoning behind this is
that we believe that a baseline for trust in XAI-methods builds on that the
methods can, at a minimum, be used to explain promoted decisions for these
types of objects.

It is from our study hard to draw wide-ranging conclusions, except that the
two GradCAM methods correlate better with human understanding than the other
XAI-methods in the study. The XAI-method Occlusion can be seen as a simpler
method since it doesn’t focus on IKR directly instead it mechanically
measures, somewhat simplified, how important squares in a grid covering image
is for a promoted decision. The other methods try, using different approaches,
to couple the internal knowledge representations in the ML-model to areas in
the image and consequently the proposed decision. As we set up the two GradCAM
methods they focus on IKR in the later layers and can thus be useful to
understand if high-level sub-concepts (‘ear’, ‘horn’) are learned and used
when a decision is promoted. The gradient methods that weighs activation, and
consequently IKR, over the layers, were in our study deemed as less useful
than the other XAI-methods.

To conclude our results, if XAI-methods are used to build trust in the ML-
model they need to be used in a structured fashion. Initially, there is a need
for a domain expert that can judge the correctness of promoted decisions
concerning the intended usage domain in a traditional manner for some test set
by using class probability, confusion matrices, etc. Given this starting
point, the XAI-methods can be used to deepen the understanding on reasons for
decisions. The XAI-methods can be used differently if the promoted decision is
perceived as correct or erroneous and consequently the explanation formulated
can be associative or contrastive (See the example with ‘stormtrooper helmet’
in Section 1). Another example is the low class probability related to the
miss-classification of the ‘komodo dragon’ as a type of snake “Sidewinder”
these evidence can together with the focus on the tail by Guided GradCAM be
weighed together by a human in order to draw conclusions around the ML-model’s
abilities.

Making designerly decisions on both the visual appearance and deciding on
parameters for the methods, as we did, subjectively influence the results. By
focusing on clearly visible mundane objects, a static well-known ML-model and
a demographically narrow group of participants we believe that it is possible
to draw some conclusions from the study. One is that methods like Occlusion
and GradCAM methods are less challenging to understand than the gradient
methods. These methods then fit to give an initial oversight over the model’s
capabilities.

Below follows a categorisation of the methods in relation to what type of
understanding they can communicate in relation to the ML-model’s abilities.

  1. 1.

XAI-methods like Occlusion and LIME (builds on initial image segmentation)
seems to be the best initial choice to verify that the ML-model focus on the
intended areas in the image in relation to the label that is perceived as
true. If the ML-model’s focus is misaligned with human focus the methods can
be used to identify the need for more training data for some classes or to add
a label that was previously o.o.d.

  2. 2.

Methods like GradCAM that extracts high-level knowledge representations from
the later layers are then useful to identify sub-concepts that should be
aligned with human concepts. We believe that these concepts have to be
organised and categorised, for example by, using necessary, sufficient
concepts and spurious correlations.

  3. 3.

We did not find the different gradient methods useful, more research is then
needed to better understand in what way and if they accentuate useful
information.

If the prediction is perceived as correct and the accentuated areas are
aligned with the concept important for an explanation, trust will likely
increase. For predictions perceived as incorrect areas focused by the XAI-
method can be used to draw conclusions either that the object is o.o.d or that
the areas accentuated can be used to understand why the image is incorrectly
classified. Although this can be important to mitigate bias or find missing
classes it is unlikely that by using XAI-methods these ML-models can be used
in any ethically challenging setting if the explanandum overlap is not
substantial. We then believe that XAI-methods can be useful to further inform
on reasons for a decision in domains with a substantial explanandum overlap,
for example, images picturing objects not reachable for our senses. This can
be medical imaging, microscope images, satellite photos, areas where the
challenge is hermeneutic and concerns identifying and naming concepts in
images. In these cases, the trained ML-model can be seen more as a scientific
instrument capable of drawing attention to image details and correlations
previously overlooked.

The explanandum overlap in a domain that pictures abstract concepts is limited
since the explanandum reachable for the ML-model will be substantially less
rich than the explanandum a human has access to. An ML system can still be
useful when the goal is to, for example, sort images based on visible concepts
like hats, dogs or cats. But as soon as the concepts are abstract and
pictures, for example, humans, the system risks being descriptive in coupling
physiognomy to concepts. If designers of these systems try to de-bias training
data they add their own cultural grounded values and the model becomes
normative. Both the descriptive and the normative effects bear risks since the
systems can be used globally and the effects on groups that do not share the,
by these systems inherited, values are hard to foresee and imagine. These
systems then bear similarities with social constructions frozen in time and
space and can perhaps be more useful as probes, to compare and understand
societal changes in the past compared to the present, than predicting the
future.

Even in a limited study like this, the lack of theory and model in a DN-sense
related to the knowledge representation in the ML-model becomes evident. For
example that three ‘mallard ducks’ (one drake and two ducks) in a pond were
classified as a “drake”. The XAI-methods can then draw attention to sub-
concepts related to the classification, for example, ‘coloured head’, ‘tail
form’, etc. What evidently is lacking concerns the purpose of the system, why
is the ‘drake’ singled out? To be able to explain a decision there is an
obvious need for a contextual grounding and a clear domain delimitation
answering why and how questions for the ML system as such. Without clarity
concerning this, explanations can only be used to improve the ML-model and not
to propose actionable decisions. The capability to name the referent “drake”
and connect the term to visual concepts severely limits the usability of the
systems outside of a biased searching and sorting of existing images. These
systems, void of reasoning and understanding, needs at a minimum to connect
existing research-grounded taxonomies to their proposed XAI-methods.

The approach we used leaves most of the interpretation of the evidence and, of
course, the formulation of explanations to the human using the system.
Consequently, selecting explanans to use for an explanation becomes subjective
guesswork. For the system to be contestable the XAI-methods needs to be more
expressive and not only mark an area as impactful for the decision but also go
behind and close in on a why-question related to an impactful sub-concept. We
see both the need for local XAI-methods, like those used in this study, and
global to be used as an ensemble to understand the strengths and weaknesses of
the ML-model. Research in this direction is increasingly attracting attention,
by either, forcing the ML-model to learn predefined concepts useful for
explanations, or, to extract vectors that bear global information related to
the internal knowledge representations Akula et al. (2020); Koh et al. (2020);
Mincu et al. (2021); Kim et al. (2018).

Local XAI-methods are then useful when the goal is to falsify an ML-model and
global methods when the goal is to get an overview of the intended
capabilities concerning the deployment domain. We then need to move towards a
situation where we do not mainly optimise on class probability for static
datasets and additionally view data drift, bias and inductive learning
processes as a solvable technical problem, instead, the main goal must be to
view contextual adaptation, continuous learning and a capability to, in a
structured fashion, expose human-understandable evidence for a decision. Using
an ensemble of XAI-methods can then open up a path to better understand the
limitations of the ML-system and thus, perhaps, avoid some of the misuse and
over-reliance resulting from a one-sided focus on the ML-systems abilities.

Finally, we believe that it is time to realise that the emperor is only partly
dressed and revisit the scepticism related to pseudoscience one hundred years
ago, actualise Hume’s problem of induction and apply scientific reasoning on
ML/AI Hempel and Oppenheim (1948); Henderson (2020). A clear-eyed view on the
state of research in ML/AI is then needed, and, a shift in focus, towards what
it cannot do and open up for anthropocentric development with humans in
command of a technology that can enhance and/or augment human capabilities.

##  6 Conclusion

In this paper, we analyse, using our theoretical lens and a case study,
consequences of the inductive statistical learning process utilised when a
neural networks is trained. Based on this, we identify and discuss the need to
complement these systems, if we expect them to be able to create explanations
that can be understood. In our study, we especially focus on local XAI-methods
and show that reasons for a decision have to be interpreted by a human with
domain knowledge in order to be explainable. The XAI-methods used in the study
produces vague and incoherent reasons for a decision, reasons that
additionally are open for different interpretations. Still, if a domain expert
knows the XAI-methods’ strengths and weaknesses, they can be useful.

The usefulness can be increased if XAI-methods can expose reasons for a
promoted decision in a human-understandable fashion. Human understandable
concepts are then a natural connection point between knowledge internalised in
the ML/AI-system and a human aiming to understand a promoted decision. We then
strongly believe that benchmarking XAI-methods, using a mundane domain, is a
central step in this direction.

Finally, if we are to trust these systems, that profoundly can affect people
and societies, we need methods that can be used to challenge promoted
decisions. If we cannot develop systems that can explain why an object in an
image is an ‘elephant’ and not a ‘zebra’ it seems strange to trust them when
the stakes are higher. We believe that a shift towards developing ML and AI
systems, with an awareness of these systems’ epistemological shortcomings, is
a point of departure for human in command ML/AI-systems that aims at
augmenting and/or enhancing human capabilities.

This work was partially financed by the Knowledge Foundation through the
Internet of Things and People research profile. I am also in debt to
colleagues at my department at Malmö University for their invaluable and
insightful comments that substantially improved the work.

## References

  * Adebayo et al. (2018) Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  In _Advances in Neural Information Processing Systems_ , volume 2018-Decem, pages 9505–9515, 2018.  URL https://goo.gl/hBmhDt.
  * Akula et al. (2020) Arjun Akula, Shuai Wang, and Song-Chun Zhu.  CoCoX: Generating Conceptual and Counterfactual Explanations via Fault-Lines.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(03):2594–2601, 2020.  ISSN 2159-5399.  10.1609/aaai.v34i03.5643.
  * Brennan (2017) Andrew Brennan.  Necessary and Sufficient Conditions.  In Edward N Zalta, editor, _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, summer 2017 edition, 2017.
  * Chollet (2019) François Chollet.  On the Measure of Intelligence.  _arXiv preprint arXiv:1911.01547_ , page 64, 2019.  URL http://arxiv.org/abs/1911.01547.
  * Couldry and Mejias (2019) Nick Couldry and Ulises A. Mejias.  Data Colonialism: Rethinking Big Data’s Relation to the Contemporary Subject.  _Television & New Media_, 20(4):336–349, 2019\.  10.1177/1527476418796632.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _IEEE conference on computer vision and pattern recognition_ , pages 248–255, Miami, 2009. IEEE.  10.1109/cvpr.2009.5206848.
  * Gebru et al. (2021) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford.  Datasheets for datasets.  In _Communications of the ACM_ , volume 64, pages 86–92, 2021.  10.1145/3458723.
  * Genone and Lombrozo (2012) James Genone and Tania Lombrozo.  Concept possession, experimental semantics, and hybrid theories of reference.  _Philosophical Psychology_ , 25(5):717–742, 2012\.  ISSN 09515089.  10.1080/09515089.2011.627538.
  * Gilpin et al. (2019) Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal.  Explaining explanations: An overview of interpretability of machine learning.  In _Proceedings - 2018 IEEE 5th International Conference on Data Science and Advanced Analytics, DSAA 2018_ , pages 80–89. IEEE, 2019.  ISBN 9781538650905.  10.1109/DSAA.2018.00018.
  * Grimm (2016) Stephen R. Grimm.  How Understanding People Differs from Understanding the Natural World.  _Nous-Supplement: Philosophical Issues_ , 26(1):209–225, 2016.  ISSN 17582237.  10.1111/phis.12068.
  * Gulrajani and Lopez-Paz (2020) Ishaan Gulrajani and David Lopez-Paz.  In Search of Lost Domain Generalization.  2020\.  URL http://arxiv.org/abs/2007.01434.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , volume 2016-Decem, pages 770–778, Las Vegas, 2016. IEEE.  ISBN 9781467388504.  10.1109/CVPR.2016.90.
  * Hempel and Oppenheim (1948) Carl G. Hempel and Paul Oppenheim.  Studies in the Logic of Explanation.  _Philosophy of Science_ , 15(2):135–175, 1948\.  ISSN 0031-8248.  10.1086/286983.
  * Henderson (2020) Leah Henderson.  The Problem of Induction.  https://plato.stanford.edu/archives/spr2020/entries/induction-problem/, 2020.
  * Holmberg (2021) Lars Holmberg.  _Human In Command Machine Learning_.  Number 16 in Studies in Computer Science. 2021.  ISBN 978-91-7877-187-5.  10.24834/isbn.9789178771875.
  * Hornik (1991) Kurt Hornik.  Approximation capabilities of multilayer feedforward networks.  _Neural Networks_ , 4(2):251–257, 1991.  ISSN 08936080.  10.1016/0893-6080(91)90009-T.
  * Hume (1896) David Hume.  _A treatise of human nature_.  Clarendon Press, 1896.
  * Hutchinson and Mitchell (2019) Ben Hutchinson and Margaret Mitchell.  50 Years of Test (Un)fairness: Lessons for machine learning.  In _FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency_ , pages 49–58. Association for Computing Machinery, Inc, 1 2019.  ISBN 9781450361255.  10.1145/3287560.3287600.
  * Kim et al. (2018) Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres.  Interpretability beyond feature attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  In _35th International Conference on Machine Learning, ICML 2018_ , 2018.  ISBN 9781510867963.
  * Ko et al. (2013) Andrew J. Ko, Thomas D. LaToza, and Margaret M. Burnett.  A practical guide to controlled experiments of software engineering tools with human participants.  _Empirical Software Engineering_ , 20(1):110–141, 2013.  ISSN 15737616.  10.1007/s10664-013-9279-3.
  * Koh et al. (2020) Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang.  Concept Bottleneck Models.  In _International Conference on Machine Learning_ , 2020.  URL https://arxiv.org/abs/2007.04612.
  * Kokhlikyan et al. (2020) Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson.  Captum: A unified and generic model interpretability library for PyTorch.  2020\.  URL http://arxiv.org/abs/2009.07896.
  * Lecun et al. (2015) Yann Lecun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _Nature_ , 521(7553):436–444, 2015.  ISSN 14764687.  10.1038/nature14539.
  * Lipton (2016) Zachary C. Lipton.  The Mythos of Model Interpretability.  _Communications of the ACM_ , 61(10):35–43, 2016\.  ISSN 15577317.  10.1145/3233231.
  * Lundberg and Lee (2017) Scott M. Lundberg and Su In Lee.  A unified approach to interpreting model predictions.  In _Advances in Neural Information Processing Systems_ , number 3, pages 4766–4775. MIT press, 2017.  URL https://github.com/slundberg/shap.
  * Margolis and Laurence (2021) Eric Margolis and Stephen Laurence.  Concepts.  In _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, 2021.
  * Miller (1998) George A Miller.  _WordNet: An electronic lexical database_.  MIT press, 1998.
  * Mincu et al. (2021) Diana Mincu, Eric Loreaux, Shaobo Hou, Sebastien Baur, Ivan Protsyuk, Martin Seneviratne, Anne Mottram, Nenad Tomasev, Alan Karthikesalingam, and Jessica Schrouff.  Concept-based model explanations for electronic health records.  In _ACM CHIL 2021 - Proceedings of the 2021 ACM Conference on Health, Inference, and Learning_ , pages 36–46, 2021.  10.1145/3450439.3451858.
  * Murphy (2004) Gregory Murphy.  _The Big Book of Concepts_.  MIT press, 2004.  10.7551/mitpress/1602.001.0001.
  * Murphy (2016) Gregory L Murphy.  Is there an exemplar theory of concepts?  _Psychonomic Bulletin and Review_ , 23(4):1035–1042, 2016.  ISSN 15315320.  10.3758/s13423-015-0834-3.
  * Myers and Newman (2007) Michael D. Myers and Michael Newman.  The qualitative interview in IS research: Examining the craft.  _Information and Organization_ , 17(1):2–26, 2007\.  ISSN 14717727.  10.1016/j.infoandorg.2006.11.001.
  * Overton (2012) James A Overton.  _Explanation in Science_.  PhD thesis, University of Western Ontario, 2012.  URL https://ir.lib.uwo.ca/etd/594.
  * Samek et al. (2021) Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, and Klaus Robert Müller.  Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications.  _Proceedings of the IEEE_ , 109(3):247–278, 3 2021.  ISSN 15582256.  10.1109/JPROC.2021.3060483.
  * Schölkopf (2019) Bernhard Schölkopf.  Causality for Machine Learning.  2019\.  URL http://arxiv.org/abs/1911.10500.
  * Seawnght and Gerring (2008) Jason Seawnght and John Gerring.  Case selection techniques in case study research: A menu of qualitative and quantitative options.  _Political Research Quarterly_ , 61(2):294–308, 2008.  ISSN 10659129.  10.1177/1065912907313077.
  * Selvaraju et al. (2017) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.  _International Journal of Computer Vision_ , 128(2):336–359, 2017.  10.1007/s11263-019-01228-7.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _34th International Conference on Machine Learning, ICML 2017_ , volume 7, pages 5109–5118, 2017.  ISBN 9781510855144.
  * Tjoa and Guan (2019) Erico Tjoa and Cuntai Guan.  A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI.  In _IEEE Transactions on Neural Networks and Learning Systems_. 2019\.  URL https://arxiv.org/abs/1907.07374.
  * Vallor (2016) Shannon Vallor.  _Technology and the virtues: A philosophical guide to a future worth wanting_.  Oxford University Press, 2016.
  * Wang and Jordan (2021) Yixin Wang and Michael I Jordan.  Desiderata for Representation Learning: A Causal Perspective.  2021\.
  * Wang et al. (2020) Zifan Wang, Piotr Mardziel, Anupam Datta, and Matt Fredrikson.  Interpreting interpretations: Organizing attribution methods by criteria.  In _IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_ , volume 2020-June, pages 48–55, 2020.  ISBN 9781728193601.  10.1109/CVPRW50498.2020.00013.
  * Woodward and Ross (2021) James Woodward and Lauren Ross.  Scientific Explanation.  In Edward N. Zalta, editor, _The Stanford Encyclopedia of Philosophy (Summer 2021 Edition)_. 2021.  URL plato.stanford.edu/archives/sum2021/entries/scientific-explanation/.
  * Yang et al. (2020) Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky.  Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the ImageNet hierarchy.  In _FAT* 2020 - Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 547–558, 2020.  ISBN 9781450369367.  10.1145/3351095.3375709.
  * Zeiler and Fergus (2014) Matthew D Zeiler and Rob Fergus.  Visualizing and Understanding Convolutional Networks.  Technical report, 2014.


# Distilling Event Sequence Knowledge From Large Language Models

Somin Wadhwa1,2 Oktie Hassanzadeh1 Debarun Bhattacharjya1 Ken Barker1&Jian Ni1
1IBM Research AI
2Northeastern University
<EMAIL_ADDRESS>{hassanzadeh, debarunb, kjbarker<EMAIL_ADDRESS>
###### Abstract

Event sequence models have been found to be highly effective in the analysis
and prediction of events. Building such models requires availability of
abundant high-quality event sequence data. In certain applications, however,
clean structured event sequences are not available, and automated sequence
extraction results in data that is too noisy and incomplete. In this work, we
explore the use of Large Language Models (LLMs) to generate event sequences
that can effectively be used for probabilistic event model construction. This
can be viewed as a mechanism of distilling event sequence knowledge from LLMs.
Our approach relies on a Knowledge Graph (KG) of event concepts with partial
causal relations to guide the generative language model for causal event
sequence generation. We show that our approach can generate high-quality event
sequences, filling a knowledge gap in the input KG. Furthermore, we explore
how the generated sequences can be leveraged to discover useful and more
complex structured knowledge from pattern mining and probabilistic event
models. We release our sequence generation code and evaluation framework, as
well as corpus of event sequence data.

##  1 Introduction

Building probabilistic models from event sequence data has numerous practical
applications across different domains when plentiful high-quality event data
is available. For example, in Finance, event models can be used to predict
stock market trends and make informed investment decisions. In Healthcare,
event models can help identify patterns and correlations in patient data to
improve diagnoses and treatment plans. In the field of Cybersecurity, these
models can be used to detect and prevent potential cyber attacks by analyzing
the sequence of events leading up to a breach. A common characteristic of the
data in these domains is that sequences are clearly associated with an entity
(e.g., a company, a person, or a device). There are however other domains
where such a clean association between events and entities may not be
possible. One such application is news event analysis Radinsky et al. (2012);
Cekinel and Karagoz (2022); Du et al. (2022); Hassanzadeh et al. (2022). While
various news sources record and describe newsworthy events, it is often not
possible to automatically put together coherent sequences of events, because
each event may involve multiple topics and actors, and many correlated and
unrelated events may be occurring simultaneously or in close proximity

Prior work has addressed this challenge by devising automated mechanisms for
extracting narratives Norambuena et al. (2023); Santana et al. (2023), topic
detection and tracking Allan (2012), and timeline summarization Gholipour
Ghalandari and Ifrim (2020). While these different categories of solutions
have been successful in a range of applications, the outcome is inherently
noisy and not in the form of structured event sequences useful for the
construction of event models. Prior work has shown little success in
automatically turning narratives into structured sequences. In this paper, we
explore a novel mechanism for creating structured event sequences in such
domains, using generative language models.

Figure 1: Examples of frequent patterns mined from LLM-generated event
sequence collections with their support (top); and some results from a simple
probabilistic event model learned over the sequences that discovers
influencing sets of event concepts and estimates probabilities based on
historical occurrences (bottom).

Large Language Models Brown et al. (2020); Raffel et al. (2020); Wei et al.
(2022) have recently become the dominant paradigm in a range of natural
language processing (NLP) tasks Chung et al. (2022) and often beat traditional
approaches on a number of challenging tasks, including complex arithmetic
reasoning Imani et al. (2023) and open-domain question answering Kamalloo et
al. (2023). In this paper, our goal is to examine the capability of LLMs to
generate structured event sequences useful for event analysis. Our aim is to
use LLMs directly for generation of event sequences, as opposed to improving
prior extraction methods that would rely on high-quality and comprehensive
sources describing detailed event sequences for a wide variety of events. Our
hypothesis is that LLMs trained on large corpora have already gathered the
required knowledge of plausible event sequences and therefore can be suitably
guided to produce diverse and high-quality event sequences. To effectively
distill this knowledge, we use event-related concepts in Wikidata Vrandečić
and Krötzsch (2014), a comprehensive general-domain knowledge graph, to guide
the sequence generation. This can be viewed as a novel mechanism for
knowledge-guided text generation Yu et al. (2022). We then use these generated
sequences for pattern mining and learning probabilistic event models, as a way
of structuring the underlying knowledge. Figure 1 shows examples from our
experiments of patterns mined from an LLM-generated event sequence collection,
as a well as a simple model learned from the collection.

In summary, we make the following contributions:

  1. 1.

We devise a new iterative in-context prompting strategy for generating high-
quality event sequences using LLMs. To the best of our knowledge, we are the
first to use LLMs to generate structured event sequences for the purpose of
analyzing various event models.

  2. 2.

We compile high-quality event sequences using our generation mechanism, based
on a curated set of high-level event concepts (classes) from Wikidata that
represent newsworthy events.

  3. 3.

We develop an evaluation framework and conduct experiments to show the value
of LLM-powered event sequence generation on replicating and augmenting
knowledge in structured representations such as knowledge graphs.

  4. 4.

We further demonstrate the practical usefulness of our approach by leveraging
downstream pattern mining and probabilistic event models.

Upon publication, we will make our code and generated sequence data publicly
available (as a separate dataset) for future research.

##  2 Methods

We split the task of distilling event sequence knowledge from LLMs into two
sub-tasks: (i) generation of new and unique event sequences given an event
trigger type; and (ii) mining high-support sequence patterns not already
derivable from the knowledge graph, and learning probabilistic models to
discover complex event sequence rules. Figure 2 describes our overall
framework for carrying out these two objectives. We frame these two subtasks
as instances of casual event prediction and event sequence mining,
respectively.

Figure 2:  An overview of our event sequence generation framework and an
example portraying a potential use case. We show that by (1) starting with a
sparse knowledge graph such as Wikidata, we can generate targeted event
sequences. Owing to the inherent sparsity in the underlying KG, we can (2) use
LLMs to carry out a portion of the evaluation (i.e. precision) to select an
optimal model. On this new generated sequence dataset, we then (3) apply
classical pattern-mining algorithms (i.e. GSP) to identify potentially
interesting has$\\_$cause and has$\\_$effect event sequence chains, and (4)
learn summary Markov models (SuMMs) to identify potential influencing events
(i.e. parent event types) for particular event types of interest; these are
both illustrations of extracting complex structured knowledge from the
generated sequences.

In real-world knowledge graphs like Wikidata, such events are represented by a
text-based identifier. For example, each event type in Figure 2 corresponds to
a high-level newsworthy concept in Wikidata. More importantly, these event
concepts are not associated with any relevant time-stamps but often do have
causal relations (e.g. has_cause or has_effect) – either directly (earthquake
causes tsunami) or through their instances (an earthquake event caused a
tsunami event).

###  2.1 LLMs as Event Sequence Generators

We first analyze the ability of LLMs to model event prediction as a
conditional generation task under zero and iterative few-shot settings. Our
targets are linearized sequences of event concepts. We begin by prompting a
large language model (with in-context exemplars) with an event trigger $y_{1}$
to generate the next concept from a defined set of labels, and repeat this
process until we get a sequence of desired length. Formally, given an event
trigger $\mathcal{T}$, we model the probability of generating linearized
string $y$ of length $T$ containing $N$ unique event concepts that follow
$\mathcal{T}$ in sequence:

$p_{\text{LM}}(y|\mathcal{T})=\prod_{t=2}^{T}p(y_{t}|\mathcal{T},y_{<t-1})$

This is the standard conditional language modeling objective. We try multiple
prompting techniques and qualitatively observe optimal results with an
iterative in-context few-shot prompting strategy (Figure 3). Specifically, we
start with a set of six randomly selected examples of the form – “What usually
follows event $X$?”. Based on the model output ($Y$) from a pre-specified
vocabulary, we append this same example to the original prompt in conjunction
(i.e. “What usually follows event $X$ and $Y$?”) with ICL examples of the same
form. As shown in Figure 3, our iterative technique serves dual purposes: (i)
it leverages in-context learning; and (ii) eliminates the need for
implementing complex resolvers to post-process model outputs. We repeat this
process until a sequence of a desired minimum length $m$ is achieved (in our
experiments, $m=3$) or we’ve exhausted a maximum number of tries ($k=10$ in
our experiments) to generate an in-domain event type.

Figure 3: Illustration of our approach to elicit has$\\_$effect and
has$\\_$cause events given a label of interest. Use of instructional in-
context exemplars substantially reduces the need for post-processing LLM
output in addition to constraining the output label space.

###  2.2 Event Concepts

We curate a new dataset of high-level event concepts (classes) from Wikidata
that represent newsworthy events. To do so, we query Wikidata for event
concepts that have links to Wikinews articles and are instances of classes
that are a subclass of the occurrence class, i.e., indicating they are
newsworthy event classes. We gathered $50$ top-level classes for these event
concepts, each having multiple labels (e.g. conflict $\rightarrow$ conflict
(psychological), dispute, disagreement, etc.). This yielded a total of $202$
unique event labels for the $50$ top-level classes. Most of these event
concepts have some causal relations (i.e. has_cause or has_effect). We use
these relation pairs as in-context exemplars to create our prompts. Full
length prompts used in all our experiments are provided in Appendix
LABEL:sec_apx:event_seq_prompts.

Algorithm 1 Generalized Sequential Pattern Algorithm

Obtain sequences $(<x>)$ as $len-1$ candidates

$F_{1}=$ set of $len-1$ sequential patterns

$k=1$

while $F_{k}\neq\emptyset$ do

Generate $C_{k+1}$ $\triangleright$ $len-(k+1)$ candidates from $F_{k}$

if $C_{k+1}\neq\emptyset$ then

Scan all sequences

Identify $F_{k+1}$ $\triangleright$ $len-(k+1)$ sequence patterns

end if

$k=k+1$

end while

###  2.3 Mining New Patterns

In order to derive new, unseen relationships (has_cause or has_effect) between
the extracted event classes, we use a large pretrained language model to
generate new event sequences and then use classical frequent itemset mining
algorithms like GSP Srikant and Agrawal (1996) and SPADE Zaki (2004) to derive
new high support patterns. This aspect of the overall workflow is illustrated
in the top right section of Figure 2, where rules are mined from event
sequences.

To generate new event sequences in a zero-shot setting, we start with an
event-trigger (e.g. a concept like workplace accident), create a prompt with
instructions describing desired relationships and a few in-context exemplars
(ICL prompt), and constrained the model output to the original $50$ event
concept labels to generate the next event in the sequence. (The full text of
the prompt is provided in the supplementary material.) We append the model
output to another ICL prompt with conjunctive event examples (i.e. questions
of the form “What happens after X, Y, and Z”). We repeat this process until we
reach a pre-defined maximum sequence length (in this case, $10$) or until the
model fails to generate an in-vocabulary response in $k$ maximum attempts (in
this case, $k=3$). We test several ablations varying prompt length, example
types, and example selection. We provide results and other details from these
ablations in the supplementary material (appendix). We repeat this sequence
generation procedure on all $202$ event concepts, generating $2,276$ event
sequences with an average length of $5.7$ labels per sequence.

The classical pattern mining algorithms (i.e. GSP and SPADE) we use to mine
for new event patterns are both a priori-based approaches (Algorithm 1). Under
both methods, given two sequences of the same event concept class

$\alpha=<a_{1},a_{2},..,a_{n}>\text{ and }\beta=<b_{1},b_{2},..,b_{n}>$

$\alpha$ is a subsequence of $\beta$ denoted as $\alpha\subseteq\beta$ iff
there exists a set of values $1\leq j_{1}<j_{2}<...<j_{n}\leq m$ such that
$a_{1}\subseteq b_{j1},a_{2}\subseteq b_{j2},...,a_{n}\subseteq b_{jn},$; then
$\beta$ is a supersequence of $\alpha$. Given multiple such sequences and a
support threshold, the task here is to find a set of frequent event
subsequences.

###  2.4 Identifying Influencing Sets through Summary Markov Models

Learning about potential influencing events that lead to a given event type in
a large set of event sequences is an important aspect of analyzing
multivariate event sequences, i.e. events without time stamps, like the ones
generated by our models. Classic $k_{th}$ order Markov chains capture these
dynamics by modeling the probability of observing a particular event type
given the preceding $k$ events in-sequence. The recent family of summary
Markov models (SuMMs) Bhattacharjya et al. (2022) generalize other well known
Markov models for event sequences by leveraging a function that summarizes
historical event occurrences, and identifying the subset of event types that
affect the probability of occurrence of event types of interest; this forms
the influencing set.

We use the LLM-generated sequences to learn two types of SuMMs: binary SuMMs
(BSuMMs) and ordinal SuMMs (OSuMMs). Briefly, in BSuMMs, it is only the
presence or absence of an event in the relevant history that has an effect on
the occurrence of other events, while in OSuMMs the order of the events is
also taken into account. We refer the readers to Sections 3.3 and 3.4 in
Bhattacharjya et al. (2022) for formal definitions and methods for learning
SuMMs over event sequence collections.

Figure 1 shows examples of learned influencing sets using BSuMMs for refugee
crisis and mass migration events. Two events mass migration and famine are
identified as influencing events for refugee crisis. The model indicates that
the occurrence of both mass migration and famine together has a $0.91$
probability of resulting in refugee crisis as a part of an event sequence. On
the other hand the occurrence of mass migration in the absence of famine has a
$0.31$ probability of resulting in refugee crisis. search strategy to
efficiently discover the minimal influencing sets. We view the discovery of
influencing sets from LLM-generated data as a mechanism of distilling complex
symbolic knowledge from the output of LLMs.

##  3 Experimental Settings

We performed all our model inference related experiments on two NVIDIA V100
GPUs. We used the Huggingface library v$4.26.1$ Wolf et al. (2020) and
publicly available model
checkpoints.111huggingface.co/docs/transformers/model_doc/flan-t5 Classical
pattern mining algorithms (GSP, SPADE) were implemented in Python through the
use of spmf222pypi.org/project/spmf/ library v$1.4$.

For generation we use in-context learning under zero shot settings for all
LLMs, with top-k sampling ($k=50$, in conjunction with top-p, with $p=0.95$).

To identify influencing events from Summary Markov Models (SuMMs), we use
implementations from Bhattacharjya et al. (2022) and split the dataset
(generated from LLMs) into train/dev/test sets ($70\%/15\%/15\%$) to generate
results reported in Table 2. BSuMMs and OSuMMs were learned with
hyperparameters of $\alpha=0.1$, $\gamma=0.5$ and a look-back ($\kappa$)
window of $4$.

##  4 Evaluation

Open ended text generation in a task like event sequencing, with extremely
sparse reference data, poses unique challenges to the evaluation of model
outputs Wadhwa et al. (2023). The traditional scheme to evaluate discrete
model outputs has been to calculate precision and recall for the generated
outputs against a predefined reference test set. However, under open-world
settings and particularly when a sparse KG like Wikidata is used as reference
data, a missing causal relation between two event classes in a sequence may
very well be a valid relation. Therefore, it is not possible to automatically
derive an accurate measure of precision and recall purely using Wikidata as
reference data.We take a multi-pronged evaluation approach to assess the
quality and usefulness of the generated event sequences across multiple tasks.

###  4.1 Evaluation of Recall

Despite the sparsity of causal relations in Wikidata, one can still reasonably
estimate recall through pairwise comparisons of generated event classes to the
existing causal relations in Wikidata. We build a reference set of causal
relations in Wikidata by curating a list of all the pairs of event concepts
that have any of the several causal
relations333www.wikidata.org/wiki/Wikidata:List_of_properties/causality in
Wikidata in any direction, including has_cause (P828), has_immediate_cause
(P1478), has_contributing_factor (P1479), and has_effect(P1542). Our results
show that this is an effective mechanism of comparing recall across methods,
as larger models that are expected to have better accuracy yield higher recall
scores.

###  4.2 Evaluation of Precision

To overcome sparsity in reference data, prior work has generally relied on
human evaluations Chiang and Lee (2023)for estimating precision, which entails
looking at pairs of events and asking annotators whether or not the events in
question have a causal relationship. Such a process for potentially thousands
of event pairs (like in our case) can be very cost prohibitive. Furthermore,
recent research Zhao et al. (2022); He et al. (2023) indicates that pretrained
language models themselves might outperform lay human annotators such as those
found on Amazon Mechanical Turk. For instance, He et al. (2023) demonstrated
that labeling data under a few-shot chain-of-thought prompt (“explain-then-
annotate” setting) surpasses crowdworker annotations on relevance assessments.
Following these results, we use LLMs for evaluating precision and for
selecting the most optimal model.

We propose evaluating precision for model selection as a binary classification
task. Given two events $e_{1},e_{2}$, an evaluator model must evaluate whether
$e_{1}$ reasonably leads to $e_{2}$ under a has_cause or has_effect
relationship. To do this, we start with a large instruction-tuned model (in
our case, Flan-T5-XXL ($11$B) Chung et al. (2022)) and adopt instructional in-
context few shot prompting to classify whether or not $e_{1}$ reasonably leads
to $e_{2}$, and for the model to provide a justification for its results.
While we hypothesize that such an approach may yield noisy results, in a
preliminary manual assessment this approach seems to yield reasonable outputs
that are sufficient for evaluating correctness of model outputs (i.e.
precision). We include a sample of model evaluated outputs in supplementary
material, and select few examples in Appendix LABEL:sec_apx:examples.

| Precision Evaluator Model (P) | R | F-1
---|---|---|---
| Flan-T5-Large | Flan-T5-XL | Flan-T5-XXL |  |
Flan-T5-Large ($783$M) Chung et al. (2022) | 0.73 | 0.72 | 0.72 | 0.47 | 0.57
Flan-T5-XL (3B) | 0.75 | 0.75 | 0.78 | 0.49 | 0.60
Open-Research LLaMA ($3$B) Touvron et al. (2023) | 0.70 | 0.69 | 0.72 | 0.45 | 0.55
Flan-T5-XXL ($11$B) | 0.79 | 0.79 | 0.81 | 0.54 | 0.65
Table 1: Evaluation of LLM-generated sequences. For the purpose of evaluating
recall (R), we count event pairs (e1, e2) if such a pair exists in Wikidata.
For evaluating precision (P), we treat correctness of all generated event
pairs (e1, e2) as a standard classification task. Best-performing model scores
are in bold.

###  4.3 Human Evaluation of Cause-Effect Prediction Accuracy of LLMs

In addition to this automated evaluation of our LLM-generated sequences, we
carry out additional evaluation of our method on an existing popular knowledge
graph – ATOMIC Hwang et al. (2021) and conduct human evaluation on a sample of
data to support our approach. ATOMIC consists of event-centered pairs of
instances of the form IsAfter (Y comes after X) and Causes (X causes Y). We
leverage these instances as input prompts to the model and use human
annotators to evaluate outputs. Specifically, we show human annotators
anonymized model outputs and true references and elicit their preferences
given a trigger event. We find that in an overwhelming majority of the cases,
the models generate output that is semantically equivalent to the reference
(even though there is no direct lexical alignment), or output that the humans
prefer over the true reference. We provide example outputs of various
evaluator models in the supplementary material (appendix).

###  4.4 Evaluation of Event Models

Apart from a qualitative manual analysis of the mined patterns, we follow
Bhattacharjya et al. (2022) and use a metric that evaluates the quality of the
learned model with respect to the data. Specifically, we use negative log loss
as the evaluation metric, which is a popular metric for probabilistic
prediction Bishop (2006). We posit that higher-quality event sequences result
in better prediction ability of the models, and so better negative log loss
scores for the models. Our results show that this is a valid assumption, as
larger models that are expected to produce higher-quality sequences yield
models with better prediction ability.

($\downarrow$) Data Generator Model | BSuMMs | OSuMMs | LSTM
---|---|---|---
Flan-T5-Large (783M) Chung et al. (2022) | -63.49 | -84.29 | -109.28
Flan-T5-XL (3B) | -63.20 | -92.65 | -121.23
Open-Research LLaMA (3B) Touvron et al. (2023) | -110.57 | -101.29 | -190.68
Flan-T5-XXL (11B) | -57.99 | -78.64 | -108.89
Table 2: Negative log likelihood (lower magnitude is better) averaged over
interest labels from LLM-generated (Flan-T5-XXL) event sequences. Lookback
window ($k$) for LSTM was fixed to $5$. Best-performing data generator model
scores are in bold.

##  5 Experimental Investigation and Results

We conducted a three-pronged quantitative and qualitative evaluation of our
results. First, we evaluated LLM-generated event sequences in absolute terms
through standard traditional metrics – precision, recall, and F1 score.
Second, we gauge how probabilistic models like SuMMs fare when trained on LLM-
generated data. Finally, we look at some qualitative examples of newly
identified high support sequences (Appendix LABEL:sec_apx:examples) not
already present in our base knowledge graph i.e. Wikidata.

###  5.1 Generated Event Sequences

We generate event sequences via LLMs through iterative in-context prompting
(Figure 3). Table 1 summarizes our results from these experiments. We find
that Flan-T5-XXL consistently outperforms its smaller counterparts, indicating
that larger models generally fare better and benefit from their larger size
(i.e. number of parameters). We posit this may also indicate that using larger
models may continue to yield performance gains.

To quantify how well model outputs correlate with human assessments, we also
conduct additional experiments and a small-scale human evaluation on a
different, independent commonsense reasoning dataset: ATOMIC Hwang et al.
(2021). We follow the same prompting strategy as above for a set of $200$
randomly selected event-centered input instances ($100$ each of the type ‘$X$
Causes $Y$’ and ‘$Y$ IsAfter $X$’)444Complete details on ATOMIC are available
in Table 1 in Hwang et al. (2021).. We then present the model output and the
true reference from ATOMIC to three human annotators. For example, a typical
instance presented to human evaluators was of the form:

Input Instance: PersonX drops out of high school

Response 1: PersonX gets a job

Response 2: PersonX turns PersonX’s life around

Type: IsAfter

One of the responses above is the model (Flan-T5-XXL) output, while the other
is the true reference. The human evaluators are then asked to answer the
following questions–

  * •

Are both responses functionally similar?

  * •

Which response do you prefer?

  * •

Which response, if any, is completely irrelevant?

Based on the responses from human evaluators555We observe strong inter-rater
agreement with a Fleiss kappa, $\kappa=0.81$; conflicting response labels were
aggregated through majority vote., we observe that humans found $65.82\%$ of
response pairs functionally equivalent. That is, even though not lexically
aligned, they meant the same thing. In $27.64\%$ of the instances the humans
preferred the model-generated event instance over the true reference. In only
$6.55\%$ of instances did the humans prefer the true reference over the model-
generated output.666Additional details on these experiments are available in
the supplementary material (appendix). These results reinforce our underlying
assertion that LLMs are capable of event-centered commonsense reasoning.

#### Precision Evaluator Models

We leverage our models for dual purposes – for generating and evaluating event
sequences. Owing to this inherent conflict, we find it prudent to
independently assess these evaluator models. To ensure robustness of our
results, we use multiple evaluator models (Table 1) and find no significant
difference between evaluated precision across different models used as
evaluators.

###  5.2 Probabilistic Prediction via SuMMs and LSTM

Following the evaluation strategy implemented in Bhattacharjya et al. (2022),
we focus on individual labels of interest and conduct an evaluation around
probabilistic prediction. Consequently, we select negative log loss as the
evaluation metric. Table 2 summarizes our results on the test set for both
BSuMMs and OSuMMs, along with a simple LSTM baseline. We observe that models
trained on event sequences generated from larger models (e.g. Flan-T5-XXL)
fare better than the ones generated from their smaller counterparts (e.g.
Flan-T5-Large). We treat this observation as a proxy for generated event
sequence quality. That is, better quality sequences lead to better predictive
models.

###  5.3 Qualitative Analysis

Figure 1 shows examples of mined patterns along with their support levels, as
well as the probabilistic association between a number of influencing events
from the output of BSuMMs. Appendix LABEL:sec_apx:examples provides two
examples of newly mined sequences derived from pattern mining over LLM-
generated sequences, and an additional two examples of influencing sets
derived from the application of SuMMs. For each example, we show evidence
supporting the accuracy of the extracted knowledge. We provide additional
qualitative examples in supplementary material. Overall, these results show
the high quality and usefulness of the distilled event sequence knowledge.

###  5.4 Error Analysis

We now describe two of the most common recurring errors in the LLM-generated
outputs and SuMMs’ discovered influence sets.

#### No Sequences Generated

Out of the $202$ unique input event-triggers in our data, $8$ event-triggers
yielded sequences of length $1$, i.e., no sequences. This means that the
underlying model could not select a relevant event concept from the provided
vocabulary that might follow the given trigger.

#### Illogical Influencing Sets

Due to the choice of the vocabulary of limited event concepts, we often find
that the identified influencing sets (i.e., parent events) for some event
concepts may not be correct or, at times, completely illogical. Qualitatively,
we observe that this occurs with rare event types that do not appear often
either in the underlying knowledge graph, or in the LLM-generated set of event
sequences. Consequently, SuMMs fail to identify logical influencing sets. For
example, for the event type United States Presidential Impeachment, BSuMM
model identified unequal treaty as the influencing event.

##  6 Related Work

#### News Event Analysis

The primary applications of our work are around news event analysis and
forecasting. Zhao (2021) presents a taxonomy of different flavors of event
prediction in the literature. Our target event prediction applications fall
under the “Semantic Prediction” category, with time and location details not
being of interest, and the primary goal being the prediction of “event
profiles" such as event types. Seminal work in this area is the work of
Radinsky et al. (2012) where causal relations between past events are
extracted from text and then generalized to make predictions. Some recent work
has explored the use of graph sequence mining Cekinel and Karagoz (2022) as a
mechanism of extracting useful relations from large and noisy outputs of
extraction. The application of building event sequence models over such
outputs is largely unexplored. Our work explores an alternative approach of
event sequence generation that is capable of generating longer, and
potentially higher-quality sequences.

#### Event Sequence Extraction from Text

There is a wealth of literature on different methods of extracting sequences
from textual corpora. Norambuena et al. (2023) present a comprehensive survey
of automated methods of narrative extraction. Narratives contain several
elements including events, participants (actors/protagonists), time, and
space. The task of event detection itself is a highly challenging task and the
topic of extensive research Chen et al. (2021); Xiang and Wang (2019), which
has its root in the Topic Detection and Tracking (TDT) task Santana et al.
(2023). This line of work started out as a DARPA-sponsored initiative with the
same name Allan et al. (1998). Another closely related task is news timeline
summarization Gholipour Ghalandari and Ifrim (2020). While pattern mining
algorithms have been applied to the output of such extractions, e.g. for
creation of “domain templates" Filatova et al. (2006), we are not aware of any
attempts to use the extractions to construct event models for analysis or
prediction.

#### Sequential Pattern Mining and Event Sequence Models

Sequential pattern mining and related approaches have been the subject of
extensive research Mannila et al. (1997); Mabroukeh and Ezeife (2010); Mooney
and Roddick (2013); Fournier-Viger et al. (2017). These algorithms take in a
set of sequences (or “sequential records") with a set of unique events (or
“items") and often a “minimum support" threshold, and return as output a
ranked set of all frequent sequences in a given sequence collection (or
database) meeting the minimum support threshold. There is also a long line of
work on statistical and probabilistic models for analysis of different kinds
of sequences. Our focus in this paper is on multivariate event sequences,
i.e., sequences of various event types without timestamps. Markov models for
sequences Raftery (1985); Begleiter et al. (2004) and long short-term memory
(LSTM) Hochreiter and Schmidhuber (1997) models are examples of prediction
models. We leverage a more recent family of models – Summary Markov models
Bhattacharjya et al. (2022) – for some of the experiments in this paper that
involve analyzing generated event sequences.

##  7 Conclusions

We evaluated the capabilities of large language models for the task of
producing event sequences from concepts derived from a knowledge graph. We
found that, through careful elicitation, LLMs are capable of producing high-
quality event sequences that can be used for the downstream tasks of pattern
mining and the construction of probabilistic event models such as SuMMs. We
also proposed methods for effectively evaluating the outputs of LLMs with
sparse reference sets and deriving useful signals for model selection through
the use of existing traditional metrics. Our hope is that this work
demonstrates the utility of large language models for extracting event-related
information and helps researchers better navigate the complex interaction
between event-related entities. While we have demonstrated some use-cases of
the resulting dataset of event sequences (mining logical rules and patterns
and extracting influencing event types), resulting datasets in this work and
in other domains may themselves be a useful resource for researchers as one
may draw upon them to inform or justify decision-making.

## References

  * Allan et al. [1998] James Allan, Jaime G. Carbonell, George Doddington, Jonathan Yamron, and Yiming Yang.  Topic Detection and Tracking Pilot Study Final Report.  In Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop, February, 1998., 1 1998.
  * Allan [2012] James Allan.  Topic Detection and Tracking: Event-Based Information Organization.  Springer Publishing Company, Incorporated, 2012.
  * Begleiter et al. [2004] Ron Begleiter, Ran El-Yaniv, and Golan Yona.  On prediction using variable order markov models.  J. Artif. Intell. Res., 22:385–421, 2004.
  * Bhattacharjya et al. [2022] Debarun Bhattacharjya, Saurabh Sihag, Oktie Hassanzadeh, and Liza Bialik.  Summary markov models for event sequences.  In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4836–4842. International Joint Conferences on Artificial Intelligence Organization, 7 2022.  Main Track.
  * Bishop [2006] Christopher M. Bishop.  Pattern Recognition and Machine Learning (Information Science and Statistics).  Springer-Verlag, Berlin, Heidelberg, 2006.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * Cekinel and Karagoz [2022] Recep Firat Cekinel and Pinar Karagoz.  Event prediction from news text using subgraph embedding and graph sequence mining.  World Wide Web, 25(6):2403–2428, 2022.
  * Chen et al. [2021] Muhao Chen, Hongming Zhang, Qiang Ning, Manling Li, Heng Ji, Kathleen McKeown, and Dan Roth.  Event-centric natural language processing.  In ACL, 2021.
  * Chiang and Lee [2023] Cheng-Han Chiang and Hung-yi Lee.  Can large language models be an alternative to human evaluations?  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models, 2022.
  * Du et al. [2022] Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Hannan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghazaleh Kazeminejad, Zhenhailong Wang, Chris Callison-Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, and Heng Ji.  RESIN-11: Schema-guided event prediction for 11 newsworthy scenarios.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations, pages 54–63, Hybrid: Seattle, Washington + Online, July 2022. Association for Computational Linguistics.
  * Filatova et al. [2006] Elena Filatova, Vasileios Hatzivassiloglou, and Kathleen McKeown.  Automatic creation of domain templates.  In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 207–214, Sydney, Australia, July 2006. Association for Computational Linguistics.
  * Fournier-Viger et al. [2017] Philippe Fournier-Viger, Jerry Chun-Wei Lin, Rage Uday Kiran, and Yun Sing Koh.  A survey of sequential pattern mining.  Data Science and Pattern Recognition, 1(1):54–77, 2017.
  * Gholipour Ghalandari and Ifrim [2020] Demian Gholipour Ghalandari and Georgiana Ifrim.  Examining the state-of-the-art in news timeline summarization.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1322–1334, Online, July 2020. Association for Computational Linguistics.
  * Hassanzadeh et al. [2022] Oktie Hassanzadeh, Parul Awasthy, Ken Barker, Onkar Bhardwaj, Debarun Bhattacharjya, Mark Feblowitz, Lee Martie, Jian Ni, Kavitha Srinivas, and Lucy Yip.  Knowledge-based news event analysis and forecasting toolkit.  In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5904–5907. ijcai.org, 2022.
  * He et al. [2023] Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen.  Annollm: Making large language models to be better crowdsourced annotators, 2023.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Comput., 9(8):1735–1780, nov 1997.
  * Hwang et al. [2021] Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi.  Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs.  In AAAI, 2021.
  * Imani et al. [2023] Shima Imani, Liang Du, and Harsh Shrivastava.  MathPrompter: Mathematical reasoning using large language models.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37–42, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * Kamalloo et al. [2023] Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei.  Evaluating open-domain question answering in the era of large language models.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5591–5606, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * Mabroukeh and Ezeife [2010] Nizar R. Mabroukeh and C. I. Ezeife.  A taxonomy of sequential pattern mining algorithms.  ACM Comput. Surv., 43(1), dec 2010.
  * Mannila et al. [1997] H. Mannila, H. Toivonen, and A. I. Verkamo.  Discovery of frequent episodes in event sequences.  Data Mining and Knowledge Discovery, 1:259–289, 1997.
  * Mooney and Roddick [2013] Carl H. Mooney and John F. Roddick.  Sequential pattern mining – approaches and algorithms.  ACM Comput. Surv., 45(2), mar 2013.
  * Norambuena et al. [2023] Brian Keith Norambuena, Tanushree Mitra, and Chris North.  A survey on event-based news narrative extraction.  ACM Comput. Surv., 55(14s), jul 2023.
  * Radinsky et al. [2012] Kira Radinsky, Sagie Davidovich, and Shaul Markovitch.  Learning causality for news events prediction.  In Proceedings of the 21st World Wide Web Conference 2012, WWW 2012, Lyon, France, April 16-20, 2012, pages 909–918. ACM, 2012.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res., 21(1), jan 2020.
  * Raftery [1985] A. Raftery.  A model for high-order Markov chains.  Journal of the Royal Statistical Society, Series B, 47(3):528–539, 1985.
  * Santana et al. [2023] Brenda Salenave Santana, Ricardo Campos, Evelin Amorim, Alípio Jorge, Purificação Silvano, and Sérgio Nunes.  A survey on narrative extraction from textual data.  Artif. Intell. Rev., 56(8):8393–8435, 2023.
  * Srikant and Agrawal [1996] Ramakrishnan Srikant and Rakesh Agrawal.  Mining sequential patterns: Generalizations and performance improvements.  In Peter Apers, Mokrane Bouzeghoub, and Georges Gardarin, editors, Advances in Database Technology — EDBT ’96, pages 1–17, Berlin, Heidelberg, 1996. Springer Berlin Heidelberg.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * Vrandečić and Krötzsch [2014] Denny Vrandečić and Markus Krötzsch.  Wikidata: A free collaborative knowledgebase.  Commun. ACM, 57(10):78–85, sep 2014.
  * Wadhwa et al. [2023] Somin Wadhwa, Silvio Amir, and Byron Wallace.  Revisiting relation extraction in the era of large language models.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15566–15589, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * Wei et al. [2022] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In International Conference on Learning Representations, 2022.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics.
  * Xiang and Wang [2019] Wei Xiang and Bang Wang.  A survey of event extraction from text.  IEEE Access, 7:173111–173137, 2019.
  * Yu et al. [2022] Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang.  A survey of knowledge-enhanced text generation.  ACM Comput. Surv., 54(11s), nov 2022.
  * Zaki [2004] Mohammed J. Zaki.  Spade: An efficient algorithm for mining frequent sequences.  Machine Learning, 42:31–60, 2004.
  * Zhao et al. [2022] Mengjie Zhao, Fei Mi, Yasheng Wang, Minglei Li, Xin Jiang, Qun Liu, and Hinrich Schuetze.  LMTurk: Few-shot learners as crowdsourcing workers in a language-model-as-a-service framework.  In Findings of the Association for Computational Linguistics: NAACL 2022, pages 675–692, Seattle, United States, July 2022. Association for Computational Linguistics.
  * Zhao [2021] Liang Zhao.  Event prediction in the big data era: A systematic survey.  ACM Comput. Surv., 54(5), may 2021.