# SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset
for Scientific Graphs

Shengzhi Li
<EMAIL_ADDRESS>
Nima Tajbakhsh
<EMAIL_ADDRESS>

###### Abstract

In this work, we present SciGraphQA, a synthetic multi-turn question-answer
dataset related to academic graphs. SciGraphQA is 13 times larger than
ChartVQA, the previously largest chart-visual question-answering dataset. It
is also the largest open-sourced chart VQA dataset with non-synthetic charts.
To build our dataset, we selected 290,000 Computer Science or Machine Learning
ArXiv papers published between 2010 and 2020, and then used Palm-2 to generate
295K samples of open-vocabulary multi-turn question-answering dialogues about
the graphs. As context, we provided the text-only Palm-2 with paper title,
abstract, paragraph mentioning the graph, and rich text contextual data from
the graph itself, obtaining dialogues with an average 2.23 question-answer
turns for each graph. We asked GPT-4 to assess the matching quality of our
question-answer turns given the paper’s context, obtaining an average rating
of 8.7/10 on our 3K test set.

We evaluated the 0-shot capability of the most popular MLLM models such as
LLaVa, mPLUGowl, BLIP-2, and openFlamingo’s on our dataset, finding LLaVA-13B
being the most performant with a CIDEr score of 0.08. We further enriched the
question prompts for LLAVA by including the serialized data tables extracted
from the graphs using the DePlot model, boosting LLaVA’s 0-shot CIDEr to 0.15.
To verify the validity of our dataset, we also fine-tuned LLaVa using our
dataset, reaching a substantially higher CIDEr score of 0.26. We anticipate
further accuracy improvement by including segmentation mask tokens and
leveraging larger LLM backbones coupled with emergent prompting techniques.
Our code and data are open-sourced at Github and HuggingFace dataset.

_Keywords_ VQA, Multi-modal large language model, Scientific Graph Question-
answering Benchmark

##  1 Introduction

Scientific literature is a rich repository, teeming with detailed graphs,
diagrams, and figures that convey intricate data narratives. The task of
comprehending, describing, and interpreting these graphs requires significant
time and effort from students, data science practitioners, and researchers.
Unlike natural images, which are interpreted based on objects, their relative
positions, and interactions, scientific graphs carry distinct semantic
meanings, illustrated through abstract components such as trend lines and
color-coded legends. Typically, readers interpret these graphs in the context
of the paper’s abstract and pertinent paragraphs. In a live presentation
scenario, the author would initiate the context, engaging the audience in a
multi-turn, question-answering dialogue.

Inspired by this interactive process, we have developed SciGraphQA, a
comprehensive, multi-turn question-answering dataset derived from scientific
literature and graphs. Leveraging a commercial large-language model, we have
generated question-answering dialogues, providing it with the figure caption,
paper title, abstract, and the initial paragraph referencing the figure. Our
objective with this dataset is to establish a scientific question-answering
benchmark for multi-modal large language models (MLLMs). In this paper, we
evaluate the performance of current MLLMs and introduce a baseline model,
SciGraphQA-baseline, which has been trained on the SciGraphQA training set.

##  2 Related Work

###  2.1 Graph Visual Answering datasets

Datasets Figure Count Data/Chart generation process Question-Answer pair count
Question generation Answer Type $\\#$ Plot types FigureQA [1] 180K Synthetic
data and charts 2.3M From 15 templates Fixed voca. 4 DVQA [2] 300K Synthetic
data and synthetic charts 3.4M From 26 templates Fixed vocab. 1 PlotQA [3]
224K Real-world data and synthetic charts 28M From 76 templates Mix of fixed
and open vocabulary answers. 3
ChartQA [4] 21.9K (4.8K human and 17.1K generated) Real-world charts from a
web crawl 32.7K (9.6K human and 23.1k generated) Human/Machine generated Open
Vocabulary Unbounded (real-world charts) SciGraphQA (ours) 295K Real-world
academic graphs 657K Machine Generated with Palm with additional textual
context Open Vocabulary Unbounded (real-world charts)

Table 1: Related open datasets on graph visual questioning.

Table 1 presented a series of open-source datasets pertinent to graph/chart
visual question answering. Earlier datasets such as FigureQA [1], DVQA [2],
and LEAF-QA [5] primarily relied on synthetic data, with template-generated
questions and answers selected from a fixed vocabulary. In contrast, PlotQA
[3] introduced open-vocabulary questions. Built on ground truth data tables
crawled online, PlotQA programmatically generated charts, and questions
answers, but the limitations stem from its synthetic nature post data crawl,
restricted chart types: bar plots, line plots, and scatter plots and
restricted question variety [3]

More recently, ChartQA capitalized on real-world, web-crawled charts to
develop its visual question-answering datasets, supplemented by human
annotators. In order to scale up, ChartQA’s authors fine-tuned a T5 model to
generate 2/3 of their questions and answers derived from human-written chart
summaries [4].

The SciGraphQA dataset is a large-scale captioning dataset that utilizes over
290,000 Arxiv papers focused on Computer Science (cs) and Machine Learning
(stat.ML) [6]. PDFFigures 2.0 was used to extract figures. Out of 2.1 million
figures extracted, the top categories are tables (23.6$\%$), graphs
(19.2$\%$), and flowcharts (8.5$\%$). Graphs were chosen as the focus for
caption prediction. Despite limited success in predicting captions based on
images (BLEU-4 $\sim$ 0.02), SciGraphQA provided valuable insights by
demonstrating that text normalization did not enhance model performance. Its
successor, SciGraphQA+, added more context to image captioning by
incorporating the first paragraph that referenced the figure and OCR-extracted
text, yielding an increase in BLEU-4 (0.064).

The SciGraphQA/SciGraphQA+ dataset, however, has inherent limitations when
framing the problem as a caption prediction task. The ground truth captions
may not always be helpful or explicit enough. One-third of captions in
SciGraphQA are single-sentence, many of which are incomplete sentences (e.g.,
"Vary the number of objects") or consist solely of single/multiple nouns
(e.g., ’IIPS’, ’Average Travel Time’). For a user’s perspective, such as a
scientific writer, automatic captioning might not reduce writing time,
especially if the system requires the writer to supply context, as in the case
with SciGraphQA+. If a full-bodied description is necessary for caption
generation to work, we posit that users would prefer to write a short caption
themselves.

We believed a dialog experience without needing lengthy textual input offers a
more natural flow of user experience. The human user would only need to upload
an image, which could range from a Tableau screenshot by a business analyst, a
matplotlib plot from a data scientist, or a multi-colored diagram from a
color-blind individual seeking to understand what a red line represents. The
human user can guide the conversation by providing feedback or additional
context. The questions could be conceptual, graphical or complex reasoning-
related.

With this in mind, we introduced the SciGraphQA dataset with multi-turn
question-answer conversations on scientific graphs. The questions and
corresponding answers were generated from Palm-2, using SciGraphQA figures,
captions, context, and paper metadata. SciGraphQA distinguishes itself from
previous datasets by using real-world academic graphs rather than synthetic
data or charts. The questions and answers inherently support open vocabulary.
By leveraging SciGraphQA and Palm2, we significantly scaled up the ChartQA-
style real-world Chart-vqa dataset to 295K question-answer pairs from 21.9
(13X) number of samples as shown in Table 1.

###  2.2 Chart VQA expert systems

Previously, when datasets only contained fixed-vocabulary questions and
answers, the standard approach was to construct classification-based VQA
models such as STL-CQA [7], LEAF-Net [5], DQVQ-baseline [2], and VisionTAPAS
[4]. These encoder-only models encoded image features and textual questions
separately, combining them later using attention blocks [5]; [7] [2].

DePlot/Matcha is a lineage of image-encoder-text-decoder models distinct from
the aforementioned encoder-only architectures. These models are trained to
’derender’ the chart, meaning they extract text and datapoints needed to
render the chart. They are based on the Pix2Struct model, which was pre-
trained on website visual understanding (from screenshot to HTML code) [8].
Matcha fine-tuned Pix2Struct on various datasets, such as Github IPython
notebooks for [chart > code], a mix of PlotQA, web-crawled data, and Wikipedia
tables for [chart > table], and math reasoning datasets for [image > answer]
[9].

DePlot, in turn, was further fine-tuned on [chart > table] datasets, including
ChartQA, to specialize in converting charts into linearized tables with
titles, legends, and interpolated data point values [10]. The output of Deplot
is a text string data table that can be fed into such large autoregressive
decoders as GPT-3 or Palm. The combination of Deplot and LLM is currently a
state-of-the-art benchmark-tuned model for the ChartQA and PlotQA datasets,
surpassing VisionTAPAS with OCR input without the need for an external OCR
system [9] [10].

We proposed to leverage Deplot as an external model to enhance the MLLMs’
capabilities with regards to charts. Even the most powerful large multi-modal
models cannot match expert OCR models by a significant margin [11]. Zero-shot
text recognition results range from 37-61% compared to the supervised SOTA of
85%. This deficiency is particularly detrimental to science graphs, where text
is key to understanding them. Deplot not only offers OCR capabilities but also
can interpolate data point values. This choice to leverage an external system
is similar to previous pipelines that incorporated external OCR systems into
their input, such as Layout-LM [12] and ChartBert [13].

###  2.3 Multi-modal large language models (MLLM)

In the domain of natural image understanding and multi-modal generative
models, there has been rapid development of vision-language foundational
models. These models are similar to the Deplot model discussed in the previous
section, as they are image-encoder-text-decoder models. However, the primary
distinction lies in the use of pre-trained image encoders, such as CLIP,
coupled with a pre-trained large language model backbone, which is typically
frozen or partially frozen during training. To bridge the two modalities,
MLLMs are trained with ’adapters’ that translate image patch embeddings into
token embeddings that can be understood by a large language model [14].
Notably, these models were trained on natural images rather than text-heavy
charts and figures.

The BLIP-2 model employed a Querying Transformer (Q-Former) to bridge the
modality gap between vision and language models. By employing learnable query
vectors, BLIP-2 is capable of extracting visual representations that are
relatable to the language model, [15].

DeepMind’s Flamingo as well as its open-source equivalent OpenFlamingo,
designed gated cross-attention-dense layers within frozen pre-trained LLMs and
perceiver resampler to support fixed-sized input with variable in-context
learning examples. By sourcing keys and values from visual features and
queries from language inputs, it efficiently conditions LLMs on visual input.
This structure is highly efficient for in-context learnings [16], [17]).

LLaVA employed self-instruction with GPT-4 to generate a high-quality multi-
modal language-image instruction-following dataset. After training a
projection layer with image-text pairs from the CC3M dataset as a feature
alignment stage, the language model and projection layer were jointly trained
for instruction-tuning. This enabled LLaVA to perform various visual
understanding and reasoning tasks [9]. LLaVA used a frozen CLIP Vit-L/14
model.

mPLUG-Owl followed a two-stage training process similar to LLaVA, with the
disinction of unfrozeing the visual encoder during the first stage and the use
of fixed learnable tokens to summarize visual information via cross-attention
with image patch features to connect image and text modalities rather an a
linear projector. [18]. The first-stage training was done on a significantly
larger data mixture that includes LAION-400M [19], COYO-700M [20], Conceptual
Captions [21], and MSCOCO [22]. The second-stage instruction dataset includes
Vicuna’s sharegpt [23] and LLaVA (multi-modal) [9]

In this study, we proposed to evaluate the abilities of the MLLMs on our
SciGraphQA datasets. We performed zero-shot evaluations to assess their
performance in this out-of-distribution domain of scientific graphs.
Furthermore, we fine-tuned a baseline MLLM using the SciGraphQA training set.

##  3 Methodology

###  3.1 Dataset Construction

We expanded the SciCap+ dataset to create the SciGraphQA dataset by generating
multi-turn conversations centered around the understanding and elaboration of
the content presented in academic graphs. This was achieved by collecting
comprehensive textual context related to the graph, which includes the title,
abstract, original captions, the first paragraphs mentioning the figure, and
OCR text. The SciCap+ dataset is an image-caption dataset derived from
scientific figures and captions; it includes additional text derived from the
Google OCR API and the paragraphs referenced in the paper [24]. As shown in
figure 1, we combined figures, captions, paragraphs, and text-from-OCR from
the SciCap+ dataset with the corresponding paper titles and abstracts, and
subsequently prompted the PALM2 Chat API to generate multi-turn conversations.
We provided this API with two in-context examples generated by GPT-4, as seen
in the Appendix 8. In the prompt, we specified that the conversation should
focus primarily on the graph, rather than the broader context. For example, we
posed questions such as, "Considering the graph structure, can you elaborate
on how the lines, axes, and legend provide a comprehensive perspective of the
interplay between source variance and mean target risk?" After a filtering
process, we were able to curate 295K high-quality, multi-turn data entries.

As depicted in Figure 3, our approach diverged from LLaVA’s methodology of
using a fixed set of questions to prompt GPT-4 [9]. We observed a greater
diversity in our question set, owing to their generation process conditioned
on text context. Some questions were broad, inquiring about textual aspects
(e.g., "What are the limitations of this graph?") or components of the
composition (e.g., "What does this axis signify?"). On the other hand, several
questions delved into specific concepts, mirroring natural conversations by
incorporating follow-up queries based on the previous question-answer
exchange. This approach seeks to capture the dynamics of organic dialogue,
introducing a layer of complexity not present in the fixed-question
methodology.

Figure 1: Illustration of multi-turn dialogue generation process. For higher
quality dialogues, we use comprehensive textual context together with in-
context learning when prompting Palm-2. Figure 2: (left) distribution of the
number of question-answer turns in our SciGraphQA dataset. (right)
distribution of GPT-4 ratings (0–10) when GPT-4 was used as a judge to measure
the matching of questions and answers from a 3k subset of the the SciGraphQA
dataset. Figure 3: Examples from our SciGraphQA dataset where questions and
answers are both generated using a commercial LLM rather than being selected
from a fixed, limited template pool. Note how the questions are specific to
the graphs and often have a conversational nature, asking to elaborate on a
concept mentioned in previous answer. For brevity, some answers are truncated,
denoted by “…“ at the end.

Our initial Palm-2 output contained 350K samples. Manual inspection showed
that a portion of the questions are not graph-related but follow-up on a
textual context in previous answers. We would like to focus on graph-related
questions to evaluate the graph-understanding capability of multi-modal large
language models. Hence we developed a heuristic rule to filter out questions
that do not contain any keywords in the following list: [’graph’, ’diagram’,
’figure’, ’chart’, ’axis’, ’plot’, ’table’, ’image’, ’visual’, ’illustrat’].
Admittedly, in naturally occurring conversations around scientific graphs,
humans ask follow-up questions on abstract concepts as well. We will release
the unfiltered question-answers in addition to filtered question-answers.

After filtering out low-quality QA turns, we curated 295K multi-turn data with
59.1 Million tokens, counted using a byte-wise BPE tokenizer. The average turn
is 2.23 while there are 111K samples with 3 or more turns, and we plotted the
number of question-answer turns in figure 2. On average, we have 143
characters in questions and 775 characters in answers, or 39 tokens in
questions and 164 tokens in answers. The token per sample is 199$\pm$98 (mean
$\pm$ std). The statistics are similar to LLaVA whose instruction tuning
dataset has 109 million tokens with an average of 276 tokens per sample.

To check the quality of our question-answer matching, we used GPT-4 to rate
the responses in the full 3K test set using GPT-4. GPT-4 was prompted with the
textual context (paragraph, caption, abstract), question, and reference answer
and asked to rate the quality and helpfulness of the reference answer to the
given questions. As shown in the figure 2, we observed that the distribution
is heavily skewed towards 10, with 86% of the answers being rated 8.5/10 or
higher. On the low end, there are 6.4% of ratings of 7/10 or lower, reflecting
imperfections in our question-answering generation process. There are 1.2% of
ratings $\leq$ 1/10, meaning the answer does not address the question. We
concluded that overall, the GPT-4 rating assessment confirmed the validity of
our question-answer generation and filtering process, with the majority of
answers accurately answering the question.

###  3.2 Zero-shot evaluation

We were interested in examining the capabilities of existing Multimodal Large
Language Models (MLLMs), originally trained with natural images typically
sourced from the CoCo or LAION datasets [14], in comprehending and answering
questions related to scientific graphs. We used the checkpoints of the
candidate models in a zero-shot evaluation setting where they received only
the image and question. The test set consisted of 3K pairs of images,
questions, and answers from SciGraphQA. We calculated NLP metrics, including
BLEU-4, CIDEr, and ROUGE, between the predicted answers and the ground truth
for the test set.

Figure 4: Illustration of how the DePlot model extracts the data table from a
figure followed by serializing the table into a string. <0x0A> is a newline
token in hexadecimal format. Note how the data table contains the interpolated
values along the x-axis and y-axis for both legends. We used the DePlot’s
output string as a prompt augmentation technique in our work.

Inspired by DePlot[10], We further augmented the question prompts by
prepending them with the data tables extracted from the graphs using the
DePlot model, which interpolates the axis legends and numerical values from
the given image, as shown in Figure 4. The authors of DePlot showed that GPT
3.5, when prompted with only the tabular data, was able to respond to
questions related to a given chart, underscoring the effectiveness of the
tabular data in enhancing the zero-shot capability of LLMs.

###  3.3 Fine-tuning

We further validate our dataset by fine tuning LLaVA-13B, which demonstrated
the highest accuracy among the models in the 0-shot evaluation setting. For
this purpose, we utilized the LLaVA-13B checkpoint, released after feature
alignment and multi-modal instruction fine-tuning [9]. We evaluated the fine-
tuned models using the 0-shot evaluation method introduced in the previous
section. We conducted a two-step fine-tuning process for LLaVA-13B on the
SciGraphQA dataset. First, we fine-tuned it for 1 epoch on the full dataset,
which consisted of 295K samples. Next, we performed fine-tuning on a 30K
subset of SciGraphQA, where the question prompts were augmented with the data
tables extracted using DePlot from the graphs.

We used a LLaVA checkpoint with LLAMA-2-13B-chat backbone as base model for
finetuning. We employeed a cosine learning rate schedule with a warmup ratio
of 3% and a learning rate of $5\times 10^{-6}$. This learning rate was
deliberately lower than the original $2\times 10^{-5}$ used for the
instruction tuning of LLaVA and also lower than the learning rate of $3\times
10^{-4}$ used for LLAMA-2-13B [9], [25]. We made this decision based on early
experiments, which indicated that end-to-end fine-tuning with higher learning
rates, like $2\times 10^{-5}$, resulted in degraded performance. Thus, it
underscored the significance of using an appropriate learning rate for
successful fine-tuning.

We utilized LoRa to enable us to train small-scale adapters on the
LLaMa-2-chat-13B backbone [25]. We used a LoRa rank of 64 and a LoRa dropout
rate of 0.05. Together with DeepSpeed Zero-2 distributed training, this setup
allowed us to train with 4 A100-80Gb GPUs on Azure with a per-device batch
size of 16 and a gradient accumulation step of 2, resulting in a effective
global batch size of 128.

##  4 Results

Model Name Finetuned on SciGraphQA? Prompt Augmented with extracted data-table
CIDEr BLEU(4) ROUGE BLIP2-2.7B No No 0.007 0.003 0.1 DePlot+mPLUG-owl-7B No
Yes 0.037 0.058 0.22 mPLUG-owl-7B No No 0.04 0.062 0.22 LLaVa-7B No No 0.048
0.07 0.18 LLaVa-13B No No 0.08 0.07 0.23 OpenFlamingo v2-7B No No 0.12 0.081
0.22 DePlot+GPT-3 No Yes 0.13 0.098 0.226 DePlot+LLaVa-13B No Yes 0.153 0.106
0.273 DePlot+SciGrahQA-baseline Yes Yes 0.268 0.123 0.31

Table 2: Evaluation of large multi-modal models using CIDEr, BLEU-4, and ROUGE
metrics on the 3K test set of our SciGraphQA dataset. The term "DePlot +
Model" denotes prompt-augmentation using DePlot-extracted data tables. The
SciGraphQA-baseline is a LLaVA-13B model fine-tuned on SciGraphQA’s training
set.

We evaluated the performance of multiple models on our dataset in zero-shot
and fine-tuning settings. We set aside a test set of 3K samples and computed
NLP metrics including BLEU-4, ROUGE, and CIDEr. As seen in Table 2,
performance of large multi-modal models increased proportionally to the size
of their Language-to-Language Model (LLM) backbones. For instance, BLIP2, with
its comparatively smaller 2.7B backbone, showed the least accuracy among the
models studied. Conversely, LLaVA-13B, equipped with a significantly larger
backbone, demonstrate superior performance.

Interestingly, within the 7B range, distinct performances emerged from models
like OpenFlamingo (Moasic-7B), LLaVA-7B (Vicuna-7B), and mPLUG-owl (Llama-7B).
This disparity in performance could be tied back to their unique multi-modal
adapter designs—ranging from linear projectors to a combination of perceived
resamplers and cross attention—and the nature of their respective training
datasets. The findings point towards a nuanced understanding that simply
scaling up model sizes might not be a one-size-fits-all solution, with the
design of adapters and choice of training data playing an equally important
role.

Despite trailing other 7B MLLMs models, OpenFlamingo’s underperformance can be
attributed to its unique optimization for in-context learning through a
retrieval-based in-context example selection (RICE) scheme, as noted by [26].
This approach retrieves images similar to the query image based on image
encodings, and subsequently incorporates top-N (image, text) examples into the
prompt, as described in [16]. However, within the constraints of our research,
we were unable to replicate this specific methodology. Instead, we augmented
the prompts by randomly selecting 3, 6, and 9 examples from the training
dataset as in-context examples. Preliminary evaluations of our randomly
selected few-shot prompting, however, did not reveal any significant
performance improvement over 0-shot evaluation, suggesting the critical role
the retrieval-based approach might play in enhancing OpenFlamingo’s
performance.

We further prompted LLaVa-13B and mPlug-owl-7B by augmenting the questions
prompts with the data tables extracted by the DePlot model (see Section 2.2).
LLaVa-13B received a considerable performance boost compared to mPLug-owl-7B,
indicating that a sufficiently large LLM backbone may better utilize the data-
table strings. In another experiment, we prompted GPT-3.5 with the same
augmented questions prompts (denoted as DePlot + GPT-3.5 in the Table 2) and
observed GPT-3.5 beats the majority of the baselines except LLaVA-13B. This
finding highlights the importance of an expert chart-to-table augmentation
technique.

Lastly, we present the how the best performing model, LLaVa-13B, improve after
being fine-tuned with the SciGraphQA dataset. We refer to this fine-tuned
model as SciGrahQA-baseline, and as seen in Table 2, outperforms the same
model without fine-tuning (CIDEr score of 0.268 vs 0.153). As shown in the
experiments on effect of training set size in the following section, this
baseline model may be constrained by the number of trainable parameters and
the linear projector between image tokens and language tokens in the LLaVA
architecture. Within the scope of this paper, we did not perform model
architecture search or hyperparameter tuning.

Figure 5: Impact of the dataset size and training with prompt augmentation on
fine-tuning performance. For the latter, we constructed 30K DePlot-enhanced
SciGraphQA samples. The LLaVA-13B checkpoint was used as the base model in our
fine-tuning experiments.

We also studied how the size of the training set affects the fine-tuning
performance. We found a positive correlation between the dataset size and
model performance, with the most significant improvement seen when using the
first 50% of the dataset, although training on the full dataset also provided
benefits. The lesser performance gain from 50% to 100% of the dataset may be
attributed to the utilization of the LoRa in the LLM backbone, rather than
full fine-tuning as done in [9]. We found that during the pre-training of the
LLaVA-style linear projector, the loss plateaued early, suggesting further
enhancements could be achieved by increasing trainable parameters in LoRa or
using a more powerful multi-modal adapter. Furthermore, as indicated in
Section 3.3, we further fine-tuned the model with a 30K subset of the dataset
augmented with the data tables. As shown in Figure
LABEL:tab:abiliation_experiment_performance_different_setup, fine-tuning with
this additional subset consistently enhanced the CIDEr score across the board.

##  5 Discussion

We introduced SciGraphQA, a dataset of 300K multi-turn question-answering
dialogues derived from actual academic publications. This dataset is
distinguished by its focus on scientific graphs, its large size (13 times
larger than ChartQA), and the comprehensive textual context provided for each
dialogue (including the paper’s title, abstract, and the paragraph containing
the graph). The quality of the responses in SciGraphQA was verified by an
impressive 8.7/10 GPT-4 average rating on a test set. We believe that this
dataset will be a valuable resource for the development of multimodal large
language models.

From a modeling perspective, we employed a fixed CLIP/ViT vision encoder for
our 0-shot evaluations and fine-tuning results, with mPLUG-owl being the only
exception. CLIP had been extensively pre-trained on a massive dataset of web-
sourced images, so enabling the encoder to adapt during training could have
potentially improved the integration of visual elements into the LLM input
domain, especially for graphs. For example, spatial awareness may be necessary
to identify specific legends, trends, or labels in order to answer
compositional questions like "where do the blue and green lines intersect?".
However, as CLIP was trained to align visual embeddings with captions, its
spatial awareness abilities may be limited. We hypothesize that segmentation-
based approaches, such as employing the Segment-Anything Model [27], to
extract masks and project them into the LLM token embedding space, could
further improve the visual-question-answering capability of MLLMs.

In this study, we confined our evaluations to conventional NLP metrics, which
only take into account pairs of ground truth reference answers and model-
generated predictions. However, GPT-4 could be harnessed as an evaluation tool
due to its capability in evaluating the model’s predicted answers based on the
entire text context in conjunction with the posed questions. In our
experimentation with GPT-4, however, we encountered three notable challenges.
First, inconsistent results were observed when asking for a rating directly
and when soliciting "step-by-step" thinking in the Chain-of-thought prompting
style. Second, limited resources constrained us to run only 100 visual
question-answering prompts for GPT-4 evaluations (for context, the authors of
LLaVa used only 30 samples for their quantitative evaluation [9]. This
hindered conclusive comparisons between the models. Third, the non-
deterministic property of GPT-4 evaluation could render subsequent work on
SciGraphQA incomparable with our published results. Therefore, we reported
only the CIDEr, ROUGE, and BLEU-4 scores, which are freely accessible, open,
and reproducible. We would like to stress that further exploration of GPT-4 as
an evaluation tool is a worthwhile endeavor and could potentially be
integrated as a reward model for reinforcement learning.

Adapting MLLMs to handle out-of-distribution data, particularly in scientific
graph question-answering is a significant challenge given the costly and
complex nature of fine-tuning large foundation models, their sensitivity to
hyperparameters and input data quality, and more importantly the risk of
catastrophic forgetting. We explored the use of LoRA to mitigate the fine-
tuning cost and further used prompt augmentation to alleviate the remaining
issues.

##  6 Conclusion

We introduced SciGraphQA, a novel large-scale dataset for multi-turn question
answering on scientific graphs. With 295K samples derived from over 290,000
academic papers, SciGraphQA represents the largest open-sourced dataset in
this domain with real-world, non-synthetic graphs and with open-ended
conversations not bound by fixed templates of questions or vocabulary. The
multi-turn dialogues of SciGraphQA were generated by prompting a commercial
LLM using comprehensive textual context including titles, abstracts, captions,
and pertinent paragraphs. This extensive context enables dialogues that
capture the nuanced process of explaining complex graphs, unlike most existing
VQA datasets focused on simple facts or graph composition questions. Looking
ahead, SciGraphQA provides a valuable resource for pre-training and
instruction tuning of MLLMs to handle real-world scientific graphs. By
releasing SciGraphQA as an open dataset, we aim to spur progress on this
frontier.

##  7 References

## References

  * Kahou et al. [2017] Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio.  Figureqa: An annotated figure dataset for visual reasoning.  _arXiv preprint arXiv:1710.07300_ , 2017.
  * Kafle et al. [2018] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan.  Dvqa: Understanding data visualizations via question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5648–5656, 2018.
  * Methani et al. [2020] Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar.  Plotqa: Reasoning over scientific plots.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1527–1536, 2020.
  * Masry et al. [2022] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.  Chartqa: A benchmark for question answering about charts with visual and logical reasoning.  _arXiv preprint arXiv:2203.10244_ , 2022.
  * Chaudhry et al. [2020] Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi.  Leaf-qa: Locate, encode & attend for figure question answering.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 3512–3521, 2020.
  * Hsu et al. [2021] Ting-Yao Hsu, C Lee Giles, and Ting-Hao’Kenneth’ Huang.  Scicap: Generating captions for scientific figures.  _arXiv preprint arXiv:2110.11624_ , 2021.
  * H and S [2020] Singh H and Shekhar S.  Stl-cqa: Structure-based transformers with localization and encoding for chart question answering.  _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_ , pages 3275–3284, 2020.
  * Lee et al. [2023] Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  In _International Conference on Machine Learning_ , pages 18893–18912. PMLR, 2023.
  * Liu et al. [2022a] Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos.  Matcha: Enhancing visual language pretraining with math reasoning and chart derendering.  _arXiv preprint arXiv:2212.09662_ , 2022a.
  * Liu et al. [2022b] Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun.  Deplot: One-shot visual language reasoning by plot-to-table translation.  _arXiv preprint arXiv:2212.10505_ , 2022b.
  * Liu et al. [2023] Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al.  On the hidden mystery of ocr in large multimodal models.  _arXiv preprint arXiv:2305.07895_ , 2023.
  * Y et al. [2020] Xu Y, Li M, Cui L, Huang S, Wei F, and Zhou M.  Layoutlm: Pre-training of text and layout for document image understanding.  _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery $\&$ Data Mining_, pages 1192–1200, 2020.  doi:10.1145/3394486.3403172.
  * Akhtar et al. [2023] Mubashara Akhtar, Oana Cocarascu, and Elena Simperl.  Reading and reasoning over chart images for evidence-based automated fact-checking.  _arXiv preprint arXiv:2301.11843_ , 2023.
  * Yin et al. [2023] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.  A survey on multimodal large language models.  _arXiv preprint arXiv:2306.13549_ , 2023.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Awadalla et al. [2023] Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt.  Openflamingo, mar 2023.  URL https://doi.org/10.5281/zenodo.7733589.
  * Ye et al. [2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.  mplug-owl: Modularization empowers large language models with multimodality.  _arXiv preprint arXiv:2304.14178_ , 2023.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Byeon et al. [2022] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset, 2022.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, 2018\.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Yang et al. [2023] Zhishen Yang, Raj Dabre, Hideki Tanaka, and Naoaki Okazaki.  Scicap+: A knowledge augmented dataset to study the challenges of scientific figure captioning.  _arXiv preprint arXiv:2306.03491_ , 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Yang et al. [2022] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang.  An empirical study of gpt-3 for few-shot knowledge-based vqa.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 3081–3089, 2022.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.

##  8 Appendix

Palm-2-chat prompt

You are an AI visual assistant that can analyze a graph in a scientific paper.
You are provided with the OCR-extracted text, the caption of the figure, and
the first paragraph that mentioned the figure.

Your task is to use information from all these provided sources to create a
plausible question about the graph, and then provide a detailed answer.

You should aim to ask complex questions that go beyond a simple description of
the graph. The answer to such questions should require understanding the graph
data, and then reasoning based on background knowledge or interpretation. You
should aim to provide guides and scholarly perspectives based on the graph’s
data and context.

Avoid directly revealing specific details in the question. Make the question
challenging in a way that the user needs to first reason about the graph data
and the context they have derived from the accompanying paper.

Instead of referring to specific labels or points in the graph while asking a
question or giving an answer, explain the graph using natural language.
Include details like data trends, prominent points, and relationships between
different data sets visualized in the graph.

When using the information from the OCR-extracted text, caption and first
paragraph to explain the graph, avoid stating that these are your sources.
Always answer as if you are directly interpreting the graph according to your
AI comprehension, understanding and reasoning. Regarding the format of the
given input: the following context described the image: "from": "human",
"value": contains OCR extracted text listed, and ’Figure (integer) caption of
the image’."from: gpt value: contains the first paragraph in the text that
mentioned the figure.

Examples:

Example 1:

User: The following context described the image: "from": "human", "value":
contains OCR extracted text listed, and ’Figure (integer) caption of the
image’."from: gpt value: contains the first paragraph in the text that
mentioned the figure.

Assistant: Question: How does the percentage of groups that have successfully
acquired a timetable change along with fluctuations in the group size, and
based on the graph, which areas are more likely to have successful timetable
allocation?

Answer: The graph presents a relationship between group size and the success
rate in acquiring a timetable. It highlights a trend where as group size
increases, the likelihood of acquiring a timetable seems to diminish. This is
the overall pattern across all the regions, including Scotland, Central UK,
and South and Central UK.

Example 2:

User:

The following context described the image: "from": "human", "value": "OCR
extracted text list, separated by ’, ’ : SNR $=$ 20DB0.5INR (dB)(a)SNR $=$
40dB1050INŘ (dB)(b)SNR $=$ 40dB1.50.5INR (dB)(c)gap (bits/sec/Hz)gap
(bits/sec/Hz)gap (bits/sec/Hz)$\%$24, SNR, $=$, 20DB, 0.5, INR, (dB), (a),
SNR, $=$, 40dB, 10, 50, INŘ, (dB), (b), SNR, $=$, 40dB, 1.5, 0.5, INR, (dB),
(c), gap, (bits/sec/Hz), gap, (bits/sec/Hz), gap, (bits/sec/Hz), $\%$24 Fig.
7. Numerical analysis: gap between achievable scheme and the outerbounds in
Corollary 1 at (a) SNR $=$ 20dB, (b) SNR $=$ 40dB, and (c) SNR $=$ 60dB for
CFB $=$ 10 bits.<image> Give an elaborate explanation of the image you see"
$\\}$, $\\{$ "from": "gpt", "value": "(i) k : k $\in$ $\\{$1, 2$\\}$, i $\in$
$\\{$1, 2, 3, 4$\\}$). In general, one can solve an optimization problem to
find the optimal choice of power level assignments that maximizes the
achievable sum-rate. We have performed numerical analysis for this
optimization problem. Figure 7 shows the gap between our proposed achievable
scheme and the outer-bounds in Corollary 1 at (a) SNR $=$ 20dB, (b) SNR $=$
40dB, and (c) SNR $=$ 60dB, for CFB $=$ 10 bits. In fact through our numerical
analysis, we can see that the gap is at most 4, 5, and 5.5 bits/sec/Hz for the
given values of SNR, respectively. Note that sharp points in Figure 7 are due
to the change of achievability scheme for different values of INR as described
before." $\\}$

Answer:

Question: What is the relationship between the gap between the achievable
scheme and the outer-bounds in Corollary 1, and the SNR and INR values?

Answer: The graph shows the gap between the achievable scheme and the outer-
bounds in Corollary 1 at (a) SNR $=$ 20dB, (b) SNR $=$ 40dB, and (c) SNR $=$
60dB, for CFB $=$ 10 bits. The gap is at most 4, 5, and 5.5 bits/sec/Hz for
the given values of SNR, respectively. The sharp points in the graph are due
to the change of achievability scheme for different values of INR.

Example 3:

User [insert prompt regarding current image] """


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

<EMAIL_ADDRESS>[1] [1]
[orcid=0000-0002-7505-2521<EMAIL_ADDRESS>[1] [1]
[orcid=0000-0002-1993-9363<EMAIL_ADDRESS>[1] [1]

# Chit-Chat or Deep Talk: Prompt Engineering for Process Mining

Urszula Jessen  Michal Sroka  Dirk Fahland

(2023)

###### Abstract

Abstract: This research investigates the application of Large Language Models
(LLMs) to augment conversational agents in process mining, aiming to tackle
its inherent complexity and diverse skill requirements. While LLM advancements
present novel opportunities for conversational process mining, generating
efficient outputs is still a hurdle. We propose an innovative approach that
amend many issues in existing solutions, informed by prior research on Natural
Language Processing (NLP) for conversational agents. Leveraging LLMs, our
framework improves both accessibility and agent performance, as demonstrated
by experiments on public question and data sets. Our research sets the stage
for future explorations into LLMs’ role in process mining and concludes with
propositions for enhancing LLM memory, implementing real-time user testing,
and examining diverse data sets.

###### keywords:

process mining large language models conversational agents

##  1 Introduction

Process mining endeavours, increasingly prominent across various industry
domain such as healthcare, manufacturing and supply chains, require
stakeholder engagement with diverse skills [1]. This paper propose a novel
framework for creating conversational agents capable of directly extracting
answers from event data, thus reducing the need for interaction with multiply
stakeholders.

Recent advancements in Large Language Models (LLM) have shown great potential
in handling tasks such as interpreting nuanced questions from non-experts.
Despite the apparent simplicity of their usage, the performance of LLMs is
largely reliant on the construction of a well-crafted, nuanced prompt [2].
This paper explores the optimal methods for automating tasks typically
performed by humans, specifically converting an end-user’s question into an
event data query. Conventionally, these tasks involve process analysts who
comprehend the issues related to process optimization, domain experts who
understand the process at hand, and data engineers who can effectively query
the background data to generate accurate answers. The key challenge lies in
assembling all necessary information to construct the query. By emulating and
automating these tasks, this study aims to streamline the workflow and deliver
results directly to the end-user.

In this study, we have constructed a framework, supported by a Large Language
Model (LLM), which emulates the tasks and skills associated with various
process mining participants. We have developed prompts that emulate individual
roles and used an orchestrator to integrate them. Finally, we employed the
corpus of process mining questions collated by Barbieri et al. [3] and
evaluated our framework using the BPI Challenge 2019 dataset [4].Our findings
indicate that in 77% of instances, LLMs were capable of fully or partially
comprehending the question and outlining the appropriate solution.
Furthermore, in 68% of cases, the model provided either the correct or a
partially correct answer.

This paper is organized as follows: Following this introduction, Chapter 2
provides an overview of the background and relevant literature in the field.
Chapter 3 describes our proposed approach for integrating Large Language
Models (LLMs) into conversational interfaces for process mining. Chapter 4
then illustrates an experimental evaluation to confirm the effectiveness of
using LLMs for conversational querying in process mining.

##  2 Background and relevant literature

Process mining a growing branch of data science, enables companies to analyze
their business processes based on their event data. It provides insights into
prevalent performance bottlenecks, inefficiencies, and compliance risks in
operational work. Nonetheless,the discovered data models provide often
complicated, underfit or Spaghetti-like diagrams[5], and the underlying event-
log data can be challenging to analyze through non-experts[3].

This study aims to address this challenge by proposing a novel approach:
integrating large language models (LLMs) into conversational interfaces for
process mining querying. A large language model (LLM) is a natural language
processing tool(NLP) that can understand and generate human-readable text[6].
This approach provides users, without technical or process-mining expertise,
direct access to the insights derived from their event data.

The success of many process mining projects often relies on effective
interaction between diverse stakeholders[7]. The complexity of data and
results, coupled with communication requirements, has amplified the interest
in enhancing usability and understandability for all participants [8, 9]
Extending this, Dumas et al. suggest the need for not only improved usability
but also the development of an Augmented Business Process Management System.
This system would facilitate a conversationally actionable interface between
humans and IT systems [10].

Recently, large language models (LLMs) such as GPT-4 have shown remarkable
progress in natural language processing tasks. They have demonstrated their
capability in generating human-like responses to complex queries and providing
accurate answers to questions [11]. LLMs learn substantial linguistic and
factual world knowledge from a vast corpora of data. They can execute multiply
task, but their performance can be highly variable and in some cases
unsatisfactory[12]. The quality of the factual information obtained from the
LLM depends on carefully designed and nuanced prompts[13]. To address this,
prompt engineering has been proving success [2].Prompt engineering in the
context of large language models involves designing and optimizing prompts to
evoke desired responses from these models. In essence, prompts serve as input
texts that direct the language model to produce specific outputs. The efficacy
of prompt engineering has been demonstrated across a range of applications.
These include knowledge-based question answering [14], essay writing [15],
sentiment analysis [15], and addressing medical challenge problems [16]. This
study investigates the feasibility of using Large Language Models (LLMs) for
conversational querying in process mining. It examines the challenges inherent
to natural language processing (NLP) and explores strategies to mitigate some
of these challenges, notably through prompt engineering and task orchestration
based on varying outcomes.

##  3 Architecture and process for prompt engineering

###  3.1 Generic approach

One of the main challenges in implementing natural language interfaces for
process mining analysis using LLMs is the creation of nuanced, context-
specific task descriptions. These must generate SQL queries that are not only
semantically correct but also precisely address the user’s question. A common
strategy employed to address such problems entails the generation of a prompt,
including the users question, followed by a request to generate SQL which
answers the question. Such prompt typically includes instructions regarding
the format and characteristics expected in the SQL statement. The SQL is then
executed against an event log database and the resulting solution, or an
error, is fed back to the system which decides whether to display it to the
user or withhold it based on predefined criteria. This approach, however,
underperforms in the following scenarios:

  1. 1.

The question contains domain-specific terms.

    1. (a)

Effect: LLM is unable to understand the specific terms and falls back to
generic meaning of the words. Often this leads to incorrect interpretation of
the questions and therefore answer is inappropriate.

  2. 2.

The SQL is malformed.

    1. (a)

Effect: Error in execution of SQL, the user does not see an answer.

  3. 3.

The question is complex and requires multiple SQL statements.

    1. (a)

Effect: The answer is malformed or meaningless SQL, alternatively, the
generated SQL makes many assumptions and does not answer the actual question.

  4. 4.

The data model is not standard or LLM does not have sufficient information
about its structure.

    1. (a)

Effect: the LLM hallucinates column names, makes assumptions about data
formatting. The ensuing SQL statements either exhibit functional inadequacies,
leading to erroneous outcomes, or generate outputs characterized by
nonsensical or stochastic properties.

In the subsequent sections, we present an architectural framework designed to
effectively address the aforementioned limitations.

###  3.2 Architecture

To illustrate the architecture of our solution, consider the following
scenario: we want an LLM to translate the question "What is the main
bottleneck in my process in department A?" into a SQL query. The task requires
specific information to execute. Necessary elements include:

  1. 1.

The data structure, such as the case_concept_name field, timestamp, or event
field names.

  2. 2.

Process-mining-specific information, such as the information on how the
eventlog is built what is the meaning of case, activity or timestamp in that
context

  3. 3.

Domain-specific information, such as the interpretation of the term "process
bottleneck", and instructions on how it is related to the underlying data.
Also as bottleneck can mean different things such as resource or time
bottleneck.

  4. 4.

Data set-specific information, such as the method for calculating duration if
there are no _start, _end timestamps for each activity.

  5. 5.

Mapping of data to domain knowledge e.g. what is the representation of
department A in the given data set.

Figure 1: Architecture of a conversational agent for process mining

Figure 1 depicts the overall architecture of our framework. To generate a
response, a customized prompt, integrating both general guidance on output
structure and specific context, is created for each query. This necessitates
an update to the eventlog DB context (field names, structure, data types) for
each new query. The LLM then establish a list of required information and
employs the context ontology to enhance the prompt. Upon receiving the prompt,
the LLM crafts an SQL query to be run against the Eventlog in the DB. If the
execution is successful, the user is provided with a response; if it fails,
the LLM is given feedback—previous results, the SQL error, and instructions
for correction. This iterative cycle continues until a satisfactory answer is
achieved or a set loop limit is met.

###  3.3 Conceptualizing Interaction with LLM as a Process

In the context of process mining, conversational agents face a significant
challenge: they must effectively integrate diverse skill sets and knowledge
domains to optimize company-specific processes. This requires data engineering
expertise for understanding data contexts, process analysis capabilities to
comprehend terms like "process model," "variant," "deviations," and to
identify bottlenecks and improvement opportunities. Moreover, responding to
domain-specific queries may require domain expert’s knowledge.

Figure 2: The general process of prompt engineering.

Figure 2 illustrates the process we have developed to address the diversity of
required skills and knowledge. One of the unique attributes of LLMs is their
effectiveness in understanding problems when specific keywords are activated,
such as "you are an experienced data engineer with expertise in databases and
SQL queries (…)." By establishing a process and assigning roles to segregate
duties and context, enabling the LLM to focus on its specific task, the
outcomes of multiple queries present a greater specificity. In order to answer
a specific question a range of different prompts has to be created in order to
query different aspects of a question. Once the context has been established,
the initial prompt to the Data Engineer can be formulated. This prompt would
only contain the necessary data and would instruct the LLM to construct the
SQL query and elucidate the reasoning behind the sequential steps of obtaining
an answer to the user’s query111 Additional prompt examples can be found at
https://tinyurl.com/chitchatdeeptalk.

The diagram in Figure 3 outlines the general process constructed in our
proposed framework. Upon receiving a user’s question, the application
(orchestrator) first checks for similar questions in the database. These
questions have previously been submitted to an LLM to generate embeddings for
each of them. If any of these vectors show a similarity greater than 0.9 with
the new question, the orchestrator verifies the success of the previous
answer. If successful, the sql query is executed and answer is immediately
forwarded to the user. Otherwise, the orchestrator forwards the execution to
prompt creation task where the context of different perspectives of the
question is assembled into one general, tailored to the question prompt.

Figure 3: Interaction between user and LLM as a process

Upon crafting the SQL query, it would be dispatched to the database via the
orchestrator. In the event of errors, the Data Engineer prompt would be
supplemented with the database’s error information and executed again. Upon
successful query execution, the user would receive an answer to their
question. After two loops the model would be changed from GPT 3.5-turbo to
GPT-4. The prompt creation component can also receive context from the most
similar questions from the database. If it is not possible to create correct
answer, the orchestrator would ask the user to provide additional feedback to
assist in answering the question.

##  4 Evaluation

We developed our proposed conversational agent architecture and prompt
engineering process using Python, employing GPT-3.5-turbo and GPT4 as Large
Language Models (LLMs). We conducted a study to assess the effectiveness of
this approach, employing a real-world dataset and real-life questions.

###  4.1 Data used in study

Questions in Barbieri et al.’s work[3, 17] were sorted into four categories:
process model, event log data, analysis, and advanced analysis, across 23
perspectives like bottleneck analysis and conformance checking. The question
corpus, primarily in Portuguese and translated by volunteers, included
malformed or unrelated queries. Rather than excluding these flawed
questions,as was done by Barbieri et al.[3, 17], we tested them all on the
LLM, adjusting only year-specific queries to "2019". This was done to explore
the LLM’s potential in comprehending complex or unclear texts, where
traditional rule-based systems may struggle.The questions has been asked
against the BPI Challenge 2019 dataset [4]222The dataset is a log of a the
purchase order handling process for a large multinational company operating
from the Netherlands in the area of coatings and paints. The log contains a
total of 1,595,923 events spread over 51,734 cases. The cases are created at
the level of the position item of a purchase order.There are 42 different
events in the eventlog. The events were executed mostly in 2018, 2019 and
cases are spread over 13,881 variants..

###  4.2 Initial Experiment Results

We executed the experiment in two stages. The initial round, focusing on the
corpus’s first 100 questions, was designed to identify and address weaknesses
in our methodology. After rectifying these, we progressed to the second round,
evaluating the entire corpus. During the first round, we also fine-tuned our
prompts and refined the architecture to minimize human intervention. Insights
gained from this preliminary round were instrumental in enhancing our
architectural approach for the second round involving the full corpus of
questions. Details of these modifications and the enhanced process are
discussed in Section 3.

###  4.3 Adjusted setup and final experiment results

To handle the complexity of defining expected answers, we manually evaluated
the responses using certain criteria, dividing correct answers into fully
answered and partially answered categories.

  * •

A fully answered question is one that correctly addresses the question,
including any additional information provided by the LLM333For example, in
response to a question like ”Which tasks have the maximum duration,” we would
consider the LLM’s answer correct if it not only identifies the tasks with the
maximum duration but also provides additional information such as the specific
duration of these tasks..

  * •

Partially answered questions are those where the executed database query
didn’t generate errors and the resulting answer required some expert
interpretation444For instance, a query requesting the top variants in a
process model might return all variants sorted by frequency, but fail to
specify the top 3, 5, or 10 variants. These types of answers, which could be
refined with extra feedback or contextual information for the LLM, were
considered partially answered..

In contrast, wrong answered questions encompassed those that did not generate
any code (resulting in SQL Server errors), as well as responses that, while
calculated, could not be considered correct.

In our evaluation process for the responses generated by the Large Language
Model (LLM), we identified two primary criteria: Understood and Partially
Understood. These criteria were assessed based on the logical chain of thought
demonstrated by the model in its responses.

  * •

An Understood question is distinguished by the LLM’s comprehensive and
accurate response, suggesting a full grasp of the query. This entails proper
problem identification and subsequent application of relevant operations to
derive a solution555For instance, when asked to identify the primary
bottlenecks in a process, the LLM correctly pinpoints them by measuring the
durations between events, identifying cases that exceed the average duration,
and ultimately, providing the specific bottleneck events. This line of
reasoning displays a thorough understanding of the question..

  * •

Conversely, a Partially Understood question is characterized by the LLM’s
partial grasp of the question or its incomplete or partially correct
response666A prime example is the query, ”How many services do not conform to
the online sales model?” Here, while the LLM rightly perceives the necessity
to contrast event sequences with a predefined online sales model, it may fall
short in comprehensively determining what qualifies as conformance to the
model, or may not produce a complete count of non-conforming services. This
yields a response that only partially addresses the question..

Our findings indicate that the Chain-Of-Thought, particularly for Understood
questions, is not just valuable for end-users but also enlightening for
process analysts. It sheds light on how to break down complex questions into
manageable parts and solve them systematically.

For the examples of "Understood" and "Partially Understood" questions with LLM
reasoning behind, refer to the file in the provided folder at
https://tinyurl.com/chitchatdeeptalk.

Table 1 summarizes the results as how many of the 795 questions were
(partially) answered or understood by the agent compared to the results of
Barbiere at al.[17].

Result | Count | Ratio | Count[17] | Ratio[17]
---|---|---|---|---
Answered | 285 | 36% | 266 | 56%
Partially answered | 254 | 32% | 42 | 9%
Understood | 155 | 19% | 304 | 64%
Partially understood | 459 | 58% | 42 | 9%
Table 1: Our experiments were conducted on a corpus of 795 questions, a
substantial increase from the 476 questions evaluated in the study by Barbieri
et al.[17]. These findings present a comparison between our results and those
from the previous research.

Though the proportion of fully or partially answered questions didn’t increase
significantly, utilizing the LLM permitted the assessment of a larger corpus
of questions. In contrast to the method employed by Barbieri et al.[17], which
solely examined whether the model comprehended the semantic essence of the
question, our study additionally assessed the model’s capacity to formulate
logical reasoning for the correct solution. This approach provides a more in-
depth perspective on the understanding capabilities of the LLM. For future
research, an important objective would be to explore ways to effectively
measure the understanding of LLMs to enhance the results.

###  4.4 Improving the results with process orchestrator and one/few shot
learning

In the large language model the model is already trained on the vast amount of
data and it is not possible to adjust the model in order to increase the
performance of the model in specific tasks. In the case of such models a set
of methods has been developed in order to fine-tune the model to the function
it should fulfill. One of the methods is already explained in the Approach
Chapter and contains of asking the LLM to explain the reasoning behind the
answer Chain-Of-Thought. Similarly the additional methods used to fine-tune
such models is zero- and few-shot learning. 777In the case of LLMs, zero-shot
learning is achieved by providing the model with a prompt that describes only
the task it needs to perform. The model then generates an output based on its
understanding of the prompt and its pre-existing knowledge. Few-shot learnings
achieved by providing the model with one or a few examples of the task it
needs to perform, along with a prompt that describes the task. The model then
generates an output based on its understanding of the prompt and the provided
examples [13]..

Zero vs Few Shot | GPT 3.5 | GPT 4.0
---|---|---
Zero Shot | 49 | 0
Few Shot | 12 | 193
Sum (Partially Answered) | 61 | 193
Zero Shot | 61 | 0
Few Shot | 46 | 178
Sum (Fully Answered) | 107 | 178
Sum (Partially and Fully Answered) | 168 | 371
Table 2: Comparison of GPT 3.5 and GPT 4.0 performance

In our procedure, we utilized various methods to enhance the LLM’s responses.
The results of our experiments are displayed in Table 2. Our application’s
orchestrator attempted to execute the SQL code after each LLM response. In 61
instances, the GPT 3.5-Turbo Model produced the correct answer without
additional shots (Zero Shot) and in 49 the answer was partially correct. If
the code failed to execute, we asked the model to correct the error, supplying
it with the error message. This method further improved 46 (Partially
Answered) and 12 (Fully Answered) responses with GPT 3.5-Turbo.If these
attempts still failed, the orchestrator involved the GPT-4 model, sending the
entire conversation to it. If this approach did not yield the desired outcome,
the orchestrator provided example code or generated additional context (one or
few-shot). This improved an additional 61 cases in total for GPT 3.5-Turbo.
This managed to enhance 193 partially correct responses for GPT 4.0 in few-
shot mode, and a further 178 fully answered questions.

It’s evident from these results that GPT-4 performs best with few-shot
learning. However, the significant cost associated with this model should not
be overlooked. The GPT-3.5 Turbo model was capable of producing satisfactory
results in 168 cases. The overall cost factor must be a part of the evaluation
as well; for our case of answering approximately 800 questions, the total
expenditure was around $60. In a real-world scenario with live data, this
could translate to considerably higher costs. Therefore, the balance between
accuracy and cost is essential when choosing between these solutions.

##  5 Conclusion

In this paper, we’ve delved into the potential of Large Language Models (LLMs)
in enhancing conversational agents for process mining. We’ve proposed a
framework that fosters data-focused conversations and highlighted techniques
that may boost model performance.

We’ve demonstrated the value of incorporating supplementary data to improve
LLM outcomes but underscore that this field still holds considerable untapped
potential. We suggest future research could explore the idea of external
memory for LLMs to retain context over extended interactions, and investigate
the effectiveness of new prompt engineering methods.

To further progress conversational agents, understanding the concept of
"understanding" is critical. Detecting answers that are technically correct
but semantically incorrect is another challenge, referred to as
’hallucination’, that needs to be addressed.

Live system testing with real users could provide valuable insights into real-
world effectiveness. Moreover, studying system responses to varied user
interactions could highlight its resilience and versatility. Examining
additional datasets within the process mining domain may also yield unique
challenges and insights, potentially refining our framework further.

In conclusion, our research paves the way for harnessing LLMs in process
mining, aiming to alleviate some barriers for non-expert users. We aspire that
our work will inspire continued exploration in this area, leading to more
intuitive, accessible, and efficient process mining tools.

###### Acknowledgements.

Thanks to the developers of ACM consolidated LaTeX styles
https://github.com/borisveytsman/acmart and to the developers of Elsevier
updated LaTeX templates https://www.ctan.org/tex-
archive/macros/latex/contrib/els-cas-templates.

## References

  * Reinkemeyer [2020] L. Reinkemeyer,  Process mining in action,  Process Mining in Action Principles, Use Cases and Outloook (2020).
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.,  Language models are few-shot learners,  Advances in neural information processing systems 33 (2020) 1877–1901.
  * Munoz-Gama and Lu [2022] J. Munoz-Gama, X. Lu, Towards a natural language conversational interface for process mining. In Process Mining Workshops: ICPM 2021 International Workshops, Eindhoven, The Netherlands, October 31–November 4, 2021, Revised Selected Papers, Springer Nature, 2022.
  * van Dongen [2019] B. van Dongen, Dataset bpi challenge 2019, 4TU.Centre for Research Data, 2019. URL: https://doi.org/10.4121/uuid:d06aff4b-79f0-45e6-8ec8-e19730c248f1. doi:10.4121/uuid:d06aff4b-79f0-45e6-8ec8-e19730c248f1.
  * van der Aalst [2022] W. M. van der Aalst,  Process mining: a 360 degree overview,  in: Process Mining Handbook, Springer, 2022, pp. 3–34.
  * Johnson et al. [2023] D. Johnson, R. Goodman, J. Patrinely, C. Stone, E. Zimmerman, R. Donald, S. Chang, S. Berkowitz, A. Finn, E. Jahangir, et al.,  Assessing the accuracy and reliability of ai-generated medical responses: an evaluation of the chat-gpt model (2023).
  * vom Brocke et al. [2021] J. vom Brocke, M. Jans, J. Mendling, H. A. Reijers, A five-level framework for research on process mining, 2021\.
  * Martin et al. [2020] N. Martin, J. De Weerdt, C. Fernández-Llatas, A. Gal, R. Gatta, G. Ibáñez, O. Johnson, F. Mannhardt, L. Marco-Ruiz, S. Mertens, et al.,  Recommendations for enhancing the usability and understandability of process mining in healthcare,  Artificial Intelligence in Medicine 109 (2020) 101962.
  * Kubrak et al. [2023] K. Kubrak, F. Milani, A. Nolte,  A visual approach to support process analysts in working with process improvement opportunities,  Business Process Management Journal 29 (2023) 101–132.
  * Dumas et al. [2023] M. Dumas, F. Fournier, L. Limonad, A. Marrella, M. Montali, J.-R. Rehse, R. Accorsi, D. Calvanese, G. De Giacomo, D. Fahland, et al.,  Ai-augmented business process management systems: a research manifesto,  ACM Transactions on Management Information Systems 14 (2023) 1–19.
  * Choi et al. [2023] J. H. Choi, K. E. Hickman, A. Monahan, D. Schwarcz,  Chatgpt goes to law school,  Available at SSRN (2023).
  * Sorensen et al. [2022] T. Sorensen, J. Robinson, C. M. Rytting, A. G. Shaw, K. J. Rogers, A. P. Delorey, M. Khalil, N. Fulda, D. Wingate,  An information-theoretic approach to prompt engineering without ground truth labels,  arXiv preprint arXiv:2203.11364 (2022).
  * Kojima et al. [2022] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa,  Large language models are zero-shot reasoners,  Advances in neural information processing systems 35 (2022) 22199–22213.
  * Yang et al. [2022] Z. Yang, Z. Gan, J. Wang, X. Hu, Y. Lu, Z. Liu, L. Wang,  An empirical study of gpt-3 for few-shot knowledge-based vqa,  in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2022, pp. 3081–3089.
  * Sezgin et al. [2022] E. Sezgin, J. Sirrianni, S. L. Linwood,  Operationalizing and implementing pretrained, large artificial intelligence linguistic models in the us health care system: outlook of generative pretrained transformer 3 (gpt-3) as a service model,  JMIR medical informatics 10 (2022) e32875.
  * Nori et al. [2023] H. Nori, N. King, S. M. McKinney, D. Carignan, E. Horvitz,  Capabilities of gpt-4 on medical challenge problems,  arXiv preprint arXiv:2303.13375 (2023).
  * Barbieri et al. [2022] L. Barbieri, E. Madeira, K. Stroeh, W. van der Aalst,  A natural language querying interface for process mining,  Journal of Intelligent Information Systems (2022) 1–30.

##  Appendix A Online Resources

The additional sources with prompt examples are available under

  * •

https://tinyurl.com/chitchatdeeptalk