11institutetext: University of Bayreuth, Bayreuth, Germany
11email<EMAIL_ADDRESS>22institutetext: University of
Vienna, Vienna, Austria
22email<EMAIL_ADDRESS>
# A Universal Prompting Strategy for Extracting Process Model Information from
Natural Language Text using Large Language Models

Julian Neuberger 11 0009-0008-4244-7659 Lars Ackermann 11 0000-0002-6785-8998
Han van der Aa 22 0000-0002-4200-4937 Stefan Jablonski 11

###### Abstract

Over the past decade, extensive research efforts have been dedicated to the
extraction of information from textual process descriptions. Despite the
remarkable progress witnessed in natural language processing (NLP),
information extraction within the Business Process Management domain remains
predominantly reliant on rule-based systems and machine learning
methodologies. Data scarcity has so far prevented the successful application
of deep learning techniques. However, the rapid progress in generative large
language models (LLMs) makes it possible to solve many NLP tasks with very
high quality without the need for extensive data. Therefore, we systematically
investigate the potential of LLMs for extracting information from textual
process descriptions, targeting the detection of process elements such as
activities and actors, and relations between them. Using a heuristic
algorithm, we demonstrate the suitability of the extracted information for
process model generation. Based on a novel prompting strategy, we show that
LLMs are able to outperform state-of-the-art machine learning approaches with
absolute performance improvements of up to 8% $F_{1}$ score across three
different datasets. We evaluate our prompting strategy on eight different
LLMs, showing it is universally applicable, while also analyzing the impact of
certain prompt parts on extraction quality. The number of example texts, the
specificity of definitions, and the rigour of format instructions are
identified as key for improving the accuracy of extracted information. Our
code, prompts, and data are publicly available111 See
https://github.com/JulianNeuberger/llm-process-generation/tree/er2024 and
https://github.com/JulianNeuberger/pet-to-bpmn-poc.

###### Keywords:

Process Information Extraction Large Language Models AI-assisted Conceptual
Modeling Business Process Modeling

##  1 Introduction

In the field of Business Process Management (BPM), process models are
established tools for designing, implementing, enacting, and analyzing
enterprise processes [12]. However, the manual creation of these models is
very time-consuming and accounts for around 60% of the total time spent on
process management [16]. In order to reduce this effort, the automatic
creation of these models based on a variety of information sources is a
research focus in the field of BPM [5, 16]. In this respect, the paper at hand
contributes to the extraction of process-relevant information from natural
language information sources.

Information on organizational processes is frequently contained in a range of
textual documents, such as process descriptions, rules and regulations, and
work instructions [1, 4]. Recognizing this, a variety of techniques has been
developed that aim to automatically extract process information from texts in
order to subsequently turn it into process models [16, 8, 2]. This two-step
procedure, in which information is extracted first and turned into a process
model second, comes with several advantages in comparison to a direct text-to-
model transformation approach: (1) The result quality can be evaluated with
established means from the information-extraction domain, (2) extracted
information can be transformed in more than one target process modeling
language222This is inspired by the paradigm of interlingua-based machine
translation [27], which reduces the number of translation systems for n
languages from $n^{2}$ to $2n$., and (3) it is possible to use extracted
process information for other purposes such as, for instance, compliance
checking, formal reasoning [3, 26], and process querying [19].

The goal, scope, and challenges of information extraction depend on the input
document type and content, as well as the desired output, i.e., the
information to be extracted. Still, the extraction of process information from
text generally involves: (1) the identification of textual mentions of process
entities, such as activities, process participants, and business objects, and
(2) relations between these entities, such as sequential dependencies,
exclusivity, and assignments (e.g., who performs which step). As an
illustration, Figure 1 shows a fragment of a textual description and two
instantiations of the extraction task, focused on the information necessary
for a model in Business Process Model and Notation
(BPMN)333https://www.omg.org/bpmn/, accessed June 2, 2024, also see Section
4.2. model [9] (upper part) and for declarative process modeling [2] (lower
part). As shown, they involve different entities and relations, which each
need to be inferred from the unstructured textual input.

Figure 1: Fragment of a larger text describing a business process of an
insurance company. Different methodologies may extract different process
relevant information, depending on the target modelling notation or use case.

A key problem is that the extraction of process information is still largely
rule-based [23]. However, crafting useful rules is complicated, requires an
extensive understanding of the process itself, and the rules are hard to
transfer across organizations or text sources. To overcome this, recent work
proposed the use of machine learning techniques [23], though these are
hampered by data scarcity. Work that strives towards using pre-trained
generative LLMs, e.g., GPT-3 [8] aims to circumvent this concern. However, the
work in [8] only presents a preliminary study, with limitations in terms of
analyzed datasets, extracted information, and result discussion. Therefore,
this paper aims to provide deeper insights into the usability of LLMs for
process information extraction and specifically includes the following core
contributions: (I) It presents challenges that make the extraction of process-
relevant information in particular a difficult task (Section 2). (II) As our
main contribution, it proposes a novel, task-specific, and rigorously
empirically validated prompting strategy for solving the aforementioned
information extraction tasks (Section 4.1). (III) For transforming the
extracted information into a process model, we propose a proof-of-concept
algorithm, illustrating that the information extracted is useful for automated
process model generation (Section 4.2). (IV) It provides the currently most
comprehensive study of using LLMs for extracting business process relevant
information from natural language text (Sections 5 and 6). To this end we
rigorously compare our prompting strategy on multiple datasets with state-of-
the-art approaches and achieve up to 7% higher absolute $F_{1}$ scores
compared to machine learning methods and up to 8% compared to rule-based
methods. (V) By testing our prompting strategy with eight state-of-the-art
LLMs, we empirically demonstrate the generality of both our results and the
applicability of our prompting strategy. (VI) An ablation study (Section 6.2)
shows that common best practices in prompt engineering are only of limited use
for process information extraction. Thus, we also define guidelines for using
LLMs for process information extraction (Section 6.3).

The rest of this paper is structured as follows. Section 2 describes the
information extraction tasks and its challenges in detail. Section 3
summarizes the current state of the art in dealing with these tasks. After
that we, describe our prompting strategy, a model generation algorithm, the
experiments, and corresponding results (Section 4–6). Section 7 describes
limitations and future work.

##  2 Task Descriptions and Challenges

In this section, we describe the three main (sub)tasks of process information
extraction (Section 2.1), before highlighting a range of challenges associated
with such extraction and with the use of LLMs for it (Sections 2.2–2.3).

###  2.1 Task descriptions

Our work focuses on three established subtasks of (process) information
extraction from text: Mention Detection (MD), Entity Resolution (ER), and
Relation Extraction (RE) [7, 2, 26, 23].

Mention Detection (MD) is concerned with finding and extracting text fragments
that contain process relevant information, such as activities (or actions),
process-relevant objects or data (i.e., business objects), or involved persons
and departments (i.e., actors). For instance, in Figure 1, the upper example
shows mentions of data, actions, and actor, whereas the lower one focuses on
activities. This definition is similar to Named Entity Recognition (NER),
though we also extract spans not covered by the traditional definition of NER,
e.g., activities.

Entity Resolution (ER) aims to recognize when different mentions refer to the
same process entity. For example, in Figure 1, successful ER would identify
that the word it in “it is examined” corresponds to the claim mentioned in the
previous phrase. Another common example is using ER to recognize that the same
actor (across mentions) performs different steps. ER is a super-set of co-
reference resolution and anaphora resolution [29] and is a crucial step when
dealing with process-related texts, which frequently involve repeated mentions
across sentences or even paragraphs [23].

Relation Extraction (RE) is the task of detecting and classifying relations
between mentions. Relations are usually directed and have one (unary
relation), or two (binary relations) arguments. For instance, the upper
example in Figure 1 shows three kinds of relations: uses signals which data
objects are used by an activity, performer captures which actor performed an
activity, and flow captures a sequential relation between two activities. RE
is crucial when it comes to information extraction in our context, given that
processes inherently involve process steps (i.e., activities) that are
connected to each other through relations. Note that we regard constraint
extraction [2, 26] (CE), which relates to declarative process modeling, as an
RE problem: constraints have one or two arguments, are directed, and carry
type information (e.g., Succession, Init).

###  2.2 Challenges of Process Information Extraction from Text

Information extraction, a common task in natural language processing (NLP),
faces general challenges, which are also well-known in BPM literature [1, 15],
and often central elements of interest in the design of rule-based and
learning-based systems alike [2, 23]. Simply using LLMs for process
information extraction solves some of these challenges and justifies the
investigation of their applicability.

In the context of process-related texts _Linguistic Variance_ means that the
same behavior or process characteristics can be described in a variety of ways
such as, for instance, active and passive voice. _Context Cues_ are a
challenge in that single words can fundamentally alter the meaning of a
process description (e.g., _“first, a claim is created” and inverted semantics
in “finally, a claim is created”_. Processes are typically described in
sequential form, although they usually contain branches (e.g. XOR decision
branches). This results in _Long-distance Relations_ that existing approaches
struggle with [23] or cannot handle [2]. _Implicit and Ambiguous Information_
such as the “examination target” in _“after registering the file in the
database, it needs to be examined”_ needs to be interpreted [3, 15]. Research
indicates a negative correlation between correctness of extracted process
information and _Text Length_ [7]. Finally, the application of deep learning
is hindered by the fact that the largest available data set contains only 45
human-annotated process descriptions [9] (_Small Datasets_).

LLMs are able to overcome the above challenges [32], which is why this paper
analyzes their suitability for process information extraction, as proposed in
previous work [8, 6]. However, LLMs require great care in the formulation of
the input (prompts) [8, 32, 33, 31, 22]. In [22] authors argue: “a good prompt
can be worth hundreds of labeled data points”. For this reason, the core of
the present work lies in the development (Section 4.1) and evaluation
(Sections 5 and 6.2) of suitable prompts for process information extraction.

###  2.3 Challenges of Process Information Extraction Using LLMs

Using LLMs for process information extraction from texts helps with linguistic
challenges, but adds itself several additional challenges. We discuss these
here and reference them later in Section 6.3 to show how we can deal with
them.

(C1) Limited output control. Input and output consist of plain text. Given
that the input for inference is raw text, it inherently suits LLMs for our
tasks (cf. Section 2.1). However, as the expected output should adhere to a
specific schema, it becomes necessary to instruct the LLM to conform to this
schema. Moreover, this principle necessitates a robust output parser, as LLMs
tend to exhibit variability in their output, which presently cannot be
entirely eradicated. Having only limited control over generated output is
especially problematic for the BPM domain, where definitions for relevant
information often overlap, e.g., actions (just predicate) versus activities
(predicate and object).

(C2) Input presentation dependencies. Although LLMs provide an interface for
natural language input, the quantity, form and level of detail must be
carefully matched to the task at hand. The LLM faces the challenge of
determining the importance of the input components. Furthermore, while LLMs
emulate human reasoning, the interpretation of inputs may diverge
significantly from that of human beings, thereby rendering prompt optimization
a trial-and-error process. This challenge is further aggravated by some
elements of process models, that are complex to explain concisely, e.g.,
parallel and exclusive workflows.

(C3) Black-box. Deep learning methods generally suffer from challenges
concerning explainability of predictions [34], which is also true for LLMs.
Contrary to classical learning methods, such as decision trees, LLMs offer no
fail safe mechanism to validate extraction rules. This is problematic for
business process information extraction in particular, since recent work
focuses on “human-in-the-loop” systems [30], where the human must be able to
follow system decisions.

(C4) Data unawareness. In contrast to generative AI models trained on task-
specific data, an LLM is usually not aware of the particular dataset it is
tasked to process. Thus, using LLMs to process a particular dataset requires
to form instructions that precisely describe all relevant details of a
dataset. The generalizing capabilities of LLMs can be an additional hurdle in
this context, especially, when declarative process models are concerned, where
a multitude of constraint types exist. The LLM is likely to know of these
through the pre-training process, and therefore may extract irrelevant ones
for a given dataset.

(C5) Costly experiments. Applying LLMs usually requires usage of commercial
APIs (e.g., OpenAI), which come with downsides: (i) a token limit restricting
the maximum input and output size and (ii) fees based on the number of tokens
processed. In view of the many possible variations in the influencing
parameters, conducting experiments can be cost-intensive. This is especially
true for the BPM domain, where the density of information in process
descriptions is very high, needing many output tokens to extract and encode
it.

##  3 Related Work

Related approaches are divided into rule-based, machine learning (ML)-based,
deep learning-based, and LLM-based process information extraction. An approach
is considered related if it solves at least one of the tasks in Section 2.1.

Rule-based approaches. Rule-based approaches leverage linguistic features to
extract information from natural language process descriptions through
explicitly coded mapping rules. For instance, Friedrich et al.’s seminal work
[16] employs syntax features and word information from a lexical database to
identify patterns at both sentence and document levels for BPMN model
creation. Other approaches like those in [28, 26] adopt similar techniques for
automatic text annotation, employing regular expressions for syntactic
dependency trees, and part-of-speech tags, showcasing superior performance on
novel datasets. Additionally, [2] presents a rule-based technique, currently
leading in extracting declarative process models from raw text using syntax
parsing and word-level features. Similar advancements are seen in [14] and
[9], the latter integrating ML-based entity MD with subsequent rule-based RE.
Furthermore, [21] focuses on extracting Dynamic Condition Response (DCR)
graphs. Recent studies suggest that while rule-based approaches can be
tailored to specific tasks and data sets, they can hardly deal with ambiguity
and linguistic variance. [8, 23].

ML-based approaches. [23] presents a ML extraction pipeline based on [9] and
is used as a baseline for our comparative evaluation (Section 6). The deep-
learning approach presented in [25] classifies text fragments analyzing the
input text on several levels of granularity. However, extracting these
fragments is not part of the approach, which simplifies the task of MD to
mention classification, i.e., locating the information to extract is omitted.
Though the work presented in [6] overcomes this limitation and outperforms the
approach, it does not support RE. In general, techniques of this paradigm
either struggle to deal with linguistic variability and ambiguity, or they
require vast amounts of training data, making them particularly unfeasible for
small datasets (see Section 2.2).

LLM-based approaches. Bellan et al. [8] utilize pre-trained LLMs to cope with
data scarcity, yet their approach exhibits three primary weaknesses: (i) It is
restricted to a subset of entity types, namely activities, participants, a
performs relation, and a direct-consequences relation, (ii) it lacks strict
output formatting, hindering automated result processing , unlike our
prompting strategy , and (iii) its evaluation is limited to 7 of the 45
process descriptions from the PET dataset, whereas we evaluate our modular
prompt on the entire dataset, plus two validation datasets. Thus, direct
comparison between [8] and our work is not feasible. Nonetheless, as in [8],
our modular prompt also descriptive instructions with input examples
accompanied by their expected outputs. Although process models are generated
using LLMs in [18], the work is not comparable to ours, as [18] requires human
involvement and only supports the extraction of activities and their
arrangement in a directed graph (e.g., actors and data are missing).

##  4 LLM-based Process Model Extraction from Text

To generate process models from natural language texts, we first extract
process information using LLMs and a novel prompting strategy (Section 4.1).
Next, we describe a baseline algorithm for generating a process model (Section
4.2).

###  4.1 Process Information Extraction with LLMs

Extracting process information with LLMs requires a _prompt design_ that
addresses the challenges mentioned in Section 2.3. Thus, a prompt structure
consisting of the three modules _Context_ , _Task Description_ and
_Restrictions_ is described below at the example of the Mention Detection
task. However, the prompting strategies for the remaining tasks (Section 2.1)
are analogous.

####  4.1.1 High-Level Prompt Structure.

LLMs take freely formulated text as input, which is called the prompt. To this
end we base our prompts on an ablation study (Section 6.2), which is used to
identify beneficial and detrimental prompt components. To do this, we first
need a modular prompt design so that we can specifically remove individual
components in the study to examine their benefits and ultimately to only keep
the advantageous components. Adhering to the best practices outlined in [31],
our initial prompt design is structured into three modules (see Figure 2): (A)
a context description framing the process information extraction task on a
high level, (B) a detailed task description, and (C) constraints that further
restrict the context and the output format, and contains disambiguation hints.
To design potentially relevant components for all three modules we rely on
general design patterns [22, 33, 31, 11]. Therefore, in the next subsection,
the three modules are specified and discussed in terms of how they address the
LLM-specific challenges outlined in Section 2.3.

Figure 2: Modular prompt structure (underlined = task-specific content, boxes
= design pattern,  = useful,  = non-useful,  = use in prompt engineering).

####  4.1.2 Context and Task Description.

In module (A) we use the persona design pattern [33] to control the language
style of generation results. We assign it the role of a process modeling
expert. This is followed by the context manager design pattern [33], which
includes a general description of the information extraction task (i.e.,
objectives and a description of the input specifics). This limits the
information basis the LLM may use, mitigating the risk of hallucination.

Module (B) is mainly concerned with defining the specifics of the process
information extraction task. Its backbone is the creation of a meta
language[33], which defines the types of elements to extract from the input
text. Figure 2 provides an example that defines activities and actors as
mention types and a relation called actor performer that associates actions
with their performers, following our running example from Figure 1. Another
widespread best practice is known as chain of thought (CoT) [32, 31], whereby
the actual task is broken down into individual steps. Thus, our prompt divides
the relation extraction task into two steps that separate the extraction of
mentions from the prediction of their relations and a third step, which
combines two more best practices, i.e., generating a list of facts about the
process and reflection about the results [33]. These cause the LLM to
elaborate both on the input and on its own output, which allows experts to
validate the extracted information and has also been shown to have a positive
impact on extraction performance [32, 31].

####  4.1.3 Restrictions.

The last prompt module (C) defines expectations towards the LLM’s output.
Additional considerations include rules for the extraction task, such as that
an actor can only be described as such if the action it performs is also named
(compare Figure 2). Disambiguation hints are particularly useful for
information types that are hard to distinguish from other information types or
irrelevant information. In Figure 2 it is intended to guide the LLM what makes
up an Activity, if the input gives additional, irrelevant specifics. Prompts
further include a schematic definition of a formal output format [33], which
for the exemplary prompt is a tuple of a relation type, a source mention and a
target mention, each separated by a pipe symbol. The definition is
complemented by an (out-of-domain) example. Finally, few-shot prompting [22,
31] dynamically adds examples for input and corresponding output. For the
current paper few-shot samples are pairs of raw textual process descriptions
and a task-dependent set of process-relevant information (e.g., actor
mentions). This design pattern and best practice is known to alleviate the
issue of data unawareness of LLMs [17, 20].

###  4.2 Process Model Generation - a Proof of Concept

Based on the format instructions contained in the prompt design (Section 4.1),
information can be extracted automatically through a parser. This information
can then be automatically converted into a process model in BPMN language,
which was chosen because it is a widely used industry standard.

Figure 3: Automatically generated model for information in document _doc-3.3_
(PET). Layout has been adjusted slightly for readability and compactness.

Our conversion algorithm works in three major stages. First, in the
_Consolidation_ phase, we assign the conditions to the mentions of their
respective decisions. Next, all mentions of gateways are merged, if they refer
to the same decision point. Finally, for all activities that are not assigned
an executing actor, we find the closest actor mention in the text left of it.
In the second stage, the _Vertex_ stage, we create process elements for all
mentions, e.g., _Tasks_ , _Data Objects_ , _Swimlanes_ , etc. The final
_Linking_ stage connects related elements, e.g., successive tasks with
_Sequence Flows_ , if they are located in the same Swimlane, or _Message
Flows_ otherwise. We also create _Data Associations_ between Data Objects and
Tasks, adding the label of the Data Object to the label of the Task, for
labels like “ _register the claim_ ”.

Figure 3 shows the resulting BPMN model after applying this algorithm on the
information contained in document _doc-3.3_ of the (PET) dataset we later use
in our experiments. Note, that the layouting is adjusted manually, as well as
the wrong assignment of Task _sent back a claim_ to the Swimlane of Actor _A
claims officer_. These issues illustrate why synthesizing a model from
extracted process information is still challenging, which warrants further
research beyond the scope of this paper.

##  5 Experiment Setup

In this section we define the experiments we run to evaluate the usefulness of
LLMs for process information extraction 444Source code:
https://anonymous.4open.science/r/llm-process-generation-2140 and
https://anonymous.4open.science/r/pet-to-bpmn-poc-B465.. It covers an overview
of the datasets we use, including the respective baselines, and a definition
of metrics we apply.

We use three well-known datasets for evaluating our prompts. One of these
(PET) represents the current state of the art, both in terms of size, as well
as the process information techniques developed for it. The other datasets
feature different characteristics, making them relevant for validation
experiments. This lets us gain insights into the robustness of an LLM as a
process information extractor, as well as how it behaves when applied to other
process modeling languages. We call the best approaches for extracting the
information from these datasets baselines, and use them in Section 6 as
comparisons with various LLMs.

PET [9]: This is the largest dataset currently available. It contains 45
documents with annotations for information especially useful for creating
process models in BPMN. These include 7 types of mentions such as activities,
actors, data objects, but also 6 relation types. These cover the behavioural
process perspective (Flow), data perspective (uses, and organizational
perspective (actor, performer). Additionally, this dataset features relations
that span multiple sentences. It therefore tests the ability of approaches to
reason across wider spans of text. We use an extended version of this dataset,
which includes data for the ER task, as presented in [23]. The currently best
approach for extracting information is using conditional random fields for MD,
a pre-trained neural co-reference resolver for ER, and a decision tree
ensemble for RE [23]. We use scores as reported in [23], which have
corresponding publicly available code and can be reproduced with it.

DECON [2]: This collection of 17 textual process descriptions is annotated
with a set of 5 Declare [24] Constraint types between business process
relevant activities. Additionally, constraints may be negated, as well as
unary, i.e., constraining a single action. Annotations are given on a sentence
level, and only sentences that describe at least one constraint are contained
in the dataset. The expected (ground-truth) activities are already transformed
into Declare-conform phrases, i.e., the activity description “The claim is
registered” should be extracted as “register claim”. It does not contain an
approach for MD in isolation, and only contains mentions of type action. The
authors of [2] propose a rule-based approach combining multiple NLP
techniques, e.g., typed dependency relations.

ATDP [26]: This dataset uses 18 textual descriptions, that largely overlap
with the ones from [2], but also contains sentences, that describe no
constraint. As such, this dataset tests approaches for their ability to judge
whether or not sentences contain relevant information, before extracting
constraints. Furthermore, the set of constraints was expanded to eight types.
Additionally, this dataset also provides annotations of actions, conditions,
entities, and events, which we used in an MD setting. Quishpi et al. proposed
a rule-based ensemble of patterns for MD and CE on typed dependency structures
[26].

We use the well established metrics Precision $P$ and Recall $R$ for our
experiments. $P$ is a measure of how well an extraction approach is able to
avoid false positives, i.e., assigning the wrong type to mentions and
relations, or extracting them, where they are not expected. $R$ on the other
hand measures how much of the expected information (true positives) is found.
The two metrics are typically aggregated via their harmonic mean
$F_{1}=2\frac{P\cdot R}{P+R}$. Following [26], we use
$P=\frac{\\#correct}{\\#pred}$ and $R=\frac{\\#correct}{\\#gold}$, with
$\\#correct$ as the number of correct predictions, $\\#pred$ the number of
total predictions, and $\\#gold$ as the number of expected mentions or
relations. For a fair assessment, we count predictions as correct in exactly
the way described by the work we compare the LLM to.

##  6 Results

Table 1 shows the results we observed when running the experiments as
described in Section 5 with an optimized prompt, that follows the
recommendations we found in our study of best practices (Section 6.2). All
results use GPT-4o, the latest version of OpenAI’s GPT with $temperature=0$.
$top\\_p$ is unchanged, as per OpenAI’s recommendation, when using temperature
based sampling. 555see OpenAI’s source code, accessed June 3, 2024

For the reference dataset PET, our experiments show that GPT-4o is capable of
an absolute $F_{1}$ score improvement of 5% for MD, 22% for ER, and 17% for
RE. Remarkably, for RE, GPT-4o is able to match and outperform the machine
learnt baseline, which was trained on 36 manually annotated documents [23],
without any labeled data (zero-shot). For MD it reaches similar scores, even
when not given any examples, compared to the machine learnt baseline, which
was trained using 36 manually annotated documents [9]. For real-world
application this means that LLMs can be used in business process information
extraction scenarios, even if the organization has not a single manually
annotated training example. This is an exciting find, as it promises
significant speed-up of model creating tasks of practitioners across business
domains.

When evaluating on the validation datasets (cf. Section 5), we found that
GPT-4o is able to match and out-perform the rule-based systems in all cases,
most notably improving $F_{1}$ scores for RE on dataset DECON by an absolute
8%. Our result for MD on dataset DECON has no corresponding baseline, as the
authors of [2] did not report values for MD in isolation. Errors and
ambiguities are common in dataset ATDP hindering machine learning methods in
learning valid extraction rules. This also adversely affects the extraction
accuracy of GPT-4, when extracting the same types of constraints in the ATDP
dataset compared to the DECON dataset. We discuss this further in Section 6.3.
Since the importance of ER only recently gained attention [26, 23], the
reference dataset PET is currently the only dataset providing data for
evaluation of this task.

| Dataset |  DECON |  ATDP |  PET
---|---|---|---|---
| Metric |  $P$ |  $R$ |  $F_{1}$ |  $P$ |  $R$ |  $F_{1}$ |  $P$ |  $R$ |  $F_{1}$
Mention Detection | Baseline | no baseline |  0.62 |  0.82 |  0.71 |  0.73 |  0.64 |  0.69
Zero-shot |  0.72 |  0.75 |  0.73 |  0.58 |  0.77 |  0.66 |  0.65 |  0.71 |  0.68
1-shot |  0.87 |  0.80 |  0.83 |  0.63 |  0.77 |  0.69 |  0.72 |  0.75 |  0.73
3-shot |  0.88 |  0.79 |  0.83 |  0.68 |  0.79 |  0.73 |  0.72 |  0.77 |  0.74
Entity Resolution | Baseline |  |  |  0.55 |  0.51 |  0.52
Zero-shot |  |  |  0.67 |  0.55 |  0.60
1-shot |  |  |  0.76 |  0.70 |  0.73
3-shot | no data | no data |  0.79 |  0.70 |  0.74
Relation Extraction | Baseline |  0.77 |  0.72 |  0.74 |  0.58 |  0.64 |  0.61 |  0.79 |  0.66 |  0.72
Zero-shot |  0.66 |  0.75 |  0.70 |  0.49 |  0.66 |  0.57 |  0.88 |  0.85 |  0.86
1-shot |  0.76 |  0.82 |  0.79 |  0.58 |  0.73 |  0.64 |  0.90 |  0.89 |  0.89
3-shot |  0.79 |  0.85 |  0.82 |  0.58 |  0.72 |  0.64 |  0.90 |  0.89 |  0.89
Table 1: Results for each dataset and the different extraction stages,
compared to baseline results using GPT-4o.

###  6.1 Model Comparison

We originally developed our prompts for GPT-4 version _GPT-4-0125-preview_ ,
to assess how well our prompting strategy generalizes to other models we
prompted a total of eight models for the MD and RE tasks on PET. We selected
models following AlpacaEval 666see https://tatsu-lab.github.io/alpaca˙eval/,
last accessed May 30, 2024., which is designed for testing the instruction
following capabilities of LMMs[13]. At the time of writing model _YI Large
Preview_ was not publicly accessible and could not be considered in our
comparison, even though it ranked third on AlpacaEval.

Results for the comparison can be found in Table 2. We set the temperature for
all models to $0$. All GPT models perform on similar levels, with the
exception of GPT3.5, which is significantly smaller compared to GPT-4 models.
For the zero-shot RE task GPT3.5 even failed to produce responses for most
documents, leading to very low recall. Claude3 Opus seems to be as capable as
GPT-4, its smaller variant Sonnet performs significantly worse on zero-shot
tasks, but is able to produce comparable results given three examples. Llama 3
70B Instruct is an open-weight model and could be run locally, i.e., it is
useful for using our prompting strategy in scenarios where sending data to an
API is not possible. Llama 3 70B Instruct seems to be nearly as capable as the
closed-weights Claude 3 Sonnet and is therefore viable in a few-shot setting.

Task |  PET MD |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
(Zero-shot) |  PET MD |  |  |  |  |  |  |  |  |
(3-shot) |  PET RE |  |  |  |  |  |  |  |  |
(Zero-shot) |  PET RE |  |  |  |  |  |  |  |  |
(3-shot) |  |  |  |  |  |  |  |  |  |  |  |
Model |  $P$ |  $R$ |  $F_{1}$ |  $P$ |  $R$ |  $F_{1}$ |  $P$ |  $R$ |  $F_{1}$ |  $P$ |  $R$ |  $F_{1}$
GPT-4o |  0.58 |  0.69 |  0.63 |  0.68 |  0.77 |  0.72 |  0.88 |  0.85 |  0.86 |  0.90 |  0.89 |  0.89
GPT-4-2024-04-09 |  0.63 |  0.67 |  0.65 |  0.73 |  0.76 |  0.74 |  0.87 |  0.79 |  0.83 |  0.89 |  0.88 |  0.88
GPT-4-0125-preview |  0.65 |  0.71 |  0.68 |  0.72 |  0.77 |  0.74 |  0.87 |  0.85 |  0.86 |  0.89 |  0.87 |  0.88
GPT-3.5-0125 |  0.35 |  0.50 |  0.42 |  0.51 |  0.70 |  0.59 |  0.51 |  0.06 |  0.11 |  0.74 |  0.64 |  0.69
Claude 3 Opus |  0.55 |  0.72 |  0.63 |  0.66 |  0.80 |  0.73 |  0.86 |  0.85 |  0.86 |  0.92 |  0.91 |  0.91
Claude 3 Sonnet |  0.46 |  0.65 |  0.54 |  0.63 |  0.78 |  0.70 |  0.78 |  0.67 |  0.72 |  0.91 |  0.87 |  0.89
Llama 3 70B Instruct |  0.56 |  0.64 |  0.59 |  0.62 |  0.71 |  0.67 |  0.76 |  0.66 |  0.70 |  0.88 |  0.81 |  0.84
Qwen1.5 72B Chat |  0.32 |  0.33 |  0.33 |  0.53 |  0.65 |  0.59 |  0.61 |  0.65 |  0.63 |  0.74 |  0.77 |  0.75
Table 2: Comparison of our prompts across different models. Best results per
task and metric are set bold.

###  6.2 Ablation Study

We conduct an ablation study to assess the usefulness of the best practices
presented in Section 4.1 and to measure the impact of the prompt’s main
components. This study is run on the reference dataset (PET), as it is the
largest one and used by recent publications [9, 8, 23]. To obtain a baseline
for the tasks of MD and RE, we use a prompt that implements the best practices
as shown in Figure 2 and run it on the _GPT-4-0125-preview_ model. We then
purposefully remove specific components from this prompt, namely the format
examples, the context manager, the persona, the definition of mention and
relation types (meta language), the instruction to think in several steps
(chain of thought), any disambiguation hints, and the instruction to generate
explanations (reflection) and a fact check list about the process.
Additionally, we also use a prompt with very short descriptions of relations
and types (balancing brevity and specificity). We run each prompt in the zero-
shot setting and record the observed $F_{1}$ score, as well as the parsing
errors that occurred.

Table 3 provides detailed results. Removing examples has a significant
negative effect ($-0.22$ for MD and $-0.07$ for RE), mainly rooted in the
number of parsing errors that are made (919 for MD), as well as directionality
of relations for the RE task (confusing source and target mentions). Removing
the context manager and persona only has minor effects ($\pm 0.01$ per task),
suggesting lower relevance for process information extraction compared to
other NLP settings.

| Mention Detection (MD) | Relation Extraction (RE) |
---|---|---|---
Experiment |  Relative $F_{1}$ |  Absolute $F_{1}$ |  Parsing Errors |  Relative $F_{1}$ |  Absolute $F_{1}$ |  Parsing Errors |  Useful
Baseline | – | $0.59$ | $0$ | – | $0.77$ | $0$ |
No Format Examples | $-0.22$ | $0.37$ | $919$ | $-0.07$ | $0.70$ | $1$ | ✓
No Context Manager | $0.01$ | $0.60$ | $0$ | $-0.01$ | $0.76$ | $0$ |
No Persona | $0.01$ | $0.59$ | $1$ | $0.01$ | $0.78$ | $0$ |
No Meta Language | $-0.09$ | $0.49$ | $2$ | $-0.05$ | $0.72$ | $0$ | ✓
No Chain of Thought | $-0.01$ | $0.57$ | $1$ | $-0.02$ | $0.75$ | $0$ | ✓
No Disambiguation | $-0.03$ | $0.55$ | $0$ | $-0.01$ | $0.76$ | $1$ | ✓
No Reflection | $0.04$ | $0.63$ | $0$ | $0.02$ | $0.79$ | $0$ | $\bigstar$
No Fact Check List | $0.03$ | $0.62$ | $1$ | $-0.02$ | $0.75$ | $0$ | $\bigstar$
Very Short Prompt | $-0.04$ | $0.54$ | $1$ | $-0.03$ | $0.74$ | $0$ | ✓
Table 3: Changes in $F_{1}$ score of GPT-4, without specific prompt components
given in Figure 2. Column relative $F_{1}$ shows difference to the baseline
prompt, Useful shows a ✓, if we recommend this component in prompts for
process information extraction and $\bigstar$ for prompt engineering and data
curating only.

In addition to removing prompt components, we also tested using GPT in an
older, less capable, but much cheaper version, GPT-3.5. Running the baseline
prompt, results in a significant drop in extraction quality, with $F_{1}=0.27$
for MD and $F_{1}=0.56$ for RE. Splitting the baseline prompt into multiple
prompts, each focusing on only one mention type, lets us prompt GPT-4
repeatedly for the same document. These highly specialized prompts are called
“agents”, which pass information between each other. For example, we instruct
the first agent to extract Actions, which are passed to other agents
extracting Actors and Business Objects respectively. This lets us exploit the
inherent dependency between these elements. If no Action is detected in a
sentence, then there is likely no relevant Actor or Business Object, even if
there are nouns that would qualify from a linguistic standpoint. This way of
prompting leads to an absolute improvement of $+0.08$ in $F_{1}$ score for
both the MD and RE tasks.

###  6.3 Lessons Learned

Using LLMs for extracting process relevant information brings with it a
category of challenges, which we already discussed in Section 2.3. Solving
these is paramount for successful application of LLMs. In this section we
discuss how we approached these challenges and what lessons we learned.

(C1) Limited output control. The expected output format, as well as the form
of extracted information, can mainly be influenced by the prompt components
Meta Language and Format Examples. Adding these results in significantly
improved $F_{1}$ scores, ($+0.22$ and $+0.05$ respectively for MD on PET).
These improvements are explained by less parsing errors (919 less for MD on
PET), and better recall and precision in detecting mentions. LLMs also run the
risk of being “stochastic parrots”, simply synthesizing linguistically correct
phrases, based on their training data [10]. In our experiments we observed
changes in $F_{1}$ of maximally $-0.02$ for rephrased prompts. This indicates
robustness of our prompts and suitability of LLMs as an tool for business
process information extraction.

(C2) Black-box. A valuable advantage of utilizing LLMs is their ability to
reflect, thereby providing justification for their generated results. Figure 4
shows three examples of justifications for extraction results in the ATDP
dataset (see Section 2.1). Case I shows the ideal outcome where prediction and
expected constraint are identical. Note, that the justification even refers to
the meta-language provided in the prompt (compare Section 4.1). In case II,
the prediction and the gold standard constraint do not match, because of an
error in the gold standard data, following [2], which defines completing a
process as a _meta action_ that can not be part of any constraint. The dataset
creators are alerted of this issue by the LLM, since it plausibly justifies
why _send out report_ marks the end of a process instance. Finally, in case
III the extraction result is controversial, since the sentence is ambiguous.
If we consider the term _immediately_ to encompass both actions, they are
constrained by an _existence_ constraint. Alternatively, viewing _check
quantity_ as a subtask of _process part list_ suggests only one action needs
modeling. Using such reflective explanations make LLMs useful for “human-in-
the-loop” systems, which are already applied in fields like process mining
[30].

(C3) Input presentation dependencies. Adding more text to prompts sometimes
has an adverse effect, reducing extraction quality (Section 2.3). This makes
optimizing prompts difficult, since it is not clear, if adding additional
disambiguation hints or longer definitions would improve the result. Using
partial extraction prompts helps with this issue, as the sections regarding
Meta Language Creation can be focused on a few types. Depending on the task,
there may even be inter-dependency between information, that can be
efficiently exploited in this way.

Figure 4: Reflection example with Predicted and Gold standard constraints: (I)
perfect match, (II) gold standard error, (III) ambiguous case

(C4) Data unawareness. This issue arises, when LLMs are used in a zero-shot
setting. There, the components Meta Language, and Format Examples are the only
ways to “teach” the model how to perform the task. Applying the pattern of
few-shot prompting, i.e., using labeled data in a few-shot setting was
beneficial. This makes the use of an LLM more akin to training a machine
learning model, but with significantly lower data requirements. In our
experiments, three examples were sufficient to achieve better extraction
results than those of machine learning models trained with more than ten times
of the data.

(C5) Costly experiments. This is a major drawback of LLM based process
information extraction. The most capable LLMs are hosted as cloud-based
solutions and are priced per token. We found that limiting the number of
examples to 1 resulted in the best cost-value ratio. Additionally, our
experiments showed that leaving out the prompt components Context Manager,
Persona, and Disambiguation is a valid way to limit the number of tokens sent
per request, albeit with potential minor decreases in extraction accuracy.
Prompting LLMs without the request for a Fact List, nor Explanations for
extracted information greatly reduces the amount of tokens as well, especially
useful after prompt engineering or data curating (during “inference”).
Alternatively one can switch to cheaper models, i.e., LLama 3 72B, if the drop
in performance is acceptable.

##  7 Conclusion

Summary. This paper presents an extensive study on the usefulness of LLMs for
the extraction of process information from natural language text. We collected
linguistic challenges and discuss how LLMs are uniquely fit for solving them.
We also discussed challenges that arise through the use of LLMs and show how
other communities propose to deal with these (or similar) concerns through
prompt engineering. We present experimental results on three process
information extraction datasets, which at least match the current state of the
art on these datasets and in most cases improve it by as much as 8% in the
$F_{1}$ metric. This shows the suitability of LLMs as a method for extracting
business process relevant information from natural language process
descriptions. To flesh out this notion, we analyze how well our prompting
strategy can be applied to different LLMs without changing them, showing their
universal nature. We make all our code, prompts, and LLM answers available, to
support further research.

Limitations. A limitation of our work is that the list of prompt components we
present may not be exhaustive, and they may have interactions that our
ablation study does not capture. Additionally, some models suffer from
hallucinations, especially Qwen1.5 and Llama 3, which hallucinate non-existent
entity and relation types – 20 and 37 instances in the worst cases
respectively. However, the severity of this problem diminishes in the few-shot
setting (0 and 4 instances respectively in the worst case). We plan on
analyzing how our prompts could be enhanced to improve instruction following
for these models. Lastly, the current pricing models prohibit large-scale
application of the most capable model to process information extraction.
Alternatives (e.g., Llama) avoid this issue, but require more labeled examples
to reach comparable levels of performance.

Future Work. In future work we aim to use LLMs as tools to support labeling of
new data. Current datasets are limited in origin, i.e., they usually describe
processes from municipalities or small service providers. We plan to analyze
the ability of LLMs to generalize beyond the domains with available labeled
data and highlight the promising flexibility observed in our current
experiments. Finally, using GPT-4o’s image processing and generation
capabilities could be a promising line of research for direct text to model
transformation.

## References

  * [1] Van der Aa, H., Carmona Vargas, J., Leopold, H., Mendling, J., Padró, L.: Challenges and opportunities of applying natural language processing in business process management. In: COLING (2018)
  * [2] van der Aa, H., Di Ciccio, C., Leopold, H., Reijers, H.A.: Extracting declarative process models from natural language. In: CAiSE (2019)
  * [3] Van der Aa, H., Leopold, H., Reijers, H.A.: Checking process compliance against natural language specifications using behavioral spaces. IS (2018)
  * [4] van der Aa, H., Leopold, H., van de Weerd, I., Reijers, H.A.: Causes and consequences of fragmented process information: Insights from a case study. In: AMCIS (2017)
  * [5] van der Aalst, W.M.P.: Process Mining - Data Science in Action, Second Edition. Springer (2016). https://doi.org/10.1007/978-3-662-49851-4, https://doi.org/10.1007/978-3-662-49851-4
  * [6] Ackermann, L., Neuberger, J., Jablonski, S.: Data-driven annotation of textual process descriptions based on formal meaning representations. In: CAiSE (2021)
  * [7] Ackermann, L., Neuberger, J., Käppel, M., Jablonski, S.: Bridging research fields: An empirical study on joint, neural relation extraction techniques. In: CAiSE (2023)
  * [8] Bellan, P., Dragoni, M., Ghidini, C.: Extracting business process entities and relations from text using pre-trained language models and in-context learning. In: EDOC (2022)
  * [9] Bellan, P., Ghidini, C., Dragoni, M., Ponzetto, S.P., van der Aa, H.: Process extraction from natural language text: the PET dataset and annotation guidelines. In: NL4AI (2022)
  * [10] Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: Can language models be too big? In: ACM FAccT (2021)
  * [11] Cui, L., Wu, Y., Liu, J., Yang, S., Zhang, Y.: Template-based named entity recognition using bart. arXiv preprint arXiv:2106.01760 (2021)
  * [12] Davies, I., Green, P., Rosemann, M., Indulska, M., Gallo, S.: How do practitioners use conceptual modeling in practice? Data & Knowledge Engineering 58(3), 358–380 (2006)
  * [13] Dubois, Y., Li, C.X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P.S., Hashimoto, T.B.: Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems 36 (2024)
  * [14] Ferreira., R.C.B., Thom., L.H., Fantinato., M.: A semi-automatic approach to identify business process elements in natural language texts. In: ICEIS (2017)
  * [15] Franceschetti, M., Seiger, R., López, H.A., Burattin, A., García-Bañuelos, L., Weber, B.: A characterisation of ambiguity in bpm. In: International Conference on Conceptual Modeling. pp. 277–295. Springer (2023)
  * [16] Friedrich, F., Mendling, J., Puhlmann, F.: Process model generation from natural language text. In: CAiSE (2011)
  * [17] Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., McHardy, R.: Challenges and applications of large language models. arXiv preprint (2023)
  * [18] Kourani, H., Berti, A., Schuster, D., van der Aalst, W.M.: Process modeling with large language models. arXiv preprint arXiv:2403.07541 (2024)
  * [19] Leopold, H., van der Aa, H., Pittke, F., Raffel, M., Mendling, J., Reijers, H.A.: Searching textual and model-based process descriptions based on a unified data format. SoSym 18, 1179–1194 (2019)
  * [20] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33, 9459–9474 (2020)
  * [21] López-Acosta, H.A., Hildebrandt, T., Debois, S., Marquard, M.: The process highlighter: From texts to declarative processes and back. In: CEUR Workshop Proceedings. pp. 66–70. CEUR Workshop Proceedings (2018)
  * [22] Min, B., Ross, H., Sulem, E., Veyseh, A.P.B., Nguyen, T.H., Sainz, O., Agirre, E., Heintz, I., Roth, D.: Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys 56(2), 1–40 (2023)
  * [23] Neuberger, J., Ackermann, L., Jablonski, S.: Beyond rule-based named entity recognition and relation extraction for process model generation from natural language text. In: CoopIS (2023)
  * [24] Pesic, M., Schonenberg, H., Van der Aalst, W.M.: Declare: Full support for loosely-structured processes. In: 11th IEEE international enterprise distributed object computing conference (EDOC 2007). pp. 287–287. IEEE (2007)
  * [25] Qian, C., Wen, L., Kumar, A., Lin, L., Lin, L., Zong, Z., Li, S., Wang, J.: An approach for process model extraction by multi-grained text classification. In: CAiSE (2020)
  * [26] Quishpi, L., Carmona, J., Padró, L.: Extracting annotations from textual descriptions of processes. In: BPM 2020 (2020)
  * [27] Richens, R.H.: Interlingual machine translation. The Computer Journal 1(3), 144–147 (1958)
  * [28] Sànchez-Ferreres, J., Burattin, A., Carmona, J., Montali, M., Padró, L., Quishpi, L.: Unleashing textual descriptions of business processes. SoSyM (2021)
  * [29] Sukthanker, R., Poria, S., Cambria, E., Thirunavukarasu, R.: Anaphora and coreference resolution: A review. Information Fusion (2020)
  * [30] Ter Hofstede, A.H., Koschmider, A., Marrella, A., Andrews, R., Fischer, D.A., Sadeghianasl, S., Wynn, M.T., Comuzzi, M., De Weerdt, J., Goel, K., et al.: Process-data quality: The true frontier of process mining. ACM JDIQ (2023)
  * [31] Törnberg, P.: Best practices for text annotation with large language models. arXiv preprint arXiv:2402.05129 (2024)
  * [32] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. NIPS (2022)
  * [33] White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., Schmidt, D.C.: A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023)
  * [34] Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., Zhu, J.: Explainable ai: A brief survey on history, research areas, approaches and challenges. In: NLPCC (2019)


# Energy dependence of $J/\psi$ production in pp collisions with the PACIAE
model

Kai-Fan Ye  School of Physics and Information Technology, Shaanxi Normal
University, Xi’an 710119, China  Key Laboratory of Quark and Lepton Physics
(MOE) and Institute of Particle Physics, Central China Normal University,
Wuhan 430079, China  Qiang Wang  School of Physics and Information Technology,
Shaanxi Normal University, Xi’an 710119, China  Jia-Hao Shi  School of Physics
and Information Technology, Shaanxi Normal University, Xi’an 710119, China
Zhi-Ying Qin  School of Physics and Information Technology, Shaanxi Normal
University, Xi’an 710119, China  Wen-Chao Zhang<EMAIL_ADDRESS>School of Physics and Information Technology, Shaanxi Normal University, Xi’an
710119, China  An-Ke Lei  Key Laboratory of Quark and Lepton Physics (MOE) and
Institute of Particle Physics, Central China Normal University, Wuhan 430079,
China  Zhi-Lei She  Wuhan Textile University, Wuhan 430200, China  Yu-Liang
Yan  China Institute of Atomic Energy, P. O. Box 275 (10), Beijing 102413,
China  Ben-Hao Sa  China Institute of Atomic Energy, P. O. Box 275 (10),
Beijing 102413, China

###### Abstract

In this work we investigate the $J/\psi$ production in proton-proton
collisions at the center-of-mass energy ($\sqrt{s}$) equal to 2.76, 5.02, 7, 8
and 13 TeV with a parton and hadron cascade model PACIAE 2.2a. It is based on
PYTHIA but extended considering the partonic and hadronic rescatterings before
and after hadronization, respectively. In the PYTHIA sector the $J/\psi$
production quantum chromodynamics processes are selected specially and a bias
factor is proposed correspondingly. The calculated total cross sections, the
differential cross sections as a function of the transverse momentum and the
rapidity of $J/\psi$ in the forward rapidity region reproduce the
corresponding experimental measurements reasonably well. In the mid-rapidity
region, the double differential cross sections at $\sqrt{s}=$ 5.02, 7 and 13
TeV are also in a good agreement with the experimental data. Moreover, we
predict the double differential cross section as well as the total cross
section of $J/\psi$ at $\sqrt{s}=$ 8 TeV, which could be validated when the
experimental data is available.

##  I Introduction

$J/\psi$ is the lightest vector charmonium meson. The suppression of $J/\psi$
production was proposed as a probe to investigate the hot and dense medium,
named quark-gluon plasma (QGP), created in ultra-relativistic nucleus-nucleus
collisions[1]. The $J/\psi$ production could also be suppressed due to the
cold nuclear matter effects, such as modifications of nuclear parton
distribution functions[2, 3]. In order to disentangle the hot and cold medium
effects, it is necessary to understand the $J/\psi$ production in proton-
proton (pp) collisions where the initial state effects are absent.

The hadronic $J/\psi$ production mainly results from the gluon-gluon
scattering into a $c\bar{c}$ pair in hard scattering and related initial- and
final-state radiations, which is described with the perturbative Quantum
Chromodynamics (pQCD). The hadronization of $c$ and $\bar{c}$ pair into
$J/\psi$ is a soft process, which cannot be dealt with pQCD. The $J/\psi$
production was extensively investigated at colliders such as the Tevatron[4,
5, 6, 7], RHIC[8] and LHC[9, 10, 11, 12, 13, 14, 15, 16]. Several theoretical
approaches, such as the color singlet model[17], the non-relativistic QCD
model [18, 19] and the color evaporation model[20, 21] have been utilized to
describe the experimental data. They differ mostly in the treatment of non-
perturbative evolution of the $c\bar{c}$ pair into the bound state $J/\psi$.
However, none of these models could simultaneously describe the polarization,
the transverse momentum ($p_{\rm T}$) spectrum and the energy dependence of
cross sections for $J/\psi$[13]. The $J/\psi$ production was also investigated
by Monte Carlo simulations. For example, in Refs.[22, 23], the $J/\psi$
production as a function of charged particle multiplicity in pp collisions at
the center-of-mass energy ($\sqrt{s}$) equal to 7 and 13 TeV was investigated
in the mid-rapidity region by PYTHIA 6.4[24] and 8.2[25], respectively. It was
found that PYTHIA 8.2 could depict the correlation between the $J/\psi$ yield
and the charged particle multiplicity ($dN_{\rm ch}/d\eta$) while PYTHIA 6.4
can not do it well. This could be due to the reason that the multiparton
interaction is described differently between them. In Ref.[23], the
correlation between the $J/\psi$ yield and $dN_{\rm ch}/d\eta$ was also
explored by EPOS3 event generator[26, 27]. In Ref.[28], the authors utilized a
modified ultra-relativistic quantum molecular dynamics (UrQMD) transport
model[29, 30] to study the $J/\psi$ suppression in high-multiplicity pp
collisions at $\sqrt{s}=$ 7 TeV.

As a complementary study to those performed in Refs.[22, 23, 26, 24, 25, 28],
we use a parton and hadron cascade model PACIAE 2.2a[31] to investigate the
$J/\psi$ production in pp collisions at $\sqrt{s}=$ 2.76, 5.02, 7, 8 and 13
TeV. In the model the $J/\psi$ production QCD process will be selected
specially and a bias factor will be introduced for the simulation sample
correspondingly. The calculated total cross sections, the differential cross
sections as a function of the transverse momentum and the rapidity of $J/\psi$
in the forward rapidity region will be, respectively, compared to the
corresponding experimental measurements. Meanwhile, in the mid-rapidity
region, the double differential cross sections at $\sqrt{s}=$ 5.02, 7 and 13
TeV will also be compared with the experimental data. Moreover, the total and
double differential cross sections of $J/\psi$ at $\sqrt{s}=$ 8 TeV will be
predicted.

This paper is organized as follows. In sect. II, we briefly introduce the
PACIAE model. In sect. III, we describe the method to produce the simulation
sample and to compare the sample with experimental data. In sect. IV, results
and discussions are presented. Finally, the conclusion is given in sect. V.

Figure 1:  (Colour online) A sketch of the physical routines in high energy pp
collisions[32].

##  II The PACIAE model

The PACIAE 2.2a model is designed for elementary collisions. This model is
based on PYTHIA 6.4 but further considers the partonic rescattering before
hadronization and the hadronic rescattering after hadronization. It divides
the ultra-relativistic energies pp collisions into four stages: parton
initiation, parton rescattering, hadronization, and hadron rescattering. Fig.1
is a sketch of the physical routines in a high energy pp collision.

In the first stage, the initial partonic states are created by temporarily
switching off the string fragmentation in PYTHIA, breaking down these strings
and splitting up the diquarks (anti-diquarks) randomly. Together with the
initial state radiation and final state radiation, this partonic matter then
undergoes hard scatterings and parton rescatterings, where the leading order
(LO) pQCD parton-parton interaction cross sections[33, 34] are employed. A $K$
factor is introduced to consider higher order effects and non-perturbative
corrections for LO-pQCD parton–parton differential cross sections. After the
parton rescattering, the partonic matter is converted into hadrons by the
string fragmentation[24] or the coalescence model[35]. Then followed is the
hadronic rescattering where the method of two-body collision[36] is utilized
to rescatter hadrons until the kinetic freeze-out happens.

##  III The method

The ALICE collaboration had published the inclusive $J/\psi$ production in the
forward rapidity region with $2.5<y<4$ in pp collisions at $\sqrt{s}=$ 2.76,
5.02, 7, 8 and 13 TeV in Refs.[9, 12, 13, 14, 16]. The collaboration also
presented the inclusive $J/\psi$ production in the mid-rapidity region with
$-0.9<y<0.9$ in pp collisions at $\sqrt{s}=$ 5.02, 7 and 13 TeV in Refs.[11,
12, 15]. Table 1 shows the references for the differential cross sections for
inclusive $J/\psi$ at a given energy. The inclusive $J/\psi$ yield contains a
prompt component, which includes the direct $J/\psi$ production and the feed-
down contribution from the decay of heavier charmonium states, as well as a
non-prompt component from the weak decay of beauty hadrons.

Table 1: Summary of references for the inclusive $J/\psi$ data at a given energy. The second (third) column is for the differential cross sections at forward rapidity (mid-rapidity). $\sqrt{s}$ | forward rapidity | mid-rapidity
---|---|---
2.76 TeV |  $2.5<y<4$[9] | –
5.02 TeV |  $2.5<y<4$[16] |  $-0.9<y<0.9$[11]
7 TeV |  $2.5<y<4$[13] |  $-0.9<y<0.9$[12]
8 TeV |  $2.5<y<4$[14] | –
13 TeV |  $2.5<y<4$[16] |  $-0.9<y<0.9$[16]
Table 2:  Summary of cross sections for each $J/\psi$ ’color-singlet’ selection process, $\sigma_{\rm sel}$, in one pp collision at a given energy in PACIAE. The last row lists the total $J/\psi$ cross-section in one event at a given energy. This cross section can be calculated by the weighted sum of the cross sections for the $J/\psi$ selection process, with the weight taken as the product of the fraction of heavier charmonium state’s radiative decay into $J/\psi$ (i.e. branching ratio (BR)) and the sampling probability. Processes | 2.76 TeV | 5.02 TeV | 7 TeV | 8 TeV | 13 TeV | BR | Sampling probablity
---|---|---|---|---|---|---|---
$\sigma_{\rm sel}(\mathrm{\mu b})$
$gg\rightarrow J/\psi g$ | 32.871 | 62.654 | 64.798 | 83.623 | 132.548 | 100.0% | 2.184%
$gg\rightarrow\chi_{0c}g$ | 241.830 | 499.519 | 533.856 | 700.603 | 1169.812 | 1.4% | 21.054%
$gg\rightarrow\chi_{1c}g$ | 138.797 | 312.055 | 345.636 | 460.827 | 823.761 | 34.3% | 5.000%
$gg\rightarrow\chi_{2c}g$ | 265.777 | 552.470 | 593.993 | 779.302 | 1302.222 | 19.0% | 23.101%
$gg\rightarrow\chi_{0c}$ | 294.944 | 554.780 | 572.512 | 744.378 | 1202.169 | 1.4% | 23.525%
$gg\rightarrow\chi_{2c}$ | 255.635 | 484.268 | 502.488 | 651.972 | 1060.098 | 19.0% | 25.083%
$gg\rightarrow J/\psi\gamma$ | 0.929 | 1.872 | 1.856 | 2.588 | 4.232 | 100.0% | 0.053%
$\sigma_{J/\psi}(\mathrm{\mu b})$ | 28.631 | 57.349 | 60.822 | 79.524 | 132.111 | – | –

In this paper, as a complementary work to the study performed by event
generators such as PYTHIA 6.4, PYTHIA 8.2, EPOS3 and UrQMD[22, 23, 28], we
investigate the $J/\psi$ production at different energies with PACIAE 2.2a. In
the model, a ’menus’ of subprocesses for the $J/\psi$ production is composed
with the sets of ’MSEL=0’ and ”MSUB($i$)=1, $i$=86, 87, 88, 89, 104,105,106’
in PYTHIA 6.4. This selection corresponds to the following ’color-singlet’
processes, $gg\rightarrow J/\psi g$, $gg\rightarrow\chi_{0c}g$,
$gg\rightarrow\chi_{1c}g$, $gg\rightarrow\chi_{2c}g$,
$gg\rightarrow\chi_{1c}$, $gg\rightarrow\chi_{2c}$, $gg\rightarrow
J/\psi\gamma$. The higher mass excited states, like the $\chi_{0c}$,
$\chi_{1c}$ and $\chi_{2c}$ may subsequently decay to $J/\psi$. The ’color-
octet’ processes are not considered in the work, as we only focus on the
production of $J/\psi$ in the low and intermediate $p_{\rm T}$ range. Relative
to the default simulation where non-single diffractive parton-parton
subprocesses are considered, the selection of the $J/\psi$ production
processes will introduce a bias sampling. In order to get rid of this bias
effect, for example, we rescale the cross sections in simulation with the
first moment of the $J/\psi$ $p_{\rm T}$ spectra and match it with the
rescaled cross section in experiment,

$\dfrac{\left.\frac{\mathrm{d}\sigma_{J/\psi}}{\mathrm{d}y}\right|_{\mathrm{sim}}}{{\int}p_{\mathrm{T}}\left.\frac{\mathrm{d^{2}}\sigma_{J/\psi}}{\mathrm{d}p_{\mathrm{T}}\mathrm{d}y}\right|_{\mathrm{sim}}\mathrm{d}p_{\mathrm{T}}}=\dfrac{\left.\frac{\mathrm{d}\sigma_{J/\psi}}{\mathrm{d}y}\right|_{\mathrm{exp}}}{{\int}p_{\mathrm{T}}\left.\frac{\mathrm{d^{2}}\sigma_{J/\psi}}{\mathrm{d}p_{\mathrm{T}}\mathrm{d}y}\right|_{\mathrm{exp}}\mathrm{d}p_{\mathrm{T}}}.$
(1)

Thus, in order to compare with the experimental data, we propose a bias factor
$B$ and apply it to the cross section from simulation,

$\displaystyle B$
$\displaystyle=\dfrac{{\int}p_{\mathrm{T}}\left.\frac{\mathrm{d^{2}}\sigma_{J/\psi}}{\mathrm{d}p_{\mathrm{T}}\mathrm{d}y}\right|_{\mathrm{exp}}\mathrm{d}p_{\mathrm{T}}}{{\int}p_{\mathrm{T}}\left.\frac{\mathrm{d^{2}}\sigma_{J/\psi}}{\mathrm{d}p_{\mathrm{T}}\mathrm{d}y}\right|_{\mathrm{sim}}\mathrm{d}p_{\mathrm{T}}}$
(2)
$\displaystyle=\dfrac{{\int}p_{\mathrm{T}}\left.\frac{\mathrm{d^{2}}\sigma_{J/\psi}}{\mathrm{d}p_{\mathrm{T}}\mathrm{d}y}\right|_{\mathrm{exp}}\mathrm{d}p_{\mathrm{T}}}{{\int}\left.p_{\mathrm{T}}\frac{\mathrm{d^{2}}N_{J/\psi}}{\mathrm{d}p_{\mathrm{T}}\mathrm{d}y}\cdot\frac{\sigma_{J/\psi}}{N_{J/\psi}}\right|_{\mathrm{sim}}\mathrm{d}p_{\mathrm{T}}}$

where $N_{J/\psi}$ is total $J/\psi$ yield in simulation. The total $J/\psi$
cross section in one pp collision at a given energy in simulation,
$\sigma_{J/\psi}$, is returned by the PYTHIA sector in PACIAE 2.2a. It is
tabulated in Table 2 in the appendix. Also listed in the table are the cross
sections for the ’color-singlet’ selection process in one event at a given
energy. The derivation of the relation between the $J/\psi$ double
differential yield and differential cross section in simulation is presented
in the appendix. Similar definition is applied to the bias factor for the
rapidity distribution of $J/\psi$.

In PACIAE, the model parameters are chosen as the default values in the PYTHIA
6.4 model, except for the $K$ factor, which is determined by fitting the
simulation to the experimental data in pp collisions at a given energy with a
least $\chi^{2}s$ method. For the $J/\psi$ differential cross section as a
function of $p_{\rm T}$ in the forward rapidity region, the $\chi^{2}$ is
defined as

$\chi^{2}=\sum_{i=1}^{N}{\frac{(y^{i}-f(p^{i}_{\rm
T};K))^{2}}{(\Delta^{i})^{2}}},$ (3)

where $N$ is the number of data points, $y^{i}$ is the experimental value in
the $i^{\rm th}$ bin, $f(p^{i}_{\rm T};K)$ is the simulation value in the same
bin, $\Delta^{i}$ is the total error of the experimental data. Table 3
tabulates the $\chi^{2}$ values divided by the number of degrees of freedom
($\chi^{2}/ndf$) for different $K$ factors at a given energy. The best $K$
factor at that energy is determined by minimizing the corresponding $\chi^{2}$
values and is shown in boldface type.

Table 3: Summary of $\chi^{2}/ndf$ for different $K$ factors at a given energy. The minimum $\chi^{2}/ndf$ is shown in boldface type. $K$ | 2.76 TeV | 5.02 TeV | 7 TeV | 8 TeV | 13 TeV
---|---|---|---|---|---
$\chi^{2}/ndf$
1.1 | 5.39/6 | 14.06/6 | 23.85/6 | 17.09/6 | 66.34/7
1.2 | 4.74/6 | 10.93/6 | 19.57/6 | 16.77/6 | 71.67/7
1.3 | 6.11/6 | 17.25/6 | 16.62/6 | 16.40/6 | 63.80/7
1.4 | 4.81/6 | 13.55/6 | 18.87/6 | 17.08/6 | 60.94/7
1.5 | 6.86/6 | 11.44/6 | 21.23/6 | 13.40/6 | 58.07/7
1.6 | 3.46/6 | 13.10/6 | 19.18/6 | 13.71/6 | 58.83/7
1.7 | 5.92/6 | 10.92/6 | 19.54/6 | 14.85/6 | 54.86/7
1.8 | 7.04/6 | 14.89/6 | 19.25/6 | 14.37/6 | 65.33/7
1.9 | 5.96/6 | 15.22/6 | 20.05/6 | 14.94/6 | 71.83/7

##  IV Results and discussions

Fig.2 shows the inclusive $J/\psi$ cross sections from experiment (solid
symbols) at forward rapidity in pp collisions with $\sqrt{s}=$ 2.76, 5.02, 7,
8 and 13 TeV, which are compared with the results from PACIAE (hollow
symbols). The error bars attached to the data points represent the total
uncertainties of the $J/\psi$ cross sections. It is found that the simulation
well describes the experimental data, except for few points in the range with
$3<p_{\rm T}<7$ GeV/c. Fig.3 presents the inclusive $J/\psi$ cross sections as
a function of rapidity in the forward region. In order to make the simulation
comparable with experimental data, we employed the same $p_{\mathrm{T}}$ cuts
in the former as those applied in the latter at each energy for $J/\psi$. It’s
observed that the simulation well reproduces the corresponding experimental
measurements.

Figure 2:  (Colour online) Inclusive $J/\psi$ cross sections as a function of
$p_{\mathrm{T}}$ at forward rapidity in pp collisions with $\sqrt{s}=$ 2.76,
5.02, 7, 8 and 13 TeV. The solid symbols are experimental data taken from
Refs. [9, 16, 13, 14, 16]. The hollow symbols are the results from the PACIAE
model.  Figure 3:  (Colour online) Inclusive $J/\psi$ cross sections as a
function of $y$ in pp collisions at $\sqrt{s}=$2.76, 5.02, 7, 8 and 13 TeV.
The solid symbols are experimental data taken from Refs. [9, 16, 13, 14, 16].
The hollow symbols are the results from the PACIAE model.

As described in sect. III , due to the selection of the $J/\psi$ production
processes in the simulation, the bias factor is introduced when comparing the
results with experimental data. The bias factors for the $J/\psi$ $y$ and
$p_{\rm T}$ distributions are presented in Fig.4. It is shown that at a given
energy the former is larger than the latter. In addition, for both cases the
bias factor slightly decreases with energy. In order to guide the eyes, we
parameterize the dependence of $B$ on energy with a fit function
$B=a+b\textrm{ln}(\sqrt{s}/\textrm{TeV})$, where $a$ and $b$ are free
parameters. The fitted curves are also presented in the figure. The upper
(lower) panel of Fig.5 shows the $J/\psi$ $p_{\rm T}$-integrated (total)
inclusive cross sections as a function of $\sqrt{s}$ in the forward rapidity
region. It is observed that the simulation from the PACIAE model is in a good
agreement with the experimental data. Moreover, there is a steady increase of
the cross sections when the collision energy increases. At a given energy, the
total cross section is about a factor of 1.5 larger than the $p_{\rm
T}$-integrated one. The bias factor for the $J/\psi$ $y$ distribution is also
1.5 times larger than that for the $p_{\rm T}$ distribution. This factor is
exactly the bin width of the forward rapidity range.

Figure 4:  The energy dependence of the bias factor in the forward region in
pp collisions at $\sqrt{s}=$ 2.76, 5.02, 7, 8 and 13 TeV. The left (right)
triangles represent the bias factors calculated from the $p_{\mathrm{T}}$
($y$) distributions. The dash and dotted curves represent the fit functions.
Figure 5:  (Colour online) Upper (Lower) panel: $J/\psi$ $p_{\rm
T}$-integrated inclusive cross section (total inclusive cross section) as a
function of $\sqrt{s}$ in the forward rapidity region in pp collisions. The
solid (hollow) symbols represent the cross sections calculated from the
experimental data (the PACIAE model). Figure 6:  (Colour online) Inclusive
$J/\psi$ cross sections as a function of $p_{\mathrm{T}}$ at mid-rapidity in
pp collisions at $\sqrt{s}=$5.02, 7 and 13 TeV. The solid symbols are
experimental data taken from Refs. [11, 12, 16]. The hollow symbols are the
results from the PACIAE model. The gray band represents the prediction of the
double differential $J/\psi$ cross section in pp collisions at $\sqrt{s}$= 8
TeV from the PACIAE model.  Figure 7:  The energy dependence of the bias
factor in the mid-rapidity region in pp collisions at $\sqrt{s}=$ 5.02, 7 and
13 TeV. The dash curve represents the parameterization. The bias factor at
$\sqrt{s}=$ 8 TeV is an interpolation with this parameterization.

Fig.6 presents the inclusive $J/\psi$ cross section as a function of
$p_{\mathrm{T}}$ in the mid-rapidity region $-0.9<y<0.9$ in pp collisions with
$\sqrt{s}=$ 5.02, 7 and 13 TeV. In order to match with the experimental data,
the simulation from the PACIAE model has to be multiplied by the bias factor,
which is defined in Eq.(LABEL:eq:Rescaledfactor). It is found that the
simulation agrees with experimental data reasonably well. Fig.7 shows the
energy dependence of the bias factor at mid-rapidity in pp collisions at
$\sqrt{s}=$5.02, 7 and 13 TeV. This dependence can be parameterized as
$B=(0.764\pm 0.159)+(-0.173\pm 0.068)\textrm{ln}(\sqrt{s}/\textrm{TeV})$. With
this parameterization, we evaluate the bias factor for the double differential
$J/\psi$ cross section at mid-rapidity in pp collisions at $\sqrt{s}=$ 8 TeV.
It is $0.404\pm 0.032$. Applying this bias factor to the simulation sample
from the PACIAE model, we could predict the double differential $J/\psi$ cross
section in pp collisions at $\sqrt{s}=$ 8 TeV. It is presented as the gray
band in Fig.6. Fig.8 shows the energy dependence of the total inclusive
$J/\psi$ cross section at mid-rapidity in pp collisions at $\sqrt{s}=$ 5.02, 7
and 13 TeV. At a given energy, the total cross section in simulation agrees
with that in experimental data within uncertainties. The inclusive $J/\psi$
cross section at $\sqrt{s}=$ 8 TeV is predicted by integrating the double
differential cross section over $p_{\mathrm{T}}$ and $y$. The predictions on
the double differential $J/\psi$ cross section as well as the total cross
section could be validated when the experimental data is available.

Figure 8:  (Colour online) Total inclusive $J/\psi$ cross section at mid-
rapidity in pp collisions with $\sqrt{s}=$ 5.02, 7 and 13 TeV. The solid
(hollow) symbols represent the cross sections calculated from the experimental
data (the PACIAE model). The cross section at $\sqrt{s}=$ 8 TeV is a
prediction based on the integration of the double differential cross section
over the mid-rapidity region.

##  V Conclusions

In this paper, we have investigated the $J/\psi$ production in pp collisions
at $\sqrt{s}=$ 2.76, 5.02, 7, 8 and 13 TeV with the parton and hadron cascade
model PACIAE 2.2a. It is based on PYTHIA but differs from the latter in the
addition of the parton rescattering before hadronization and the hadron
rescattering after hadronization. In the model the $J/\psi$ production QCD
processes are selected specially and the bias factor is proposed and applied
to the simulation sample correspondingly. The calculated $J/\psi$ total cross
section, the differential cross section as a function of $p_{\rm T}$ and $y$
in the forward rapidity region agree with the corresponding experimental
measurements reasonably well. In the mid-rapidity region, the double
differential cross sections of $J/\psi$ at $\sqrt{s}=$ 5.02, 7 and 13 TeV also
reproduce the experimental data. Moreover, the double differential cross
section as well as the total cross section of $J/\psi$ at $\sqrt{s}=$ 8 TeV
are predicted. They could be validated when the experimental data is
available. We realize the necessary of the individually investigating the
partonic and hadronic rescattering effects, which will be presented in our
next work.

###### Acknowledgements.

We would like to thank Prof. X.-M. Zhang and D.-M. Zhou at Central China
Normal University for their valuable discussions. This work is supported by
the research fund from the School of Physics and Information Technology at
Shaanxi Normal University, by the Scientific Research Foundation for the
Returned Overseas Chinese Scholars, State Education Ministry, by Natural
Science Basic Research Plan in Shaanxi Province of China (program No. 2023-JC-
YB-012) and by the National Natural Science Foundation of China under Grant
Nos. 11447024, 11505108 and 12375135.

## Appendix

In this appendix, we present a detailed derivation of the relation between the
$J/\psi$ double differential yield and differential cross section in
simulation. The total $J/\psi$ cross section in one event at a given energy in
simulation, $\sigma_{J/\psi}$, can be expressed as follows:

$\sigma_{J/\psi}=\sigma_{J/\psi}^{ev}\dfrac{N_{J/\psi}}{N_{ev}}=\sigma_{J/\psi}^{ev}N_{J/\psi}^{per},$
(A1)

where $N_{J/\psi}$ ($N_{ev}$) is the number of $J/\psi$s (events) in the
simulation sample. In this work, for each energy, $N_{ev}$ is set to be
$10^{6}$. $N_{J/\psi}^{per}$ is the number of $J/\psi$s in one event,
$N_{J/\psi}^{per}=N_{J/\psi}/N_{ev}$. $\sigma_{J/\psi}^{ev}$ is the cross
section of producing one $J/\psi$ in one event,
$\sigma_{J/\psi}^{ev}=\sigma_{J/\psi}/N_{J/\psi}^{per}$. With Eq.(A1), the
double differential cross section $d^{2}\sigma_{J/\psi}/dp_{\rm T}\mathrm{d}y$
is written as

$\dfrac{\mathrm{d^{2}}\sigma_{J/\psi}}{\mathrm{d}p_{\mathrm{T}}\mathrm{d}y}=\dfrac{1}{N_{\mathrm{ev}}}\dfrac{\mathrm{d^{2}}N_{J/\psi}}{\mathrm{d}p_{\mathrm{T}}\mathrm{d}y}\sigma_{J/\psi}^{\mathrm{ev}}=\dfrac{\mathrm{d^{2}}N_{J/\psi}}{\mathrm{d}p_{T}\mathrm{d}y}\dfrac{\sigma_{J/\psi}}{N_{J/\psi}}.$
(A2)

This is exactly the relation between the double differential yield and
differential cross section in simulation.

## References

  * Matsui and H.Satz [1986] T. Matsui and H.Satz, Phys. Lett. B 178, 416 (1986).
  * Eskola  _et al._ [2017] K. J. Eskola _et al._ , Eur. Phys. J. C 77, 163 (2017).
  * Kovarik  _et al._ [2016] K. Kovarik _et al._ , Phys. Rev. D 93, 085037 (2016).
  * Acosta  _et al._ [2005] D. Acosta _et al._ (CDF Collaboration), Phys. Rev. D 71, 032001 (2005).
  * Abulencia  _et al._ [2007] A. Abulencia _et al._ (CDF Collaboration), Phys. Rev. Lett. 99, 132001 (2007).
  * Abachi  _et al._ [1996] S. Abachi _et al._ (D0 Collaboration), Phys. Lett. B 370 (1996).
  * Abbott  _et al._ [1999] B. Abbott _et al._ (D0 Collaboration), Phys. Rev. Lett. 82 (1999).
  * Adare  _et al._ [2007] A. Adare _et al._ (PHENIX Collaboration), Phys. Rev. Lett. 98, 232002 (2007).
  * Abelev  _et al._ [2012a] B. Abelev _et al._ (ALICE Collaboration), Phys. Lett. B 718, 295 (2012a).
  * Abelev  _et al._ [2015] B. Abelev _et al._ (ALICE Collaboration), Phys. Lett. B 748, 472 (2015).
  * Acharya  _et al._ [2019] S. Acharya _et al._ (ALICE Collaboration), J. High Energy Physics 2019(10), 84 (2019).
  * Aamodt  _et al._ [2011] K. Aamodt _et al._ (ALICE Collaboration), Phys. Lett. B 704, 442 (2011).
  * Acharya  _et al._ [2014] S. Acharya _et al._ (ALICE Collaboration), Eur. Phys. J. C 74, 2974 (2014).
  * Adam  _et al._ [2016] J. Adam _et al._ (ALICE Collaboration), Eur. Phys. J. C 76, 184 (2016).
  * Acharya  _et al._ [2021] S. Acharya _et al._ (ALICE Collaboration), Eur. Phys. J. C 81, 1121 (2021).
  * Acharya  _et al._ [2017] S. Acharya _et al._ (ALICE Collaboration), Eur. Phys. J. C 77, 392 (2017).
  * Baier and Ruckl [1981] R. Baier and R. Ruckl, Phys. Lett. B 102, 364 (1981).
  * Bodwin  _et al._ [1995a] G. T. Bodwin, E. Braaten, and G. P. Lepage, Phys. Rev. D 51, 1125 (1995a).
  * Bodwin  _et al._ [1995b] G. T. Bodwin, E. Braaten, and G. P. Lepage, Phys. Rev. D 55, 5853 (1995b).
  * Fritzsch [1977] H. Fritzsch, Phys. Lett. B 67, 217 (1977).
  * Amundson  _et al._ [1997] J. F. Amundson _et al._ , Phys. Lett. B 390, 323 (1997).
  * Abelev  _et al._ [2012b] B. Abelev _et al._ (ALICE Collaboration), Phys. Lett. B 712, 165 (2012b).
  * Abelev  _et al._ [2020] B. Abelev _et al._ (ALICE Collaboration), Phys. Lett. B 810, 135758 (2020).
  * Sjostrand  _et al._ [2006] T. Sjostrand, S. Mrenna, and P. Z. Skands, J. High Energy Physics 2006(05), 026 (2006).
  * Sjostrand  _et al._ [2015] T. Sjostrand _et al._ , Comput. Phys. Commun. 191, 159 (2015).
  * Werner  _et al._ [2010] K. Werner _et al._ , Phys. Rev. C 82, 044904 (2010).
  * Werner  _et al._ [2014] K. Werner _et al._ , Phys. Rev. C 89, 064903 (2014).
  * Lang and Bleicher [2013] T. Lang and M. Bleicher, Phys. Rev. C 87, 024907 (2013).
  * Bass  _et al._ [1998] S. A. Bass _et al._ , Prog. Part. Nucl. Phys. 41, 255 (1998).
  * Bleicher  _et al._ [1999] M. Bleicher _et al._ , J. Phys. G: Nucl. Part. Phys. 25, 1859 (1999).
  * Zhou  _et al._ [2015] D. M. Zhou _et al._ , Comput. Phys. Commun. 193, 89 (2015).
  * Lei  _et al._ [2023] A.-K. Lei _et al._ , arXiv: 2309.05110 (2023).
  * Combridge  _et al._ [1977] B. L. Combridge _et al._ , Phys. Lett. B 70, 234 (1977).
  * Field [1989] R. D. Field,  _Applications Of Perturbative QCD_ (Addison-Wesley Publishing Company, 1989).
  * Sa  _et al._ [2012] B. H. Sa _et al._ , Comput. Phys. Commun. 183, 333 (2012).
  * Sa and Tai [1995] B. H. Sa and A. Tai, Comput. Phys. Commun. 90, 121 (1995).