# ProMoAI: Process Modeling with Generative AI

Humam Kourani1,2 0000-0003-2375-2152 Alessandro Berti1,2 0000-0002-3279-4795
Daniel Schuster1,2 0000-0002-6512-9580 Wil M. P. van der Aalst1,2
1Fraunhofer Institute for Applied Information Technology FIT, Sankt Augustin,
Germany
2RWTH Aachen University, Aachen, Germany
{humam.kourani, alessandro.berti, daniel.schuster,
<EMAIL_ADDRESS>0000-0002-0955-6940

###### Abstract

ProMoAI is a novel tool that leverages Large Language Models (LLMs) to
automatically generate process models from textual descriptions, incorporating
advanced prompt engineering, error handling, and code generation techniques.
Beyond automating the generation of complex process models, ProMoAI also
supports process model optimization. Users can interact with the tool by
providing feedback on the generated model, which is then used for refining the
process model. ProMoAI utilizes the capabilities LLMs to offer a novel, AI-
driven approach to process modeling, significantly reducing the barrier to
entry for users without deep technical knowledge in process modeling.

##  1 Introduction

Process modeling is a fundamental task in analyzing and optimizing
organizational workflows. Traditional process modeling approaches require
significant effort to capture workflow details and deep expertise in process
modeling languages Indulska et al. (2009). Such approaches often require
specialized knowledge, making them inaccessible to non-experts.

In recent years, Large Language Models (LLMs) Feuerriegel et al. (2024) have
shown advanced capabilities in a wide array of applications, ranging from
content generation to complex problem-solving. By leveraging vast amounts of
data, LLMs have demonstrated remarkable advancements in understanding and
generating human-like text Li et al. (2021); Zhou et al. (2023b). Their
ability to interpret natural language and generate coherent, contextually
relevant responses makes them particularly suited for tasks that involve
understanding complex textual descriptions and translating them into
structured outputs.

ProMoAI introduces an innovative approach to process modeling by utilizing the
capabilities of state-of-the-art LLMs, such as GPT-4 OpenAI (2023), and
employing advanced prompt engineering, error handling, and code generation
techniques. The primary contribution of our tool is the automation of process
model generation. Furthermore, a feedback loop allows users to interactively
refine the generated process models, ensuring they accurately reflect the
intended processes. ProMoAI is available under https://promoai.streamlit.app/,
and a demonstration video is available under https://youtu.be/t8fkx9rJmBE.

Figure 1: A screenshot of ProMoAI.

Fig. 1 shows a screenshot of ProMoAI. The user initially sets the
LLM111Currently, only OpenAI LLMs (https://openai.com/) are supported. We are
planning to support more LLMs in the future as our system’s architecture is
independent of the used LLM (cf. Fig. 2)., enters the corresponding API key,
and provides a textual description of the process. After the initial process
model generation, the user can provide feedback, and the process model is
updated accordingly. The generated process model can be viewed and exported in
the standard modeling notation used in the fields of business process
management and process mining (BPMN von Rosing et al. (2015) and PNML Hillah
et al. (2010)).

##  2 Application Domain

ProMoAI allows for a more intuitive and efficient creation and optimization of
process models, marking a novel application of LLMs in the field of process
modeling. The tool can be employed in a wide range of domains including
business process management, workflow automation, and systems engineering. It
is particularly beneficial in environments where rapid prototyping of process
models is required to visualize and optimize workflows, making it a valuable
asset for business analysts, systems engineers, and process managers.

##  3 Related Work

Previous works have explored utilizing natural language processing and text
mining for the generation of process models (e.g., Friedrich et al. (2011) and
de A. R. Gonçalves et al. (2011)). An overview of different techniques for the
extraction of processes from text is provided in Bellan et al. (2020).

Recently, LLMs have been evaluated on business process management (BPM) tasks.
In Busch et al. (2023) and Vidgof et al. (2023), the authors discuss the
potentials and challenges of using LLMs for BPM tasks. In Berti et al. (2023),
LLMs are evaluated on various process mining tasks. In Klievtsova et al.
(2023), the authors discuss the potential of LLMs in generating process models
through conversational modeling. In Grohs et al. (2023), the authors
illustrated LLMs capabilities in translating textual statements into
procedural and declarative process model constraints. In Fill et al. (2023),
the authors explored potential applications of LLMs for conceptual modeling.

Figure 2: System architecture. Figure 3: BPMN generated by ProMoAI for the
order process using GPT-4. This model conforms to the reference model of the
process, and it additionally captures the feedback requests for modeling the
item selection as a loop and allowing for skipping the reward selection.

##  4 System Overview

This section provides an overview of our methodologies.

ProMoAI implements our framework for LLM-based process modeling Kourani et al.
(2024). The high-level architecture of the underlying system is illustrated in
Fig. 2. The user submits a textual description of a process in natural
language. Our system creates a prompt that guides the LLM in generating Python
code that utilizes a predefined set of functions we designed for the
generation of process models (cf. Section 4.2). This prompt is then sent to
the LLM, and based on the LLM’s answer, the system attempts to generate a
process model within a secure execution environment, designed to safely
execute the generated code while protecting against malicious inputs or
unintended operations (cf. Section 4.3). If errors are encountered during this
process, an error handling prompt is sent back to the LLM, utilizing self-
refinement techniques to fix the code (cf. Section 4.4). Finally, the
generated model is presented to the user using standard process modeling
notations (BPMN and Petri nets), with an option for exporting the process
model in these formats. Additionally, the user can provide feedback, which is
integrated into the system to optimize and refine the process model in an
iterative refinement loop (cf. Section 4.5).

###  4.1 Utilization of POWL

ProMoAI employs the Partially Ordered Workflow Language (POWL) Kourani and van
Zelst (2023) for the generation of process models. POWL represents a subclass
of Petri nets that allows for the generation of block-structured models where
submodels are combined to generate larger ones (i.e., a hierarchy is
identified in the model). POWL and other hierarchical modeling languages are
employed in the automated discovery of process models (e.g., in Kourani et al.
(2023) and Leemans (2022)) due to the quality guarantees they inherently
provide. For example, it is not possible to generate a POWL model with dead
parts that can never be reached.

POWL allows for modeling complex, non-hierarchical process constructs while
preserving the quality guarantees of hierarchical process modeling languages.
POWL stands out by assuming that every task is concurrent by default unless
explicitly constrained, a principle that aligns with the inherent nature of
many real-world processes. This foundational assumption of concurrency
facilitates the direct translation of natural language statements into POWL
models, eliminating the need to specify the order of tasks unless necessary.

###  4.2 Prompt Engineering Techniques

ProMoAI uses sophisticated prompt engineering techniques to guide the LLM to
understand the process descriptions accurately and generate the desired
process models, without the extensive resource requirements and computational
costs of fine-tuning Li et al. (2022); Thangarasa et al. (2023). First, we
employ _role prompting_ Xu et al. (2023) by establishing a clear role for the
LLM as an expert in process modeling and as a process owner who is familiar
with the process context at the same time. Second, we provide a comprehensive
knowledge base about POWL, offering detailed insights into its hierarchical
structure, the semantics of the different POWL components, and the way POWL
models are generated. Recently, LLMs have demonstrated strong capabilities in
solving programming tasks and generating executable code Vidan and Fiedler
(2023). ProMoAI leverages these capabilities by instructing the LLM to
transform natural language descriptions into Python code that utilizes a
predefined set of functions we designed for the safe generation of POWL
models. Our strategy incorporates a _least-to-most prompting_ Zhou et al.
(2023a) design, which outlines the task of generating a POWL model through
recursive model generation, starting from simpler submodels and combining them
into larger ones. Moreover, we reinforce the LLM with instructions for self-
evaluation to avoid common mistakes and ensure the process model’s conformance
with the process description and POWL’s modeling principles. Finally, we apply
_few-shot learning_ Brown et al. (2020) by providing the LLM with concrete
examples that illustrate how to generate POWL models starting from process
descriptions (including descriptions from the PET data set Bellan et al.
(2022)).

The prompt created for our example process in Section 6 can be found under
https://github.com/humam-
kourani/ProMoAI/blob/main/example_online_shop/prompt_1.txt.

###  4.3 Process Model Generation

To ensure a high level of security and reliability, we restrict the execution
of the generated Python code to a controlled environment. We implement a
validation process to confirm that the generated code strictly adheres to our
defined validation rules and syntax, expressly prohibiting the use of any
external libraries or constructs that could introduce security risks. This
methodology ensures a secure execution environment for the generation of valid
process models.

ProMoAI transforms the generated POWL models into Petri nets and BPMN models
Kourani and van Zelst (2023). The tool provides options for viewing and
exporting the generated models in these notations, which are widely recognized
in the field of business process management.

###  4.4 Error Handling

ProMoAI features an error-handling mechanism that utilizes the self-evaluation
and refinement capabilities of LLMs through _iterative prompting_ Jha et al.
(2023). If an error occurs during the execution of the validated Python code
(e.g., due to a violation of our validation rules), the system produces an
informative feedback message detailing the nature of the error. This message
is then incorporated into the conversation history with the LLM, enabling the
LLM to refine the generated code in response to the identified error.

###  4.5 Process Model Refinement

A key feature of ProMoAI is its model optimization mechanism, realized through
an interactive refinement loop. Users can provide feedback on the generated
process models, which the system uses to generate a refinement prompt. This
prompt is integrated into the conversation history with the LLM, allowing for
the iterative improvement of the process model to ensure it accurately meets
the user’s requirements.

##  5 Future Work and Extensions

Recognizing the rapid advancement in LLM technology, ProMoAI is designed to be
forward-compatible, enabling users to utilize newer OpenAI models as they
become available. This flexibility ensures that ProMoAI remains at the cutting
edge of AI-driven process modeling, capable of delivering improved results by
leveraging the latest advancements. We also plan to support further LLMs
beyond those offered by OpenAI (e.g., Google Gemini Anil et al. (2023)).

We are exploring the direct generation of BPMN models from textual process
descriptions without the intermediate POWL representation. This will offer
more flexibility in representing complex process structures and will allow us
to enrich the process models with informative, context-rich annotations.
However, moving away from POWL’s structured guarantees requires the
development of more advanced process model generation and validation
techniques to maintain the quality of the generated process models.

##  6 Example Application

We applied ProMoAI using GPT-4 on the process used in Kourani and van Zelst
(2023) to describe the user’s journey to order items through an online shop.
This process contains typical process constructs (choice, loop, concurrency,
sequence) as well as more complex non-hierarchical dependencies. After
generating the initial process model, we applied two feedback comments to
refine the model (cf. Fig. 1). The initial process description, the feedback
comments, all generated prompts, and GPT-4’s responses are available under
https://github.com/humam-kourani/ProMoAI/tree/main/example_online_shop.

The BPMN view of the generated process model is shown in Fig. 3. The generated
model conforms to the reference process model, and it additionally integrates
the feedback requests correctly into the model. This process contains complex
non-hierarchical dependencies between selecting the items, setting a payment
method, the reward selection, and the payment choice. As discussed in Kourani
and van Zelst (2023), traditional hierarchical process modeling notations such
as process trees Leemans (2022) fail to model such structures. The use of POWL
enables ProMoAI to detect such structures, offering a level of flexibility and
precision that traditional hierarchical process modeling languages lack.

Rerunning the application on the same input may lead to different results,
e.g., GPT-4 might place the reward selection after setting the payment method.
With the continuous development and advances added to LLMs, we expect future
LLMs to perform better in understanding and modeling complex dependencies,
leading to more consistency in the results.

##  7 Conclusion

ProMoAI represents a significant advancement in the field of process modeling,
offering a user-friendly, efficient, and interactive approach to generating
and refining process models. The tool allows users to describe workflows in
natural language, which are then automatically translated into formal process
models. By utilizing the capabilities of large language models, ProMoAI
simplifies the creation of complex process models, opening up new
possibilities for optimizing organizational workflows.

## References

  * Anil et al. [2023] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al.  Gemini: A family of highly capable multimodal models.  CoRR, abs/2312.11805, 2023.
  * Bellan et al. [2020] Patrizio Bellan, Mauro Dragoni, and Chiara Ghidini.  A qualitative analysis of the state of the art in process extraction from text.  In Giuseppe Vizzari, Matteo Palmonari, and Andrea Orlandini, editors, Proceedings of the AIxIA 2020 Discussion Papers Workshop co-located with the the 19th International Conference of the Italian Association for Artificial Intelligence (AIxIA2020), Anywhere, November 27th, 2020, volume 2776 of CEUR Workshop Proceedings, pages 19–30. CEUR-WS.org, 2020.
  * Bellan et al. [2022] Patrizio Bellan, Han van der Aa, Mauro Dragoni, Chiara Ghidini, and Simone Paolo Ponzetto.  PET: an annotated dataset for process extraction from natural language text tasks.  In Cristina Cabanillas, Niels Frederik Garmann-Johnsen, and Agnes Koschmider, editors, Business Process Management Workshops - BPM 2022 International Workshops, Münster, Germany, September 11-16, 2022, Revised Selected Papers, volume 460 of Lecture Notes in Business Information Processing, pages 315–321. Springer, 2022.
  * Berti et al. [2023] Alessandro Berti, Daniel Schuster, and Wil M. P. van der Aalst.  Abstractions, scenarios, and prompt definitions for process mining with llms: A case study.  In Jochen De Weerdt and Luise Pufahl, editors, Business Process Management Workshops - BPM 2023 International Workshops, Utrecht, The Netherlands, September 11-15, 2023, Revised Selected Papers, volume 492 of Lecture Notes in Business Information Processing, pages 427–439. Springer, 2023.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * Busch et al. [2023] Kiran Busch, Alexander Rochlitzer, Diana Sola, and Henrik Leopold.  Just tell me: Prompt engineering in business process management.  In Han van der Aa, Dominik Bork, Henderik A. Proper, and Rainer Schmidt, editors, Enterprise, Business-Process and Information Systems Modeling - 24th International Conference, BPMDS 2023, and 28th International Conference, EMMSAD 2023, Zaragoza, Spain, June 12-13, 2023, Proceedings, volume 479 of Lecture Notes in Business Information Processing, pages 3–11. Springer, 2023.
  * de A. R. Gonçalves et al. [2011] João Carlos de A. R. Gonçalves, Flávia Maria Santoro, and Fernanda Araújo Baião.  Let me tell you a story - on how to build process models.  J. Univers. Comput. Sci., 17(2):276–295, 2011.
  * Feuerriegel et al. [2024] Stefan Feuerriegel, Jochen Hartmann, Christian Janiesch, and Patrick Zschech.  Generative AI.  Bus. Inf. Syst. Eng., 66(1):111–126, 2024.
  * Fill et al. [2023] Hans-Georg Fill, Peter Fettke, and Julius Köpke.  Conceptual modeling and large language models: Impressions from first experiments with chatgpt.  Enterp. Model. Inf. Syst. Archit. Int. J. Concept. Model., 18:3, 2023.
  * Friedrich et al. [2011] Fabian Friedrich, Jan Mendling, and Frank Puhlmann.  Process model generation from natural language text.  In Haralambos Mouratidis and Colette Rolland, editors, Advanced Information Systems Engineering - 23rd International Conference, CAiSE 2011, London, UK, June 20-24, 2011. Proceedings, volume 6741 of Lecture Notes in Computer Science, pages 482–496. Springer, 2011.
  * Grohs et al. [2023] Michael Grohs, Luka Abb, Nourhan Elsayed, and Jana-Rebecca Rehse.  Large language models can accomplish business process management tasks.  In Jochen De Weerdt and Luise Pufahl, editors, Business Process Management Workshops - BPM 2023 International Workshops, Utrecht, The Netherlands, September 11-15, 2023, Revised Selected Papers, volume 492 of Lecture Notes in Business Information Processing, pages 453–465. Springer, 2023.
  * Hillah et al. [2010] Lom-Messan Hillah, Fabrice Kordon, Laure Petrucci, and Nicolas Trèves.  PNML framework: An extendable reference implementation of the petri net markup language.  In Johan Lilius and Wojciech Penczek, editors, Applications and Theory of Petri Nets, 31st International Conference, PETRI NETS 2010, Braga, Portugal, June 21-25, 2010. Proceedings, volume 6128 of Lecture Notes in Computer Science, pages 318–327. Springer, 2010.
  * Indulska et al. [2009] Marta Indulska, Jan Recker, Michael Rosemann, and Peter F. Green.  Business process modeling: Current issues and future challenges.  In Pascal van Eck, Jaap Gordijn, and Roel J. Wieringa, editors, Advanced Information Systems Engineering, 21st International Conference, CAiSE 2009, Amsterdam, The Netherlands, June 8-12, 2009. Proceedings, volume 5565 of Lecture Notes in Computer Science, pages 501–514. Springer, 2009\.
  * Jha et al. [2023] Susmit Jha, Sumit Kumar Jha, Patrick Lincoln, Nathaniel D. Bastian, Alvaro Velasquez, and Sandeep Neema.  Dehallucinating large language models using formal methods guided iterative prompting.  In IEEE International Conference on Assured Autonomy, ICAA 2023, Laurel, MD, USA, June 6-8, 2023, pages 149–152. IEEE, 2023.
  * Klievtsova et al. [2023] Nataliia Klievtsova, Janik-Vasily Benzin, Timotheus Kampik, Juergen Mangler, and Stefanie Rinderle-Ma.  Conversational process modelling: State of the art, applications, and implications in practice.  In Chiara Di Francescomarino, Andrea Burattin, Christian Janiesch, and Shazia W. Sadiq, editors, Business Process Management Forum - BPM 2023 Forum, Utrecht, The Netherlands, September 11-15, 2023, Proceedings, volume 490 of Lecture Notes in Business Information Processing, pages 319–336. Springer, 2023.
  * Kourani and van Zelst [2023] Humam Kourani and Sebastiaan J. van Zelst.  POWL: partially ordered workflow language.  In Chiara Di Francescomarino, Andrea Burattin, Christian Janiesch, and Shazia Sadiq, editors, Business Process Management - 21st International Conference, BPM 2023, Utrecht, The Netherlands, September 11-15, 2023, Proceedings, volume 14159 of Lecture Notes in Computer Science, pages 92–108. Springer, 2023.
  * Kourani et al. [2023] Humam Kourani, Daniel Schuster, and Wil M. P. van der Aalst.  Scalable discovery of partially ordered workflow models with formal guarantees.  In 5th International Conference on Process Mining, ICPM 2023, Rome, Italy, October 23-27, 2023, pages 89–96. IEEE, 2023.
  * Kourani et al. [2024] Humam Kourani, Alessandro Berti, Daniel Schuster, and Wil M. P. van der Aalst.  Process Modeling With Large Language Models.  In Proceedings of the 29th International Working Conference on Exploring Modeling Methods for Systems Analysis and Development (EMMSAD 2024). Springer, 2024.  In press.
  * Leemans [2022] Sander J. J. Leemans.  Robust Process Mining with Guarantees - Process Discovery, Conformance Checking and Enhancement, volume 440 of Lecture Notes in Business Information Processing.  Springer, 2022.
  * Li et al. [2021] Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen.  Pretrained language model for text generation: A survey.  In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4492–4499. International Joint Conferences on Artificial Intelligence Organization, 8 2021.  Survey Track.
  * Li et al. [2022] Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, and Junjie Bai.  Parameter-efficient sparsity for large language models fine-tuning.  In Luc De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 4223–4229. ijcai.org, 2022.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  CoRR, abs/2303.08774, 2023.
  * Thangarasa et al. [2023] Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, and Shreyas Saxena.  SPDF: sparse pre-training and dense fine-tuning for large language models.  In Robin J. Evans and Ilya Shpitser, editors, Uncertainty in Artificial Intelligence, UAI 2023, July 31 - 4 August 2023, Pittsburgh, PA, USA, volume 216 of Proceedings of Machine Learning Research, pages 2134–2146. PMLR, 2023.
  * Vidan and Fiedler [2023] Andy Vidan and Lars H. Fiedler.  A composable just-in-time programming framework with llms and FBP.  In IEEE High Performance Extreme Computing Conference, HPEC 2023, Boston, MA, USA, September 25-29, 2023, pages 1–8. IEEE, 2023.
  * Vidgof et al. [2023] Maxim Vidgof, Stefan Bachhofner, and Jan Mendling.  Large language models for business process management: Opportunities and challenges.  In Chiara Di Francescomarino, Andrea Burattin, Christian Janiesch, and Shazia W. Sadiq, editors, Business Process Management Forum - BPM 2023 Forum, Utrecht, The Netherlands, September 11-15, 2023, Proceedings, volume 490 of Lecture Notes in Business Information Processing, pages 107–123. Springer, 2023.
  * von Rosing et al. [2015] Mark von Rosing, Stephen White, Fred Cummins, and Henk de Man.  Business process model and notation - BPMN.  In Mark von Rosing, Henrik von Scheel, and August-Wilhelm Scheer, editors, The Complete Business Process Handbook: Body of Knowledge from Process Modeling to BPM, Volume I, pages 429–453. Morgan Kaufmann/Elsevier, 2015.
  * Xu et al. [2023] Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao.  Expertprompting: Instructing large language models to be distinguished experts.  CoRR, abs/2305.14688, 2023.
  * Zhou et al. [2023a] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi.  Least-to-most prompting enables complex reasoning in large language models.  In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023\.
  * Zhou et al. [2023b] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.  Large language models are human-level prompt engineers.  In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023\.


# A Distributed Inference System for Detecting Task-wise Single Trial Event-
Related Potential in Stream of Satellite Images ††thanks: This work was
supported by the Agency For Defense Development Grant Funded by the Korean
Government (UI233001TD).

Sung-Jin Kim
Dae-Hyeok Lee  Dept. of Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Dept. of Brain and Cognitive Engineering
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Heon-Gyu Kwak
Ji-Hoon Jeong  Dept. of Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Dept. of Computer Science
Chungbuk National University
Cheongju, Republic of Korea
<EMAIL_ADDRESS>Hyeon-Taek Han
Seong-Whan Lee  Dept. of Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Dept. of Artificial Intelligence
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

Brain-computer interface (BCI) has garnered the significant attention for
their potential in various applications, with event-related potential (ERP)
performing a considerable role in BCI systems. This paper introduces a novel
Distributed Inference System tailored for detecting task-wise single-trial
ERPs in a stream of satellite images. Unlike traditional methodologies that
employ a single model for target detection, our system utilizes multiple
models, each optimized for specific tasks, ensuring enhanced performance
across varying image transition times and target onset times. Our experiments,
conducted on four participants, employed two paradigms: the Normal paradigm
and an AI paradigm with bounding boxes. Results indicate that our proposed
system outperforms the conventional methods in both paradigms, achieving the
highest $F_{\beta}$ scores. Furthermore, including bounding boxes in the AI
paradigm significantly improved target recognition. This study underscores the
potential of our Distributed Inference System in advancing the field of ERP
detection in satellite image streams.

###### Index Terms:

event-related potential, target detection, deep learning,
electroencephalogram, satellite images

##  I INTRODUCTION

Brain-computer interface (BCI), also known as brain-machine interfaces,
represents a direct communication pathway between the brain and external
devices. This technology has been at the forefront of neuroscience and
engineering research, aiming to restore lost sensory and motor functions,
enhance human capabilities, or even create new forms of interaction with the
world. BCI decodes neural signals, translating them into commands that can
control various devices or software applications [1, 2, 3, 4]. They have been
employed in medical settings to assist patients with severe motor
disabilities, allowing them to communicate or control prosthetic limbs using
only their brain activity [5, 6]. In the realm of rehabilitation, BCI aids in
restoring motor functions for stroke victims or individuals with spinal cord
injuries [7, 8]. Beyond medical applications, BCI has ventured into the
entertainment industry, enabling users to play video games or operate virtual
reality environments using their thoughts [9, 10]. Furthermore, in the domain
of defense and aerospace, BCI is being explored for piloting drones or even
potentially controlling advanced machinery [11, 12]. The integration of event-
related potential (ERP) within BCI, especially in tasks like satellite image
analysis, underscores the potential of this technology in rapidly processing
vast amounts of visual data, capturing the brain’s instantaneous reactions,
and making timely decisions.

In the realm of ERP detection, various methodologies have been proposed and
explored over the years. The conventional approaches primarily relied on
theory-driven methods, where the extraction of ERP components was based on
predetermined temporal and spatial filters [13]. These methods, such as
independent component analysis and time-frequency representations, have been
instrumental in isolating specific ERP components from the background EEG
activity. Their strength lies in their ability to leverage prior knowledge
about the expected ERP waveforms and their temporal dynamics.

However, with the advent of deep learning, the landscape of ERP detection has
witnessed a paradigm shift. Deep learning-based methods, particularly
convolutional neural networks and recurrent neural networks, have shown
promise in automatically extracting intricate patterns from raw EEG data
without the need for manual feature engineering [14, 15, 16]. These models are
trained on enormous datasets, enabling them to capture the subtle nuances of
ERP signals, even in noisy environments. Recent studies have demonstrated the
superiority of deep learning techniques over the conventional methods,
especially in scenarios with high inter-subject variability or when dealing
with single-trial ERP detection [17].

Detecting single-trial ERP presents a unique set of challenges. The inherent
variability in neural responses, even to identical stimuli, could lead to the
significant differences in ERP features across trials. This variability is
further increased when considering the transition times between images. As the
brain processes rapidly changing visual stimuli, the ERP elicited by one image
can be influenced by the preceding image, leading to a temporal smearing
effect. This effect complicates the task of isolating the ERP corresponding to
a specific image, especially when the transition times between pictures are
inconsistent.

In recent years, several methods have been proposed to address these
challenges. Techniques such as adaptive filtering and advanced artifact
rejection algorithms have been introduced to enhance the clarity of single-
trial ERP [18]. Furthermore, deep learning architectures incorporating
temporal attention mechanisms have been developed to focus on relevant time
points in the ERP, mitigating the effects of varying transition times [19].
However, while these methods have shown promise, they often require extensive
training data and may not generalize well across different subjects or ERP
paradigms. The investigation for a robust and universally applicable single-
trial ERP detection method remains an active area of research.

We have developed a novel approach, the Distributed Inference System, to
address the inherent difficulties faced by the conventional single-trial ERP
detection methods. Our proposed strategy diverges from the conventional
approaches that attempt to process all diverse scenarios, such as varying
image transition times and ERP stimuli, and tasks, like target recognition and
target onset time inference, within a single model. Such an approach often
restricts the model’s ability to learn features from the data effectively. Our
system adopts a strategy where each task is learned and inferred
independently. By segmenting the learning process, each model in the system
could learn about specific features relevant to each task, enhancing its
overall accuracy and efficiency. This methodology has proven particularly
effective in tasks involving the detection of targets within a stream of
satellite images compared with the conventional methods.

##  II METHODS

###  II-A Subjects and Experimental Environment

We collected data from a total of four healthy subjects, comprising three
males and one female. Our experiment received approval from the Institutional
Review Board at Korea University [KUIRB-2022-0373-01]. Before the actual
experiment, all participants were thoroughly briefed on the experimental
paradigm and underwent practice sessions to familiarize themselves with the
procedure. For EEG signal acquisition, we utilized a signal amplifier
(BrainAmp, Brain Products GmbH, Germany). The EEG signals were sampled at a
frequency of 250 Hz, and a 60 Hz notch filter was employed to eliminate DC
noise. The EEG data was acquired using 32 channels positioned on the subjects’
scalps following the international 10-20 system. To ensure optimal signal
quality, we applied a conductive gel to the subjects’ scalps, reducing the
impedance of the EEG electrodes to 15 k$\Omega$ or below.

Figure 1: Experimental paradigm applied in the target recognition. (a) and (b)
indicate the normal paradigm with the image transition time of 0.5 sec. and
0.1 sec., respectively. (c) indicates the AI paradigm containing a bounding
box at images.

Figure 2: Overall flow to decide the decisions of target recognition using the
proposed Distributed Inference System

###  II-B Experimental Paradigm

Our data acquisition paradigm centered on detecting targets within a sequence
of 10 consecutively displayed satellite images. We conducted two distinct
paradigms for this purpose. In the first paradigm (Normal paradigm),
participants were presented with 16 trials, each comprising a series of
satellite images. These trials were divided into two categories based on the
transition time between images: 0.1 sec. and 0.5 sec. Of the 16 trials, 8 had
a transition time of 0.1 sec., while the remaining 8 had a transition time of
0.5 sec. Targets appeared a total of 8 times across these 16 trials, with
their appearance in the 0.1-sec. and 0.5-sec. trials being randomized. The
second paradigm (AI paradigm) also consisted of 16 trials but with a
consistent image transition time of 0.5 sec. across all trials. Unlike the
first paradigm, this one simulated a scenario where the images had been
processed by a computer vision model. As a result, targets within the images
were marked with a red bounding box. However, considering the simulated
computer vision inference, some non-target areas resembling the target were
also mistakenly boxed in red. As with the first paradigm, targets appeared 8
times across the 16 trials. Each trial is separated by a beep sound that is
inaudible to the subject in both paradigms. For both paradigms, one session
consists of 16 trials, and four sessions are conducted in each paradigm.
During both paradigms, participants were instructed to manually indicate the
presence or absence of a target at the end of each trial in 5 sec., ensuring
an interactive and attentive engagement with the presented images.

###  II-C Distributed Inference System

Our primary goal through the paradigm was to determine the existence of a
target within a given trial and, if present, to ascertain the exact time of
its appearance. While the conventional methods employing a singular model for
target detection share the overarching goal of capturing the characteristics
of the ERP, our paradigm, which contains varying image transition times (0.1
sec. and 0.5 sec.) and the inference of target onset times, presents unique
challenges. The differences in features that each task-specific model could
learn make it difficult for a single model to optimize for all tasks
simultaneously. To address this issue, we developed the Distributed Inference
System to integrate the outputs of separated models tailored for each task.

Our system is structured around three main components: EEG data acquisition,
Distributed target recognition, and Inference result aggregation. The EEG data
acquisition component is responsible for collecting data from a participant
over a session, which includes 16 trials. Following the previous part, the
Distributed target recognition part takes over, segmenting the acquired EEG
signals based on beep triggers, distinguishing between the 0.1-sec. and
0.5-sec. transition times. The data split by trial and the original data
before the split are entered into the 0.1 sec. Target detector, 0.5 sec.
Target detector, and Appearance time detector, respectively, to get the
respective results. Each of these models independently processes the data and
generates its inference. Finally, the Inference result aggregation component
converges the outputs from the models. It determines the existence of a target
in each trial using the results from each target detector model. In addition,
it detects the exact onset time of the target with the Appearance time
detector. By conducting all these processes, we could facilitate the
concurrent extraction of results for all 16 trials within the session.

To include the P300 feature, which occurs when the target appears in the last
image, each trial was split into one trial by including an additional 0.5 sec.
of data after the 10-image stream. The 0.1 sec. Target detector and 0.5 sec.
Target detector were trained on their respective segmented data. In contrast,
the Appearance time detector utilized the entire EEG data, employing a sliding
window approach with a window size of 1 sec. and a 0.5-sec. overlap for both
training and inference. The optimized DeepConvNet [20] architecture was used
for all detector models.

###  II-D Performance Evaluation

Our evaluation was conducted in a subject-dependent environment. Each of the
four sessions was designated as test data in turn, with the remaining three
sessions serving as training data. This process was repeated four times,
ensuring each session was used as test data once. The final performance was
determined by averaging the results from these four evaluations. We conducted
separate evaluations for the two paradigms. To focus on the recall of target
detection, we employed the $F_{\beta}$ score as our evaluation metric, setting
$\beta$ to 2. This choice emphasizes the importance of recall in our task. The
formula for calculating $F_{\beta}$ score is shown below:

${F}_{\beta}=(1+{\beta}^{2})\frac{(\text{precision}\times\text{recall})}{({\beta}^{2}\cdot\text{precision})+\text{recall}}$
(1)

Additionally, to evaluate the variability across subjects, we also computed
the standard deviation of the $F_{\beta}$ scores for each participant. For the
training process, each model was trained for 20 epochs with a learning rate of
0.001. AdamW optimizer [21] was used for optimization with the weight decay of
0.01. In addition, the cosine learning rate scheduler was utilized for
training stability.

##  III RESULTS AND DISCUSSION

Our proposed method was evaluated against the conventional methods including
ShallowConvNet [22], DeepConvNet [20], and EEGNet [23]. The evaluation method
for the conventional methods was consistent with the conditions applied to the
Appearance Time Detector in our system. The conventional methods assessed the
performances by sliding a window of 1-sec. duration, moving in 0.5-sec.
increments, and determining whether the segment contained a target. If a
target was detected at least once within a trial by these methods, the entire
trial was classified as containing a target. For both the traditional models
and our Appearance Time Detector, a target score exceeding 0.5 was considered
indicative of target presence.

Our proposed method, as shown in Table I, outperformed the other comparison
methods in both paradigms, achieving the highest $F_{\beta}$ scores of 0.6875
and 0.7266 for the Normal and AI paradigms, respectively. We verified that all
methodologies generally achieved better performance in the AI paradigm, where
bounding boxes were present in the images, compared to the Normal paradigm. We
attribute this performance enhancement in the AI paradigm to the presence of
the red bounding boxes, which likely made the targets more conspicuous to the
subjects, aiding in target detection. However, when examining the standard
deviation value, a measure of variability, the EEGNet model recorded the
lowest values of 0.0928 and 0.0881 for the two paradigms. Our proposed method
achieved values of 0.1308 and 0.1481, indicating the more significant
performance variance across subjects compared to other methods. The increased
variance in our method’s performance is attributed to notably enhanced results
in specific subjects rather than a lack of stability. Furthermore, we won
first place in the 1st BCI Target Recognition Technology Competition using our
proposed method.

TABLE I: Comparison of performances among the conventional methods across
different paradigms

Method | Normal paradigm | AI paradigm
---|---|---
$F_{\beta}$ | std. | $F_{\beta}$ | std.
ShallowConvNet [22] | 0.5314 | 0.1314 | 0.6481 | 0.1421
DeepConvNet [20] | 0.6328 | 0.1753 | 0.6899 | 0.1641
EEGNet [23] | 0.6425 | 0.0928 | 0.6841 | 0.0881
Proposed | 0.6875 | 0.1308 | 0.7266 | 0.1481

∗std.: standard deviation

##  IV CONCLUSIONS AND FUTURE WORKS

In this study, we introduced the Distributed Inference System, a novel
approach designed to address the challenges of detecting single-trial ERP in a
stream of satellite images. Our method, distinct from the conventional single-
model approaches, employs multiple models tailored for specific tasks,
ensuring optimized performance across varying image transition times and
target onset times. The results demonstrate the superiority of our proposed
system over existing methods in both the Normal and AI paradigms. Notably, the
inclusion of bounding boxes in the AI paradigm enhanced target recognition,
underscoring the potential benefits of visual aids in ERP detection tasks.
However, while our system excels in post-session inference, it currently lacks
the capability for real-time analysis. In future work, a primary objective
will be developing a real-time inference system. Such an advancement would not
only elevate the practicality of our approach but also pave the way for
broader applications in real-world scenarios.

## References

  * [1] S.-H. Lee, M. Lee, J.-H. Jeong, and S.-W. Lee, “Towards an EEG–based intuitive BCI communication system using imagined speech and visual imagery,” in _Conf. Proc. IEEE Int. Conf. Syst. Man Cybern. (SMC)_ , 2019, pp. 4409–4414.
  * [2] S.-H. Lee, M. Lee, and S.-W. Lee, “Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 28, no. 12, pp. 2647–2659, 2020.
  * [3] K.-H. Thung _et al._ , “Conversion and time–to–conversion predictions of mild cognitive impairment using low–rank affinity pursuit denoising and matrix completion,” _Med. Image Anal._ , vol. 45, pp. 68–82, 2018.
  * [4] D.-H. Lee, J.-H. Jeong, B.-W. Yu, T.-E. Kam, and S.-W. Lee, “Autonomous system for EEG-based multiple abnormal mental states classification using hybrid deep neural networks under flight environment,” _IEEE Trans. Syst. Man Cybern. Syst._ , vol. 53, no. 10, pp. 6426–6437, 2023.
  * [5] J. Kim _et al._ , “Abstract representations of associated emotions in the human brain,” _J. Neurosci._ , vol. 35, no. 14, pp. 5655–5663, 2015.
  * [6] M. Lee, C.-B. Song, G.-H. Shin, and S.-W. Lee, “Possible effect of binaural beat combined with autonomous sensory meridian response for inducing sleep,” _Front. Hum. Neurosci._ , vol. 13, pp. 425–440, 2019.
  * [7] L. F. Nicolas-Alonso and J. Gomez-Gil, “Brain computer interfaces, a review,” _Sensors_ , vol. 12, no. 2, pp. 1211–1279, 2012.
  * [8] S.-B. Lee _et al._ , “Comparative analysis of features extracted from EEG spatial, spectral and temporal domains for binary and multiclass motor imagery classification,” _Inf. Sci._ , vol. 502, pp. 190–200, 2019.
  * [9] B. Kerous, F. Skola, and F. Liarokapis, “EEG-based BCI and video games: A progress report,” _Virtual Real._ , vol. 22, pp. 119–135, 2018.
  * [10] L. Bonnet, F. Lotte, and A. Lécuyer, “Two brains, one game: Design and evaluation of a multiuser BCI video game based on motor imagery,” _IEEE Trans. Comput. Intell. AI Games_ , vol. 5, no. 2, pp. 185–198, 2013.
  * [11] D.-H. Lee, D.-K. Han, S.-J. Kim, J.-H. Jeong, and S.-W. Lee, “Subject–independent brain–computer interface for decoding high–level visual imagery tasks,” in _Conf. Proc. IEEE Int. Conf. Syst. Man Cybern. (SMC)_ , 2021, pp. 3396–3401.
  * [12] S.-J. Kim, D.-H. Lee, and S.-W. Lee, “An EEGgram-based neural network enhancing the decoding performance of visual imagery EEG signals to control the drone swarm,” in _Conf. Proc. IEEE Int. Conf. Syst. Man Cybern. (SMC)_ , 2022, pp. 2281–2286.
  * [13] H.-I. Suk, S. Fazli, J. Mehnert, K.-R. Müller, and S.-W. Lee, “Predicting BCI subject performance using probabilistic spatio–temporal filters,” _PLoS One_ , vol. 9, no. 2, p. e87056, 2014.
  * [14] R. Mane _et al._ , “FBCNet: A multi–view convolutional neural network for brain–computer interface,” _arXiv preprint arXiv:2104.01233_ , 2021.
  * [15] J.-S. Bang, M.-H. Lee, S. Fazli, C. Guan, and S.-W. Lee, “Spatio–spectral feature representation for motor imagery classification using convolutional neural networks,” _IEEE Trans. Neural Netw. Learn. Syst._ , vol. 33, no. 7, pp. 3038–3049, 2021.
  * [16] K.-T. Kim, C. Guan, and S.-W. Lee, “A subject–transfer framework based on single–trial EMG analysis using convolutional neural networks,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 28, no. 1, pp. 94–103, 2019.
  * [17] B. Zang, Y. Lin, Z. Liu, and X. Gao, “A deep learning method for single-trial EEG classification in RSVP task based on spatiotemporal features of ERPs,” _J. Neural Eng._ , vol. 18, no. 4, p. 0460c8, 2021.
  * [18] H. Woehrle _et al._ , “An adaptive spatial filter for user-independent single trial detection of event-related potentials,” _IEEE. Trans. Biomed. Eng._ , vol. 62, no. 7, pp. 1696–1705, 2015.
  * [19] G. Chen, X. Zhang, J. Zhang, F. Li, and S. Duan, “A novel brain-computer interface based on audio-assisted visual evoked EEG and spatial-temporal attention CNN,” _Front. Neurorob._ , vol. 16, p. 995552, 2022.
  * [20] R. T. Schirrmeister _et al._ , “Deep learning with convolutional neural networks for EEG decoding and visualization,” _Hum. Brain Mapp._ , vol. 38, no. 11, pp. 5391–5420, 2017.
  * [21] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _Int. Conf. Learn. Repr. (ICLR)_ , 2018.
  * [22] S.-J. Kim, D.-H. Lee, and S.-W. Lee, “Rethinking CNN architecture for enhancing decoding performance of motor imagery-based EEG signals,” _IEEE Access_ , vol. 10, pp. 96 984–96 996, 2022.
  * [23] V. J. Lawhern _et al._ , “EEGNet: A compact convolutional neural network for EEG–based brain–computer interfaces,” _J. Neural Eng._ , vol. 15, no. 5, p. 056013, 2018.