# SoK: Exploring the Potential of Large Language Models for Improving Digital
Forensic Investigation Efficiency

Akila Wickramasekara  University College DublinIreland
<EMAIL_ADDRESS>,  Frank Breitinger  University of
LausanneSwitzerland<EMAIL_ADDRESS>and  Mark Scanlon  University
College DublinIreland<EMAIL_ADDRESS>
(2024)

###### Abstract.

The growing number of cases requiring digital forensic analysis raises
concerns about law enforcement’s ability to conduct investigations promptly.
Consequently, this systemisation of knowledge paper delves into the potential
and effectiveness of integrating Large Language Models (LLMs) into digital
forensic investigation to address these challenges. A thorough literature
review is undertaken, encompassing existing digital forensic models, tools,
LLMs, deep learning techniques, and the utilisation of LLMs in investigations.
The review identifies current challenges within existing digital forensic
processes and explores both the obstacles and possibilities of incorporating
LLMs. In conclusion, the study asserts that the adoption of LLMs in digital
forensics, with appropriate constraints, holds the potential to enhance
investigation efficiency, improve traceability, and alleviate technical and
judicial barriers faced by law enforcement entities.

Digital Forensics, Large Language Models, Investigative Process, Challenges

††journalyear: 2024††copyright: acmlicensed††conference: The 19th
International ARES Conference on Availability, Reliability and Security; July
30–August 2, 2024; Vienna, Austria††booktitle: The 19th International ARES
Conference on Availability, Reliability and Security, July 30–August 2, 2024,
Vienna, Austria††doi: https://doi.org/10.1145/3407023.3407068††isbn:
978-1-4503-8833-7/20/08††copyright: none††conference:  ; ;

##  1\. Introduction

With the pervasive growth of Information and Communication Technology (ICT)
and information systems, cybercrimes have witnessed a significant surge in
recent years (Ali, 2019). As a further compounding factor, the number of
“traditional” police investigations including digital evidence is also ever-
increasing (Du and Scanlon, 2019). Addressing and investigating this volume of
cases presents substantial challenges.

Artificial General Intelligence (AGI) and Large Language Models (LLMs) have
become prominent topics of global discussion, prompting researchers to
intensify their investigations by leveraging the capabilities of LLMs. The
usage of LLMs within the scientific community experienced a rapid surge after
2022, notably with the advent of OpenAI’s ChatGPT platform. In a remarkably
short period, this topic has garnered attention from academia, industry, and
the research community at large (Ray, 2023). Simultaneously, researchers are
exploring the potential of LLMs across various domains and assessing their
impact on the future of science and society. This inquiry also includes an
examination of the potential harmfulness associated with the deployment of
LLMs (Brown et al., 2020; Nozza et al., 2022). In other words, the use of LLMs
in various tasks can be a double-edged sword, necessitating careful
consideration depending on the specific situations and contexts.

Given the swiftly evolving landscape of LLMs, it is prudent to look into
various types and their unique capabilities. A nuanced understanding of the
strengths and characteristics of different LLMs can contribute to more
informed and effective applications within the dynamic field of digital
forensics (DF).

The main objective of this article is to explore the viability of integrating
LLMs into the fundamental phases of the modern DF process. This exploration
involved a meticulous and systematic review, adhering to the standards set by
Kitchenham (2004). The literature selected for this review was exclusively in
English, and non-article sources such as books were excluded. The focus of the
review encompassed empirical studies, case studies, and literature reviews
within the domain of DF investigation methods, models, frameworks, efficiency,
and challenges, as well as LLM capabilities, challenges, and architectures.
Additionally, the study underscores the utilisation of autonomous agents based
on LLMs, emphasising their capabilities and the potential efficiency gains
achievable in the field of DF. In light of the fast-paced advancements and
recent explosion in LLM-focused research, a substantial influx of LLM-focused
research papers has occurred since the launch of ChatGPT in late 2022. Due to
this fast pace, a large number of research articles exist solely as preprints
on preprint services, e.g., arXiv or OpenReview. To give two examples, the
initial papers for GPT-4 (Achiam et al., 2023) and LLaMA (Touvron et al.,
2023) are both only published on arXiv, but have garnered thousands of
citations each. Despite their preprint status, these papers offer essential
insights critical for contemporary research and dialogue within the domain,
making their incorporation into this paper necessary to provide the most up-
to-date knowledge and perspectives.

The paper is structured as follows: Section 2 provides a comprehensive
background for the review, delving into existing DF process models, the
challenges inherent in DF, and a detailed overview of the current work
conducted with the utilisation of LLMs within DF. In Section 3 the paper
delves into the realm of Natural Language Processing (NLP), elucidating the
working principles of LLMs, their architectural foundations, and the specifics
surrounding specially trained LLMs. Section 4 provides an in-depth review
focusing on the capabilities and benchmark information of LLMs trained for
coding tasks, as well as those tailored for vision assistance. Section 4
explores the synergy between DF and LLMs, detailing how LLMs can be
effectively employed in each phase of the DF process model. Finally, in
Section 6, the paper summarises the future challenges associated with
integrating LLMs with automated agents within the DF domain. It outlines
potential avenues for future research and development, shedding light on the
path of future DF investigations employing LLMs. The discussion encompasses
not only the potential negative impacts but also the practical difficulties
and risks in real-world environments.

##  2\. Background

DF is a process to identify, preserve, analyse, and document digitally
recorded data, which originate in electronic devices such as computers,
servers, smartphones, and IoT devices (Baryamureeba and Tushabe, 2004). This
exercise is required in most criminal cases. Data collected in this process
are kept unchanged and safe to present in a court case or to support future
investigations conducted by law enforcement agencies (Mukherjee and Haque,
2018).

###  2.1. Digital Forensic Process Models

DF process models consist of a series of activities that help to standardise
the investigative process (Du et al., 2017) and outline the phases;
collection, preservation of evidence, examination or analysis, and reporting.

Figure 1. Traditional digital forensic process model (Casey, 2011)

DF encompasses various subdisciplines such as computer forensics, mobile
device forensics, memory forensics, network forensics, and cloud forensics,
each employing distinct processes reflected in a plethora of models within the
literature (Scanlon et al., 2023a; Wu et al., 2020). These models often share
phases but differ in their focus and execution. For instance, Al-Dhaqm et al.
(2020) proposed a mobile forensic model that adds a preparatory phase and
bifurcates the analysis stage into examination and analysis phases. To
accommodate the complexities of computer, network, cloud, and smart device
forensics, Lutui (2016) introduced a multidisciplinary model that necessitates
diverse skills for effective investigation, covering incident detection to
evidence storage.

Casey’s model, as shown in Figure 1, includes phases like incident
recognition, evidence collection, preservation, and presentation, with the
examination phase detailed into recovery, harvesting, reduction, and
classification (Casey, 2011). This model emphasises the critical nature of
maintaining evidence integrity and the requirement for expert analysis to
extract and interpret pertinent information, culminating in a report suitable
for legal scrutiny (Casey, 2011). Notably, the analysis or examination stage
is pivotal in all models, demanding specialised knowledge in the relevant DF
area (Mir et al., 2016; Prayudi et al., 2020).

The advent of cloud computing has led to the Digital Forensic as a Service
(DFaaS) model by van Baar et al., integrating evidence preservation and
analysis into an automated, secure software service, marking a significant
evolution in forensic methodologies (van Beek et al., 2015; Du et al., 2017;
van Beek et al., 2020).

###  2.2. Existing Challenges in Digital Forensics

DF is an evolving field, yet the literature highlights that it still undergoes
changes to address ongoing challenges and advancements. Dubey et al. (2023)
assert that DF faces key challenges, including the complexity of data and its
volume, a lack of standardisation, inadequacies in the power of existing tools
to support investigations and issues related to timelines.

In addition to the previously mentioned challenges, issues such as scope creep
in cases due to complexity and vast data, as well as the critical tasks of
selecting and prioritising the right set of evidence, and efficiently
allocating time and investigators for the chosen evidence, were highlighted
(Kalaimannan et al., 2013). Koper et al. (2014) focus on certain issues from
the investigator’s perspective, including challenges in adapting to a system,
unexpected time-saving negations, and frustrations among officers arising from
timelines and the adoption of complex systems.

Automating the DF process using existing technology appears to be a promising
solution for addressing issues related to time management and effectiveness
(Michelet et al., 2023). However, an ongoing challenge revolves around
measuring the accuracy of investigations and ensuring the verification of the
automated process. This aspect remains an open area that requires further
attention and resolution (James and Gladyshev, 2013).

###  2.3. Existing Work With LLMs in Digital Forensics

Scanlon et al. (2023a) analysed using ChatGPT for DF. In their assessment, the
authors evaluated the programming, incident narration, keyword list creation,
and DF teaching abilities of ChatGPT. Their conclusion highlighted that while
ChatGPT exhibited some hallucinations in the output results, it still serves
as an effective assistant for code generation.

Timeline reconstruction aids investigators in deducing the behaviour of an
event. In line with timeline regeneration, Silalahi et al. (2023) proposed a
method to detect anomalies in a drone flight by employing sentiment analysis
with the assistance of a pre-trained LLM. Their approach successfully
discerned differences between normal and abnormal events with an accuracy of
92.5%.

Hansken, a DFaaS platform created by the Netherlands Forensic Institute, is
designed to assist investigators in handling evidence and conducting
investigations more efficiently (van Beek et al., 2015). ChatGPT has been
utilised as an assistance for the Hansken DFaaS system, contributing to
streamlined processes and improved support for investigators (Henseler and van
Beek, 2023). In a successful experiment, evidence traces were provided to
ChatGPT, and it was tasked with analysing the case, ultimately providing
conclusions. This utilisation of ChatGPT showcases its potential in assisting
with the analytical aspects of investigations, highlighting its ability to
process and interpret evidence data (Henseler and van Beek, 2023).

##  3\. Large Language Models

This section explores LLMs, concentrating on three principal aspects.
Initially, it explores the architecture of LLMs, detailing their design and
function. Next, it assesses the usability of LLMs, underscoring the features
and capabilities that render them apt for a wide range of tasks. Finally, it
showcases the versatility of LLMs by discussing their application across
various fields, demonstrating their wide-reaching impact and the extensive
scope of their applications.

###  3.1. Natural Language Processing

Popular LLMs such as Generative Pre-trained Transformer (GPT) (Dale, 2021),
Language Model for Dialogue Applications (LaMDA) (Thoppilan et al., 2022),
Pathways Language Model (PaLM) (Chowdhery et al., 2023), Bidirectional Encoder
Representations from Transformers (BERT) (Devlin et al., 2019), and Large
Language Model Meta AI (LLaMA) (Touvron et al., 2023) stem from advancements
in Natural Language Processing (NLP). NLP, focusing on language-based tasks,
utilises both traditional and deep learning models to enable applications such
as language translation, text processing, and speech recognition (Reshamwala
et al., 2013).

Deep learning, a branch of machine learning, uses complex computational layers
and adaptive weights to enhance prediction accuracy, offering a more refined
analysis than conventional machine learning (LeCun et al., 2015). It has
excelled in image and speech recognition, and natural language understanding,
mimicking the decision-making process of the human brain through artificial
neurons. These neurons form networks capable of intricate pattern recognition
and data analysis. Central to deep learning are neural networks with multiple
hidden layers, which autonomously learn and feature-extract from data,
bypassing the need for manual variable selection. This automatic feature
extraction makes them exceptionally adept at handling complex tasks (Wang et
al., 2017).

###  3.2. LLMs

An LLM is a language model employing neural networks with billions of
parameters, trained on extensive text data. These models are engineered to
comprehend and generate human language. Fundamentally, they rely on multiple
neural network architectures, enabling them to recognise the relationships
between words and phrases within sentences (Fan et al., 2023; Shen et al.,
2023a). These architectures have been a transformative force in the field of
natural language processing. Its capability to excel across a diverse array of
language-related tasks distinguishes it as a game-changer, in contrast to
being tailored for a singular, specific task.

###  3.3. Architecture of LLMs

LLMs utilise deep learning, particularly neural networks, to process and
produce human language. Fundamentally, a language model operates with letters
or words, but since machine learning algorithms and neural networks require
vector inputs, words are vectorised. Each word in the vocabulary is assigned a
unique numerical value for input into neural networks. Through initial random
weight assignments and subsequent backpropagation, words acquire numerical
positions reflecting their semantic similarity, culminating in a word
embedding model (Lai et al., 2016).

####  3.3.1. Word to Vectors

Word embeddings, as introduced by Mikolov et al. (2013), entail precise and
high-dimensional vector representations for words, particularly suited for
extensive datasets comprising billions of text entries. The authors explored
model architectures for word vectorisation, achieving substantial enhancements
in accuracy while requiring lower computational resources and reduced training
time (Mikolov et al., 2013). In the realm of LLMs, the primary objective is to
generate new text based on the extensive dataset, on which it was trained. For
this purpose, Vaswani et al. (2017) introduced the transformer model, aided by
the word-to-vector model. This architecture incorporates a self-attention
mechanism, as well as encoder and decoder processes, enabling the model to
rapidly and simultaneously focus on pertinent information.

####  3.3.2. Transformer Models

The transformer model initially aimed at machine translation, translating
input words into another language, begins with word embedding, where inputs,
termed as tokens, are vectorised. Recognising word order is achieved through
positional encoding, with two main techniques: absolute and relative. Absolute
positional encoding assigns unique vectors to each position, enhancing the
model’s ability to recognise word placement and facilitate position-specific
attention (Ke et al., 2021). Relative positional encoding, on the other hand,
calculates the relative positions of words by introducing a bias term that
quantifies distances between positions, improving the model’s capacity to
understand word relationships within a sequence (Ke et al., 2021).

Self-attention, a core mechanism within the transformer, calculates the
relationship among words in a sentence, allowing the model to assess each
word’s similarity to others and generate unique representations for each (Al-
Rfou et al., 2019). The decoder mirrors the encoder’s steps but uses different
weights, starting with positional encoding and computing self-attention values
to identify the sentence’s initial translation word.

This transformer process, leveraging stacked self-attention and unique
positional encoding, has significantly advanced NLP tasks, including machine
translation, text generation, and summarisation, by executing these processes
in parallel and optimising weights for both encoder and decoder (Vaswani et
al., 2017; Acheampong et al., 2021).

###  3.4. Specifically Trained LLMs

The transformer model and the self-attention mechanism have paved the way for
researchers to train language models on trillions of tokens with billions of
parameters. Several LLMs have been trained and harnessed, each tailored with
specific capabilities for diverse fields such as security, chemistry,
engineering, medicine, business, tourism, and language-related applications.
These models are employed in tasks ranging from detecting security threats,
analysing data, and generating synthetic actions to teaching, code generation,
structured query generation, planning, assisting in medical education,
clinical decision-making, leveraging clinical settings, clinical validation,
understanding general patterns and decision-making, bias detection, addressing
ethical issues, language translations, question answering, information
extraction, and business process automation, among others (Chang et al., 2024;
Xuan-Quy et al., 2023; Tsai et al., 2023; Bran and Schwaller, 2023;
Thirunavukarasu et al., 2023; Karabacak and Margetis, 2023; Eggmann et al.,
2023; Vidgof et al., 2023; Carvalho and Ivanov, 2023; Hou et al., 2023; White
et al., 2023; Odena et al., 2021; Glukhov et al., 2023; Moore et al., 2023;
Caines et al., 2023; Bonner et al., 2023; Zhu et al., 2023; Rajkumar et al.,
2022). The fine-tuning and retraining capabilities of LLMs empower them to be
adapted to specific tasks or behaviours in a predefined manner. Fine-tuning
involves taking an already trained language model and retraining its existing
weights and bias values using a new dataset specific to a particular domain.
This process allows the LLM to be customised and refined for tasks beyond its
original training, enhancing its applicability in specific contexts (Ziegler
et al., 2020). This process results in a new model that is more tailored and
focused on the specified domain. In existing literature, it is frequently
observed that LLMs are fine-tuned with a particular emphasis on engineering
and research-related fields. This targeted fine-tuning ensures that the model
is adept at handling tasks and generating content specifically relevant to the
intricacies of these domains (Chang et al., 2024).

##  4\. Capabilities of Large Language Models

This section focuses on the abilities and capabilities of Language Model
Models (LLMs) as outlined in Section 3.4. Additionally, it discusses the
currently available fine-tuned LLMs that exhibit potential for application in
DF.

###  4.1. Programming/Coding

The ability to generate source code within a specific context is a crucial
proficiency inherent to a language model (Alon et al., 2020). Xu et al. (2022)
conducted a systematic evaluation of six LLMs for code generation across
twelve diverse programming languages. The benchmarking process involved the
use of the HumanEval benchmark and an evaluation dataset designed to assess
the functional correctness of programs generated by an LLM (Chen et al.,
2021). Initially, the dataset was employed for benchmarking Code, a GPT
language model fine-tuned on publicly available code sourced from GitHub (Chen
et al., 2021). The Mostly Basic Programming Problems (MBPP) stands as another
benchmark, comprising 974 programming tasks. It serves as a frequently
employed evaluation dataset for LLMs specialising in code-related tasks (Wei
et al., 2022). Several LLMs explicitly trained for code generation include
Code LLaMA, CodeGen, StarCoder, PanGu-Coder, PanGu-Coder2, WizardCoder,
InCoder 6B, CodeGen-Mono 16B, Code-Davinci-001, Code-Davinci-002, PaLM-
Coder-540B, CodeT5+, InstructCodeT5+, GPT-4 with Reflexion, CodeGeeX,
AlphaCode, Santa-Coder (Rozière et al., 2024; Nijkamp et al., 2022; Li et al.,
2023a; Shen et al., 2023b; Luo et al., 2024; Zhou et al., 2022; Chowdhery et
al., 2023; Wang et al., 2023c; Shinn et al., 2023; Zheng et al., 2023; Li et
al., 2022; Allal et al., [n.d.]; Christopoulou et al., 2022; Chen et al.,
2023c; Ye et al., 2023; Brown et al., 2020; Fried et al., 2023). A higher
value for both HumanEval and MBPP indicates greater accuracy in code
generation for a given task. For detailed information, refer to Table 1, which
presents the counts for HumanEval and MBPP, along with the trained parameter
size for each LLM. Even with the benchmarking rates from HumanEval, Liu et al.
(2023d) raised questions about the accuracy of the functionality of the
generated code. To address this concern, they proposed EvalPlus, an automated
test generation engine, to verify the functional correctness of LLM-generated
code.

Table 1. Trained parameter count, Humaneval and MBPP count for specifically
trained LLMs for code generation.

Model | Parameters |  HumanEval | MBPP
---|---|---|---
Code LLaMA | $34B$ |  62.2 | 61.2
CodeGen | $16.1B$ |  29.28 | 35.28
CodeGen-MONO | $16.1B$ |  29.7 | 42.4
StarCoder | $15.5B$ |  40.8 | 49.5
PanGu-Coder | $2.6B$ |  27.78 | 23
PanGu-Coder2 | $15B$ |  61.64 | -
WizardCoder | $15B$ |  57.3 | 51.8
Code-Davinci-001 (GPT3) | $175B$ |  39.0 | 51.8
Code-Davinci-002 (GPT3.5) | $175B$ |  47.0 | 58.1
PaLM-Coder | $540B$ |  36.0 | 47.0
CodeT5+ | $16B$ |  30.9 | -
InstructCodeT5+ | $16B$ |  35.0 | -
GPT-4 with Reflexion | $1.76T*$ |  91.0 | 77.1
Sanata-Coder | $1.1B$ |  18 | 35
AlphaCode | $1.1B$ |  17.1 | -
Codex-12B | $12B$ |  28.81 | -
code-cushman-001 | $12B$ |  33.5 | 45.9
InCoder 6B | $6.7B$ |  16.4 | 21.3

††* Estimated parameter count as value is not officially released (Rizzo et
al., 2024).

###  4.2. Vision Assistance

Traditional vision assistant systems face limitations in image processing or
recognition, as they are typically trained on fixed types of datasets.
However, with the emergence of LLMs, this paradigm has shifted towards
utilising raw text as the source of supervision (Radford et al., 2021a; Tewel
et al., 2022). Research on visual recognition language models is experiencing
exponential growth, with the number of models surpassing 1,500 in 2023 (Zhang
et al., 2023b). Radford et al. (2021a) introduced a novel method called
Contrastive Language-Image Pre-training (CLIP). This method is efficient and
capable of performing a wide range of tasks during pre-training. It enables
the training of a model to learn a shared representation space for both images
and text, facilitating a deeper understanding of the relationships between the
two modalities. Ramesh et al. (2021) proposes a model for text-to-image
generation, capable of generating images as combinations derived from textual
input or sentences. Moreover, with the model named Generating Images with
Large Language Models (GILL), it becomes feasible to generate text, retrieve
images, generate novel images, and interleave the results into coherent
multimodal dialogues (Koh et al., 2023). VisionLLM is a framework leveraging
LLMs for diverse vision tasks with unified language instruction, demonstrating
generality and flexibility (Wang et al., 2023a). It incorporates a language-
guided image tokeniser and an LLM-based task decoder, capable of handling
open-ended tasks based on provided language instructions (Wang et al., 2023a).

Visual instruction tuning leverages language-only models, such as GPT-4, to
generate multimodal language-image instruction following data. This data is
then utilised to instruction-tune large multimodal models, such as Large
Language and Vision Assistant (LLaVA) (Liu et al., 2023c, b; Achiam et al.,
2023). The open-source LLaVA project introduces an end-to-end trained model,
integrating a vision encoder with an LLM. Notably, LLaVA showcases multimodal
chat capabilities. LLaVa exhibits the capability to engage with images,
providing detailed descriptions and responding to queries with a reported
accuracy of 92.53% (Liu et al., 2023c). This demonstrates its effectiveness in
understanding and generating contextually relevant information about visual
content (Liu et al., 2023c). MiniGPT-4 is an open-source, powerful visual
instruction-tuned LLM, and it demonstrates versatility by generating stories
and poems inspired by provided images and teaching users how to cook based on
visual cues from food photos. This showcases its ability to understand and
respond creatively to diverse visual stimuli (Zhu et al., 2024).

Position-enhanced Visual Instruction Tuning (PVIT) represents an extended
version of Multimodal Large Language Models (MLLMs). It facilitates region-
level encoding in an image, enabling the model to discern and identify
information within specific regions (Chen et al., 2023a). Like PVIT, GPT-4
Region of Interest (ROI) is an instruction-tuned LLM capable of extracting
information such as colour, shape, material, and action from a specified
region (Zhang et al., 2024). This model enables users to interact with both
language and drawing bounding boxes to indicate the area of interest within an
image (Zhang et al., 2024). Other MLLMs, such as Visual ChatGPT, InternGPT,
Flamingo, BLIP-2, and Kosmos, are noted in the literature for their capacity
to assist users in visual-related information (Wu et al., 2023b; Liu et al.,
2023a; Alayrac et al., 2022; Li et al., 2023b; Radford et al., 2021b; Zhao et
al., 2023b; Peng et al., 2023).

Video information is gaining prominence in vision assistance, and Zhao et al.
(2023a) has introduced a novel approach to automatically narrate lengthy
videos using LLMs. UniVL is another language pre-trained model designed for
both multimodal understanding and generation. It is capable of retrieving a
video segment based on text descriptions, generating captions for given video
clips, segmenting a video according to a provided text input, and conducting
multimodal sentiment analysis of a video segment (Luo et al., 2020). VidIL and
MOV are additional MLLMs with similar capabilities, demonstrating proficiency
in video classification and video-language operations such as video
captioning, video question answering, video caption retrieval, and video
future event prediction (Wang et al., 2022; Qian et al., 2022).

These MLLMs adhere to a shared task set, encompassing visual question
answering, visual captioning, visual common-sense reasoning, visual
generation, multimodal affective computing, visual retrieval, vision-language
navigation, multimodal machine translation, visual question generation, and
visual dialoguing, as summarised in Table 2 (Kiros et al., 2014; Uppal et al.,
2022).

Table 2. Capabilities of MLLMs trained for vision assistance Capabilities |  |  GILL |  VisionLLM |  GPT-4 |  LLaVa |  MiniGPT-4 |  Visual ChatGPT |  InternGPT with Husky |  Flamingo |  Kosmos |  UniVL |  VidIL
---|---|---|---|---|---|---|---|---|---|---|---|---
| Image | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |
Visual question answering (Task of providing an answer to a visual input.) | Video |  |  |  |  |  |  |  |  |  | $\checkmark$ | $\checkmark$
| Image | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |
Visual captioning (Task of generate fitting visual descriptions.) | Video |  |  |  |  |  |  |  |  |  | $\checkmark$ | $\checkmark$
| Image | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |
Visual common-sense reasoning (Task of infer understanding from images or video clip.) | Video |  |  |  |  |  |  |  |  |  | $\checkmark$ | $\checkmark$
| Image | $\checkmark$ |  |  |  |  | $\checkmark$ | $\checkmark$ |  |  |  |
Visual generation (Task of generating image or video from a given textual input.) | Video |  |  |  |  |  |  |  |  |  |  |
| Image | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |
Multimodal affective computing (Task of automatically recognition of emotions and causes.) | Video |  |  |  |  |  |  |  |  |  |  |
| Image | $\checkmark$ | $\checkmark$ |  |  |  | $\checkmark$ | $\checkmark$ |  |  |  |
Visual retrieval (Task of language and vision understanding and retrieval.) | Video |  |  |  |  |  |  |  |  |  | $\checkmark$ |
| Image | $\checkmark$ |  |  |  |  | $\checkmark$ | $\checkmark$ |  |  |  |
Vision-language navigation (Task of navigation based on linguistic instructions.) | Video |  |  |  |  |  |  |  |  |  | $\checkmark$ |
| Image |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |
Multimodal machine translation (Task of translation from a video or an image) | Video |  |  |  |  |  |  |  |  |  |  |
| Image |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |
Visual question generation (Task of generating questions for given image or video) | Video |  |  |  |  |  |  |  |  |  |  | $\checkmark$
| Image | $\checkmark$ |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |
Visual dialoguing (Task of automating a conversation about a video or image) | Video |  |  |  |  |  |  |  |  |  |  | $\checkmark$

###  4.3. Conversation

Specific LLMs are trained explicitly for meaningful and coherent dialogues
with humans. An example is Dialogue Generative Pre-trained Transformer
(DialoGPT), a fine-tuned model trained on 174 million conversations from
Reddit (Zhang et al., 2020). DialoGPT exhibits the ability to provide human-
like answers in tested conversations (Zhang et al., 2020). Dettmers et al.
(2023) introduced a fine-tuning mechanism for LLMs named Quantized Pre-trained
Language Model into Low-Rank Adapters (QLoRA). This allows the fine-tuning of
large-parameter LLMs with low training costs. They introduced Guanaco, a fine-
tuned LLM with 65 billion parameters, which achieved a performance level of
99.3%. Falcon-180B and Falcon-40B represent another set of open-source LLMs
with 180 billion and 40 billion parameters. These models are trained to
communicate in multiple languages, allowing users to engage in conversations
in languages other than English (Penedo et al., 2023). To evaluate the
accuracy of human-like dialogue systems, Ou et al. (2023) proposed a dialogue
evaluation benchmark named DialogBench, comprising 12 dialogue tasks to assess
the capabilities of LLMs. In their evaluation, they assessed 28 pre-trained
and instruction-tuned LLMs, demonstrating that GPT-4, ChatGPT, and
KwaiYii-13B-Chat emerged as the top three models for conversations in domains
related to daily life and professional knowledge.

In DF chat conversations, the significance lies in facilitating non-technical
investigators to elucidate terminologies and areas lacking understanding. This
serves a dual purpose, acting as an interactive teacher to enhance
comprehension in discussions (Scanlon et al., 2023a).

###  4.4. Prompt Engineering

Achieving quality outputs from LLMs often relies on providing well-crafted,
meaningful, and precise input queries, known as input prompts. However, even
human-defined natural language instructions may not consistently yield the
best results. Prompt engineering is a methodology that involves carefully
defining and instructing LLMs to generate more accurate and desirable outputs.
Through thoughtful refinement of input prompts, prompt engineering aims to
enhance the performance and effectiveness of LLMs in generating outputs that
align more closely with user expectations and requirements (White et al.,
2023; Zhou et al., 2023). This plays a crucial role in biasing LLMs towards
specific domains or topics, enabling a more targeted and nuanced response. By
carefully crafting prompts, users can guide LLMs to delve deeper into the
nuances of their queries, leading to more accurate and relevant outputs. This
approach enhances the model’s responsiveness to specific areas of interest,
allowing users to fine-tune and tailor their interactions with the LLM for
more precise and meaningful outcomes. Prompt engineering with LLMs is employed
across various sectors, including but not limited to medical, engineering,
construction, and healthcare (B, 2022; Polak and Morgan, 2023).

ChainForge is an open-source Graphical User Interface (GUI) tool developed
specifically for prompt engineering and hypothesis testing derived from LLMs
that can be used in the above-mentioned fields to generate accurate and swift
outputs (Arawjo et al., 2023).

###  4.5. Autonomous Agents

The evolution of LLMs, with their capacity to generate information and
communicate in a manner resembling human interaction, has led to the
development of autonomous agents. The expectation is that these agents will
effectively execute a wide array of tasks, capitalising on the human-like
capabilities inherent in LLMs (Wang et al., 2023d). These autonomous agents
follow a four-stage architecture, encompassing profiling, memory, planning,
and action. Profiling defines the agent’s role, privileges, domain, and
expertise (Wang et al., 2023d). Memory stores information about tasks and
profile data relevant to the environment. Planning involves breaking down
given tasks into subtasks and solving them individually. The action stage is
the final phase where all decisions and subtasks are translated into actions
executed by the agent. Zhang et al. (2023a) developed a framework designed to
facilitate collaboration between AGI agents and humans. This framework enables
planning and communication for specific tasks, leveraging the capabilities of
LLMs Similarly, AgentSims, ToolBench, GameGPT, ChatDev, Voyager, and RecMind
represent a diverse array of autonomous AGI agents developed with distinct
goals and objectives (Wang et al., 2023b, e; Lin et al., 2023; Qin et al.,
2024; Chen et al., 2023b; Qian et al., 2023). Certainly, AutoGen stands out as
a multiagent framework with the capability to autonomously perform tasks or
collaborate with human feedback. This flexibility makes it a versatile tool
for various applications (Wu et al., 2023a).

###  4.6. Limitations and Risks

As explored in the preceding sections, it appears that LLMs possess a vast
range of capabilities. However, it is crucial to acknowledge that they are not
without limitations and risks. In multimodal LLMs, it is a common issue that
they are over-reliant (Wolf et al., 2024). Hadi et al. (2023) highlighted
significant drawbacks associated with LLMs, including issues such as bias,
explainability challenges, reasoning errors, logical errors, hallucinations,
vulnerability to prompt injections, and spelling and grammar errors. These
limitations underscore the importance of a cautious and critical approach when
utilising LLMs in various applications. Additionally, the literature shows
limitations in LLMs, including statistical inconsistency, the absence of
emotional attributes in linguistic responses, and challenges related to fact
verification (Tang et al., 2023; Fröhling and Zubiaga, 2021). These factors
contribute to a comprehensive understanding of the constraints and potential
shortcomings when working with LLMs. Thapa et al. (2023) contend that while
LLMs can indeed diminish the time and costs associated with annotation tasks,
they fall short of completely supplanting human annotation. This is because
they struggle with intricate linguistic constructions, such as idioms, irony,
sarcasm, and metaphor, which can potentially impact the precision of
annotations.

Similar limitations are associated with MLLMs. Issues such as over-reliance on
training data, sensitivity to word order in the input prompts, and
vulnerability to prompts containing extra knowledge represent challenges in
the capabilities of MLLMs (Qi et al., 2023). There are several more concerns
associated with LLMs including restricted input and output text lengths,
limited comprehension of syntax, ethical considerations with the generated
information, constraints with multilingual capabilities, elevated costs
associated with training and maintenance, inadequate understanding of human
behaviours, and limited ability to learn incrementally (Hadi et al., 2023;
Chiang and yi Lee, 2023; Yang et al., 2023).

Despite their considerable capabilities, LLMs are not without risks. Bommasani
et al. (2022); Lund and Wang (2023); Rahman and Santacana (2023) provide
comprehensive overviews of risks linked to LLMs. These include the
homogenisation of outcomes, wherein defects or biases from the foundation
model are inherited by all downstream models. There is also the risk of a
monopoly for the owners of the foundation models, leading to increased
centralisation of power in a single entity. Ethical and legal concerns are
intertwined with privacy and intellectual property issues. Additionally, there
are economic and environmental impacts, raising concerns about the potential
displacement of human workers. Furthermore, inequity and misuse of LLMs, such
as the creation of deepfakes and their application in criminal and unethical
activities, pose additional challenges. Given that LLMs do not inherently
prioritise the accuracy of information, Bender et al. (2021) have underscored
the risk of generating social turbulence, especially, when employed on social
media platforms. Additionally, the use of LLMs is associated with significant
costs, leading to a direct environmental impact due to their substantial
energy consumption (Rillig et al., 2023).

The risks associated with LLMs are predominantly emphasised within Information
Communication and Technology (ICT) and cyberspace. Primary concerns include
the disclosure of personal information, the generation of malicious text, and
the creation of malicious code (Derner and Batistič, 2023).

In response to the increasing prominence of LLMs and Artificial General
Intelligence (AGI), the EU Artificial Intelligence Act (AIA) presently
classifies AI systems into four risk categories based on broad fields of
application. However, this categorisation approach may result in inaccurate
risk estimation and ineffective enforcement. To rectify this issue, the
authors advocate for a risk assessment model that aligns the risk categories
with specific AI scenarios. This model integrates the AIA with the risk
assessment approach employed by the Intergovernmental Panel on Climate Change
(IPCC) and relevant literature, facilitating a more precise estimation of the
magnitude of AI-related risks (Novelli et al., 2023).

The Beyond the Imitation Game benchmark (BIG-bench), serves as an evaluation
framework for LLMs. It encompasses 204 distinct language-related tasks,
spanning contextual and context-free question-answering, reading
comprehension, etc. (Srivastava et al., 2023). It is acknowledged that the
challenge of social biases and English language dependency persists across
nearly all LLMs.

##  5\. Large Language Models For Digital Forensics

Section 5 summarises existing work with LLMs in DF, the feasibility of
employing them, and potential future directions. As discussed in Sections 3
and 4, despite the widespread use of LLMs in various fields to enhance the
efficiency and accuracy of tasks within specific domains, their application in
the field of DF is still relatively new.

Conducting a thorough analysis of the utilisation of LLMs in conjunction with
the stages of the DF process model, as highlighted in Section 2.1, proves to
be a valuable undertaking.

###  5.1. Incident Recognition Phase

In the initial phase of Casey’s DF process model, which delineates the
recognition of an incident, LLMs can serve as a valuable detection mechanism.
In cases of cybercrime, the primary artefacts often involve data logs, data
dumps and network dumps. Fine-tuning an LLM to monitor text-based logs and
related files enables it to discern and identify potential or ongoing
incidents within the environment. In network-related activities, anomaly
detection plays a pivotal role in initiating an incident response. Various
existing anomaly detection techniques are employed in systems for this
purpose. Leveraging its capability to identify patterns in a series of text
data sets, LLMs exhibit potential as an Intrusion Detection System (IDS)
within such systems (Li et al., 2023c).

###  5.2. Collection Phase

While evidence collection or seizure traditionally involves physical tasks
requiring human interaction, LLMs can play a role in identifying and listing
potential pieces of evidence at a crime scene. For instance, in the
examination of photographs or video records from a crime scene, an
investigator can enlist the assistance of a MLLM like LLaVa, GPT-4, or
VisionLLM. These models are capable of processing information within the
images and generating a text-based output, facilitating the interpretation and
categorisation of visual data. While this task may seem simple and within the
capabilities of a human agent, the efficiency becomes particularly evident,
when dealing with a massive-scale investigation involving thousands of
collected artefacts and photographs. Utilising an MLLM for initial processing
can significantly save time, with human agents then focusing on the crucial
task of verification and validation.

###  5.3. Preservation/Acquisition Phase

Preserving evidence is centred on maintaining integrity. To achieve this,
various tools such as EnCase and FTK Imager have been employed, aiding
investigators in streamlining their work processes (Shah et al., 2017). In the
context of preserving disk evidence, it becomes feasible for an investigator
to articulate their requirements in natural language, LLM to generate source
code tailored to the specific needs. Indeed, LLMs specialised in code
generation, such as StarCoder, Code LLaMA, and others, can be fine-tuned and
retrained for specific tasks, including those related to preserving disk
evidence through customised code generation. In certain instances, gathering
live data for forensic investigations becomes crucial, particularly data
collected at the crime scene. For this purpose, investigators may leverage
DFaaS platforms like Hansken. Hansken possesses the capability to amalgamate
custom extraction APIs for data extractions, and these APIs can be developed
using code-generative LLMs (van Beek et al., 2015). This approach enhances the
adaptability and efficiency of the investigative process.

As stated in Section 4.5, the automation of code generation and unit testing
can be facilitated by autonomous agents that utilise LLMs as their core.
AutoGen, being an open-source framework, provides the means to develop AI
agents tailored for specific tasks. These AutoGen agents are not only
customizable and conversational but can also operate in diverse modes,
employing combinations of LLMs, human inputs, and various tools (Wu et al.,
2023a). Positively, automated agents, particularly those developed within
frameworks like AutoGen, can be effectively employed in the preservation
phase. Their capabilities will enable the automated preservation within this
phase, streamlining and enhancing efficiency in the preservation of digital
evidence (Wu et al., 2023a).

###  5.4. Examination Phase

This phase constitutes a pivotal component of the investigation, playing a
crucial role in elucidating the case through activities such as data recovery,
collection, reduction, and classification. For each of these components, LLMs
fine-tuned for scripting can significantly assist, especially at a larger
scale. Within these components, tasks such as keyword search, file recovery,
pattern matching, and fragment reassembly can be achieved with minimal
technical knowledge using LLMs. LLMs can provide valuable assistance in these
tasks by generating new codes, crafting regular expressions, generating
passwords and/or password hash lists for decryption, and creating sample logs
or files. LLMs can generate a set of instructions, queries, and Application
Programming Interface (API) validations from natural language provided by a
human. This opens up the possibility of integrating third-party tools like
Scapy, tshark, John the Ripper, and others seamlessly into the investigative
process, enhancing the toolkit available for DF investigations and the ability
to automate these processes enhances efficiency and effectiveness in the
examination phase of the investigation.

###  5.5. Analysis Phase

The analysis phase involves comprehending the incident and arriving at a
conclusive understanding based on the information gathered during the
examination phase. As also highlighted in Section 2.3, it has been
demonstrated that LLMs are effective in case analysis (Henseler and van Beek,
2023). The use of MLLMs, which possess the capability to interpret images,
broadens the scope for analysing a crime case more comprehensively. LLMs can
be specifically fine-tuned for the analysis of various data types, including
log files, email contents, chat transcripts, call records, file metadata, hex
dumps, memory dumps, and registry hives. Incorporating contents such as event
logs, timestamps, and network traffic captures further enables the effective
recreation of incidents by correlating each data set with the assistance of
LLMs. Also, audio and video-specific MLLMs can assist in analysing the
contents within these formats. This specialised capability has the potential
to significantly save time for investigators in their analyses of audio and
video data during investigations.

The use of automated agents can distribute the analysis workload.
Simultaneously, leveraging Augmented Large Language Models (ALLM) and
Retrieval Augmented Generation (RAG) techniques can enhance real-time
knowledge searching, thereby improving the accuracy of analysis and decision-
making processes (Lewis et al., 2020).

###  5.6. Reporting Phase

The quality and validity of evidence, along with the thoroughness of the
analysis, are encapsulated in the final report. The reporting phase holds
significant weight, as the entire judgement may hinge on this crucial stage.
Notably, DF is experiencing heightened scrutiny with regard to the quality of
the reports, emphasising the importance of precision and clarity in this phase
(Karie et al., 2019). As pointed out by Champod et al. (2016), there is no
standard framework for evaluating and reporting scientific findings to
authorities and stakeholders. To provide assistance and alleviate scrutiny,
incorporating LLMs for report creation is a viable solution. While by
definition, LLMs do not produce deterministic outputs, they could be retrained
using standards such as ISO/IEC 27043:2015, the international standard for the
DF investigation process (Valjarević et al., 2016). A first feasibility study
has already been released by Michelet and Breitinger (2023). Naturally, LLMs
could also help to automate forensic reports and alternative formats (i.e.,
not MS Word) such as HTML or Latex.

###  5.7. Other Possibilities

In general, LLMs can be effectively utilised in each phase of the DF process
model. By integrating the DF process life cycle with automated agents, it
becomes feasible to establish a fully automated DF process throughout an
investigation. For instance, if automated agents are assigned specific roles
and maintain synchronisation within each agent, employing frameworks such as
AutoGen, there is a substantial potential to expedite the completion of one or
multiple DF cases compared to relying solely on human agents. This approach
enhances efficiency, speed, and consistency in the DF investigative process.

Scanlon et al. (2023a) highlights that LLMs can play a significant role in DF
teaching scenarios. This involvement extends to activities such as
storyboarding, synthetic content creation, and synthetic character profiling.
Additionally, fine-tuned LLMs prove to be valuable tools for establishing
evaluation criteria for DF agents, facilitating the creation of exams,
measuring the accuracy and efficiency of investigations through an automated
weighing system, and effectively presenting a case to non-technical or non-
domain individuals, such as judges.

###  5.8. Challenges for LLMs in Digital Forensics

To optimise results, LLMs likely need to be trained with specific forensic
data (i.e., previous case data) to achieve the best results. Given the
complexity and variation of cases, it is questionable how good the training
data is and whether there is sufficient data (Breitinger et al., 2024). Biases
present in training data can lead to skewed interpretations and unjust
outcomes.

Investigative findings and drawing conclusions are also heavily based on
investigator experience and intuition, which are difficult to replicate using
LLMs. Initially, they will only prove effective for certain subtasks within
digital forensics, which raises questions about their superiority over
deterministic programs in certain contexts. For instance, data may be
parsed/converted using a regular piece of software or an LLM. While the former
works deterministically, the latter is more general, but requires the
practitioners to validate the output. This leads to the fundamental necessity
for explainability (i.e., not using blackboxes) for investigations. The
explainability of LLM-generated results remains a daunting challenge, as
understanding the reasons for their, frequently non-deterministic, results is
often elusive (Michelet and Breitinger, 2023).

Maintaining the energy- and resource-intensive infrastructure required to
train and deploy LLMs is a significant financial burden, making it impractical
for smaller forensic laboratories to utilise such technologies. Centralised
systems may be an option but require clear guidelines on how data can/may be
shared and exchanged. While LLMs can serve as valuable tools to support
forensic investigations, it must be recognised that they currently function
best as an aid, not a substitute for human expertise (Scanlon et al., 2023b).
There is a risk that people may place too much trust in the results generated
by LLMs (over-reliance), which could lead to complacency and overlook the need
for detailed human expert analysis and validation.

Finally, ethical and legal considerations must also be discussed. Determining
accountability in cases where LLMs produce false information or are
compromised by hacking. Clarifying responsibilities between developers, users
and regulators is crucial to establish a framework for accountability. If LLM-
generated DF results lead to incorrect information, the responsibility may lie
with the developers for ensuring the accuracy of the model and with the users
for the appropriate interpretation and validation of the results.

###  5.9. Risks of Integration

The integration of LLMs within the DF process comes with inherent risks –
extra to those general LLM limitations outlined in Section 4.6. Notably, in
the examination, analysis, and reporting phases, the utilisation of LLMs
introduces the risk of producing inaccurate information, primarily due to the
phenomenon of inheritance hallucinations associated with these models
(Michelet and Breitinger, 2023; Scanlon et al., 2023b). Additionally, the
biases and obscurities present in an inheritance model may significantly
impact the performance of a DF-focused LLM – potentially leading to the
unacceptable generation of biased or inaccurate information within the DF
process.

It is also crucial to acknowledge that DF LLMs, like any complex models, are
susceptible to adversarial manipulation (Zou et al., 2023). This vulnerability
poses a substantial risk in the context of sensitive domains such as DF, where
the integrity of information gained is paramount. Adversarial attacks can
compromise the reliability of LLM-generated outputs, potentially influencing
the outcomes of various phases within the DF process.

Indeed, despite incorporating human verification, outputs and reports
generated by LLMs within DF applications may encounter challenges regarding
acceptance within the legal systems of different countries. This highlights a
significant usability risk associated with LLM-based DF applications, but one
that can be carefully mitigated by limiting the technology’s deployment as a
human-in-the-loop investigative aid as opposed to directly feeding into any
investigative/judicial decision-making processes.

##  6\. Conclusion

The convergence of LLMs with an array of technologies represents an exciting
synergy. Although the utilisation of LLMs in the realm of DF is still in its
nascent stages, there is evidence of their substantial potential to
significantly augment the efficiency of investigations. The exploration of
investment for LLMs across the entire DF process is considered, aiming to
enhance the productivity and efficiency of investigations. Additionally, the
integration of LLMs into current DF tools is posited to reduce user training
times, as these models comprehend natural language input and provide outputs
accordingly. In the dynamic landscape of LLM applications within DF, promising
avenues for further exploration and advancement unfold.

While the surge in LLM research is promising, it is crucial to balance
enthusiasm with an awareness of existing challenges. The propensity for LLMs
to produce hallucinations highlights the need for human oversight in critical
decision-making processes, underscoring the irreplaceable value of human
judgement, intuition and expertise. A notable limitation is the language
dependency issue, as most LLMs are predominantly trained on English data,
reducing their effectiveness with non-English content. Additionally, the
deployment of LLMs in DF involves significant costs related to infrastructure
for processing evidence. Questions also arise about the validation of task
correctness and quality when automated by LLMs, as well as the legal and
professional acceptance of results obtained with limited human intervention.

Integrating LLMs with automated agents offers a promising path to automating
DF processes, potentially allowing multiple cases to be handled concurrently
for more timely and precise outcomes. This integration could significantly
streamline investigations. Future research should explore the role of LLMs and
AI in DF decision-making. It is essential to focus on validating LLM-generated
outputs to ensure their scope, accuracy, reliability, and trustworthiness in
investigations. Further studies comparing DF outcomes with and without LLM
integration are critical, as they could highlight LLMs’ benefits and
controlled applicability in DF and similar fields.

In essence, while LLMs offer exciting prospects for the future of digital
forensics, a balanced approach that integrates their strengths with human
oversight is essential for harnessing their full potential. Inevitably, LLM-
facilitated DF processes will themselves become the focus of future
investigation.

## References

  * (1)
  * Acheampong et al. (2021) Francisca Adoma Acheampong, Henry Nunoo-Mensah, and Wenyu Chen. 2021.  Transformer models for text-based emotion detection: a review of BERT-based approaches.  _Artificial Intelligence Review_ (2021), 1–41.
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. 2023\.  GPT-4 Technical Report.  _CoRR_ abs/2303.08774 (2023).  https://doi.org/10.48550/ARXIV.2303.08774 arXiv:2303.08774
  * Al-Dhaqm et al. (2020) Arafat Al-Dhaqm, Shukor Abd Razak, Richard Adeyemi Ikuesan, Victor R. Kebande, and Kamran Siddique. 2020.  A Review of Mobile Forensic Investigation Process Models.  _IEEE Access_ 8 (2020), 173359–173375.  https://doi.org/10.1109/ACCESS.2020.3014615
  * Al-Rfou et al. (2019) Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2019.  Character-level language modeling with deeper self-attention. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 3159–3166.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, et al. 2022\.  Flamingo: a Visual Language Model for Few-Shot Learning. In _Advances in Neural Information Processing Systems_ , Vol. 35. Curran Associates, Inc., 23716–23736.
  * Ali (2019) Liaqat Ali. 2019.  Cyber crimes-A constant threat for the business sectors and its growth (A study of the online banking sectors in GCC).  _The Journal of Developing Areas_ 53, 1 (2019).
  * Allal et al. ([n.d.]) Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, et al. [n.d.].  SantaCoder: don’t reach for the stars!  _Deep Learning for Code (DL4C) Workshop_ ([n. d.]).  https://par.nsf.gov/biblio/10416454
  * Alon et al. (2020) Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. 2020.  Structural Language Models of Code. In _International Conference on Machine Learning_. PMLR, 245–256.
  * Arawjo et al. (2023) Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023.  ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing.  arXiv:cs.HC/2309.09128
  * B (2022) Meskó B. 2022.  Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial.  _J Med Internet Res 2023;25:e50638_ (2022).  https://doi.org/10.2196/50638
  * Baryamureeba and Tushabe (2004) Venansius Baryamureeba and Florence Tushabe. 2004.  The enhanced digital investigation process model.  _Digital Investigation_ (2004).
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ (Virtual Event, Canada) _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 610–623.  https://doi.org/10.1145/3442188.3445922
  * Bommasani et al. (2022) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, et al. 2022\.  On the Opportunities and Risks of Foundation Models.  (2022).  arXiv:cs.LG/2108.07258
  * Bonner et al. (2023) Euan Bonner, Ryan Lege, and Erin Frazier. 2023.  Large Language Model-Based Artificial Intelligence in the Language Classroom: Practical Ideas for Teaching.  _Teaching English with Technology_ 23, 1 (2023).
  * Bran and Schwaller (2023) Andres M Bran and Philippe Schwaller. 2023.  Transformers and Large Language Models for Chemistry and Drug Discovery.  (2023).  arXiv:cs.LG/2310.06083
  * Breitinger et al. (2024) Frank Breitinger, Jan-Niclas Hilgert, Christopher Hargreaves, John Sheppard, Rebekah Overdorf, and Mark Scanlon. 2024.  DFRWS EU 10-Year Review and Future Directions in Digital Forensic Research.  _Forensic Science International: Digital Investigation_ (03 2024).
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. 2020\.  Language Models Are Few-Shot Learners. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ (Vancouver, BC, Canada) _(NIPS’20)_. Curran Associates Inc., Red Hook, NY, USA, Article 159.
  * Caines et al. (2023) Andrew Caines, Luca Benedetto, Shiva Taslimipoor, Christopher Davis, Yuan Gao, et al. 2023\.  On the Application of Large Language Models for Language Teaching and Assessment Technology.  (2023).  arXiv:cs.CL/2307.08393
  * Carvalho and Ivanov (2023) Inês Carvalho and Stanislav Ivanov. 2023.  ChatGPT for tourism: applications, benefits and risks.  _Tourism Review_ (2023).
  * Casey (2011) Eoghan Casey. 2011.  _Digital Evidence and Computer Crime: Forensic Science, Computers, and the Internet_.  Academic press.
  * Champod et al. (2016) Christophe Champod, Alex Biedermann, Joëlle Vuille, Sheila Willis, and Jan De Kinder. 2016.  ENFSI guideline for evaluative reporting in forensic science: A primer for legal practitioners.  _Criminal Law and Justice Weekly_ 180, 10 (2016), 189–193.
  * Chang et al. (2024) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024.  A Survey on Evaluation of Large Language Models.  _ACM Trans. Intell. Syst. Technol._ (jan 2024).  https://doi.org/10.1145/3641289
  * Chen et al. (2023c) Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023c.  CodeT: Code Generation with Generated Tests. In _The Eleventh International Conference on Learning Representations_.
  * Chen et al. (2023a) Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu. 2023a.  Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models.  arXiv:cs.CV/2308.13437
  * Chen et al. (2023b) Dake Chen, Hanbin Wang, Yunhao Huo, Yuzhao Li, and Haoyang Zhang. 2023b.  GameGPT: Multi-agent Collaborative Framework for Game Development.  arXiv:cs.AI/2310.08067
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, et al. 2021\.  Evaluating Large Language Models Trained on Code.  _ArXiv_ abs/2107.03374 (2021).
  * Chiang and yi Lee (2023) Cheng-Han Chiang and Hung yi Lee. 2023.  Can Large Language Models Be an Alternative to Human Evaluations?  arXiv:cs.CL/2305.01937
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, et al. 2023\.  PaLM: Scaling Language Modeling with Pathways.  _Journal of Machine Learning Research_ 24, 240 (2023), 1–113.  http://jmlr.org/papers/v24/22-1144.html
  * Christopoulou et al. (2022) Fenia Christopoulou, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, et al. 2022\.  PanGu-Coder: Program Synthesis with Function-Level Language Modeling.  _ArXiv_ abs/2207.11280 (2022).
  * Dale (2021) Robert Dale. 2021.  GPT-3: What’s it good for?  _Natural Language Engineering_ 27, 1 (2021), 113–118.
  * Derner and Batistič (2023) Erik Derner and Kristina Batistič. 2023.  Beyond the Safeguards: Exploring the Security Risks of ChatGPT.  (2023).  arXiv:cs.CR/2305.08005
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  QLoRA: Efficient Finetuning of Quantized LLMs.  (2023).  arXiv:cs.LG/2305.14314
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171–4186.  https://doi.org/10.18653/V1/N19-1423
  * Du et al. (2017) Xiaoyu Du, Nhien-An Le-Khac, and Mark Scanlon. 2017.  Evaluation of Digital Forensic Process Models with Respect to Digital Forensics as a Service. In _Proceedings of the 16th European Conference on Cyber Warfare and Security (ECCWS 2017)_. ACPI, Dublin, Ireland, 573–581.
  * Du and Scanlon (2019) Xiaoyu Du and Mark Scanlon. 2019.  Methodology for the Automated Metadata-Based Classification of Incriminating Digital Forensic Artefacts. In _Proceedings of the 14th International Conference on Availability, Reliability and Security_ (Canterbury, CA, United Kingdom) _(ARES ’19)_. Association for Computing Machinery, New York, NY, USA, Article 43.  https://doi.org/10.1145/3339252.3340517
  * Dubey et al. (2023) Himanshu Dubey, Shobha Bhatt, and Lokesh Negi. 2023.  Digital Forensics Techniques and Trends: A Review.  _The International Arab Journal of Information Technology (IAJIT)_ 20, 4 (2023), 644–654.
  * Eggmann et al. (2023) Florin Eggmann, Roland Weiger, Nicola U Zitzmann, and Markus B Blatz. 2023.  Implications of large language models such as ChatGPT for dental medicine.  _Journal of Esthetic and Restorative Dentistry_ (2023).
  * Fan et al. (2023) Lizhou Fan, Lingyao Li, Zihui Ma, Sanggyu Lee, Huizi Yu, and Libby Hemphill. 2023.  A Bibliometric Review of Large Language Models Research from 2017 to 2023.  _ArXiv_ abs/2304.02020 (2023).
  * Fried et al. (2023) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023.  InCoder: A Generative Model for Code Infilling and Synthesis. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.  https://openreview.net/pdf?id=hQwb-lbM6EL
  * Fröhling and Zubiaga (2021) Leon Fröhling and Arkaitz Zubiaga. 2021.  Feature-Based Detection of Automated Language Models: Tackling GPT-2, GPT-3, and Grover.  _PeerJ Computer Science_ 7 (2021), e443.
  * Glukhov et al. (2023) David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, and Vardan Papyan. 2023.  LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?  (2023).  arXiv:cs.AI/2307.10719
  * Hadi et al. (2023) Muhammad Usman Hadi, qasem al tashi, Rizwan Qureshi, Abbas Shah, amgad muneer, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, and Seyedali Mirjalili. 2023.  Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects.  (11 2023).  https://doi.org/10.36227/techrxiv.23589741.v4
  * Henseler and van Beek (2023) Hans Henseler and Harm van Beek. 2023.  ChatGPT as a Copilot for Investigating Digital Evidence. In _Proceedings of the Third International Workshop on Artificial Intelligence and Intelligent Assistance for Legal Professionals in the Digital Workplace (LegalAIIA 2023) co-located with the 19th International Conference on Artificial Intelligence and Law (ICAIL 2023), Braga, Portugal, June 19, 2023_ _(CEUR Workshop Proceedings)_ , Jack G. Conrad, Daniel W. Linna Jr., Jason R. Baron, Hans Henseler, Paheli Bhattacharya, Aileen Nielsen, Jyothi K. Vinjumur, Jeremy Pickens, and Amanda Jones (Eds.), Vol. 3423. CEUR-WS.org, 58–69.  https://ceur-ws.org/Vol-3423/paper6.pdf
  * Hou et al. (2023) Xinying Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John C. Grundy, and Haoyu Wang. 2023.  Large Language Models for Software Engineering: A Systematic Literature Review.  _ArXiv_ abs/2308.10620 (2023).
  * James and Gladyshev (2013) Joshua James and Pavel Gladyshev. 2013.  Challenges with Automation in Digital Forensic Investigations.  _ArXiv_ abs/1303.4498 (2013).
  * Kalaimannan et al. (2013) Ezhil Kalaimannan, Jatinder N.D. Gupta, and Seong-Moo Yoo. 2013.  Maximizing Investigation Effectiveness in Digital Forensic Cases. In _2013 International Conference on Social Computing_. 618–623.  https://doi.org/10.1109/SocialCom.2013.93
  * Karabacak and Margetis (2023) Mert Karabacak and Konstantinos Margetis. 2023.  Embracing Large Language Models for Medical Applications: Opportunities and Challenges.  _Cureus_ 15, 5 (2023).
  * Karie et al. (2019) Nickson. M. Karie, Victor R. Kebande, H.S. Venter, and Kim-Kwang Raymond Choo. 2019.  On the Importance of Standardizing the Process of Generating Digital Forensic Reports.  _Forensic Science International: Reports_ 1 (2019), 100008.  https://doi.org/10.1016/j.fsir.2019.100008
  * Ke et al. (2021) Guolin Ke, Di He, and Tie-Yan Liu. 2021.  Rethinking Positional Encoding in Language Pre-training.  (2021).  https://openreview.net/forum?id=09-528y2Fgf
  * Kiros et al. (2014) Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014.  Multimodal Neural Language Models. In _Proceedings of the 31st International Conference on Machine Learning_ _(Proceedings of Machine Learning Research)_ , Vol. 32. PMLR, Bejing, China, 595–603.  https://proceedings.mlr.press/v32/kiros14.html
  * Kitchenham (2004) Barbara Kitchenham. 2004.  Procedures for Performing Systematic Reviews.  _Keele, UK, Keele University_ 33, 2004 (2004), 1–26.
  * Koh et al. (2023) Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. 2023.  Generating Images with Multimodal Language Models. In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.).  http://papers.nips.cc/paper_files/paper/2023/hash/43a69d143273bd8215578bde887bb552-Abstract-Conference.html
  * Koper et al. (2014) Christopher S. Koper, Cynthia Lum, and James J. Willis. 2014.  Optimizing the Use of Technology in Policing: Results and Implications from a Multi-Site Study of the Social, Organizational, and Behavioural Aspects of Implementing Police Technologies.  _Policing: A Journal of Policy and Practice_ 8, 2 (05 2014), 212–221.  https://doi.org/10.1093/police/pau015
  * Lai et al. (2016) Siwei Lai, Kang Liu, Shizhu He, and Jun Zhao. 2016.  How to generate a good word embedding.  _IEEE Intelligent Systems_ 31, 6 (2016), 5–14.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015.  Deep learning.  _Nature_ 521, 7553 (2015), 436–444.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, et al. 2020\.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In _Advances in Neural Information Processing Systems_ , Vol. 33. Curran Associates, Inc., 9459–9474.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  , Article 814 (2023).
  * Li et al. (2023a) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, et al. 2023a.  StarCoder: May the Source Be with You!  _Transactions on Machine Learning Research_ (2023).  https://openreview.net/forum?id=KoFOg41haE Reproducibility Certification.
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, et al. 2022\.  Competition-Level Code Generation with AlphaCode.  _Science_ 378, 6624 (2022), 1092–1097.  https://doi.org/10.1126/science.abq1158 arXiv:https://www.science.org/doi/pdf/10.1126/science.abq1158
  * Li et al. (2023c) Yuanze Li, Haolin Wang, Shihao Yuan, Ming Liu, Debin Zhao, Yiwen Guo, Chen Xu, Guangming Shi, and Wangmeng Zuo. 2023c.  Myriad: Large Multimodal Model by Applying Vision Experts for Industrial Anomaly Detection.  arXiv:cs.CV/2310.19070
  * Lin et al. (2023) Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping, and Qin Chen. 2023.  AgentSims: An Open-Source Sandbox for Large Language Model Evaluation.  arXiv:cs.AI/2308.04026
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023b.  Improved Baselines with Visual Instruction Tuning.  _CoRR_ abs/2310.03744 (2023).  https://doi.org/10.48550/ARXIV.2310.03744 arXiv:2310.03744
  * Liu et al. (2023c) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023c.  Visual Instruction Tuning. In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.).  http://papers.nips.cc/paper_files/paper/2023/hash/6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html
  * Liu et al. (2023d) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023d.  Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.  arXiv:cs.SE/2305.01210
  * Liu et al. (2023a) Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, et al. 2023a.  InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language.  (2023).  arXiv:cs.CV/2305.05662
  * Lund and Wang (2023) Brady D Lund and Ting Wang. 2023.  Chatting About ChatGPT: How AI and GPT May Impact Academia and Libraries?  _Library Hi Tech News_ 40, 3 (2023), 26–29.
  * Luo et al. (2020) Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou. 2020.  UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation.  _ArXiv_ abs/2002.06353 (2020).
  * Luo et al. (2024) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024.  WizardCoder: Empowering Code Large Language Models with Evol-Instruct.  (2024).  https://openreview.net/forum?id=UnUwSIgK5W
  * Lutui (2016) Raymond Lutui. 2016.  A multidisciplinary digital forensic investigation process model.  _Business Horizons_ 59, 6 (2016), 593–604.
  * Michelet and Breitinger (2023) Gaëtan Michelet and Frank Breitinger. 2023.  ChatGPT, Llama, can you write my report? An experiment on assisted digital forensics reports written using (Local) Large Language Models.  _CoRR_ abs/2312.14607 (2023).  https://doi.org/10.48550/ARXIV.2312.14607 arXiv:2312.14607
  * Michelet et al. (2023) Gaëtan Michelet, Frank Breitinger, and Graeme Horsman. 2023.  Automation for Digital Forensics: Towards a definition for the community.  _Forensic Science International_ (2023), 111769.
  * Mikolov et al. (2013) Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient Estimation of Word Representations in Vector Space. In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1301.3781
  * Mir et al. (2016) Sara Sarwar Mir, Umar Shoaib, and Muhammad Shahzad Sarfraz. 2016.  Analysis of digital forensic investigation models.  _Int. J. Comput. Sci. Inform. Secur_ 14, 11 (2016).
  * Moore et al. (2023) Steven Moore, Richard Tong, Anjali Singh, Zitao Liu, Xiangen Hu, et al. 2023\.  Empowering Education with LLMs: The Next-Gen Interface and Content Generation. In _International Conference on Artificial Intelligence in Education_. Springer, 32–37.
  * Mukherjee and Haque (2018) Shaaswat Mukherjee and Shazia Haque. 2018.  Review Paper on Digital Forensics Practices: A Road Map for Building Digital Forensics Capability.  _Iconic Research and Engineering Journals_ 1, 9 (2018), 96–99.
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, et al. 2022\.  CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In _The Eleventh International Conference on Learning Representations_.
  * Novelli et al. (2023) Claudio Novelli, Federico Casolari, Antonino Rotolo, Mariarosaria Taddeo, and Luciano Floridi. 2023.  Taking AI Risks Seriously: A New Assessment Model for the AI Act.  _AI & Society_ (12 07 2023).  https://doi.org/10.1007/s00146-023-01723-z
  * Nozza et al. (2022) Debora Nozza, Federico Bianchi, Anne Lauscher, Dirk Hovy, et al. 2022\.  Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals. In _Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion_. Association for Computational Linguistics.
  * Odena et al. (2021) Augustus Odena, Charles Sutton, David Martin Dohan, Ellen Jiang, Henryk Michalewski, et al. 2021\.  Program Synthesis with Large Language Models.  (2021).  arXiv:2108.07732
  * Ou et al. (2023) Jiao Ou, Junda Lu, Che Liu, Yihong Tang, Fuzheng Zhang, Di Zhang, Zhongyuan Wang, and Kun Gai. 2023.  DialogBench: Evaluating LLMs as Human-like Dialogue Systems.  (2023).  arXiv:cs.CL/2311.01677
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, et al. 2023\.  The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only.  (2023).  https://openreview.net/forum?id=kM5eGcdCzq
  * Peng et al. (2023) Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023.  Kosmos-2: Grounding Multimodal Large Language Models to the World.  _CoRR_ abs/2306.14824 (2023).  https://doi.org/10.48550/ARXIV.2306.14824 arXiv:2306.14824
  * Polak and Morgan (2023) Maciej P. Polak and Dane Morgan. 2023.  Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering - Example of ChatGPT.  _CoRR_ abs/2303.05352 (2023).  https://doi.org/10.48550/ARXIV.2303.05352 arXiv:2303.05352
  * Prayudi et al. (2020) Yudi Prayudi, Ahmad Ashari, and Tri Kuntoro Priyambodo. 2020.  The Framework to Support the Digital Evidence Handling: A Case Study of Procedures for the Management of Evidence in Indonesia.  _Journal of Cases on Information Technology (JCIT)_ 22, 3 (2020), 51–71.
  * Qi et al. (2023) Shuhan Qi, Zhengying Cao, Jun Rao, Lei Wang, Jing Xiao, and Xuan Wang. 2023.  What Is the Limitation of Multimodal LLMs? A Deeper Look into Multimodal LLMs Through Prompt Probing.  _Information Processing & Management_ 60, 6 (2023), 103510.  https://doi.org/10.1016/j.ipm.2023.103510
  * Qian et al. (2023) Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, Maosong Sun, and Wei Liu. 2023.  Communicative Agents for Software Development.  _ArXiv_ abs/2307.07924 (2023).
  * Qian et al. (2022) Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, and Yin Cui. 2022.  Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models.  arXiv:cs.CV/2207.07646
  * Qin et al. (2024) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024.  ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs.  https://openreview.net/forum?id=dHng2O0Jjr
  * Radford et al. (2021a) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, et al. 2021a.  Learning Transferable Visual Models From Natural Language Supervision. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research)_ , Vol. 139. PMLR, 8748–8763.
  * Radford et al. (2021b) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, et al. 2021b.  Learning Transferable Visual Models From Natural Language Supervision. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research)_ , Vol. 139. PMLR, 8748–8763.
  * Rahman and Santacana (2023) Noorjahan Rahman and Eduardo Santacana. 2023.  Beyond Fair Use: Legal Risk Evaluation for Training LLMs on Copyrighted Text. In _ICML Workshop on Generative AI and Law_.
  * Rajkumar et al. (2022) Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022.  Evaluating the Text-to-SQL Capabilities of Large Language Models.  arXiv:cs.CL/2204.00498
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, et al. 2021\.  Zero-Shot Text-to-Image Generation. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research)_ , Vol. 139. PMLR, 8821–8831.
  * Ray (2023) Partha Pratim Ray. 2023.  ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope.  _Internet of Things and Cyber-Physical Systems_ 3 (2023), 121–154.  https://doi.org/10.1016/j.iotcps.2023.04.003
  * Reshamwala et al. (2013) Alpa Reshamwala, Dhirendra Mishra, and Prajakta Pawar. 2013.  Review on natural language processing.  _IRACST Engineering Science and Technology: An International Journal (ESTIJ)_ 3, 1 (2013), 113–116.
  * Rillig et al. (2023) Matthias C. Rillig, Marlene Ågerstrand, Mohan Bi, Kenneth A. Gould, and Uli Sauerland. 2023.  Risks and Benefits of Large Language Models for the Environment.  _Environmental Science & Technology_ 57, 9 (2023), 3464–3466.  https://doi.org/10.1021/acs.est.3c01106 PMID: 36821477.
  * Rizzo et al. (2024) Michael G. Rizzo, Nathan Cai, and David Constantinescu. 2024.  The performance of ChatGPT on orthopaedic in-service training exams: A comparative study of the GPT-3.5 turbo and GPT-4 models in orthopaedic education.  _Journal of Orthopaedics_ 50 (2024), 70–75.  https://doi.org/10.1016/j.jor.2023.11.056
  * Rozière et al. (2024) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, et al. 2024\.  Code Llama: Open Foundation Models for Code.  (2024).  arXiv:cs.CL/2308.12950
  * Scanlon et al. (2023a) Mark Scanlon, Frank Breitinger, Christopher Hargreaves, Jan-Niclas Hilgert, and John Sheppard. 2023a.  ChatGPT for digital forensic investigation: The good, the bad, and the unknown.  _Forensic Science International: Digital Investigation_ 46 (2023), 301609.  https://doi.org/10.1016/j.fsidi.2023.301609
  * Scanlon et al. (2023b) Mark Scanlon, Bruce Nikkel, and Zeno Geradts. 2023b.  Digital forensic investigation in the age of ChatGPT.  _Forensic Science International: Digital Investigation_ 44 (03 2023), 301543.  https://doi.org/10.1016/j.fsidi.2023.301543
  * Shah et al. (2017) Makhdoom Syed Muhammad Baqir Shah, Shahzad Saleem, and Roha Zulqarnain. 2017.  Protecting digital evidence integrity and preserving chain of custody.  _Journal of Digital Forensics, Security and Law_ 12, 2 (2017), 12.
  * Shen et al. (2023b) Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, and Qianxiang Wang. 2023b.  PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback.  (2023).  arXiv:cs.CL/2307.14936
  * Shen et al. (2023a) Yiqiu Shen, Laura Heacock, Jonathan Elias, Keith D. Hentel, Beatriu Reig, George Shih, and Linda Moy. 2023a.  ChatGPT and Other Large Language Models Are Double-edged Swords.  _Radiology_ 307, 2 (2023), e230163.  https://doi.org/10.1148/radiol.230163 PMID: 36700838.
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023.  Reflexion: Language Agents with Verbal Reinforcement Learning. In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Silalahi et al. (2023) Swardiantara Silalahi, Tohari Ahmad, and Hudan Studiawan. 2023.  Transformer-based Sentiment Analysis for Anomaly Detection on Drone Forensic Timeline. In _2023 11th International Symposium on Digital Forensics and Security (ISDFS)_. 1–6.  https://doi.org/10.1109/ISDFS58141.2023.10131749
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, et al. 2023\.  Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_ (2023).  https://openreview.net/forum?id=uyTL5Bvosj
  * Tang et al. (2023) Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2023.  The Science of Detecting LLM-Generated Texts.  (2023).  arXiv:cs.CL/2303.07205
  * Tewel et al. (2022) Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. 2022.  ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 17918–17928.
  * Thapa et al. (2023) Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023.  From Humans to Machines: Can ChatGPT-like LLMs Effectively Replace Human Annotators in NLP Tasks. In _Workshop Proceedings of the 17th International AAAI Conference on Web and Social Media_.
  * Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023.  Large language models in medicine.  _Nature Medicine_ 29, 8 (2023), 1930–1940.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, et al. 2022\.  LaMDA: Language Models for Dialog Applications.  _ArXiv_ abs/2201.08239 (2022).
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, et al. 2023\.  LLaMA: Open and Efficient Foundation Language Models.  _ArXiv_ abs/2302.13971 (2023).
  * Tsai et al. (2023) Meng-Lin Tsai, Chong Wei Ong, and Cheng-Liang Chen. 2023.  Exploring the use of large language models (LLMs) in chemical engineering education: Building core course problem models with Chat-GPT.  _Education for Chemical Engineers_ 44 (2023), 71–95.
  * Uppal et al. (2022) Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumder, Soujanya Poria, et al. 2022\.  Multimodal research in vision and language: A review of current and emerging trends.  _Information Fusion_ 77 (2022), 149–171.  https://doi.org/10.1016/j.inffus.2021.07.009
  * Valjarević et al. (2016) Aleksandar Valjarević, Hein Venter, and Ranko Petrović. 2016.  ISO/IEC 27043:2015 — Role and Application. In _2016 24th Telecommunications Forum (TELFOR)_. 1–4.  https://doi.org/10.1109/TELFOR.2016.7818718
  * van Baar et al. (2014) R.B. van Baar, H.M.A. van Beek, and E.J. van Eijk. 2014.  Digital Forensics as a Service: A game changer.  _Digital Investigation_ 11 (2014), S54–S62.  https://doi.org/10.1016/j.diin.2014.03.007 Proceedings of the First Annual DFRWS Europe.
  * van Beek et al. (2015) H.M.A. van Beek, E.J. van Eijk, R.B. van Baar, M. Ugen, J.N.C. Bodde, and A.J. Siemelink. 2015.  Digital Forensics as a Service: Game on.  _Digital Investigation_ 15 (2015), 20–38.  https://doi.org/10.1016/j.diin.2015.07.004 Special Issue: Big Data and Intelligent Data Analysis.
  * van Beek et al. (2020) Harm MA van Beek, Jeroen van den Bos, Abdul Boztas, EJ Van Eijk, R Schramp, and M Ugen. 2020.  Digital forensics as a service: Stepping up the game.  _Forensic Science International: Digital Investigation_ 35 (2020), 301021.
  * van Beek et al. (2015) Harm MA van Beek, EJ Van Eijk, Ruud B van Baar, Mattijs Ugen, JNC Bodde, and AJ Siemelink. 2015.  Digital forensics as a service: Game on.  _Digital Investigation_ 15 (2015), 20–38.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need. In _Advances in Neural Information Processing Systems_ , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc.
  * Vidgof et al. (2023) Maxim Vidgof, Stefan Bachhofner, and Jan Mendling. 2023.  Large Language Models for Business Process Management: Opportunities and Challenges. In _Business Process Management Forum_ , Chiara Di Francescomarino, Andrea Burattin, Christian Janiesch, and Shazia Sadiq (Eds.). Springer Nature Switzerland, Cham, 107–123.
  * Wang et al. (2023e) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023e.  Voyager: An Open-Ended Embodied Agent with Large Language Models.  arXiv:cs.AI/2305.16291
  * Wang et al. (2017) Haohan Wang, Bhiksha Raj, and Eric P. Xing. 2017.  On the Origin of Deep Learning.  _CoRR_ abs/1702.07800 (2017).  arXiv:1702.07800 http://arxiv.org/abs/1702.07800
  * Wang et al. (2023d) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. 2023d.  A Survey on Large Language Model based Autonomous Agents.  _CoRR_ abs/2308.11432 (2023).  https://doi.org/10.48550/ARXIV.2308.11432 arXiv:2308.11432
  * Wang et al. (2023a) Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. 2023a.  VisionLLM: Large Language Model Is Also an Open-Ended Decoder for Vision-Centric Tasks.  arXiv:cs.CV/2305.11175
  * Wang et al. (2023b) Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Xiaojiang Huang, Yanbin Lu, and Yingzhen Yang. 2023b.  RecMind: Large Language Model Powered Agent For Recommendation.  arXiv:cs.IR/2308.14296
  * Wang et al. (2023c) Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. 2023c.  CodeT5+: Open Code Large Language Models for Code Understanding and Generation.  (Dec. 2023), 1069–1088.  https://doi.org/10.18653/v1/2023.emnlp-main.68
  * Wang et al. (2022) Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, et al. 2022\.  Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners. In _Advances in Neural Information Processing Systems_ , Vol. 35. Curran Associates, Inc., 8483–8497.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, et al. 2022\.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  _Advances in Neural Information Processing Systems_ 35 (2022), 24824–24837.
  * White et al. (2023) Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023.  A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT.  (2023).  arXiv:cs.SE/2302.11382
  * Wolf et al. (2024) Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, and Amnon Shashua. 2024.  Fundamental Limitations of Alignment in Large Language Models.  (2024).  arXiv:cs.CL/2304.11082
  * Wu et al. (2023b) Chenfei Wu, Sheng-Kai Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023b.  Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models.  _ArXiv_ abs/2303.04671 (2023).
  * Wu et al. (2023a) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang (Eric) Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023a.  _AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation_.  Technical Report MSR-TR-2023-33. Microsoft.  https://www.microsoft.com/en-us/research/publication/autogen-enabling-next-gen-llm-applications-via-multi-agent-conversation-framework/
  * Wu et al. (2020) Tina Wu, Frank Breitinger, and Stephen O’Shaughnessy. 2020.  Digital forensic tools: Recent advances and enhancing the status quo.  _Forensic Science International: Digital Investigation_ 34 (2020), 300999.
  * Xu et al. (2022) Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022.  A Systematic Evaluation of Large Language Models of Code. In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_ (San Diego, CA, USA) _(MAPS 2022)_. Association for Computing Machinery, New York, NY, USA, 1–10.  https://doi.org/10.1145/3520312.3534862
  * Xuan-Quy et al. (2023) Dao Xuan-Quy, Le Ngoc-Bich, Ngo Bac-Bien, Phan Xuan-Dung, et al. 2023\.  LLMs’ Capabilities at the High School Level in Chemistry: Cases of ChatGPT and Microsoft Bing Chat.  (2023).  https://doi.org/10.26434/chemrxiv-2023-kxxpd
  * Yang et al. (2023) Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023.  Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization.  arXiv:cs.CL/2302.08081
  * Ye et al. (2023) Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023.  A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models.  _ArXiv_ abs/2303.10420 (2023).
  * Zhang et al. (2023a) Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua Tenenbaum, Tianmin Shu, and Chuang Gan. 2023a.  Building Cooperative Embodied Agents Modularly with Large Language Models.  https://openreview.net/forum?id=oBQVCTpKXW
  * Zhang et al. (2023b) Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2023b.  Vision-Language Models for Vision Tasks: A Survey.  arXiv:cs.CV/2304.00685
  * Zhang et al. (2024) Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. 2024.  GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest.  (2024).  https://openreview.net/forum?id=DzxaRFVsgC
  * Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, et al. 2020\.  DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Online, 270–278.  https://doi.org/10.18653/v1/2020.acl-demos.30
  * Zhao et al. (2023b) Bo Zhao, Boya Wu, and Tiejun Huang. 2023b.  SVIT: Scaling up Visual Instruction Tuning.  arXiv:cs.CV/2307.04087
  * Zhao et al. (2023a) Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. 2023a.  Learning Video Representations From Large Language Models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6586–6597.
  * Zheng et al. (2023) Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, et al. 2023\.  CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X.  (2023), 5673–5684.  https://doi.org/10.1145/3580305.3599790
  * Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. 2022\.  Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In _The Eleventh International Conference on Learning Representations_.
  * Zhou et al. (2023) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023.  Large Language Models are Human-Level Prompt Engineers.  (2023).  https://openreview.net/forum?id=92gvk82DE-
  * Zhu et al. (2024) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2024.  MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models.  https://openreview.net/forum?id=1tZbq88f27
  * Zhu et al. (2023) Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji rong Wen. 2023.  Large Language Models for Information Retrieval: A Survey.  _ArXiv_ abs/2308.07107 (2023).
  * Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, et al. 2020\.  Fine-Tuning Language Models from Human Preferences.  (2020).  arXiv:cs.CL/1909.08593
  * Zou et al. (2023) Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023.  Universal and Transferable Adversarial Attacks on Aligned Language Models.  _CoRR_ abs/2307.15043 (2023).  https://doi.org/10.48550/ARXIV.2307.15043 arXiv:2307.15043