# DivLog: Log Parsing with Prompt Enhanced In-Context Learning

Junjielong Xu<EMAIL_ADDRESS>School of Data Science, The
Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China ,  Ruichun
Yang<EMAIL_ADDRESS>School of Data Science, The Chinese
University of Hong Kong, Shenzhen (CUHK-Shenzhen), China ,  Yintong Huo
<EMAIL_ADDRESS>Dept. of Computer Science & Engineering, The Chinese
University of Hong Kong, China ,  Chengyu Zhang<EMAIL_ADDRESS>ETH Zurich,Switzerland and  Pinjia He<EMAIL_ADDRESS>School of Data
Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen),
ChinaShenzhen Research Institute of Big Data, China

(2024)

###### Abstract.

Log parsing, which involves log template extraction from semi-structured logs
to produce structured logs, is the first and the most critical step in
automated log analysis. However, current log parsers suffer from limited
effectiveness for two reasons. First, traditional data-driven log parsers
solely rely on heuristics or handcrafted features designed by domain experts,
which may not consistently perform well on logs from diverse systems. Second,
existing supervised log parsers require model tuning, which is often limited
to fixed training samples and causes sub-optimal performance across the entire
log source. To address this limitation, we propose DivLog, an effective log
parsing framework based on the in-context learning (ICL) ability of large
language models (LLMs). Specifically, before log parsing, DivLog samples a
small amount of offline logs as candidates by maximizing their diversity.
Then, during log parsing, DivLog selects five appropriate labeled candidates
as examples for each target log and constructs them into a prompt. By mining
the semantics of examples in the prompt, DivLog generates a target log
template in a training-free manner. In addition, we design a straightforward
yet effective prompt format to extract the output and enhance the quality of
the generated log templates. We conducted experiments on 16 widely-used public
datasets. The results show that DivLog achieves (1) 98.1% Parsing Accuracy,
(2) 92.1% Precision Template Accuracy, and (3) 92.9% Recall Template Accuracy
on average, exhibiting state-of-the-art performance.

Log Parsing, Large Language Model, In-Context Learning

††conference: 46th International Conference on Software Engineering; April
2024; Lisbon, Portugal††journalyear: 2024††copyright: acmlicensed††conference:
2024 IEEE/ACM 46th International Conference on Software Engineering; April
14–20, 2024; Lisbon, Portugal††booktitle: 2024 IEEE/ACM 46th International
Conference on Software Engineering (ICSE ’24), April 14–20, 2024, Lisbon,
Portugal††price: 15.00††doi: 10.1145/3597503.3639155††isbn:
979-8-4007-0217-4/24/04††ccs: Software and its engineering Software creation
and management

##  1\. Introduction

Modern software systems, including online services such as Google Search and
Bing Search, and system software such as Android and Windows, have become an
essential part of our lives, serving millions of users globally. These systems
produce valuable software logs continuously, providing a rich resource for
maintainers to perform downstream tasks, such as anomaly detection (Du et al.,
2017; He et al., 2016a; Lou et al., 2010; Zhang et al., 2019), root cause
analysis (Amar and Rigby, 2019; He et al., 2018; Lin et al., 2016), and
program verification (Ding et al., 2015; Shang et al., 2013). The first step
of log analysis is log parsing, i.e., converting semi-structured log messages
into structured log messages. Manual log parsing is impractical due to the
enormous volume of logs generated (He et al., 2021). Therefore, numerous data-
driven automatic log parsers have been proposed, including traditional
unsupervised parsers (Fu et al., 2009a; Du and Li, 2016; Dai et al., 2020; He
et al., 2017b) and deep learning (DL) based supervised parsers (Liu et al.,
2022; Huo et al., 2021; Le and Zhang, 2023a). Existing log parsers primarily
distinguish between constants and variables in a log message without the guide
of logging statements. As shown in Fig. 1, constants are the tokens written by
developers in the logging statements (e.g., a description of a software
operation), while variables are tokens that record run-time environments
(e.g., a directory path). These constants make up a log template, while
variables are treated as parameters.

Figure 1. A simple example of log parsing. The logging statements are
typically not accessible in industrial scenarios.

However, existing log parsers still face limited robustness (Zhu et al., 2019;
Le and Zhang, 2023a), leading to unsatisfactory accuracy in diverse logs for
two reasons:

First, unsupervised log parsers utilize specially designed features or
heuristics (e.g., n-gram (Dai et al., 2020), prefix tree (Du and Li, 2016; He
et al., 2017b)) based on domain knowledge to extract common patterns for log
parsing. As a result, they often fall short in log sources whose template
design does not match well with the handcrafted features. For example, the
most widely-used parser, Drain, (He et al., 2017b) assumes that leading tokens
are constants, and logs from the same template share the same length. However,
they are inadequate for logs from E8 event in Proxifier (Zhu et al., [n. d.]),
e.g., "ss.bdimg.com:80 close, 89652 bytes (87.5 KB) sent, 599249 bytes (585
KB) received, lifetime 37:51", where the prefix is a variable (i.e., URL) and
log length is flexible due to the optional tokens in brackets (i.e., no
bracket tokens when the transferred bytes less than 1KB). Moreover,
unsupervised parsers usually need preprocessing (He et al., 2021), wherein
regex replace common variable tokens with the wildcard "<*>". Developers
typically invest extra effort in designing distinct regex for different
datasets with their domain knowledge due to the significant impact of regex
quality on parsing performance (He et al., 2016b), which further limits their
usability.

Second, existing supervised log parsers typically need to train models to mine
the data characteristics in target log samples (Huo et al., 2021; Le and
Zhang, 2023a). This training process may restrict the models to the specific
training data, leading to sub-optimal generalization performance across the
overall dataset, especially when the dataset exhibits significant sample
diversity. For instance, the state-of-the-art parser, LogPPT (Le and Zhang,
2023a), extracts a few of the most frequent variable tokens from the
adaptively selected logs to generate virtual labels for model training.
However, for diverse and complex datasets such as Mac (Zhu et al., [n. d.]),
there might be no representative logs or variables for overall dataset, as the
dataset contains large variance of logs and variables (e.g., E253 event only
contains one token without variable, while E277 event contains 78 tokens with
36 variables) and imbalanced template distribution (e.g., the number of logs
in each template ranging from 1 to 166, with over 50% of templates containing
only one log). Overall, there is a pressing need for a more effective log
parsing technique that matches the following requirements: (1) Sufficiently
automated to reduce the human effort and domain knowledge involved. (2)
Demonstrates decent results on complex data and ensures stability across
various log datasets.

To this end, this paper proposes DivLog, an effective and training-free log
parsing framework without requiring human effort in feature design or
hyperparameter tuning. DivLog adopts in-context learning (ICL) ability of
large language models (LLMs) (Liu et al., 2023) to generate log templates via
mining the common features of related log examples in the prompt.
Specifically, DivLog employs GPT-3 (Brown et al., 2020) as the backbone, the
first LLM that exhibits the ability to learn semantics in prompt contexts.
Before log parsing, DivLog selects a fixed amount of logs by explicitly
maximizing their diversity to build a candidate set for subsequent prompting.
During the parsing process, it retrieves the top five most relative log
examples and their labels from candidate set for each target log based on
their similarity. Then, DivLog organizes them in a special prompt format,
which is designed to enhance ICL performance for log parsing. With the
guidance of examples and their labels in the prompt, DivLog generates log
templates without necessitating model tuning. DivLog has been evaluated on 16
public datasets from LogPAI. (Zhu et al., 2019) The results show that DivLog
achieves (1) 98.1% Parsing Accuracy, (2) 92.1% Precision Template Accuracy,
and (3) 92.9% Recall Template Accuracy when using the prompt examples from the
same log source of the target log message, outperforms the current best method
by 6.5%, 24.7%, 18.8%, respectively. Furthermore, DivLog exhibits stable
parsing quality across 16 different datasets, with standard deviations of only
3.58%, 7.66%, and 7.65% for the three accuracy metrics.

This paper makes the following main contributions:

  * •

It proposes DivLog, the first general log parsing framework that exploits the
in-context learning ability for log parsing by mining common patterns in
prompt examples.

  * •

It designs a general prompt format to explicitly control the output structure
and ensure the quality of the generated log template from LLMs, which can be
generalized to enhance other prompting methods.

  * •

It presents the evaluation of DivLog on 16 public datasets using three
different metrics. The results show that DivLog achieves the SOTA performance,
outperforming all existing log parsing tools.

##  2\. Motivation And Background

Figure 2. An example of ICL on commonsense reasoning task.

###  2.1. Log Parsing

Log parsing is the initial and most critical step in automated log analysis
(He et al., 2021; Zhu et al., 2019), which aims to extract log templates from
semi-structured logs to produce structured logs. Specifically, as shown in
Fig. 1, in log parsing, a log parser is required to first extract the log
headers, typically consisting of timestamps (i.e., specific date and time) and
verbosity level (e.g., ERROR, INFO, and DEBUG). Since log headers are
automatically generated by the logger, their format is fixed and can usually
be extracted easily. Therefore, log parsers primarily focus on extracting
constants from log messages as log templates. An intuitive method to extract
constants from log messages is to manually design regular expressions for each
log template. However, this method is impractical for modern software systems
due to the continuously increasing volume of log templates in industrial
scenarios (Wang et al., 2022b; Zhang et al., 2019), resulting in an
unaffordable effort to maintain log parsing rules (He et al., 2021).
Therefore, numerous data-driven log parsers have been proposed, including
unsupervised parsers (Fu et al., 2009a; Du and Li, 2016; Dai et al., 2020; He
et al., 2017b) and supervised parsers (Liu et al., 2022; Huo et al., 2021; Le
and Zhang, 2023a). However, they suffer from unsatisfactory effectiveness in
diverse log sources due to two reasons in design as stated in Sec. 1: First,
unsupervised parsers utilize specially designed features or heuristics based
on domain knowledge for pattern extraction, leading to failure in logs that do
not match well with the features. Second, supervised parsers typically require
model training on target log samples to learn features, making it open to the
pitfalls of unsatisfactory generalization ability, resulting in sub-optimal
performance on logs with a large sample diversity. Therefore, we believe a
properly evaluated effective log parsing methodology that is suitable for
diverse logs is needed.

###  2.2. Large Language Model

Large language models (LLMs) are deep neural networks that typically consist
of 100 million to over 1 billion parameters and are obtained through self-
supervised pre-training on massive valuable corpora, e.g., Wikipedia, GitHub,
arXiv, .etc. Existing LLMs are predominantly based on the Transformer
architecture. Depending on their specific structures, LLMs can be categorized
into three main types: encoder-decoder LLMs (e.g., T5 (Raffel et al., 2020),
BART (Lewis et al., 2019)), encoder-only LLMs (e.g., BERT (Devlin et al.,
2018), RoBERTa (Liu et al., 2019)), and decoder-only LLMs (e.g., GPT series
(Radford et al., 2018, 2019; Brown et al., 2020)). These different LLM
structures are often suited for various Natural Language Processing (NLP)
tasks. For instance, encoder-only LLMs are commonly employed for tasks like
text classification, while decoder-only LLMs are usually used for text
generation.

The emergence of LLMs has fundamentally transformed the paradigm of NLP from
training models from scratch to fine-tuning existing pre-trained models, i.e.,
pre-train & fine-tune paradigm. This shift has had profound implications,
benefiting numerous tasks in NLP and other domains. Specifically, in software
engineering, fine-tuned LLMs have significant impacts on various tasks, e.g.,
log parsing (Le and Zhang, 2023a), code review generation (Li et al., 2022),
fuzzing test case generation (Deng et al., 2022), automated program repair
(Xia and Zhang, 2023; Xia et al., 2023), and root cause analysis (Ahmed et
al., 2023).  However, this task-specific fine-tuning could excessively
restrict the model to the training data, diminishing its pre-training derived
generalization ability, despite improving downstream task performance. (Kumar
et al., 2022)

###  2.3. In-Context Learning

In recent years, with the increase in model sizes and richer training corpora,
LLMs have notably grown in power. Researchers have discovered that, in
numerous instances, merely presenting the task description as a textual prompt
to pre-trained LLMs suffices for performing various tasks, eliminating the
necessity for task-specific fine-tuning. (Liu et al., 2023) The approach is
known as pre-training, prompting, and predicting. (Liu et al., 2023) For
example, when classifying the sentiment of the sentence “I feel good.”, we can
append a prompt “I am experiencing $[MASK]$ emotion.” and ask the LLM to fill
the blank $[MASK]$ with a polar word.

Furthermore, since the advent of GPT-3, researchers have observed that LLMs
possess the capability of learning in the prompt context to perform inference.
This few-shot prompting is known as in-context learning (ICL). (Brown et al.,
2020; Dong et al., 2022). Specifically, if a prompt contains (1) a clear
instruction specifying a particular task, (2) a few examples with ground-truth
labels providing task-specific knowledge, and (3) a query from this particular
task, the LLM can generate an answer for the query by mining the semantics
between the examples and the query based on the instruction. For instance,
Fig. 2 illustrates an example of ICL on commonsense reasoning. Multiple
studies have demonstrated that LLMs perform well on various complex problems
with ICL, such as fact retrieval (Zhao et al., 2021) and mathematical
reasoning (Wei et al., 2022). People typically believe that LLMs have gained
the ability to complex reason from a large amount of pre-training data, which
enables LLMs to generate the expected answer of the query based on the
demonstration examples in the prompt (Dong et al., 2022).

Since the ICL directly uses the knowledge obtained from pre-trained LLMs along
with the information in the instruction and examples in the prompt, there is
no significant gap between pre-training and downstream tasks (Wang et al.,
2022a), making the output more consistent with the human expectation. It
avoids the issues of over-reliance on specific data and enables high-quality
predictions on the overall target log sources. To this end, we proposed
DivLog, the first attempt to automatically and efficiently accomplish the log
parsing task within the framework of ICL.

Figure 3. The workflow of DivLog framework. The candidates need to be sampled
and labeled before log parsing.

##  3\. DivLog

###  3.1. Overveiw

Our method, DivLog, performs high-quality log parsing under the ICL paradigm
without training. As depicted in Fig. 3, DivLog is composed of an offline
sampling stage and an online parsing stage. Specifically, DivLog uses GPT-3
(Brown et al., 2020), a pre-trained LLM that excels in multiple natural
language tasks, as the backbone. Before parsing begins, DivLog samples a small
number of logs from offline log data, maximizing the differences between
samples, and provides them to developers for annotation (i.e., supplying
templates for each log) to construct a candidate set. During the parsing
process, DivLog selects the five most similar labeled examples from the
candidate set based on the semantics of the target log. These examples are
then assembled in a specially designed prompt format and provided for in-
context inference. Subsequently, the LLM learning semantics in the prompt
context to analogy the template of the target log. Since DivLog does not
necessitate model training or the handcrafted features in both the offline and
online stages, it mitigates the potential limitations as discussed in Sec. 1.

###  3.2. Problem Definition

Figure 4. DPP Sampler: Sampling the most diverse logs as the candidates of the
prompt examples from offline logs for subsequent online parsing.

This paper explicitly defines the log parsing task as a log template
generation task for log messages. Specifically, for a raw log message $x$
containing $n$ tokens after tokenization (denoted as
$x=[t^{r}_{1},t^{r}_{2},...,t^{r}_{n}]$) as input, our proposed method,
DivLog, are required to generate a sequence $y$ consisting $n$ tokens (denoted
as $y=[t^{p}_{1},t^{p}_{2},...,t^{p}_{n}]$) within the locator pair (defined
in Sec. 3.7) as the log template of $x$. The difference between $x$ and $y$ is
that all variables in $x$ are substituted by wildcard $\langle$*$\rangle$ in
$y$. The relationship $\mathcal{F}$ of $t^{r}_{i}$ and $t^{p}_{i}$ can be
written as follows:

(1) $t^{p}_{i}=\mathcal{F}(t^{r}_{i})=\left\\{\begin{aligned}
&``\langle*\rangle"&\ \ \text{if $t^{r}_{i}$ is variable}\\\ &\ \ t^{r}_{i}&\
\ \text{if $t^{r}_{i}$ is constant}\end{aligned}\right.$

For example, as shown in Fig. 1, for a raw log message "Setting block size to
1919810", the generated log template is "Setting block size to
$\langle$*$\rangle$". In the online parsing (inference) stage, the generated
sequence will be considered as the log template, and the different tokens
between the raw log and the generated sequence will be served as parameters.

###  3.3. Model Backbone

Figure 5. KNN Selector: Selecting the most similar logs along with their
labels as examples from the candidate set for the subsequent prompting.

The performance of the large language model is a key factor in the success of
ICL. Considering that log messages are semi-structured sentences that are
mainly composed of natural language descriptions (i.e., log template), we
chose GPT-3 (Brown et al., 2020), an LLM that is pre-trained on an extremely
large amount of semantic information from the open-source corpus, as the
backbone for DivLog. We will not delve into the architectural details of GPT-3
in this paper. However, it is worth noting that from GPT-3 onwards, it was
gradually discovered that LLMs emerged with the ability to perform ICL, which
enables LLMs to explore new knowledge from a small number of demonstration
examples and to exploit it for the query to generate an appropriate answer in
the prompt (Dong et al., 2022). It is the reason that motivates us to adopt
GPT-3 as our backbone, as stated in Sec. 2.3. Since DivLog utilizes the LLM in
a black-box manner, the backbone can be replaced at will as long as the model
or relevant API is accessible.

###  3.4. Candidate Sampling

Algorithm 1 DPP: Enhanced Candidate Sampling

0: logs from target sources $\mathcal{X}=\\{x_{i}\\}^{N}_{i=1}$, encoder
$\epsilon(\cdot)$ ,candidate number: $K$

1: Init: $\boldsymbol{c_{i}}=[]$, $j=\arg\max_{i\in Z}\log(d_{i}^{2})$,
$\mathcal{C}=\mathcal{C}\cup\\{x_{j}\\}$

2: $V=\epsilon(\mathcal{X})$

3: $L=\text{sim}(V,V)$, $d_{i}^{2}$ = $L_{ii}$

4: while $\mathcal{C}\text{.}size\leq K$ do

5: for $i\in\mathcal{X}\backslash\mathcal{C}$ do

6: $e_{i}=(L_{ji}-c_{j}^{T}c_{i})/d_{j}$

7: $\boldsymbol{c_{i}}=\begin{bmatrix}\boldsymbol{c_{i}}&e_{i}\end{bmatrix}$

8: $d_{i}^{2}=d_{i}^{2}-e_{i}^{2}$

9: end for

10: $j=\arg\max_{i\in Z}\log(d_{i}^{2})$,
$\mathcal{C}=\mathcal{C}\cup\\{x_{j}\\}$

11: end while

11: Candidate set $\mathcal{C}$

Before log parsing, DivLog needs to construct a set of labeled log samples as
the candidates of prompt examples for subsequent ICL inference. However, since
the collected logs are unlabeled and manual labeling is labor-intensive,
people usually only sample a subset of logs for labeling. To enhance the
robustness of DivLog and reduce the effort of manual labeling, we use
Determinantal Point Process (DPP) (Chen et al., 2018), a classic algorithm for
maximizing the sample diversity, to select a small number of logs as
candidates from the offline data for labeling. By explicitly optimizing the
diversity of the candidate set, DivLog can achieve an equitable selection of
unlabeled log samples from different templates, thereby reducing the potential
risk of inductive bias that may arise from the unbalanced sample distribution.
Moreover, using DPP for sampling yields stable results for each offline log
dataset, which could mitigate the impact of randomness in the entire DivLog
workflow.

The detailed algorithm is shown in Algo. 1. Specifically, DivLog first encodes
log samples in the provided log dataset $\mathcal{X}=\\{x_{i}\\}^{N}_{i=1}$
from text sequence $x_{i}$ to vector $v_{i}$ via OpenAI embedding model, and
then concatenates them to an embedding matrix
$V=\epsilon(\mathcal{X})=[v_{1},v_{2},...,v_{N}]$. Then, DivLog calculates the
similarity matrix $L=\text{sim}(V,V)$ (i.e., kernel matrix in original paper
(Chen et al., 2018)). Based on the similarity matrix $L$, DivLog can
iteratively select samples that maximize the marginal gain in total
dissimilarity for the current candidate set in a greedy manner until satisfies
the maximum candidate set capacity $K$. In our implementation, we use the
cosine distance to evaluate the similarity between any two tensors
$\boldsymbol{v}_{i}$ and $\boldsymbol{v}_{j}$, which is shown in Eq. 2

(2)
$\text{sim}(\boldsymbol{v}_{i},\boldsymbol{v}_{j}):=\cos(\boldsymbol{v}_{i},\boldsymbol{v}_{j})=\frac{\boldsymbol{v}_{i}^{T}\boldsymbol{v}_{j}}{\|\boldsymbol{v}_{i}\|_{2}\|\boldsymbol{v}_{j}\|_{2}},$

Therefore, we can obtain a stable candidate set $C$ for the following in-
context inference stage.

###  3.5. Example Selection

During log parsing, DivLog required selecting several log examples in the
candidate set as the prompt examples for each target log (i.e., the query).
Recently, several studies (Rubin et al., 2021; Liu et al., 2021) have pointed
out that the selection of examples greatly influences the performance of ICL-
based approaches. For instance, Liu et al. (Liu et al., 2021) found that using
examples closer to the query can significantly enhance the quality of
inference. To strengthen the accuracy of DivLog, we adopt $k$-Nearest Neighbor
($k$NN), a straightforward clustering algorithm, to select several log
examples that are most similar to the query as demonstration examples. Through
deliberate selection of log samples that resemble the query as examples,
DivLog effectively learns the semantics and common patterns of the query from
data that aligns closely with its characteristics. Furthermore, the examples
chosen from $k$NN exhibit higher similarity to one another, leading to a more
compact data distribution in the prompt. This facilitates the LLM in
comprehending the log semantics within the context.

The detailed algorithm is shown in Algo. 2 Specifically, DivLog also begins by
encoding all log candidates $x_{i}\in\mathcal{C}$ as well as the query $x_{q}$
into embedded vectors $v_{i}$ and $v_{q}$. Then, for each vectorized query
$v_{q}$, it calculates the similarity sim$(v_{q},v_{i})$ between it and all
candidates $v_{i}$. After iterating over the entire candidate set, DivLog
generates a distance map $\mathcal{D}$ to record the similarity between the
query vector $v_{q}$ and all candidate queries $v_{i}$. Subsequently, by
querying the distance map $\mathcal{D}$, the top-$k$ ($k$=5 by default) most
similar log samples can be extracted as demonstration examples $\mathcal{E}$.
In our implementation, the similarity metric sim$(v_{i},v_{j})$ also
represents the cosine distance as shown in Eq. 2.

###  3.6. Example Permutation

To further strengthen the parsing performance, DivLog deliberately reorders
the selected examples $\mathcal{E}$ in the ascending order based on their
similarity to the query $\text{sim}(v_{q},v_{i})$ during prompt construction.
This is because recent research (Zhao et al., 2021) suggests that LLM’s ICL
exhibits a recency bias characteristic, whereby the LLM is more susceptible to
the inductive bias of the examples closer to the query when generating an
answer prediction. By ordering examples based on similarity, the example
closer to the query is more similar to the query, making it easier for the LLM
to learn the hidden relations between the last example and its label, and
generate a prediction closer to the query’s ground-truth label. The advantages
of this approach will be further elaborated in subsequent experimental results
5.2.

Algorithm 2 $k$NN: Augmented Example Selection

0: query $x_{q}$, candidate set $\mathcal{C}=\\{x_{i}\\}^{K}_{i=1}$, encoder
$\epsilon(\cdot)$, example number:$k$

1: Init: distance map $\mathcal{D}=\emptyset$ and $\mathcal{E}=\emptyset$

2: $v_{q}=\epsilon(x_{q})$

3: for $x_{i}\in\mathcal{C}$ do

4: $v_{i}=\epsilon(x_{i})$

5: $d_{qi}=\text{sim}(v_{q},v_{i})$

6: $\mathcal{D}=\mathcal{D}\cup\\{d_{qi}:x_{i}\\}$

7: end for

8: extract values with top-$k$ largest keys from $\mathcal{D}$ to
$\mathcal{E}$

8: Example set $\mathcal{E}$

###  3.7. Prompt Format

Figure 6. An example of a prompt and its expected output.

A carefully designed prompt format that consists of a clear and concise
instruction along with several relevant examples are necessary for LLM to
comprehend the log parsing task, as discussed in Sec. 2.3. However, providing
a naive prompt without output restriction could be impractical for log parsing
since the LLM may produce extraneous text beyond the query answer, potentially
compromising parsing quality. For instance, in the sentiment classification
task depicted in Fig.2, the output might include explanatory content, such as
“It is negative. Because awful is not a positive word”, whereas only the word
“negative” is relevant to the task. Thus, even if the anticipated response is
present in the generated text, automatically extracting the template precisely
is still challenging.

To filter the redundant contents in the raw output, we include a locator pair
$``\langle$START$\rangle"$ and $``\langle$END$\rangle"$ in the prompt to limit
the answer we require within the locators. Specifically, in DivLog, each
example’s log template is enclosed by the locator pair. With the guide of the
restricted format for demonstration, LLM is inclined to generate the log
template within the same locator pair via analogy (Dong et al., 2022). An
example of a complete prompt is presented in Fig.6, where the last example is
the most similar one to the query, and the expected raw output is
"$\langle$START$\rangle$ JVM with ID: jvm_$\langle$*$\rangle$ given task:
attempt_$\langle$*$\rangle$ $\langle$END$\rangle$" along with other redundant
output generated by the model. Then, DivLog can simply utilize regular
expressions to extract the expected log template in the output. Notably, even
with locators, the output often generates redundant text after the expected
answer. However, we can easily extract the expected answer with the help of
locators and just simply ignore the rest of redundant text.

##  4\. Experiment Setup

###  4.1. Research Question Design

We conducted extensive experiments on 16 public datasets to answer the
following research questions:

  * •

RQ1: How effective and stable is DivLog?

  * •

RQ2: How do different settings affect DivLog?

Specifically, RQ1 investigates DivLog’s parsing effectiveness (i.e., accuracy
and robustness (Zhu et al., 2019; Le and Zhang, 2023a)) on various log sources
through comparison to other cutting-edge log parsers. RQ2 further discusses
the contribution of each component of DivLog through ablation study.
Furthermore, it also explores the impact of different configurations through
comparative experiments.

###  4.2. Environment and Implementation

In our experiments, we utilized HTTP requests to invoke the OpenAI APIs and
interact with the LLM (i.e., GPT-3 curie) and encoder (i.e., text-search-
babbage-query-001) to get the raw output response of the origin prompt. We
also employed Python 3.9 to implement Algo. 1, Algo. 2, and various evaluation
scripts to construct the complete prompt construction in a local machine with
Ubuntu 20.04.5 LTS. Specifically, in Algo. 1, DivLog samples 200 logs from
each log dataset to construct the candidate set, while in Algo. 2, it selects
5 labeled examples from the candidate set to create prompt for each query.
These settings remain consistent throughout the evaluation process.

###  4.3. Datasets

Our experiments are conducted on 16 public log datasets from LogPAI (Zhu et
al., 2019), the most widely-used benchmark in log analysis. The LogPAI
datasets cover a variety of log types, including distributed systems,
standalone software, supercomputers, PC operating systems, mobile systems, and
microservices. In each dataset, all logs are labeled with ground-truth
templates in advance with a unique Event ID. The logPAI datasets have been
widely used in multiple log parsers’ evaluations, and we will keep following
them to use LogPAI datasets as standard benchmarks.

###  4.4. Metrics

Following recent studies (Khan et al., 2022; Le and Zhang, 2023a), our
evaluation uses three metrics: Parsing Accuracy (PA), Precision Template
Accuracy (PTA), and Recall Template Accuracy (RTA), where the last two metrics
are also known as Template Accuracy (TA) (Khan et al., 2022). Specially, we
use PA to evaluate the parsing performance at the message level and use PTA
and RTA to evaluate the parsing performance at the template level. The
concepts of these metrics are as follows:

####  4.4.1. Parsing Accuracy (PA)

Parsing Accuracy (PA), also known as Message Level Accuracy (MLA) (Liu et al.,
2022), is defined as the ratio of ”correctly parsed” log messages over the
total number of log messages, where a log is considered to be ”correctly
parsed” if and only if all constants and variables are exactly divided in the
log template. Since PA is not related to the number of templates, but only to
the total number of parsed log messages, we use PA as a message-level
evaluation metric to assess the most basic parsing capability of the parsers.

####  4.4.2. Template Accuracy (TA)

Template Accuracy (TA) is proposed by Kan et al. (Khan et al., 2022), which
gives template-level guidance of parsing quality. Specifically, TA comprises
two metrics, Precision Template Accuracy (PTA) and Recall Template Accuracy
(RTA). PTA measures the ratio of correctly identified templates to the total
number of identified templates, while RTA measures the ratio of correctly
identified templates to the total number of ground-truth templates. The
concept of ”correctly identified” templates means that all log messages from
this log template are ”correctly parsed” (defined in the PA context).
Therefore, it is obvious that PTA and RTA are more stringent metrics than PA,
and we aim to use them to showcase the strong parsing ability of DivLog.

Additionally, we do not continue to adopt Grouping Accuracy (GA), which is
widely used for unsupervised parser evaluation in the past. This is mainly due
to two reasons. Firstly, GA has a broader definition of ”correct parsing,”
which reduces its reference value for supervised parsers. Specifically, GA
only requires that logs belonging to the same template be correctly clustered,
without considering whether constants and variables in the logs are
distinguished correctly. Secondly, although GA aims to evaluate the clustering
ability of logs at the template level, its value is affected by the message
number in each cluster, rather than the total number of templates (Khan et
al., 2022). Compared to GA, TA can provide more effective evaluation
capabilities at the template level. 111We also provide GA evaluation in
anonymous repository as a reference.

###  4.5. Baselines

We selected LenMa (Shima, 2016), Spell (Du and Li, 2016), Drain (He et al.,
2017b), and Logram (Dai et al., 2020), the top-performing unsupervised log
parsers, and LogPPT (Le and Zhang, 2023a), the current state-of-the-art
supervised log parser, as our baselines for comparison. Specifically, these
unsupervised log parsers are based on handcrafted features, i.e., log length,
longest common subsequence, prefix tree, and n-gram, while LogPPT fine-tunes a
pre-trained language model, RoBERTa (Liu et al., 2019), and adapts it to log
parsing. Notably, supervised parsers typically perform better than
unsupervised parsers, and the only parser that DivLog can compare in an apple-
to-apple manner is LogPPT.  However, as we cannot reproduce the results of
LogPPT from its implementation on GitHub (Le and Zhang, 2023b), we solely rely
on the reported paper results for comparison. Specifically, in the GitHub
implementation, LogPPT employs the Adaptive Random Testing algorithm to select
32 labeled logs for model training and evaluates on all 2,000 logs in each
dataset. Notably, LogPPT’s accuracy saturates after using 32 logs as
supervision for training (see Fig. 5 in the LogPPT paper (Le and Zhang,
2023a)), while DivLog only uses 5 examples selected from the DPP sampled 200
candidates as supervision for online parsing without training. Thus, we
believe the comparison is fair.  The baselines’ results are consistent with
the reported data in previous research. (Le and Zhang, 2023a)

##  5\. Evaluation Results

Table 1. Accuracy comparison with the cutting-edge log parsers on LogPAI
datasets (%).

Dataset | LenMa | Spell | Drain | Logram | LogPPT | DivLog
---|---|---|---|---|---|---
PA | PTA | RTA | PA | PTA | RTA | PA | PTA | RTA | PA | PTA | RTA | PA | PTA | RTA | PA | PTA | RTA
Andriod | 72.2 | 59.0 | 60.1 | 24.1 | 24.4 | 27.2 | 73.0 | 56.6 | 62.0 | 42.8 | 38.7 | 27.2 | 76.7 | 58.4 | 68.4 | 94.3 | 84.6 | 86.1
Apache | 29.3 | 42.9 | 50.0 | 28.5 | 25.0 | 16.7 | 100.0 | 100.0 | 100.0 | 50.9 | 33.3 | 83.3 | 99.4 | 83.3 | 83.3 | 100.0 | 100.0 | 100.0
BGL | 15.4 | 9.6 | 25.8 | 32.9 | 12.1 | 12.5 | 44.4 | 33.9 | 30.8 | 17.0 | 15.7 | 16.7 | 97.0 | 68.6 | 78.3 | 98.7 | 91.7 | 92.5
Hadoop | 24.2 | 20.6 | 28.9 | 19.6 | 20.8 | 23.7 | 43.9 | 36.8 | 34.2 | 37.0 | 18.7 | 17.5 | 89.5 | 54.0 | 58.8 | 99.6 | 98.3 | 99.1
HDFS | 12.5 | 37.5 | 42.9 | 48.7 | 57.1 | 57.1 | 95.9 | 81.3 | 92.9 | 1.8 | 19.4 | 42.9 | 90.2 | 85.7 | 85.7 | 99.9 | 86.7 | 92.9
HealthApp | 28.9 | 3.4 | 46.7 | 15.2 | 21.5 | 18.7 | 24.1 | 8.3 | 34.7 | 26.3 | 3.5 | 25.3 | 78.9 | 85.3 | 85.3 | 99.6 | 94.7 | 96.0
HPC | 67.1 | 12.9 | 50.0 | 53.2 | 38.3 | 39.1 | 67.2 | 38.8 | 41.3 | 67.9 | 37.7 | 43.5 | 94.7 | 73.6 | 84.8 | 99.7 | 91.5 | 93.5
Linux | 13.2 | 41.0 | 41.4 | 10.9 | 17.3 | 14.7 | 19.4 | 43.4 | 42.2 | 18.5 | 13.5 | 4.3 | 94.9 | 47.5 | 49.1 | 99.7 | 95.8 | 95.8
Mac | 15.5 | 14.6 | 19.4 | 3.3 | 5.7 | 5.9 | 27.7 | 21.2 | 24.9 | 25.2 | 16.4 | 20.2 | 67.3 | 43.6 | 53.4 | 86.3 | 67.6 | 66.6
OpenSSH | 15.5 | 25.0 | 23.1 | 12.7 | 25.0 | 23.1 | 53.4 | 52.0 | 50.0 | 48.2 | 29.0 | 34.6 | 97.6 | 48.9 | 84.6 | 93.9 | 96.2 | 92.6
OpenStack | 19.1 | 18.7 | 60.5 | 0.0 | 0.0 | 0.0 | 18.7 | 5.5 | 39.5 | 11.2 | 9.3 | 9.3 | 90.7 | 84.4 | 88.4 | 99.9 | 95.3 | 95.3
Proxifier | 50.6 | 10.6 | 87.5 | 47.8 | 22.2 | 25.0 | 52.7 | 26.9 | 87.5 | 0.0 | 0.0 | 0.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 100.0
Spark | 2.3 | 12.4 | 33.3 | 33.6 | 34.5 | 27.8 | 37.6 | 50.0 | 41.7 | 27.5 | 3.1 | 22.2 | 99.1 | 60.0 | 58.3 | 99.9 | 97.2 | 97.2
Thunderbird | 17.1 | 27.1 | 34.2 | 3.9 | 19.1 | 16.8 | 19.1 | 29.9 | 36.9 | 12.8 | 20.8 | 6.7 | 92.6 | 50.6 | 59.1 | 98.5 | 91.8 | 95.3
Windows | 26.6 | 37.9 | 44.0 | 0.4 | 11.5 | 12.0 | 69.6 | 46.3 | 50.0 | 37.4 | 15.2 | 10.0 | 98.3 | 55.4 | 72.0 | 99.8 | 91.8 | 90.0
Zookeeper | 45.7 | 30.0 | 36.0 | 45.3 | 22.0 | 26.0 | 49.8 | 39.1 | 36.0 | 51.6 | 26.1 | 24.0 | 99.0 | 74.1 | 86.0 | 99.8 | 88.7 | 94.0
Average | 28.5 | 25.2 | 42.7 | 23.8 | 22.3 | 21.6 | 49.8 | 41.9 | 50.3 | 29.8 | 18.8 | 24.2 | 91.6 | 67.4 | 74.1 | 98.1 | 92.1 | 92.9

###  5.1. RQ1: How effective and stable is DivLog?

This section compares the accuracy and robustness of DivLog with other state-
of-the-art log parsers on 16 publicly available log datasets. The study
evaluates Parsing Accuracy (PA), Precision Template Accuracy (PTA), and Recall
Template Accuracy (RTA) to demonstrate the accuracy of the parsers. Following
the previous research (Zhu et al., 2019; Le and Zhang, 2023a), the accuracy
distribution across the 16 datasets is also reported to assess the methods’
stability and robustness.

Notably, log data might contain multiple completely identical log instances.
If one such log is included in the candidate set, it could result in data
leakage when parsing other identical logs, as these identical logs might serve
as both the prompt example and the query simultaneously (even though they are
not exactly the ”same” log). To ensure fair evaluation, in this experiment, we
ask DivLog to avoid selecting example logs that are identical to the query. It
continues selecting examples from the candidate set in order of similarity
until no identical candidate is included in the examples. This also enables
DivLog to be directly evaluated on all 2,000 logs in each dataset without data
leakage in the prompt examples, ensuring the same evaluation data used for
other unsupervised parsers and LogPPT, as discussed in Sec.4.5. Note the
accuracy of DivLog in practice could be higher because it is common to observe
identical logs during the continuous log collection process.

####  5.1.1. Accuracy

Accuracy is the most critical characteristic to evaluate the effectiveness of
a log parser (Zhu et al., 2019). Any minor parsing errors can lead to a
significant impact on the performance of downstream tasks of log analysis (He
et al., 2016b). In this part, we focus on evaluating the accuracy metrics of
the log parser on each dataset, as well as its overall average accuracy
metrics. These metrics provide a distinct representation of the global
strengths and weaknesses of each parser for each metric, as well as the
relative strengths and weaknesses of each parser on each dataset.

The results are shown in Table. 1. To visualize the advantage of DivLog, we
marked the best results in each evaluation metric on each dataset, as well as
their average value on all datasets in bold font. It is clear that DivLog
demonstrates state-of-the-art Parsing Accuracy (PA), Precision Template
Accuracy (PTA), and Recall Template Accuracy (RTA) on all 16 LogPAI log
datasets, outperforming the best DL-based parser, LogPPT (Le and Zhang,
2023a), which is proposed recently. Specifically, regarding the widely used PA
metric, DivLog achieved an average PA of 98.1% across 16 datasets, with 100%
PA on 4 datasets and PA exceeding 95% on 13 datasets (including 2 datasets
with 100% PA), and no dataset had a PA below 80%. This outperforms any
previous log parser by far. For instance, DivLog outperformed the current best
log parser LogPPT on 14 out of 16 datasets in terms of PA, with less than 4%
PA gap on the only dataset where it was not superior, and another dataset tied
at 100% accuracy. Additionally, DivLog’s advantage is more evident in PTA and
RTA metrics. It achieved an average PTA and RTA of 92.1% and 92.9%, across 16
datasets, with 12 and 14 datasets having PTA and RTA metrics not lower than
90%, respectively. This performance significantly exceeds that of LogPPT by
21.0% and 16.5%, respectively, where LogPPT only achieved an average PTA and
RTA of 67.4% and 74.1%. These findings demonstrate the significant accuracy
advantage of DivLog over current parsers.

Furthermore, DivLog’s accuracy advantage over other traditional log parsers is
even more apparent. For instance, compared to the strongest traditional log
parser, Drain, DivLog exceeded its average PA, PTA, and RTA by 48.3%, 50.2%,
and 42.6%, respectively. We believe that this phenomenon is due to the fact
that traditional log parsers, which are based on expert-designed heuristic
rules or handcrafted features, struggle to correctly distinguish between
variables and constants in different types of log data, and can only coarsely
cluster log messages with the same template. Therefore, to fairly demonstrate
DivLog’s accuracy, we primarily compared it with the current SOTA supervised
log parser, LogPPT.

####  5.1.2. Robustness

Figure 7. Accuracy on different datasets (%). Our approach, DivLog, achieves
the most stable performance among 16 datasets.

Robustness is an important characteristic to measure the versatility of a log
parser (Zhu et al., 2019). A practical log parser should perform consistently
on logs generated in multiple production environments (Zhu et al., 2019). In
this part, we focus on evaluating the robustness of each log parser through
the analysis of the distribution of metrics among 16 log datasets. By
analyzing the distribution of metrics for each log parser, we can identify the
stability of their parsing ability on different datasets.

The results are shown in Fig. 7. The results demonstrate that DivLog
outperforms existing log parsers by exhibiting the narrowest accuracy
distribution among 16 datasets on three metrics. This indicates that DivLog
possesses strong robustness across various datasets. Specifically, among all
16 datasets, DivLog has a standard deviation of only 3.58%, 7.66%, and 7.65%
on the three metrics, outperforming the currently most robust log parser,
LogPPT, which has standard deviations of 9.2%, 16.56%, and 14.57%.  In
addition, compared to traditional unsupervised log parsers, DivLog has a more
pronounced advantage. We believe this advantage of DivLog stems from two
reasons: (1) DivLog learns features of different log sources from relative log
examples in prompts and generates the expected template via these features,
without the need for manual feature selection or heuristic design. (2) Since
DivLog does not train or fine-tune a model but rather leverages LLM’s ICL
capabilities, it is not subject to the training set, which can reduce model
performance stability on various log datasets.

###  5.2. RQ2: How do different settings affect DivLog?

Table 2. Ablation study of components on Mac dataset (%).

| PA | PTA | RTA
---|---|---|---
$\text{Full}_{\text{{DivLog}}}$ | 86.3 | 67.6 | 66.6
$\text{w/o}_{\text{{DPP Sampling}}}$ | $75.3_{(\downarrow 11.0)}$ | $39.4_{(\downarrow 28.2)}$ | $41.3_{(\downarrow 24.9)}$
$\text{w/o}_{\text{{$k$NN Comparison}}}$ | $17.0_{(\downarrow 69.3)}$ | $2.3_{(\downarrow 65.3)}$ | $7.3_{(\downarrow 59.3)}$
$\text{w/o}_{\text{{Locators}}}$ | $70.3_{(\downarrow 16.0)}$ | $50.5_{(\downarrow 17.1)}$ | $31.7_{(\downarrow 34.9)}$

This section conducts ablation experiments to discuss the contributions of
each key component and the impact of different configurations in DivLog.
Specifically, we use the Mac dataset as the benchmark for the following
experiments, as its complex and diverse data (as illustrated in Sec.1) can
better illustrate the contributions of different settings to log parsing,
compared to other simpler datasets.

####  5.2.1. Components

Figure 8. Analysis of candidate number on Mac dataset (%). Figure 9. Analysis
of example number on Mac dataset (%).

As shown in Fig. 3, DivLog comprises three main components: DPP sampling,
$k$NN comparison, and prompt locators. To evaluate the individual
contributions of these components to DivLog’s log parsing capability, we
removed each component sequentially and measured its necessity and importance
by observing the extent of the resulting accuracy drop. Specifically, we (1)
replaced the DPP algorithm with random sampling to construct the candidate
set, (2) replaced the $k$NN algorithm with random selection to choose
examples, and (3) removed the locators in the prompt, relying solely on the
first-line output of LLM as the log template prediction. To mitigate the
impact of random bias, we repeated each experimental configuration five times
and calculated their mean as the final result.

The results are shown in Table. 2. It is clear that removing DPP sampling,
$k$NN selection, or prompt locators from DivLog leads to a significant
decrease in all three accuracy metrics. For instance, without DPP sampling or
prompt locators, DivLog achieves an 11.1% and 20.4% lower PA than the full
version, respectively. The reason for this drop is that DPP sampling enhances
the diversity of the candidate set, providing more related examples to assist
in generating the log template, while prompt locators help to extract the
expected log template from the raw output. Notably, $k$NN selection plays the
most crucial role in DivLog; without it, DivLog can hardly continue log
parsing, achieving only 16.8% PA, 1.8% PTA, and 6.8% RTA. This is because
there are enormous dissimilarities between each template and the scarcity of
log samples in each template in the Mac dataset. Without similar log messages
in prompt context, LLMs can hardly mine valid semantics and common patterns
among irrelevant log examples through analogy. Thus, we conclude that these
components are indispensable for the effectiveness of DivLog.

####  5.2.2. Configurations

In addition to the above-mentioned three key components, there are several
configurations that significantly affect DivLog’s parsing performance, i.e.,
(1) the number of examples, (2) the permutation method, and (3) the model
backbone. Specifically, we first analyze the impact of DivLog’s performance
with different numbers of examples ranging from 1 to 9. Then, we compare the
effectiveness of different permutation methods, i.e., ascending, descending,
and random order. Finally, we discuss how different model backbones in GPT-3
(Brown et al., 2020) series (i.e., Ada, Babbage, and Curie) affect parsing
performance. We also repeat each experiment five times and report their mean
as the final result.

Candidate Number: As shown in Fig. 8, all metrics improves with an increased
number of candidates. The reason is that the sample space for examples
available to each query expands, making the potential examples more suitable
for the query. However, after reaching 200 candidates, the improvement in
performance metrics slows down significantly. Given the considerable increase
in manual labeling efforts with more candidates in practice, we limit the use
to 200 candidates.

Example Number: As shown in Fig. 9, all metrics show an increasing trend when
the number of examples is less than 5. However, when the number exceeds 5, all
metrics gradually decrease. The reason for this is that too few examples do
not provide enough semantics for LLM to learn log patterns, and too many
examples include irrelevant logs that do not provide effective guidance for
parsing the target log message. Thus, in our experiments, 5 examples are
sufficient, and more examples do not help significant improvement.

Figure 10. Analysis of permutation on Mac dataset (%).

Example Permutation: As shown in Fig. 10, using ascending order performs
better on all metrics than descending order and random order. This is
consistent with the analysis in Sec. 3.5, which suggests that due to recency
bias (Zhao et al., 2021), LLMs are more likely to learn from examples that are
closer to the query. Hence, if the most recent examples have patterns similar
to the query, it enhances parsing accuracy. Notably, random order sometimes
coincidentally creates an ascending permutation, resulting in better message-
level metrics than descending order on average. However, due to its unstable
randomness, incorrectly parsed log messages may be evenly distributed among
multiple templates, leading to inferior template-level performance.
Conversely, descending order’s stable ordering may not significantly affect
the parsing ability on logs belonging to some templates, resulting in slightly
higher PTA and RTA.

Table 3. Analysis of model backbone on Mac dataset (%).

| PA | PTA | RTA
---|---|---|---
Curie | 86.3 | 67.6 | 66.6
Babbage | $78.5_{(\downarrow 7.8)}$ | $58.3_{(\downarrow 9.2)}$ | $65.7_{(\downarrow 0.9)}$
Ada | $74.1_{(\downarrow 12.2)}$ | $43.6_{(\downarrow 24.0)}$ | $36.1_{(\downarrow 30.5)}$

Model Backbone: As shown in Table.3, the accuracy increases with the LLM scale
in DivLog framework. Specifically, GPT-3 consists of four different versions:
Ada-350M, Babbage-3B, Curie-13B, and Davinci-175B. Due to limited research
resources, we were only able to test DivLog’s performance with the first three
versions as backbones. Nevertheless, we believe that with the emergence of
more powerful LLMs, DivLog’s parsing quality will further excel.

##  6\. Discussion

###  6.1. Efficiency and Practicality

Efficiency is also an aspect of evaluating the performance of log parsers.
Typically, researchers measure a log parser’s efficiency by analyzing the
execution time required for finishing parsing. However, in this paper, we
adopt OpenAI’s API to interact with LLM and the encoder, resulting in an
unmeasurable efficiency due to network latency (i.e., the parsing time for
each log message is almost exactly twice the network latency). Thus, we can
only qualitatively discuss potential efficiency issues in practice.

Specifically, DivLog’s time cost consists of (1) constructing a candidate set,
(2) selecting prompt examples, (3) encoding logs, and (4) LLM inference. The
cost of (1) and (2) mainly comes from the DPP and the $k$NN, which are
efficient and classic algorithms. The time complexities are $O(K^{2}N)$ and
$O(KM)$ respectively, where $K$, $N$, and $M$ are the log scale of (i)
candidate set $\mathcal{C}$, (ii) offline logs from target source
$\mathcal{X}$, and (iii) incoming logs in online parsing stage. The cost of
(3) and (4) mainly depends on the model inference speed. For large enterprises
and organizations that may face large volumes of log data, increasing the
parallel computing resources might be a potential solution to mitigate the
efficiency threat in LLM inference. Furthermore, in industrial scenarios,
DivLog only needs to collect the newly produced logs as candidates for ICL
inference. It does not require additional resource consumption for retraining
the models on these new log data. Therefore, we believe that the DivLog
framework can be practical with specialized development.

###  6.2. Threat to Validity

We identified the following major threats to validity:

  * •

Data Leakage: As discussed in Sec. 4.5, there is a possibility of data leakage
in the prompt example due to the presence of multiple completely identical log
instances. To eliminate this threat and ensure fair evaluation, in our
experiments, we ask DivLog to avoid selecting example logs that are identical
to the query. It continues selecting examples from the candidate set based on
their similarity until no identical candidate is included, enabling direct
evaluation on all 2000 logs in each dataset, consistent with the evaluation
data used for other baselines. In addition, due to the opacity of GPT-3’s
training data, it is unclear whether it was exposed to the log parsing task
during pre-training, potentially leading to data leakage. To investigate its
impact, we conducted zero-shot tests by removing all the examples in the
prompts before evaluation, observing GPT-3 producing only invalid output and
being unable to parse the templates. Therefore, we consider this threat
negligible.

  * •

Randomness: Randomness may affect the performance through two aspects: (1) the
randomness in LLMs and (2) the randomness introduced during the experiments in
Sec. 5.2, including a random sampling of candidates, selection of examples,
and permutation of examples. To mitigate the former threat, we reduced the
randomness of the LLM by configuring it to generate consistent outputs for the
same input text (i.e., initialize temperature=0). To mitigate the latter
threat, we conducted five independent experiments for each experimental
setting and used the mean of the results as the final outcome.

##  7\. Related Works

Log parsing methods can be categorized into unsupervised and supervised
methods according to the parsing algorithms.

Unsupervised log parsers Unsupervised log parsers leverage manually designed
features to extract log templates, which have been widely explored in the
past. There are three main categories of unsupervised log parsers: frequent
pattern mining-based, clustering-based, and heuristic-based methods. (1)
Frequent pattern mining-based methods regard the mined frequent patterns
(e.g., n-grams) as log templates. For example, SLCT (Vaarandi, 2003), LFA
(Nagappan and Vouk, 2010), LogCluster (Vaarandi and Pihelgas, 2015), and
Logram (Dai et al., 2020) try to use different methods to extract frequent
patterns in logs. (2) Clustering-based methods aim to group similar logs
first, assuming that logs in the same cluster share the same template, and
then extract the common tokens as the template in each cluster. Some
clustering-based methods can perform in an online manner because they adopt an
online grouping strategy rather than clustering all the offline logs at once.
Specifically, LKE (Fu et al., 2009b), LogSig (Tang et al., 2011), LogMine
(Hamooni et al., 2016) are offline methods, SHISO (Mizutani, 2013), and LenMa
(Shima, 2016) are online methods. (3) Heuristic-based methods encode expert
domain knowledge into general and effective heuristic rules. For example, AEL
(Fu et al., 2009a), IPLoM (Makanju et al., 2009), Spell (Du and Li, 2016), and
Drain (He et al., 2017b) utilize different heuristic rules to extract
templates from logs. In particular, Drain (He et al., 2017b) achieved SOTA in
all open-source traditional unsupervised parsers with a parse tree structure
to perform log parsing in an online manner. Based on Drain’s architecture,
there are multiple updated log parsers: POP (He et al., 2017a) improves Drain
and provides a parallel implementation on Spark for distributed deployment.
SPINE (Wang et al., 2022b) improves Drain and proposes a progressive
clustering step for log template correction via human feedback.

Supervised log parsers Supervised log parsers are a recently emerging class of
parsers. They typically use deep learning and use data sampled from a target
log source to supervise mine patterns in log templates. For example, UniParser
(Liu et al., 2022) utilizes a contrast learning strategy to overcome the
pattern difference in heterogeneous log sources based on a BiLSTM (Graves et
al., 2013) model. SemParser (Huo et al., 2021) utilizes a special semantic
miner for template extraction along with the concept-instance pair using a
BiLSTM (Graves et al., 2013) model. Both of the above log parsers require
training a neural network from scratch using labeled training data. Recently,
LogPPT (Le and Zhang, 2023a) has proposed a new paradigm for log parsing,
which uses template-free prompt-tuning (Ma et al., 2021) to fine-tune the pre-
trained language model, RoBERTa (Liu et al., 2019). By mining the semantic
information contained in the pre-trained model itself and fine-tuning it with
a small amount of log data (i.e., using 32 log samples for each dataset),
LogPPT is able to achieve the current SOTA performance on multiple datasets
and multiple metrics, outperforming all existing log parsers in accuracy and
robustness. These results show that mining the pre-trained model for log
parsing is feasible and effective.

Unlike other log parsers, DivLog doesn’t require heuristic rules, handcrafted
features, or fine-tuning on specific training data. Instead, it generates the
target log template by leveraging the ICL ability of LLMs and mining semantics
from related log examples in the prompts.

##  8\. Conclusion

In conclusion, our paper introduces a new log parsing framework, named DivLog,
that utilizes the in-context learning capability of large language models to
enhance log parsing effectiveness. Our method mines semantics from several log
examples provided in a prompt, enabling precise log template generation
without requiring model tuning. Additionally, we design a prompt format to
constrain the output and ensure the quality of the generated log templates
from LLMs, which we believe can generalize to other ICL applications. Through
experiments on 16 public datasets using three different metrics, we
demonstrate that DivLog outperforms current log parsers by 3.86%, 20.4%, and
16.7%, in terms of PA, PTA, and RTA, respectively. In a broader sense, we
believe this framework has significant potential to improve the effectiveness
and practicality of various downstream automated log analysis applications.

##  9\. Data Availability

We uploaded our algorithm implementation and datasets in GitHub repository at
https://github.com/logpai/logparser

###### Acknowledgements.

This paper was supported by the National Natural Science Foundation of China
(No. 62102340) and Shenzhen Science and Technology Program.

## References

  * (1)
  * Ahmed et al. (2023) Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, and Saravan Rajmohan. 2023\.  Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models.  _ICSE_ (2023).
  * Amar and Rigby (2019) Anunay Amar and Peter C Rigby. 2019.  Mining historical test logs to predict bugs and localize faults in the test logs. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 140–151.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chen et al. (2018) Laming Chen, Guoxin Zhang, and Eric Zhou. 2018.  Fast greedy map inference for determinantal point process to improve recommendation diversity.  _Advances in Neural Information Processing Systems_ 31 (2018).
  * Dai et al. (2020) Hetong Dai, Heng Li, Che Shao Chen, Weiyi Shang, and Tse-Hsun Chen. 2020.  Logram: Efficient log parsing using n-gram dictionaries.  _IEEE Transactions on Software Engineering_ (2020).
  * Deng et al. (2022) Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2022\.  Fuzzing Deep-Learning Libraries via Large Language Models.  _arXiv preprint arXiv:2212.14834_ (2022).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Ding et al. (2015) Rui Ding, Hucheng Zhou, Jian-Guang Lou, Hongyu Zhang, Qingwei Lin, Qiang Fu, Dongmei Zhang, and Tao Xie. 2015\.  Log2: A cost-aware logging mechanism for performance diagnosis. In _2015 $\\{$USENIX$\\}$ Annual Technical Conference ($\\{$USENIX$\\}$$\\{$ATC$\\}$ 15)_. 139–150.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A Survey for In-context Learning.  _arXiv preprint arXiv:2301.00234_ (2022).
  * Du and Li (2016) Min Du and Feifei Li. 2016\.  Spell: Streaming parsing of system event logs. In _2016 IEEE 16th International Conference on Data Mining (ICDM)_. IEEE, 859–864.
  * Du et al. (2017) Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017\.  Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In _Proceedings of the 2017 ACM SIGSAC conference on computer and communications security_. 1285–1298.
  * Fu et al. (2009a) Qiang Fu, Jian-Guang Lou, Yi Wang, and Jiang Li. 2009a.  Execution anomaly detection in distributed systems through unstructured log analysis. In _2009 ninth IEEE international conference on data mining_. IEEE, 149–158.
  * Fu et al. (2009b) Qiang Fu, Jian-Guang Lou, Yi Wang, and Jiang Li. 2009b.  Execution anomaly detection in distributed systems through unstructured log analysis. In _2009 ninth IEEE international conference on data mining_. IEEE, 149–158.
  * Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013\.  Speech recognition with deep recurrent neural networks. In _2013 IEEE international conference on acoustics, speech and signal processing_. Ieee, 6645–6649.
  * Hamooni et al. (2016) Hossein Hamooni, Biplob Debnath, Jianwu Xu, Hui Zhang, Guofei Jiang, and Abdullah Mueen. 2016\.  Logmine: Fast pattern recognition for log analytics. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_. 1573–1582.
  * He et al. (2016b) Pinjia He, Jieming Zhu, Shilin He, Jian Li, and Michael R Lyu. 2016b.  An evaluation study on log parsing and its use in log mining. In _2016 46th annual IEEE/IFIP international conference on dependable systems and networks (DSN)_. IEEE, 654–661.
  * He et al. (2017a) Pinjia He, Jieming Zhu, Shilin He, Jian Li, and Michael R Lyu. 2017a.  Towards automated log parsing for large-scale log data analysis.  _IEEE Transactions on Dependable and Secure Computing_ 15, 6 (2017), 931–944.
  * He et al. (2017b) Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R Lyu. 2017b.  Drain: An online log parsing approach with fixed depth tree. In _2017 IEEE international conference on web services (ICWS)_. IEEE, 33–40.
  * He et al. (2021) Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R Lyu. 2021\.  A survey on automated log analysis for reliability engineering.  _ACM computing surveys (CSUR)_ 54, 6 (2021), 1–37.
  * He et al. (2018) Shilin He, Qingwei Lin, Jian-Guang Lou, Hongyu Zhang, Michael R Lyu, and Dongmei Zhang. 2018\.  Identifying impactful service system problems via log analysis. In _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 60–70.
  * He et al. (2016a) Shilin He, Jieming Zhu, Pinjia He, and Michael R Lyu. 2016a.  Experience report: System log analysis for anomaly detection. In _2016 IEEE 27th international symposium on software reliability engineering (ISSRE)_. IEEE, 207–218.
  * Huo et al. (2021) Yintong Huo, Yuxin Su, Baitong Li, and Michael R Lyu. 2021\.  SemParser: A Semantic Parser for Log Analysis.  _arXiv preprint arXiv:2112.12636_ (2021).
  * Khan et al. (2022) Zanis Ali Khan, Donghwan Shin, Domenico Bianculli, and Lionel Briand. 2022.  Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques. In _Proceedings of the 44th International Conference on Software Engineering (ICSE’22)_. ACM.
  * Kumar et al. (2022) Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  _arXiv preprint arXiv:2202.10054_ (2022).
  * Le and Zhang (2023a) Van-Hoang Le and Hongyu Zhang. 2023a.  Log Parsing with Prompt-based Few-shot Learning.  _arXiv preprint arXiv:2302.07435_ (2023).
  * Le and Zhang (2023b) Van-Hoang Le and Hongyu Zhang. 2023b.  Repository of LogPPT.  https://github.com/LogIntelligence/LogPPT.  [Online; accessed 19-March-2023].
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ (2019).
  * Li et al. (2022) Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang, and Chun Zuo. 2022\.  AUGER: automatically generating review comments with pre-training models. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1009–1021.
  * Lin et al. (2016) Qingwei Lin, Hongyu Zhang, Jian-Guang Lou, Yu Zhang, and Xuewei Chen. 2016.  Log clustering based problem identification for online service systems. In _2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C)_. IEEE, 102–111.
  * Liu et al. (2021) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021\.  What Makes Good In-Context Examples for GPT-$3$?  _arXiv preprint arXiv:2101.06804_ (2021).
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023\.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _Comput. Surveys_ 55, 9 (2023), 1–35.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Liu et al. (2022) Yudong Liu, Xu Zhang, Shilin He, Hongyu Zhang, Liqun Li, Yu Kang, Yong Xu, Minghua Ma, Qingwei Lin, Yingnong Dang, et al. 2022\.  UniParser: A Unified Log Parser for Heterogeneous Log Data. In _Proceedings of the ACM Web Conference 2022_. 1893–1901.
  * Lou et al. (2010) Jian-Guang Lou, Qiang Fu, Shenqi Yang, Ye Xu, and Jiang Li. 2010.  Mining invariants from console logs for system problem detection. In _2010 USENIX Annual Technical Conference (USENIX ATC 10)_.
  * Ma et al. (2021) Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. 2021.  Template-free prompt tuning for few-shot NER.  _arXiv preprint arXiv:2109.13532_ (2021).
  * Makanju et al. (2009) Adetokunbo AO Makanju, A Nur Zincir-Heywood, and Evangelos E Milios. 2009.  Clustering event logs using iterative partitioning. In _Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining_. 1255–1264.
  * Mizutani (2013) Masayoshi Mizutani. 2013\.  Incremental mining of system log format. In _2013 IEEE International Conference on Services Computing_. IEEE, 595–602.
  * Nagappan and Vouk (2010) Meiyappan Nagappan and Mladen A Vouk. 2010.  Abstracting log lines to log event types for mining software system logs. In _2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)_. IEEE, 114–117.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018\.  Improving language understanding by generative pre-training.  (2018).
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _J. Mach. Learn. Res._ 21, 1, Article 140 (jan 2020), 67 pages.
  * Rubin et al. (2021) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021\.  Learning to retrieve prompts for in-context learning.  _arXiv preprint arXiv:2112.08633_ (2021).
  * Shang et al. (2013) Weiyi Shang, Zhen Ming Jiang, Hadi Hemmati, Brain Adams, Ahmed E Hassan, and Patrick Martin. 2013\.  Assisting developers of big data analytics applications when deploying on hadoop clouds. In _2013 35th International Conference on Software Engineering (ICSE)_. IEEE, 402–411.
  * Shima (2016) Keiichi Shima. 2016\.  Length matters: Clustering system log messages using length of words.  _arXiv preprint arXiv:1611.03213_ (2016).
  * Tang et al. (2011) Liang Tang, Tao Li, and Chang-Shing Perng. 2011.  LogSig: Generating system events from raw textual logs. In _Proceedings of the 20th ACM international conference on Information and knowledge management_. 785–794.
  * Vaarandi (2003) Risto Vaarandi. 2003\.  A data clustering algorithm for mining patterns from event logs. In _Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003)(IEEE Cat. No. 03EX764)_. Ieee, 119–126.
  * Vaarandi and Pihelgas (2015) Risto Vaarandi and Mauno Pihelgas. 2015.  Logcluster-a data clustering and pattern mining algorithm for event logs. In _2015 11th International conference on network and service management (CNSM)_. IEEE, 1–7.
  * Wang et al. (2022a) Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R Lyu. 2022a.  No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 382–394.
  * Wang et al. (2022b) Xuheng Wang, Xu Zhang, Liqun Li, Shilin He, Hongyu Zhang, Yudong Liu, Lingling Zheng, Yu Kang, Qingwei Lin, Yingnong Dang, et al. 2022b.  SPINE: a scalable log parser with feedback guidance. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1198–1208.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ (2022).
  * Xia et al. (2023) Chunqiu Steven Xia, Yifeng Ding, and Lingming Zhang. 2023\.  Revisiting the Plastic Surgery Hypothesis via Large Language Models.  _arXiv preprint arXiv:2303.10494_ (2023).
  * Xia and Zhang (2023) Chunqiu Steven Xia and Lingming Zhang. 2023.  Conversational automated program repair.  _arXiv preprint arXiv:2301.13246_ (2023).
  * Zhang et al. (2019) Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li, et al. 2019\.  Robust log-based anomaly detection on unstable log data. In _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 807–817.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models. In _International Conference on Machine Learning_. PMLR, 12697–12706.
  * Zhu et al. ([n. d.]) Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R Lyu. [n. d.].  Repository of LogPAI.  https://github.com/logpai/loghub.
  * Zhu et al. (2019) Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R Lyu. 2019.  Tools and benchmarks for automated log parsing. In _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 121–130.