# LogSD: Detecting Anomalies from System Logs through Self-supervised Learning
and Frequency-based Masking

Yongzheng Xie  yongzheng.xie@@adelaide.edu.au The University of
AdelaideAdelaideAustralia ,  Hongyu Zhang  The University of
NewcastleNewcastleAustralia<EMAIL_ADDRESS>and  Muhammad Ali
Babar  The University of AdelaideAdelaideAustralia<EMAIL_ADDRESS>
(2024)

###### Abstract.

Log analysis is one of the main techniques that engineers use for
troubleshooting large-scale software systems. Over the years, many supervised,
semi-supervised, and unsupervised log analysis methods have been proposed to
detect system anomalies by analyzing system logs. Among these, semi-supervised
methods have garnered increasing attention as they strike a balance between
relaxed labeled data requirements and optimal detection performance,
contrasting with their supervised and unsupervised counterparts. However,
existing semi-supervised methods overlook the potential bias introduced by
highly frequent log messages on the learned normal patterns, which leads to
their less than satisfactory performance. In this study, we propose LogSD, a
novel semi-supervised self-supervised learning approach. LogSD employs a dual-
network architecture and incorporates a frequency-based masking scheme, a
global-to-local reconstruction paradigm and three self-supervised learning
tasks. These features enable LogSD to focus more on relatively infrequent log
messages, thereby effectively learning less biased and more discriminative
patterns from historical normal data. This emphasis ultimately leads to
improved anomaly detection performance. Extensive experiments have been
conducted on three commonly-used datasets and the results show that LogSD
significantly outperforms eight state-of-the-art benchmark methods.

Log Analysis, Anomaly Detection, Self-supervised Learning, Deep Learning,
AIOps

††copyright: acmcopyright††journalyear: 2024††doi: XXXXXXX.XXXXXXX††booktitle:
Proceedings of the 32nd ACM Symposium on the Foundations of Software
Engineering (FSE ’24), July 15–19, 2024, Porto de Galinhas, Brazil††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Software and its engineering
Software maintenance tools††ccs: Computer systems organization
Reliability††ccs: Computer systems organization Availability††ccs: Computer
systems organization Maintainability and maintenance

##  1\. Introduction

Modern software-intensive systems have grown notably in scale and complexity
(Meng et al., 2019; Yang et al., 2021; Le and Zhang, 2021; Xie et al., 2022b;
Zhang et al., 2022). While these systems offer users a wealth of services,
they also introduce new operational and maintenance challenges. Among these
challenges is the task of detecting system issues and uncovering potential
risks through the analysis of large amounts of system log data (Zhang et al.,
2019; Meng et al., 2020; Le and Zhang, 2021; Zhang et al., 2022). Manually
identifying anomalies based on massive log data has become an impractical
endeavor (Lin et al., 2016; Yang et al., 2021; Xie et al., 2021, 2022b).
Therefore, automatically detecting system anomalies from logs has become
essential.

Over the past decades, a variety of log-based anomaly detection methods have
been proposed (Bodik et al., 2010; Liang et al., 2007; Chen et al., 2004;
Makanju et al., 2009; Lin et al., 2016; Du et al., 2017; Zhang et al., 2019;
Meng et al., 2020; Zhang et al., 2020; Xie et al., 2021; Yang et al., 2021;
Guo et al., 2021; Xie et al., 2022b; Zhang et al., 2022), which can be broadly
classified into supervised, unsupervised, and semi-supervised categories.
Supervised methods often achieve superior performance when a large amount of
normal and abnormal data are available for training. However, their utility is
limited by the labor-intensive requirements for data labeling and the scarcity
of anomalous instances (Yang et al., 2021; Guo et al., 2021; Zhang et al.,
2022). Unsupervised methods eliminate the requirement for labeled data,
including both normal and abnormal, but often yield sub-optimal performance
due to the complete absence of label-based guidance (Yang et al., 2021). Semi-
supervised methods offer a balanced solution, relaxing the need for scarce
anomalous instances by leveraging more readily available normal data. This
balance allows for a trade-off between stringent data requirements and optimal
detection performance. In this paper, we focus on semi-supervised log anomaly
detection.

Semi-supervised anomaly detection methods typically start by training a model
to capture normal patterns from historical data devoid of anomalies using a
variety of techniques (Wang et al., 2021; Yang et al., 2021; Farzad and
Gulliver, 2020; Du et al., 2017; Zhang et al., 2022; Guo et al., 2021). They
then identify anomalies based on deviations from these normal patterns learned
by the model. However, the effectiveness of the learned patterns in these
methods is often influenced by the occurrence frequency of the corresponding
normal data in a dataset. In particular, the model tends to exhibit a stronger
bias towards learning patterns associated with normal data that occurs more
frequently. In the context of log-based anomaly detection, this phenomenon is
common, which is reflected in the fact that some log events occur much more
frequently than others. For example, in the BGL dataset (Zhu et al., 2023b),
the average occurrence ratio of frequent events to infrequent events is 120:1
111Further details on the meaning and setting of this frequency threshold are
discussed in Section 4.. In the Spirit dataset (USENIX, 2008), this average
ratio is 32:1. In such cases, frequently occurring patterns receive undue
emphasis, while those that appear less often may be neglected or downplayed.
Consequently, anomaly detection carried out with these biased patterns may
encounter difficulties in distinguishing between normal instances containing
less frequent events and those anomalous ones (Xu et al., 2023).

Figure 1a gives an illustrative example of this issue. Consider a log dataset
comprising six normal sequences, denoted as $S_{1}$ through $S_{6}$, along
with an anomalous sequence $S_{a}$. Each sequence contains four events, which
are listed in the ’Events’ column. Sequences $S_{1}$ and $S_{6}$ contain four
distinct events, while each of the sequences $S_{2}$ to $S_{5}$ comprises a
unique event and three common events ($E_{5}$). The anomalous sequence $S_{a}$
shares three common events with $S_{1}$ ($E_{1},E_{2}$ and $E_{3}$) and also
includes an anomalous event ($E_{9}$). Let’s consider that $S_{1}$ to $S_{5}$
are used for training, while $S_{6}$ and $S_{a}$ are designated for testing.
During the training phase, when a model is used to learn representations of
these sequences, the learned representations of sequences $S_{2}$ through
$S_{5}$ stay closely in the latent space due to their common event $E_{5}$.
However, the representation of $S_{1}$ diverges from others because of its
dissimilarity to the sequences of $S_{2}$ to $S_{5}$. That is, the learned
normal pattern, primarily represented by the representations cluster of
$S_{2}$ to $S_{5}$, is dominated by the frequent occurrence of event $E_{5}$,
while the normal pattern associated with $S_{1}$ is downplayed. During the
testing phase, the representations of both $S_{6}$ and $S_{a}$ are similar and
stay far from the learned normal pattern, largely due to their shared events
with $S_{1}$. Their similarity introduces challenges in distinguishing the
normal sequence $S_{6}$ from the anomalous sequence $S_{a}$.

(a) Issue: Biased Normal Pattern.

(b) LogSD Solution.

Figure 1. Illustrative example of the issue identified and the solution
provided by our proposed method, LogSD. In both sub-figures, $S_{1}$ to $S5$
are normal training sequences. For testing, $S_{6}$ represents a normal
sequence, and $S_{a}$ is anomalous. Sub-figure (a) demonstrates how frequent
event $E_{5}$ in training sequences $S_{2}$ to $S_{5}$ can obscure the
distinction between normal sequence and anomalous sequence in latent space;
Sub-figure (b) depicts how LogSD addresses this issue by utilizing the
contrast between two subnets: one to emphasize infrequent events and the other
to focus on frequent events.

In this paper, we propose LogSD, a Log-based Self-supervised learning method
for anomaly Detection. LogSD addresses the problem of biased normal patterns
by encouraging the model to focus on less frequent events during training.
This strategy ensures that the representations of all normal sequences
generated by two subnets, irrespective of the frequency of their events, are
closely clustered in the respective latent space. Anomaly detection is then
performed based on the contrast between these learned representations.
Specifically, LogSD employs a dual-network architecture, consisting of an
Auto-Encoder (AE) subnet and an Encoder-Only (EO) subnet. We also introduce a
frequency-based masking scheme and a Global-to-Local (G2L) reconstruction
paradigm to guide the model training focusing specifically on less frequent
events. For an input sequence, two representations are generated: one from the
AE subnet and another from the EO subnet. For a normal sequence, the two
representations should be similar, whereas for an anomalous sequence, they
will differ. The degree of difference between the two representations serves
as an anomaly score to determine whether the input sequence is anomalous or
not.

To validate the efficacy of LogSD, extensive experiments have been conducted
on three commonly used public log datasets. The results show that LogSD
consistently outperforms eight state-of-the-art benchmark methods on the three
datasets across varying window size settings. Specifically, LogSD improves the
F1-score by at least 1.39%, 12.7%, and 82.6% over the second-best method on
the three respective datasets. Our ablation studies further confirm the
importance of the key components in LogSD.

In summary, our main contributions are as follows:

  1. (1)

A frequency-based masking scheme: We propose an innovative masking scheme that
leverages the occurrence frequency of log events in sequences to guide the
masking of sequences. Our ablation study confirms that frequency-based masking
can facilitate the model to achieve more stable and better performance in log-
based anomaly detection.

  2. (2)

A novel global-to-local reconstruction paradigm: We introduce a novel global-
to-local reconstruction paradigm that effectively incorporates overlooked
information from existing self-supervised methods leveraging masked sequences
as input. This paradigm enables LogSD to better capture the underlying normal
pattern present in sequences.

  3. (3)

A dual-network self-supervised anomaly detection approach: We propose an
anomaly detection approach LogSD, which utilizes a dual-network architecture
with three self-supervised learning tasks. One subnet is designed to emphasize
infrequent events, while the other focuses on frequent events. By further
integrating the frequency-based masking and global-to-local reconstruction
techniques, LogSD achieves superior performance in log anomaly detection. To
our best knowledge, this is the first dual-network self-supervised learning
approach for log anomaly detection.

  4. (4)

Extensive experiments: We compare LogSD with eight SOTA baselines on three
widely used public datasets. The results confirm the effectiveness of LogSD.
In addition, extensive ablation experiments are conducted to show the efficacy
of each major component in LogSD.

##  2\. Background

###  2.1. Log Data

A log message denotes a text sentence that is printed in the log file, which
is used to record the run-time behaviors of systems (Zhang et al., 2020; Xie
et al., 2021; Li et al., 2023a; Fu et al., 2023). Log messages are usually
semi-structured texts. Each log message consists of two parts: a constant part
known as the log event or log template, and a variable part known as the log
parameter. Log parser is a tool that is used to transform given log messages
from semi-structured texts into structured log events and corresponding log
parameters. Figure 2 shows a snippet of raw logs and the results after they
are parsed.

An event sequence, also referred to as an instance, is parsed and grouped from
a set of consecutive log messages that conform to chronological order. This
succession of log events captures specific execution flows of the program that
generate the log messages and is usually organized based on sessions or
entries. Session-based log partition utilizes certain log identifiers to
generate log sequences that encapsulate a series of interrelated operations.
Alternatively, entry-based log partitioning often generate sequences by two
strategies: fixed time window and fixed entry window.

  * •

The fixed time window approach entails the use of a consistent time window
size, such as 15 minutes or 1 hour, for the creation of log sequences. Events
within a sequence are timestamped within the predefined time frame. It is
worth noting that when employing the fixed time window grouping strategy, the
lengths of resulting sequences may vary in response to variations in system
workloads (Liao et al., 2020).

  * •

The fixed entry window approach adopts a predetermined entry window size, such
as 60 or 100 log messages, to produce log sequences. The sequences generated
by this method maintain a consistent length.

In addition, both fixed time window grouping and fixed entry window grouping
have respective variants known as the time-sliding window and entry-sliding
window strategies, which generate log sequences by overlapping two consecutive
time or entry windows, respectively. For example, a 15-minute time window with
a step size of 1, and an entry window of 100 logs with a step size of 1.

Figure 2. A snippet of HDFS (Hadoop Distributed File System) raw logs and
events after parsing

###  2.2. Relevant Techniques

####  2.2.1. Self-supervised Learning

Self-supervised learning (SSL) paradigm aims to harness intrinsic signals
presented within the data itself for guiding model optimization (Xie et al.,
2022a), culminating in the creation of representations tailored for diverse
downstream tasks. This paradigm has achieved promising successes in natural
language processing (Devlin et al., 2018; Ni et al., 2022), image processing
(Huang et al., 2022), and graph processing (Tan et al., 2023). Existing self-
supervised learning approach can be roughly categorized into two groups:
predictive methods and contrastive methods (Xie et al., 2022a). Predictive
methods train deep learning models to predict certain labels obtained from the
input data, such as masked input, statistical properties, and domain
knowledge-based targets. In contrast, contrastive methods utilize pair-wise
discrimination as their learning tasks. They apply transformations or
augmentations to generate multiple views from given data samples and train
models to distinguish between jointly sampled view pairs and independently
sampled view pairs.

####  2.2.2. Knowledge Distillation

Knowledge Distillation (KD) is a model training paradigm. It is originally
designed to enhance a simplified model by transferring distilled insights from
a more complex model. After the transferring, the simplified model is able to
maintain a comparable level of accuracy to that complex mode. This paradigm is
first introduced in the work (Hinton et al., 2015), and has been extended to
the field of anomaly detection (Xiao et al., 2021; Salehi et al., 2021; Ma et
al., 2022). Knowledge distillation-based methods for anomaly detection
typically involve two distinct sets of networks: the student network and the
teacher network. The teacher network tends to have a higher capacity to learn
knowledge from the dataset or additional data sources. These methods (Xiao et
al., 2021; Salehi et al., 2021; Ma et al., 2022) aim to train the student
network to forecast the outputs of the teacher network. Anomalies are then
identified when disparities emerge between the student network’s outputs and
those of the teacher network.

####  2.2.3. One-class Classification

One-class classification technique is based on the assumption that a trained
model can encapsulate all instances of normal data within a compact space,
while anomalies inherently deviate from this norm. In the case of OC-SVM
(Schölkopf et al., 2001), normal samples are projected into a higher-
dimensional space through a kernel function. This process establishes a
hyperplane that optimally maximizes the separation between normal instances
and anomalies. To enhance the aggregation of mapped data within the latent
space, Deep SVDD (Ruff et al., 2018) refines the neural network by minimizing
the volume of a hyper-sphere encompassing the representations of the normal
data. Consequently, this hyper-sphere serves as a boundary, effectively
distinguishing anomalous instances from the normal ones.

##  3\. Proposed Approach

###  3.1. Problem Definition

This proposed method, LogSD, aims to address the problem of end-to-end log-
based anomaly detection at the sequence level in a semi-supervised manner.
Specifically, given a set of normal sequences
$\mathcal{X}=\\{X_{1},...,X_{n}\\}$, where $X_{i}$ denotes a normal sequence,
we aim at learning an anomaly scoring function
$f:\mathcal{X}\rightarrow\mathbb{R}$, parameterized by $\theta$, such that for
$\forall
X^{\prime}_{i},X^{\prime}_{j}\in\mathcal{X}^{\prime},\mathcal{X}\bigcap\mathcal{X}^{\prime}\neq\emptyset$,
$f(X^{\prime}_{i};\theta)<f(X^{\prime}_{j};\theta)$ if $X^{\prime}_{i}$
conforms to $\mathcal{X}$ better than $X^{\prime}_{j}$. Within the given
sequences, each sequence can be denoted as $X_{i}=[E_{1},...,E_{|X_{i}|}]$,
where ${E_{k}}$ indicates the $k$-th event within the sequence $X_{i}$ and
$|X_{i}|$ represents the overall count of events within a session window or a
predefined entry window. Moreover, every event can be further represented as
$E_{k}=[token_{1},...,token_{|E_{k}|}]$, where $token_{t}$ corresponds to the
$t$-th tokenized word, symbol, or numerical value within $E_{k}$, and
$|E_{k}|$ signifies the total count of tokens encapsulated by $E_{k}$. To
employ the underlying semantic relationships across $token_{t}\in
V_{vocabulary}$, where $V_{vocabulary}$ denotes a token dictionary mapping
each $token_{t}$ to a vector $x_{i}\in\mathbb{R}^{m}$, derived from some
specific word embedding techniques utilized in Natural Language Processing
(NLP) such as Word2Vec (Nguyen et al., 2016), Glove (Pennington et al., 2014),
FastText (Joulin et al., 2016), and BERT model (Devlin et al., 2018).

###  3.2. Approach Overview

As shown in Fig LABEL:fig:framework_overview, LogSD consists of an auto-
encoder subnet (AE) and an encoder-only (EO) subnet. The EO subnet has the
same architecture as the encoder in the AE subnet. LogSD incorporates two key
techniques to guide its training focus: frequency-based masking and a Global-
to-Local (G2L) reconstruction paradigm.

####  3.2.1. Intuition of the Design

To clarify how LogSD works, let us apply it to the example depicted in Fig. 1.
LogSD first categorizes events in the input sequences into frequent and
infrequent events using a dynamically adjusted masking ratio, as elaborated in
Section 3.3. Utilizing both the frequency-based masking and G2L reconstruction
techniques, the AE subnet concentrates on reconstructing the infrequent events
in the sequences, i.e., events $E_{1}$ to $E_{4}$ in this case, while ignoring
the reconstruction of frequent event $E_{5}$. By doing so, the latent
representations learned by AE emphasize the infrequent events present in input
sequences.

For the EO subnet, the training sequences are first modified by masking out
the infrequent events, i.e., setting their values to zero. In this example,
the values of sequences $S_{2}$ to $S_{5}$ become $[0,E_{5},E_{5},E_{5}]$
after masking, and are then used for training. The EO subnet is designed to
learn the latent representations of the frequent events in an input sequence,
when those infrequent events are missing in the input. During the training
phase, the representations from the AE and EO subnets are optimized to be
similar.

During testing, for a normal sequence like $S_{6}$, its AE and EO
representations are expected to be close to the normal pattern in the latent
space, as depicted in the two plots in Fig. 1b. Thus, its anomaly score,
calculated as the difference between the two representations, would be low. On
the other hand, for the anomalous sequence $S_{a}$, its representations
generated from the two subnets will diverge. The EO representation is expected
to be close to the normal pattern as the anomalous (also infrequent) event
$E_{9}$ is set to zero during the masking process. While the AE representation
diverges from the normal pattern because the AE subnet is unable to
reconstruct the anomalous event $E_{9}$. This divergence results in a high
anomaly score for $S_{a}$, making it easily detectable.

####  3.2.2. Outline of the Design

In LogSD, frequency-based masking essentially categorizes events in the input
sequences into two groups based on their occurrence frequencies: the context,
containing frequent events, and the focus, containing infrequent events. For a
given set of input sequences $X$, the AE subnet produces the reconstructions
$\hat{X}$ and a set of representations $Z_{a}$ from its bottleneck layer.
Frequency-based masking is then applied to both $X$ and $\hat{X}$, yielding
the respective focuses, $X_{f}$ and $\hat{X}_{f}$, which are used to compute
the reconstruction loss $L_{r}$. This loss guides the learning task of the AE
subnet to better reconstruct $X_{f}$. Additionally, a one-class loss $L_{o}$
is computed as the difference between $Z_{a}$ and its mean $\overline{Z_{a}}$,
encouraging the learned AE representations to cluster closely.

For the EO subnet, frequency-based masking is applied to $X$ to obtain its
context $X_{c}$, which is then fed into the EO subnet to yield a set of
representations $Z_{e}$. A prediction loss $L_{p}$ is calculated as the
difference between $Z_{a}$ and $Z_{e}$. This loss serves to ensure that the
representations from the AE and EO subnets are similar for normal sequences.

Finally, the three losses $L_{r}$, $L_{o}$, and $L_{p}$ are combined to form
the final loss for training. During the inference phase, $L_{p}$ serves as the
anomaly score, indicating the abnormality of a given sequence.

Next, we elaborate on each major component of LogSD in the following
subsections.

###  3.3. Frequency-based Masking Scheme

In this paper, we introduce a frequency-based masking scheme tailored
specifically to the log analysis domain. In LogSD, event occurrence
frequencies are calculated at the batch level. When the occurrence frequency
of each event is available, the events with the lowest $\kappa$ occurrence
frequency in the batch are selected to be the focus, while the remaining
events constitute the context. The parameter $\kappa$ can be configured as a
fixed value or as a ratio of unique events in a batch, serving as a hyper-
parameter. Inspired by randomly masking scheme, the frequency-based masking
can be further extended to a random frequency-based masking scheme. This
entails dynamically generating a unique mask ratio in real-time for each
batch. Specifically, the parameter $\kappa$ is sampled from a set
$K=\\{k_{i}\\}_{i=1:N_{k}}$, where ${N_{k}}$ is the set cardinality. In this
study, we empirically set $K=\\{0.05,0.1,0.15,0.2,0.3\\}$, where ${N_{k}}$ is
5. Subsequently, the randomly determined ratio $\kappa$ is utilized to
facilitate the masking processing.

Based on the grouped focus events and context events, two distinct masking
processing are employed: focus masking and context masking. In focus masking,
infrequent events, termed as focus, are masked out from the input sequences,
resulting in what we refer to as the context. Conversely, in context masking,
frequent events, or the context, are masked out to produce the focus. As
depicted in Fig LABEL:fig:framework_overview, in LogSD, context masking is
performed prior to the computation of the reconstruction loss $L_{r}$. This
ensures that the loss is calculated based on the focus, guiding the AE subnet
to learn the reconstruction of infrequent events. In contrast, focus masking
is applied to the sequences $X$ to produce the context $X_{c}$, which serves
as the input to the EO subnet. This enables the EO subnet to focus on learning
the representations of $X$ with the absence of input information about these
events. By doing so, the frequency-based masking strategy ensures LogSD learn
the less biased normal patterns in both AE and EO representations by
emphasizing the learning of infrequent events.

###  3.4. Global-to-Local Reconstruction Paradigm

The proposed Global-to-Local (G2L) reconstruction paradigm distinguishes
itself from existing reconstruction approaches, including both global and
local paradigms as described in Section 7.2. As depicted in Fig. 3a, global
reconstruction employs the complete input sequence (referred to as ‘global’)
as the input to the AE network, and the reconstruction loss is calculated
based on all events in both the original and reconstructed sequences. In local
reconstruction, illustrated in Fig. 3b, only the context of the input sequence
(referred to as ‘local’) serves as the input to the AE network, and the
reconstruction loss is similarly computed using only this focus. In contrast,
G2L reconstruction, shown in Fig. 3c, takes the complete input sequence as its
input (similar to global approaches) but computes the reconstruction loss
using only the focus of the original and reconstructed sequences (akin to
local approaches).

(a) Global Reconstruction.

(b) Local Reconstruction.

(c) G2L reconstruction.

Figure 3. Different Reconstruction Paradigms.

The G2L reconstruction paradigm leverages all events (global) within a
sequence to reconstruct the focus events (local), which forces the trained
model focus on the reconstruction of the focus events rather than the
reconstruction of all the events in sequences. Therefore, this design enables
the model to enhance its learning of the normal pattern reflected by focus
events during the training phase. During the inference stage, when applied to
normal sequences, the insights drawn from normal focus events excel at self-
reconstruction, leading to a small reconstruction error. Conversely, in the
context of anomalous sequences, the presence of the abnormal focus events
amplifies the reconstruction error due to the poor self-reconstruction ability
of abnormal focus events. The heightened disparity in reconstruction error
renders anomalous sequences more discernible under the G2L reconstruction
paradigm.

###  3.5. Three Self-supervised Tasks

LogSD employs a dual-network architecture incorporating three self-supervised
tasks. In this framework, one network utilizes an auto-encoder structure,
while the other follows an encoder-only design. Both networks implement their
encoders and decoders using CNNs as fundamental building blocks.

LogSD utilizes three carefully designed self-supervised tasks, i.e.,
reconstruction task, one-class task, and prediction task, to effectively guide
model optimization during training. These tasks corresponds to three specific
losses: the G2L reconstruction loss $L_{r}$, the one-class classification loss
$L_{o}$, and the prediction loss $L_{p}$.

Task I: G2L reconstruction Loss Following the proposed G2L reconstruction
paradigm described in Section 3.4, the reconstruction task of LogSD takes the
focus $X_{f}$ and $\hat{X}_{f}$ as the inputs to generate the G2L
reconstruction error $L_{r}$. As $X_{f}$ and $\hat{X}_{f}$ are derived through
a context masking, elucidated in Section 3.3, this implies that the
reconstruction task is focused on those infrequent events within the
sequences. Here, the Mean Square Error(MSE) is utilized to derive the G2L
reconstruction error. The loss function of reconstruction task is defined as

(1) $\displaystyle
L_{r}=\frac{1}{|N|*|M|*|d|}\sum_{i=1}^{N}\sum^{M}_{j=1}{\|x_{ij}-\hat{x}_{ij}\|^{2}_{2}}$

where $x_{ij},\hat{x}_{ij}\in\mathbb{R}^{d}$ denotes the $d$-dimensional
semantic features of the $j$-th event and its reconstructed event in $i$-th
sequence of a batch, and $M$ is the total number of the focus events in the
given sequence, while $N$ is the total number of log sequences in a batch.
Here, $\|\cdot\|^{2}_{2}$ refers to as the squared $L_{2}$ norm.

Task II: One Class Loss Inspired by LogBERT (Guo et al., 2021) and CAT (Zhang
et al., 2022), we also adopt an one-class classification task to govern the
distribution of normal sequences representations in the latent space generated
by the auto-decoder network. In order to facilitate the convergence of
representations for normal sequences in the training set, we continually
update the central representation, denoted as $c$. This is accomplished
through a mean operation of the generated representations in each epoch during
the training phase. Consequently, the optimization process entails minimizing
Equation 2 over the course of training. This leads to a desired outcome during
inference, where the representation of a given normal log sequence should
closely align with the center, while anomalous log sequences are expected to
exhibit a substantial distance from this central reference point.
Additionally, leveraging the one-class loss yields a supplementary advantage:
it may aid in the convergence of the global-to-local reconstruction loss by
contracting the sphere that encapsulates the latent representations of
sequences. The mathematical definition of the one-class loss is as follows:

(2) $\displaystyle L_{o}=\frac{1}{N}\sum_{i=1}^{N}{\|z_{i}-c\|^{2}_{2}}$

where $z_{i}\in\mathbb{R}^{l}$ denotes the $l$-dimensional representation of
the i-th event sequence generated from the encoder of the AE subnet, $c$
represents the center representation of normal log sequences, calculated as
$c=\frac{1}{N}\sum^{N}_{i=1}{z_{i}}$, and $N$ is the total number of log
sequences in a batch.

Task III: Prediction Loss Taking inspiration from Knowledge Distillation, we
employ the AE subnet within LogSD as the teacher network, while the EO subnet
serves as the student. Notably, the student network takes the context $X_{c}$,
generated by a focus masking, as input, while the teacher network operates on
the input sequences $X$ without any masking. This design allows the teacher
network to gather extra insights into normal patterns in comparison to the
student network. Subsequently, we utilize the representations $Z_{e}$
extracted from the student network’s latent space to predict the corresponding
representations $Z_{a}$ generated by the teacher network. The loss for the
prediction task is calculated using the Mean Square Error of predictive
errors, referred to as the prediction loss. The formal definition of the
prediction loss is as follows:

(3) $\displaystyle
L_{p}=\frac{1}{|N|*|l|}\sum^{N}_{i=1}{\|z_{i}-z^{\prime}_{i}\|^{2}_{2}}$

where $z_{i},{z}_{i}^{\prime}\in\mathbb{R}^{l}$ denotes the $l$-dimensional
representation of the $i$-th sequence generated from the encoder of the AE
subnet and that from the EO subnet, and $N$ is the total number of log
sequences in a batch.

Finally, the objective function for training LogSD is defined as shown in
Equation 4, where a hyper-parameter $\alpha$ is used to balance the
reconstruction loss with other losses. Following the practice in (Akcay et
al., 2019) and the principle of minimizing hyper-parameters, we maintain a
ratio of 50:1 for the reconstruction loss based on sequences in the original
space to the other losses based on the representations in the latent space,
i.e, $\alpha$ is set to 50.

(4) $\displaystyle{L}=\alpha{L}_{r}+{L}_{p}+{L}_{o}$

###  3.6. Anomaly Detection

The loss value obtained from Equation 3 is utilized as the anomaly score.
Following a procedure similar to training, given event sequences are input
into the trained model to compute anomaly score for each sequence. These
anomaly scores are then ranked, with higher scores indicating a higher
likelihood of the corresponding sequences as anomalies. To evaluate LogSD, We
follow the studies (Wang et al., 2021; Liu et al., 2023) to employ a simple
threshold-moving algorithm (Haibo and Yunqian, 2013) to determine the optimal
threshold. The optimal threshold would be the one resulting in the best result
based on the predefined evaluation metric. In this work, we choose F1 score as
the measurement to locate the threshold with the optimal balance between
precision and recall.

Table 1. Overview of datasets used in the experiments.

Datasets | #Events | Window | Training Set (80%) | Testing Set (20%)
---|---|---|---|---
#Sequences | #Events | #Norm. | %Norm. | #Sequences | #Events | #Anom. | %Anom.
HDFS | 48 | session | 460,048 | 48 | 446,663 | 97.07% | 115,012 | 42 | 3,453 | 3.00%
|  | 100-logs | 5,019 | 440 | 4,111 | 81.91% | 1,254 | 226 | 129 | 10.29%
BGL | 450 | 60-logs | 8,365 | 440 | 7,110 | 85.00% | 2,091 | 226 | 150 | 7.17%
|  | 20-logs | 25,095 | 440 | 22,401 | 89.26% | 6,273 | 226 | 198 | 3.16%
|  | 100-logs | 33,325 | 1,209 | 19,744 | 59.25% | 8,331 | 847 | 409 | 4.91%
Spirit | 1,228 | 60-logs | 55,541 | 1,209 | 34,387 | 61.91% | 13,885 | 847 | 434 | 3.13%
|  | 20-logs | 166,624 | 1,209 | 110,524 | 66.33% | 41,656 | 847 | 465 | 1.12%

  * 1

#Events: number of unique events; #Sequences: number of sequences; #Norm.:
number of normal sequences; %Norm.: percentage of normal sequences;#Anom.:
number of anomalous sequences; %Anom.: percentage of anomalies.

  * 2

For BGL and Spirit, consecutive duplicate events are removed from parsed logs
before being grouped, which ensures that sequences with only a single unique
event are eliminated, making model evaluation more realistic and reliable.

  * 3

The data splitting strategy for BGL and Spirit follows a chronological order,
with the initial 80% of log messages used for training and the remaining 20%
for testing. This approach prevents data leakage during model training.

##  4\. Experimental Setting

###  4.1. Datasets

In our experiments, three public log datasets HDFS, BGL, and Spirit are used
to evaluate the proposed approach and the relevant baselines. These datasets
are extensively utilized in log analysis research (Le and Zhang, 2021; Zhang
et al., 2020; Du et al., 2017; Lou et al., 2010; Xu et al., 2009a; Landauer et
al., 2022) due to their nature of real-world datasets with comprehensive
labeling. All these log datasets are sourced from publicly available websites
(Zhu et al., 2023a) and (USENIX, 2008). The details of the resulting log
datasets in our experiments are summarized in Table 1.

HDFS dataset is generated by running Hadoop-based map-reduce jobs on more than
200 Amazon’s EC2 nodes. Given that the annotations available for this dataset
are associated with the BlockId, the resultant log event sequences are
organized into sessions using the BlockId. Regarding data segmentation, we
adopt the methodology from (Le and Zhang, 2021, 2022; Xie et al., 2022b),
allocating 80% of the log sequences randomly for training and reserving the
remaining 20% for testing purposes.

BGL dataset is an open dataset comprising logs obtained from a BlueGene/L
supercomputer system at Lawrence Livermore National Labs (LLNL) in Livermore,
California. This supercomputer is equipped with 131,072 processors and 32,768
GB of memory. Annotations for the BGL dataset are applied to individual log
messages, resulting in the grouping of log event sequences based on entries.
We specifically opt for fixed entry window grouping for the following reasons.
Firstly, it enhances the generalizationability of LogSD across datasets, as
not all datasets have a universally applicable identifier for session
grouping. Additionally, fixed entry window grouping offers storage efficiency
advantages over the entry sliding window approach. The dataset is generated
under three fixed entry window size settings, namely 100 logs, 60 logs, and 20
logs. Fixed Time window grouping setting and fixed entry window grouping under
larger window size settings were not chosen because they may lead to potential
delays in fault discovery within real-world scenarios. For data splitting, we
adhere to a chronological order strategy for the BGL dataset, wherein the
initial 80% of log messages are used for training, and the subsequent 20% are
allocated for testing. This approach aligns with real-world scenarios,
ensuring that no data leakage occurs during the model training process.

Spirit dataset originates from a high-performance cluster located at Sandia
National Labs (SNL). This dataset was collected from systems equipped with
1,028 processors and 1,024 GB of memory. In this study, we utilize 1-gigabyte
continuous log lines from the Spirit dataset for computational efficiency. The
grouping strategy and data splitting approach for the Spirit dataset are
identical to those used for the BGL dataset.

It’s essential to highlight that before aggregating log events into sequences,
we apply a specific pre-processing step to the BGL and Spirit datasets.
However, this pre-processing is not performed on the HDFS dataset since its
labels are annotated at the session level, and any event removal to a sequence
might affect its label. The specific pre-processing involves removing
consecutive duplicate events from the parsed logs before grouping them into
sequences. For instance, consider a sequence of parsed log events represented
as $S=[...E_{1},E_{2},,E_{2},,E_{2},,E_{1}...]$. After this pre-processing,
the sequence is transformed to $S=[...E_{1},E_{2},E_{1}...]$, effectively
eliminating consecutive duplicates of $E_{2}$ that appeared in the original
sequence. Subsequently, the resulting events are grouped based on entries.
This pre-processing step serves a dual purpose: firstly, it reduces the
overall dataset size, thus minimizing the time required for model training.
Secondly, it effectively addresses potential issues with inflated detection
performance, especially when a sequence consists solely of a single unique
event. By removing such duplicates, the evaluation of the model’s performance
becomes more realistic and reliable.

###  4.2. Baseline Methods

To evaluate the effectiveness of the proposed method, we compare LogSD with
eight state-of-the-art deep semi-supervised methods, a trivial detector, and
one state-of-the-art supervised method on the aforementioned public log
datasets. Specifically, AE (Farzad and Gulliver, 2020) and OC4Seq (Wang et
al., 2021) are reconstruction based and one-class classification based
methods, respectively. PLElog (Yang et al., 2021) is an approach using
Probabilistic Label Estimation technique, while DeepLog/Log2Vec (Meng et al.,
2020), LogAnomaly (Meng et al., 2019), Logsy (Nedelkoski et al., 2020),
LogBERT (Guo et al., 2021), and CAT (Zhang et al., 2022) belong to self-
supervised methods. The random detector, as a trivial detector, randomly
assigns labels to each sequence, acting as a basic baseline reference. CNN (Lu
et al., 2018) is chosen as a representative state-of-the-art (SOTA) supervised
baseline method for comparison, providing a reference to assess performance
disparities between supervised and semi-supervised approaches. This selection
is particularly pertinent because CNN employs the same network building blocks
as LogSD, enabling a direct comparison of methodologies built upon identical
fundamental components. In terms of the neural networks, these methods cover
the utilization of GRU, LSTM, BiLSTM, Transformer, and CNN models,
respectively, which also provide us with another perspective to examine the
efficacy of different neural network building blocks.

We did not compare our method with those traditional machine learning-based
methods, such as PCA (Xu et al., 2009b), iForest (Liu et al., 2008),
LogCluster (Lin et al., 2016), OC-SVM(Schölkopf et al., 2001), Invariants
Mining (Lou et al., 2010), among others. This decision was made because some
studies (Andonov et al., 2022; Guo et al., 2021; Le and Zhang, 2022; Zhang et
al., 2022) had demonstrated deep learning-based methods more effective than
traditional machine learning approaches in anomaly detection effectiveness.
Additionally, we refrained from comparing LogSD with two GNN-based semi-
supervised methods, namely GLAD-PAW (Wan et al., 2021) and Logs2Graphs
(Andonov et al., 2022), due to the unavailability of their implementations. We
leave the comparisons with them as future work.

###  4.3. Evaluation Metrics

To evaluate the effectiveness of the approaches, we utilize three groups of
evaluation metrics.

  * •

The first group of metrics is Matthews Correlation Coefficient (MCC) (Baldi et
al., 2000), which is considered as a reliable metric of the quality of
prediction models (Yao and Shepperd, 2021; Lavazza and Morasca, 2022). This
metric takes into account true and false positives and negatives and is
generally regarded as a balanced measure. The MCC metric is defined as:
$MCC=\frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$

  * •

To align with the evaluation method of prior studies (Farzad and Gulliver,
2020; Wang et al., 2021; Yang et al., 2021; Meng et al., 2019, 2020;
Nedelkoski et al., 2020; Guo et al., 2021; Le and Zhang, 2022; Zhang et al.,
2022), therefore, precision/recall/F1 score are used as the second group of
metrics in our study. Specifically, the metrics are calculated as follows:

Precision: the percentage of correctly detected abnormal log sequences amongst
all detected abnormal log sequences by the model. $Precision=\frac{TP}{TP+FP}$

Recall: the percentage of log sequences that are correctly identified as
anomalies over all real anomalies. $Recall=\frac{TP}{TP+FN}$

F1 score: the harmonic mean of Precision and Recall.
$F1=\frac{2*Precision*Recall}{Precision+Recall}$

  * •

The third group is ranking metrics, including ROC score (Area Under Receiver
Operating Characteristics Curve) and PRC score (Area Under the Precision-
Recall Curve). Both metrics are widely used to quantify the detection accuracy
of anomaly detection (Huang and Ling, 2005; Zhang et al., 2022; Andonov et
al., 2022). Thus, we also leverage them to assess the baseline methods and
LogSD.

For all the aforementioned metrics, a value closer to 1 indicates superior
performance.

Table 2. Performance Comparison of LogSD and Baseline methods.

Method | Metrics | HDFS | BGL | Spirit
---|---|---|---|---
|  | session | 100-logs | 60-logs | 20-logs | 100-logs | 60-logs | 20-logs
AE (Farzad and Gulliver, 2020) | MCC | 0.7570 | 0.5764 | 0.5536 | 0.3903 | 0.1968 | 0.1651 | 0.0713
F1 | 0.7432 | 0.6165 | 0.5475 | 0.3911 | 0.2101 | 0.1631 | 0.0719
Precision | 0.8445 | 0.7450 | 0.5090 | 0.3129 | 0.1214 | 0.0970 | 0.0549
Recall | 0.6735 | 0.5297 | 0.5933 | 0.5236 | 0.7889 | 0.5922 | 0.1835
PRC | 0.6175 | 0.5974 | 0.4697 | 0.2489 | 0.1112 | 0.0800 | 0.0247
| ROC | 0.7191 | 0.8189 | 0.8623 | 0.8898 | 0.7924 | 0.7653 | 0.5963
OC4Seq (Wang et al., 2021) | MCC | 0.9287 | 0.7528 | 0.6939 | 0.4809 | 0.4217 | 0.3268 | 0.1232
F1 | 0.9315 | 0.7894 | 0.6859 | 0.4764 | 0.4074 | 0.3069 | 0.1257
Precision | 0.9362 | 0.7944 | 0.6254 | 0.3986 | 0.2754 | 0.1890 | 0.0812
Recall | 0.9268 | 0.7959 | 0.7711 | 0.7138 | 0.8354 | 0.8487 | 0.2875
PRC | 0.9582 | 0.7117 | 0.5522 | 0.4390 | 0.2416 | 0.1634 | 0.0423
| ROC | 0.9988 | 0.9634 | 0.9604 | 0.9617 | 0.9168 | 0.9142 | 0.7539
PLELog (Yang et al., 2021) | MCC | 0.9447 | 0.8122 | 0.7557 | 0.7817 | 0.5283 | 0.4349 | 0.4943
F1 | 0.9451 | 0.8249 | 0.7684 | 0.7853 | 0.4903 | 0.3664 | 0.4192
Precision | 0.9617 | 0.7441 | 0.6468 | 0.7637 | 0.3322 | 0.2302 | 0.2722
Recall | 0.9299 | 0.9259 | 0.9477 | 0.8157 | 0.9544 | 0.9531 | 0.9384
PRC | 0.8949 | 0.6969 | 0.6167 | 0.6275 | 0.3193 | 0.2207 | 0.2559
| ROC | 0.9647 | 0.9452 | 0.9534 | 0.9036 | 0.9254 | 0.9196 | 0.9542
DeepLog/Log2Vec (Meng et al., 2020) | MCC | 0.9393 | 0.6059 | 0.5606 | 0.4573 | 0.1126 | 0.0962 | 0.1321
F1 | 0.9295 | 0.6358 | 0.5889 | 0.4543 | 0.1643 | 0.1261 | 0.1398
Precision | 0.9716 | 0.6767 | 0.5169 | 0.3845 | 0.1073 | 0.1243 | 0.1400
Recall | 0.8911 | 0.6006 | 0.6856 | 0.5567 | 0.3537 | 0.1289 | 0.1395
PRC | 0.8691 | 0.6569 | 0.6121 | 0.4776 | 0.0696 | 0.0432 | 0.0292
| ROC | 0.9452 | 0.7858 | 0.8190 | 0.7639 | 0.6004 | 0.5497 | 0.5650
LogAnomaly (Meng et al., 2019) | MCC | 0.7524 | 0.6320 | 0.4283 | 0.3313 | 0.0737 | 0.0853 | 0.0796
F1 | 0.7333 | 0.6404 | 0.4515 | 0.3116 | 0.0797 | 0.0860 | 0.0908
Precision | 0.9985 | 0.6952 | 0.3314 | 0.2102 | 0.1245 | 0.1214 | 0.0758
Recall | 0.5856 | 0.5938 | 0.7136 | 0.6531 | 0.0587 | 0.0668 | 0.1133
PRC | 0.6390 | 0.5208 | 0.3817 | 0.2438 | 0.1003 | 0.0373 | 0.0185
| ROC | 0.7927 | 0.8909 | 0.8686 | 0.8689 | 0.7274 | 0.5255 | 0.5488
Logsy (Nedelkoski et al., 2020) | MCC | 0.9156 | 0.6827 | 0.6033 | 0.4796 | 0.1823 | 0.1453 | 0.1310
F1 | 0.9225 | 0.7155 | 0.6282 | 0.4735 | 0.1873 | 0.1536 | 0.1337
Precision | 0.9405 | 0.6630 | 0.5762 | 0.3879 | 0.1094 | 0.1196 | 0.1242
Recall | 0.9054 | 0.7855 | 0.6978 | 0.6094 | 0.6504 | 0.2220 | 0.1455
PRC | 0.8549 | 0.5415 | 0.4222 | 0.2498 | 0.0883 | 0.0506 | 0.0276
| ROC | 0.9518 | 0.8697 | 0.8288 | 0.7889 | 0.6886 | 0.5841 | 0.5669
LogBERT (Guo et al., 2021) | MCC | 0.7473 | 0.7419 | 0.6782 | 0.6533 | 0.4117 | 0.3905 | 0.3771
F1 | 0.7255 | 0.7646 | 0.6873 | 0.6652 | 0.4266 | 0.3939 | 0.3535
Precision | 0.9435 | 0.8408 | 0.5983 | 0.5830 | 0.3236 | 0.2950 | 0.2544
Recall | 0.5894 | 0.7028 | 0.8578 | 0.7828 | 0.6259 | 0.5929 | 0.5792
PRC | 0.4794 | 0.8314 | 0.5601 | 0.5035 | 0.1975 | 0.1752 | 0.1340
| ROC | 0.9370 | 0.9624 | 0.9337 | 0.9472 | 0.8662 | 0.8870 | 0.9187
CAT (Zhang et al., 2022) | MCC | 0.6235 | 0.6861 | 0.4987 | 0.2745 | 0.3276 | 0.2513 | 0.0511
F1 | 0.6458 | 0.7164 | 0.4835 | 0.2942 | 0.3248 | 0.2535 | 0.0537
Precision | 0.7909 | 0.7396 | 0.7677 | 0.2100 | 0.5059 | 0.2438 | 0.0492
Recall | 0.5457 | 0.6810 | 0.4167 | 0.5069 | 0.3238 | 0.3220 | 0.0789
PRC | 0.5124 | 0.5679 | 0.4132 | 0.1441 | 0.2320 | 0.1315 | 0.0131
| ROC | 0.8164 | 0.8257 | 0.7058 | 0.7319 | 0.7482 | 0.6845 | 0.3779
LogSD | MCC | 0.9559 | 0.9483 | 0.8712 | 0.9047 | 0.8954 | 0.8714 | 0.8676
F1 | 0.9583 | 0.9627 | 0.8664 | 0.8906 | 0.8957 | 0.8861 | 0.8707
Precision | 0.9587 | 0.9600 | 0.8753 | 0.9118 | 0.8386 | 0.8287 | 0.9209
Recall | 0.9580 | 0.9664 | 0.8578 | 0.8704 | 0.9650 | 0.9616 | 0.8258
PRC | 0.9840 | 0.9716 | 0.8736 | 0.9416 | 0.7346 | 0.7609 | 0.8013
| ROC | 0.9993 | 0.9977 | 0.9911 | 0.9967 | 0.9927 | 0.9961 | 0.9987

  * 1

All the models were trained for up to 100 epochs and an early stopping
strategy was used for 20 consecutive iterations without performance
improvement.

  * 2

The figures in the table that were underlined represent the second-best
metrics, while those in bold font indicated the best metrics.

  * 3

Each method was executed three times for each dataset configuration, and the
resulting values were averaged to report the final results.

###  4.4. Implementation and Environment

In the implementation of LogSD, we chose the state-of-the-art tool, Drain (He
et al., 2017) for log parsing because of its proven effectiveness, robustness,
and efficiency (Zhu et al., 2019; Dai et al., 2023; Fu et al., 2023). We
implemented LogSD in PyTorch 1.11.0. The encoder and decoder in LogSD are
single-layer 2D-convolution and 2D-transposed convolution modules,
respectively. Following the practice in (Chen et al., 2021), we extracted a
32-dimensional vector as the semantic embedding for each log event. The hidden
dimension of all convolution networks in LogSD was set to 128, with three
kernel sizes specified as [3, 4, 5]. LogSD was trained using the AdamW
optimizer with polynomial learning rate decay at $1\times 10^{-4}$. The
learning rate started at $1\times 10^{-2}$ and gradually decreased to $1\times
10^{-4}$. We set the mini-batch size to 64. The model underwent training for
up to 100 epochs, with an early stopping strategy applied when performance
failed to improve for 20 consecutive iterations. The hyper-parameter $\kappa$
for the frequency-based masking ratio was dynamically selected from the set
$K=\\{0.05,0.1,0.15,0.2,0.3\\}$. Additionally, the hyper-parameter for the
reconstruction loss weight, $\alpha$, was set to 50.

In our comparative analysis, the implementations of baseline approaches were
obtained from public repositories (Dee, 2021; OC4, 2021; PLE, 2021; Log, 2019;
LBc, 2021; CAT, 2022). For consistency, we used the parameters provided by
their respective implementations unless specified otherwise. In our
experiments, each method was executed three times for each dataset
configuration, and the resulting values were averaged to report the final
results. All experiments were conducted on a Linux server Ubuntu 20.04
equipped with an AMD Ryzen 3.5GHz CPU, 96 GB of memory, and an RTX2080Ti with
11 GB of GPU memory.

##  5\. Results and Analysis

###  5.1. Comparison with State-of-the-art Methods

In this experiment, our objective is to assess the effectiveness of LogSD
through a comparative study with baseline methods on the three aforementioned
public log datasets. To ensure a fair comparison, all methods that utilize
semantic embeddings as inputs, including AE, DeepLog/Log2Vec, Logsy, and LogSD
(excluding PLELog and CAT), employ the identical semantic extraction scheme as
detailed in (Chen et al., 2021). PLELog and CAT maintain their original
semantic scheme. LogAnomaly, OC4Seq, and LogBERT utilize log indices as input,
without any modifications to their original implementations.

The experimental results, as presented in Table 2, consistently highlight the
superior performance of LogSD across three datasets with varying window
settings, in terms of the MCC, F1 score and ranking metrics. Notably, LogSD
achieves superior results on the HDFS dataset, outperforming all the baseline
methods that also demonstrate commendable performance. On the BGL and Spirit
datasets, LogSD significantly outperforms the baseline methods. In terms of
F1-score, it achieves an improvement of 16.7%, 12.7%, and 13.4% under the
100-logs, 60-logs, and 20-logs settings on the BGL dataset over the second-
best baseline method, and remarkable improvements of 82.6%, 141.8%, and 107.7%
under the Spirit dataset’s corresponding settings. These results provide
empirical evidence for the overall effectiveness of our proposed LogSD. In
contrast, PLELog secures the second-best performance across the three datasets
in most settings. This observation suggests that combining probabilistic label
estimation with supervised learning can be a promising solution for anomaly
detection in semi-supervised scenarios. Moreover, LogBERT exhibits better
performance on the BGL and Spirit datasets compared to other baselines,
specifically outperforming the other self-supervised approaches (Meng et al.,
2019, 2020; Nedelkoski et al., 2020; Zhang et al., 2022) working under the
predefined positional masking scheme and local reconstruction paradigm
described in Section 7. Despite this, it is still far inferior to our LogSD
that combines the G2L reconstruction paradigm with frequency-based masking
scheme. In addition, the reconstruction-based method, AE, exhibits lower
performance compared to the one-class classification-based method, OC4Seq.
This observation implies that global reconstruction may limit the model’s
capacity to effectively differentiate anomalies from normal samples.

When considering the selection of neural networks, the results in Table 2
demonstrate that our CNN-based LogSD outperforms all baseline methods using
other neural networks,including AE (BiLSTM), OC4Seq (LSTM), PLELog (LSTM),
DeepLog/Log2Vec (LSTM), LogAnomaly (LSTM), Logsy (Transformer), LogBERT
(Transformer), and CAT (Transformer). This solidifies the advantage of CNN
networks, consistent with findings in previous studies (Lu et al., 2018; Chen
et al., 2021; Le and Zhang, 2022). Another noteworthy observation pertains to
the methods categorized under the self-supervised type. The three methods
based on Transformer networks—Logsy, CAT, and LogBERT—consistently demonstrate
superior performance compared to LSTM-based methods like DeepLog/Log2Vec and
LogAnomaly in most cases. This observation strengthens the perception of
Transformer networks holding an advantage over LSTM networks. However, upon
examining methods across different technique categories, drawing deterministic
conclusions from their performance comparison becomes intricate. The LSTM-
based methods, OC4Seq and PLELog, occasionally outperform the Transformer-
based methods, Logsy, CAT, and LogBERT. This suggests that while the choice of
neural network types does influence detection performance, its impact may be
less pronounced compared to selecting appropriate anomaly detection techniques
among reconstruction, one-class classification, probabilistic label
estimation, and self-supervised learning.

Additionally, we also compare LogSD with a random detector and a SOTA
supervised method, CNN (Lu et al., 2018). The experimental results in Table
LABEL:tab:extra_baseline_comparison, reveal that LogSD significantly
outperforms the random detector in all experimental settings. While the CNN
method outperforms LogSD, it’s important to note that CNN is a supervised
method, whereas LogSD operates as a semi-supervised method. This indicates
that the additional prior knowledge employed in the supervised method enhances
the detection performance, particularly when there is an ample amount of
labeled data available.

Table 3. Ablation Experiments for Network, Masking Schemes, and Reconstruction
Paradigms.

Dataset | HDFS | BGL | Spirit
---|---|---|---
| MCC | F1 | PRC | ROC | MCC | F1 | PRC | ROC | MCC | F1 | PRC | ROC
LogSDsng | 0.7880 | 0.7858 | 0.7345 | 0.8793 | 0.9126 | 0.9152 | 0.9474 | 0.9935 | 0.7801 | 0.7949 | 0.7074 | 0.9781
LogSDsrl | 0.3872 | 0.3895 | 0.3071 | 0.5999 | 0.8357 | 0.8226 | 0.8336 | 0.9209 | 0.3335 | 0.3340 | 0.2590 | 0.6771
LogSDsfl | 0.7122 | 0.7143 | 0.6583 | 0.8468 | 0.9095 | 0.8939 | 0.9377 | 0.9921 | 0.8769 | 0.8846 | 0.7731 | 0.9634
LogSDsrf | 0.9101 | 0.9067 | 0.9019 | 0.9977 | 0.9264 | 0.9339 | 0.9616 | 0.9965 | 0.8032 | 0.8012 | 0.7055 | 0.9824
LogSDsff | 0.9213 | 0.9153 | 0.9403 | 0.9954 | 0.9596 | 0.9534 | 0.9644 | 0.9977 | 0.8930 | 0.8921 | 0.7509 | 0.9837
LogSDdng | 0.9471 | 0.9462 | 0.9821 | 0.9995 | 0.9384 | 0.9366 | 0.9489 | 0.9905 | 0.8913 | 0.8936 | 0.7343 | 0.9925
LogSDdrl | 0.4223 | 0.4120 | 0.3450 | 0.5979 | 0.8382 | 0.8371 | 0.8631 | 0.9515 | 0.3577 | 0.3420 | 0.2669 | 0.7270
LogSDdfl | 0.7688 | 0.7597 | 0.7473 | 0.9458 | 0.9418 | 0.9281 | 0.9489 | 0.9905 | 0.8812 | 0.8886 | 0.7368 | 0.9833
LogSDdrf | 0.9319 | 0.9338 | 0.9812 | 0.9995 | 0.9329 | 0.9498 | 0.9697 | 0.9974 | 0.8807 | 0.8911 | 0.6930 | 0.9914
LogSDdff | 0.9559 | 0.9583 | 0.9840 | 0.9993 | 0.9483 | 0.9627 | 0.9716 | 0.9977 | 0.8954 | 0.8957 | 0.7346 | 0.9927

  * 1

Notes: the first character in the subscript signifies the network type, where
”s” denotes a single network, and ”d” indicates a dual-network. The second
character represents the masking type, where ”n” implies no masking, ”r”
stands for randomly masking, and ”f” corresponds to frequency-based masking.
The third character indicates the reconstruction type, with ”g” standing for
global reconstruction, ”l” indicating local reconstruction, and ”f”
representing global-to-local reconstruction. Notably, LogSDdff is identical to
LogSD, bearing the name LogSDdff for the purpose of convenient correspondence
with its counterpart within a single network group.

###  5.2. Ablation Studies

In this section, our objective is to comprehensively examine the effectiveness
of each major component within LogSD on the final results. To achieve this, we
conduct ablation studies that categorize the LogSD variants into three
distinct groups:

  * •

Network Comparison: We compare the variants with and without a dual-network
architecture.

  * •

Masking Scheme Comparison: We evaluate the performance of variants with
different masking schemes: no masking, randomly masking, and frequency-based
masking.

  * •

Reconstruction Paradigm Comparison: We assess and contrast the variants with
distinct reconstruction paradigms, including global, local, and G2L
reconstruction paradigms.

Due to space constraints, we only present experimental results for the HDFS
session setting and both BGL and Spirit datasets under the 100-logs window
setting. The results are shown in Table 3.

####  5.2.1. Single-network Vs. Dual-network

From Table 3, it is evident that LogSD variants employing dual-network
(subscripts starting with ”d”) consistently outperform their counterparts with
a single network (subscripts starting with ”s”). These results empirically
confirm our hypothesis that integrating a self-supervised prediction task
between dual networks in LogSD enhances the model’s capability to learn
patterns from normal sequences, aiding in better distinguishing anomalies from
normal sequences.

####  5.2.2. No masking Vs. Randomly Masking Vs. Frequency-Based Masking

Table 3 provides a clear comparison between LogSD variants denoted by ”$n$”
(for no masking), ”$r$” (for randomly masking) and ”$f$” (for frequency-based
masking) across different reconstruction paradigms, using either single or
dual networks. The results consistently demonstrate the superiority of
frequency-based masking in terms of F1-score. Moreover, it’s evident that
LogSD variants with random masking exhibit significantly higher performance
fluctuations compared to their frequency-based masking counterparts. This
variance suggests a potential drawback associated with the randomly masking
scheme. Overall, the experimental results empirically support our claim that
frequency-based masking facilitates the trained model in effectively learning
normal patterns through the reconstruction of low-frequency events.
Consequently, when faced with new sequences, our approach excels in
distinguishing these anomalies from normal sequences, particularly those with
rare normal events.

####  5.2.3. Global Vs. Local Vs. G2L reconstruction

Furthermore, we investigate the performance difference of three reconstruction
paradigm on log anomaly detection. From Table3, we can see that those LogSD
variants subscript ending with ”$f$” outperform their counterparts with
subscript ending with ”$l$” or ”$g$” in most cases. Another interesting thing
we can see is that those methods under local reconstruction paradigm are not
necessarily superior to those under global reconstruction paradigm. This may
imply that feeding masked inputs into reconstruction networks in the local
reconstruction paradigm carries a potential risk of information loss, which
leads to sub-optimal detection performance for the corresponding variant
methods. Conversely, the global-to-local paradigm might be able to avoid this
shortcoming in the local reconstruction paradigm.

(a) Loss Weight Sensitivity.

(b) Masking Ratio Sensitivity.

Figure 4. Sensitivity analysis on BGL under 100 logs window setting.

####  5.2.4. The impact of consecutive duplicate events preprocessing

Two types of sequences in the original datasets may influence the performance
of LogSD. The first is the sequences containing only a single unique event.
The presence of such sequences can undermine LogSD’s frequency-based masking
mechanism, diminishing the model’s effectiveness in learning infrequent norm
event patterns. The second is the sequences with consecutive duplicate events
but more than one event. These sequences introduce higher fluctuations in the
distribution of events, posing challenges in applying a specific masking ratio
or a small random masking ratio set that ensures competitive model performance
across diverse datasets. The experimental results for the BGL and Spirit
datasets, categorized by original sequences, sequences without single events,
and preprocessed sequences, are provided in Table
LABEL:tab:preprocessing_experiments. The results clearly indicate that the
removal of consecutive duplicate events leads to an enhancement in the
performance of LogSD.

###  5.3. Sensitivity Studies

####  5.3.1. Loss Weights Sensitivity

In the previous experiments, all assessments were conducted with the hyper-
parameter value set at $\alpha=50$. To delve deeper into the impact of these
weights, a sensitivity analysis was conducted. The hyper-parameter was varied
within the range of $10^{-3}$ to $10^{2}$. As illustrated in Figure 4a, the
influence of hyper-parameter choices for the loss weight on the model’s
performance varies from 0.8043 to 0.9627. Specifically, increasing the weight
$\alpha$ for the reconstruction loss generally corresponds to higher accuracy,
peaking at 50. Therefore, we empirically opt to maintain the current hyper-
parameter setting.

####  5.3.2. Masking Ratio Sensitivity

Regarding the frequency-based masking ratio hyper-parameter, we investigated
its influence on detection performance. This involved setting the masking
ratio to a constant value ranging from 0.05 to 0.9, as well as utilizing a
random value chosen from our empirically determined set
$K=\\{0.05,0.1,0.15,0.2,0.3\\}$. From Figure 4b, we observe a slight impact of
random masking ratios on LogSD’s performance, attributed to event distribution
characteristics within a specified window size. In the comparison between
fixed masking ratios and randomly chosen ratios from $K$, the latter revealed
commendable results, albeit not the optimal. Opting for a unified set of
randomly chosen ratios mitigates challenges in hyper-parameter selection for
diverse datasets, leading us to adopt this approach in this study.

(a) Training Time.

(b) Testing Time.

Figure 5. Scalability analysis on BGL under 100 logs window setting.

###  5.4. Scalability Studies

To evaluate the scalability of LogSD, we conducted runtime comparisons with
the baseline methods, showcasing the efficiency of our CNN-based LogSD. Figure
5 presents the average training and testing times per epoch for these
approaches across varying window size settings. Due to a consistent trend in
time efficiency, we just present the experimental results on the BGL dataset
with a window size of 100 logs for conciseness. LogSD demonstrates comparable
training efficiency in comparison to the baseline methods. In terms of testing
time, LogSD is shorter than LogAnomaly and LogBERT, similar to CAT, yet
slightly longer than other baseline methods.

##  6\. Discussion

###  6.1. The Usability of the Proposed Method

In the log analysis domain, beyond the effectiveness of detection, the
usability of a detection method also plays a pivotal role in real-world
scenarios. The usability of a detection method typically encompass:
integration capability and cost with existing software systems generating
logs, adaptability to system evolution, and root cause localization
capability. (1) LogSD showcases robust integration capabilities as it doesn’t
depend on any prerequisite prior knowledge about the system generating the
logs. LogSD can perform well in both session-based and time-window-based
datasets, allowing it to adapt effectively to different blackbox applications.
(2) Regarding the data collection, preprocessing and model training cost,
LogSD is similar to existing methods except an additional preprocessing step
to remove consecutive events, whose time cost is negligible. Therefore, it is
not costly to integrate LogSD with an existing system. (3) In terms of
adaptability, regular model retraining is a usual essential strategy when
significant changes occur in the business workflow of a system. The efficient
model training/retraining time, depicted in Figure 5, contributes to LogSD’s
adaptability in dynamic environments. (4) Root cause analysis is crucial for
operators to reduce the impact of a system failure. Our proposed method,
incorporating a masking mechanism, can be enhanced for fine-grained root cause
analysis by integrating a progressive masking refinement mechanism. We will
leave it to the future work.

###  6.2. Threats To Validity

In this study, we identify the following threats to validity:

External Threats: This threat pertains to factors affecting the generalization
of results. One potential external threat to validity of this study is the
subject selection bias introduced by the choice of datasets. We utilized three
log datasets for our experiments, all of them are drawn from real-world
industrial systems and are prevalent in existing literature (Xie et al., 2021;
Le and Zhang, 2021, 2022; Li et al., 2023b). Our future work will encompass a
more diverse range of datasets and systems to further evaluate the
generalizability of our approach.

Internal Threats: This refers to potential threats that might affect the
results and have not been adequately considered. In our study, the internal
threat to validity primarily lies in the implementations of LogSD and the
compared approaches, as well as the design of experiments. To mitigate the
threat stemming from the implementation, we conducted rigorous inspection and
testing of the program. We assumed that all bugs revealed during testing were
fixed in our approach. LogSD was implemented based on popular libraries, and
we made our source code available for replication purposes. Concerning the
compared methods, we utilized their open-source implementations. To address
potential threats related to the experimental design, we took several
measures. First, we ran all experiments three times and reported the averaged
results. Second, we ensured a consistent setting for all methods, such as
using 100 epochs for training and a patience of 20 for early stopping.
Additionally, for the baselines, we selected the learning rate and relevant
configurations as per the best-reported value in their respective papers. We
believe this careful experimental design facilitates fair comparisons among
all evaluated methods.

##  7\. Related Work

###  7.1. Existing Masking Schemes

The utilization of masking as a data augmentation technique has gained
widespread recognition in many self-supervised learning based log anomaly
detection methods (Du et al., 2017; Meng et al., 2019; Nedelkoski et al.,
2020; Guo et al., 2021; Zhang et al., 2022). In these methods, a masking
scheme serves as one of pre-processing steps to enable the trained model to
utilize the unmasked event to reconstruct/predict the masked events for log
anomaly detection. The masking schemes in existing log anomaly detection
methods can be roughly categorized into predefined positional masking scheme
and randomly masking scheme. Predefined positional masking scheme involves the
selection of specific positional indices where events occur within a sequence
for input sequence event masking. Typically, these indices are chosen in
advance and remain unchanged throughout the model training and inference
processes. The methods using predefined positional masking scheme include (Du
et al., 2017; Meng et al., 2019; Nedelkoski et al., 2020; Zhang et al., 2022).
In contrast, the positional indices in randomly masking scheme are selected on
the fly in a random manner, which makes sequence event masking more flexible
and diverse. (Guo et al., 2021) is the representative of the method using this
masking scheme. However, both of the aforementioned masking schemes fail to
take the occurrence frequency of events in sequences into consideration,
making it challenging for the model to effectively distinguish between
anomalous sequences and normal sequences that comprise infrequent events.

###  7.2. Existing Reconstruction Paradigms

The reconstruction paradigm in existing methods can be classified into: global
reconstruction paradigm and local reconstruction paradigm. Some existing
reconstruction based methods like AE (Farzad and Gulliver, 2020) often employ
the following reconstruction paradigm: sequences are fed into a reconstruction
network featuring an auto-encoder architecture. Subsequently, reconstruction
errors are computed by comparing the reconstructed sequences with the original
ones. Given that the reconstruction covers all events within a sequence, we
refer to it as the ”global reconstruction” paradigm. As overemphasizing the
reconstruction of each event within sequences, the global reconstruction
paradigm may hinder model convergence, especially when dealing with sequences
exhibiting complex distributions (Xu et al., 2023). Consequently, it may lead
to sub-optimal outcomes for those methods using the global reconstruction
paradigm.

The local reconstruction paradigm involves inputting sequences with
intentionally masked specific events into a reconstruction/prediction network.
Subsequently, the masked events are reconstructed using the unmasked events
within the sequences. We name this paradigm as local reconstruction paradigm
because they leverage one portion of events in sequences to reconstruct the
other portion. This paradigm is widely employed in many models like BERT
(Devlin et al., 2018) and Masked Autoencoder (He et al., 2022). In the log
analysis domain, this paradigm is also widely used in many studies, such as
DeepLog/LogVec, LogAnomaly, Logsy, CAT, and LogBERT. However, the local
reconstruct paradigm does not fully exploit the potential information
contained within the masked events themselves to generate informative
indicators for anomaly detection. This limitation may result in sub-optimal
performance in the context of log sequence anomaly detection.

###  7.3. Existing Deep Semi-supervised Approaches to Log-based Anomaly
Detection

From the perspective of techniques, existing deep semi-supervised log-based
anomaly detection approaches can be broadly classified as follows:

  * •

One class classification-based approach, which commonly utilizes machine
learning or deep learning techniques to encapsulate normal data or their
latent representations within a compact space. Then, any given instance
positioned beyond the boundaries of the learned compact space is classified as
anomalous. OC4Seq (Wang et al., 2021) is the representative of this approach.
Nonetheless, the effectiveness of one-class models can be compromised when
confronted with datasets exhibiting intricate distributions within normal
samples (Huang et al., 2022; Xu et al., 2023).

  * •

Probabilistic label estimation-based approach, which employs the idea of
Positive and Unlabeled Learning (PU Learning) (Elkan and Noto, 2008) to
initially derive pseudo labels for given unlabeled instances based on known
normal log sequences in the training set. Subsequently, a supervised deep
learning model is trained on these data with pseudo labels to classify these
instances as anomalous or normal. A notable exemplar within this approach is
PLELog (Yang et al., 2021). However, the effectiveness of this approach is
intrinsically linked to the accuracy of probabilistic label estimation. The
estimation outcome is, in turn, influenced by the inherent characteristics of
the data and the techniques employed for probabilistic label estimation.

  * •

Reconstruction-based approach works on the assumption that a normal data
instance can be well reconstructed from its low-dimensional latent
representation, while anomalies exhibit higher reconstruction errors compared
to their normal counterparts. This idea is utilized in the method of AE
(Farzad and Gulliver, 2020). However, this approach still struggles to achieve
satisfactory performance due to its overemphasis on reconstructing the entire
input (Xu et al., 2023).

  * •

Self-supervised learning-based approach, which leverages designed self-
supervised tasks to steer model optimization (Xie et al., 2022a), consequently
yielding specialized representations for log-based anomaly detection. DeepLog
(Du et al., 2017), LogBERT (Guo et al., 2021) and CAT (Zhang et al., 2022)
serve as notable exemplars of this approach. However, as mentioned in Section
7.1 and 7.2, the existing masking schemes and the reconstruction paradigm may
potentially impede their models’ capacity to learn patterns from normal
instances effectively. This could lead to a compromise in the overall
detection accuracy.

##  8\. Conclusion

In this paper, we have proposed LogSD, a novel semi-supervised log-based
anomaly detection approach. LogSD encompasses a frequency-based masking
scheme, a G2L reconstruction paradigm, and a dual-network framework with three
self-supervised learning tasks. The frequency-based masking scheme empowers
the model to emphasize the learning of normal patterns associated with
infrequent events. The global-to-local reconstruction paradigm capitalizes on
all the events in sequences for the reconstruction of those infrequent events.
The dual-network framework with three self-supervised tasks not only bolsters
the model by leveraging information from both the original space and the
latent space but amalgamates the strengths of reconstruction, one-class-
classification, and knowledge distillation techniques. Through the
collaboration of these components, LogSD efficiently captures less biased and
highly discriminative normal patterns from infrequent log messages. The
acquired less biased normal patterns enable LogSD to better discriminate
between anomalous sequences and those containing sporadic events. Our
extensive experiments on three widely-used public datasets illustrate the
effectiveness of LogSD.

Our source code and experimental data are available at
https://github.com/ilwoof/logsd/.

###### Acknowledgements.

This research was supported by an Australian Government Research Training
Program (RTP) Scholarship, and by the Australian Research Council’s Discovery
Projects funding scheme (project DP200102940). The work was also supported
with super-computing resources provided by the Phoenix High Powered Computing
(HPC) service at the University of Adelaide.

## References

  * (1)
  * Log (2019) 2019\.  LogAnomaly Code Repository.  https://github.com/donglee-afar/logdeep.
  * Dee (2021) 2021\.  DeepLoglizer Code Repository.  https://github.com/logpai/deep-loglizer.
  * LBc (2021) 2021\.  LogBert Code Repository.  https://github.com/HelenGuohx/logbert.
  * OC4 (2021) 2021\.  OC4Seq Code Repository.  https://github.com/wzwtrevor/Multi-Scale-One-Class-Recurrent-Neural-Networks.
  * PLE (2021) 2021\.  PLELog Code Repository.  https://github.com/LeonYang95/PLELog.
  * CAT (2022) 2022\.  CAT Code Repository.  https://github.com/mmichaelzhang/CAT.
  * Akcay et al. (2019) Samet Akcay, Amir Atapour-Abarghouei, and Toby P Breckon. 2019\.  Ganomaly: Semi-supervised anomaly detection via adversarial training. In _Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part III 14_. Springer, 622–637.
  * Andonov et al. (2022) Stefan Andonov, Viktor Jovev, Aleksandar Kitanovski, Aleksandar Krsteski, and Gjorgji Madjarov. 2022\.  logs2graphs: Data-driven graph representation and visualization of log data.
  * Baldi et al. (2000) Pierre Baldi, Søren Brunak, Yves Chauvin, Claus AF Andersen, and Henrik Nielsen. 2000.  Assessing the accuracy of prediction algorithms for classification: an overview.  _Bioinformatics_ 16, 5 (2000), 412–424.
  * Bodik et al. (2010) Peter Bodik, Moises Goldszmidt, Armando Fox, Dawn B Woodard, and Hans Andersen. 2010.  Fingerprinting the datacenter: automated classification of performance crises. In _Proceedings of the 5th European conference on Computer systems_. 111–124.
  * Chen et al. (2004) Mike Chen, Alice X Zheng, Jim Lloyd, Michael I Jordan, and Eric Brewer. 2004.  Failure diagnosis using decision trees. In _International Conference on Autonomic Computing, 2004\. Proceedings._ IEEE, 36–43.
  * Chen et al. (2021) Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael R Lyu. 2021.  Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection.  _arXiv preprint arXiv:2107.05908_ (2021).
  * Dai et al. (2023) Hetong Dai, Yiming Tang, Heng Li, and Weiyi Shang. 2023\.  PILAR: Studying and Mitigating the Influence of Configurations on Log Parsing. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_. IEEE, 818–829.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Du et al. (2017) Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017\.  Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In _Proceedings of the 2017 ACM SIGSAC conference on computer and communications security_. 1285–1298.
  * Elkan and Noto (2008) Charles Elkan and Keith Noto. 2008.  Learning classifiers from only positive and unlabeled data. In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_. 213–220.
  * Farzad and Gulliver (2020) Amir Farzad and T Aaron Gulliver. 2020.  Unsupervised log message anomaly detection.  _ICT Express_ 6, 3 (2020), 229–237.
  * Fu et al. (2023) Ying Fu, Meng Yan, Zhou Xu, Xin Xia, Xiaohong Zhang, and Dan Yang. 2023\.  An empirical study of the impact of log parsers on the performance of log-based anomaly detection.  _Empirical Software Engineering_ 28, 1 (2023), 6\.
  * Guo et al. (2021) Haixuan Guo, Shuhan Yuan, and Xintao Wu. 2021.  Logbert: Log anomaly detection via bert. In _2021 international joint conference on neural networks (IJCNN)_. IEEE, 1–8.
  * Haibo and Yunqian (2013) He Haibo and Ma Yunqian. 2013.  Imbalanced learning: foundations, algorithms, and applications.  _Wiley-IEEE Press_ 1, 27 (2013), 12.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022\.  Masked autoencoders are scalable vision learners. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 16000–16009.
  * He et al. (2017) Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R Lyu. 2017\.  Drain: An online log parsing approach with fixed depth tree. In _2017 IEEE international conference on web services (ICWS)_. IEEE, 33–40.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ (2015).
  * Huang et al. (2022) Chaoqin Huang, Qinwei Xu, Yanfeng Wang, Yu Wang, and Ya Zhang. 2022.  Self-supervised masking for unsupervised anomaly detection and localization.  _IEEE Transactions on Multimedia_ (2022).
  * Huang and Ling (2005) Jin Huang and Charles X Ling. 2005.  Using AUC and accuracy in evaluating learning algorithms.  _IEEE Transactions on knowledge and Data Engineering_ 17, 3 (2005), 299–310.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016.  Fasttext. zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_ (2016).
  * Landauer et al. (2022) Max Landauer, Sebastian Onder, Florian Skopik, and Markus Wurzenberger. 2022.  Deep Learning for Anomaly Detection in Log Data: A Survey.  _arXiv preprint arXiv:2207.03820_ (2022).
  * Lavazza and Morasca (2022) Luigi Lavazza and Sandro Morasca. 2022.  Comparing $\phi$ and the F-measure as performance metrics for software-related classifications.  _Empirical Software Engineering_ 27, 7 (2022), 185\.
  * Le and Zhang (2021) Van-Hoang Le and Hongyu Zhang. 2021.  Log-based anomaly detection without log parsing. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 492–504.
  * Le and Zhang (2022) Van Hoang Le and Hongyu Zhang. 2022.  Log-based Anomaly Detection with Deep Learning: How Far Are We?  _arXiv preprint arXiv:2202.04301_ (2022).
  * Li et al. (2023a) Zhenhao Li, An Ran Chen, Xing Hu, Xin Xia, Tse-Hsun Chen, and Weiyi Shang. 2023a.  Are They All Good? Studying Practitioners’ Expectations on the Readability of Log Messages.  _arXiv preprint arXiv:2308.08836_ (2023).
  * Li et al. (2023b) Zhong Li, Jiayang Shi, and Matthijs van Leeuwen. 2023b.  Graph Neural Network based Log Anomaly Detection and Explanation.  _arXiv preprint arXiv:2307.00527_ (2023).
  * Liang et al. (2007) Yinglung Liang, Yanyong Zhang, Hui Xiong, and Ramendra Sahoo. 2007\.  Failure prediction in ibm bluegene/l event logs. In _Seventh IEEE International Conference on Data Mining (ICDM 2007)_. IEEE, 583–588.
  * Liao et al. (2020) Lizhi Liao, Jinfu Chen, Heng Li, Yi Zeng, Weiyi Shang, Jianmei Guo, Catalin Sporea, Andrei Toma, and Sarah Sajedi. 2020\.  Using black-box performance models to detect performance regressions under varying workloads: an empirical study.  _Empirical Software Engineering_ 25 (2020), 4130–4160.
  * Lin et al. (2016) Qingwei Lin, Hongyu Zhang, Jian-Guang Lou, Yu Zhang, and Xuewei Chen. 2016.  Log clustering based problem identification for online service systems. In _2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C)_. IEEE, 102–111.
  * Liu et al. (2008) Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008\.  Isolation forest. In _2008 eighth ieee international conference on data mining_. IEEE, 413–422.
  * Liu et al. (2023) Jinyang Liu, Tianyi Yang, Zhuangbin Chen, Yuxin Su, Cong Feng, Zengyin Yang, and Michael R Lyu. 2023.  Practical Anomaly Detection over Multivariate Monitoring Metrics for Online Services.  _arXiv preprint arXiv:2308.09937_ (2023).
  * Lou et al. (2010) Jian-Guang Lou, Qiang Fu, Shengqi Yang, Ye Xu, and Jiang Li. 2010.  Mining Invariants from Console Logs for System Problem Detection.. In _USENIX Annual Technical Conference_. 1–14.
  * Lu et al. (2018) Siyang Lu, Xiang Wei, Yandong Li, and Liqiang Wang. 2018\.  Detecting anomaly in big data system logs using convolutional neural network. In _2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing_. IEEE, 151–158.
  * Ma et al. (2022) Rongrong Ma, Guansong Pang, Ling Chen, and Anton van den Hengel. 2022.  Deep graph-level anomaly detection by glocal knowledge distillation. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_. 704–714.
  * Makanju et al. (2009) Adetokunbo AO Makanju, A Nur Zincir-Heywood, and Evangelos E Milios. 2009.  Clustering event logs using iterative partitioning. In _Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining_. 1255–1264.
  * Meng et al. (2020) Weibin Meng, Ying Liu, Yuheng Huang, Shenglin Zhang, Federico Zaiter, Bingjin Chen, and Dan Pei. 2020.  A semantic-aware representation framework for online log analysis. In _2020 29th International Conference on Computer Communications and Networks (ICCCN)_. IEEE, 1–7.
  * Meng et al. (2019) Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, et al. 2019\.  Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs.. In _IJCAI_ , Vol. 19. 4739–4745.
  * Nedelkoski et al. (2020) Sasho Nedelkoski, Jasmin Bogatinovski, Alexander Acker, Jorge Cardoso, and Odej Kao. 2020.  Self-attentive classification-based anomaly detection in unstructured logs. In _2020 IEEE International Conference on Data Mining (ICDM)_. IEEE, 1196–1201.
  * Nguyen et al. (2016) Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016\.  Integrating distributional lexical contrast into word embeddings for antonym-synonym distinction.  _arXiv preprint arXiv:1605.07766_ (2016).
  * Ni et al. (2022) Chao Ni, Wei Wang, Kaiwen Yang, Xin Xia, Kui Liu, and David Lo. 2022\.  The best of both worlds: integrating semantic features with expert features for defect prediction and localization. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 672–683.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Ruff et al. (2018) Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. 2018.  Deep one-class classification. In _International conference on machine learning_. PMLR, 4393–4402.
  * Salehi et al. (2021) Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad H Rohban, and Hamid R Rabiee. 2021\.  Multiresolution knowledge distillation for anomaly detection. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 14902–14912.
  * Schölkopf et al. (2001) Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. 2001\.  Estimating the support of a high-dimensional distribution.  _Neural computation_ 13, 7 (2001), 1443–1471.
  * Tan et al. (2023) Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. 2023.  S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking. In _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining_. 787–795.
  * USENIX (2008) USENIX. 2008.  CFDR DATA.  GitHub.  https://www.usenix.org/cfdr-data Accessed: 3/3/2023.
  * Wan et al. (2021) Yi Wan, Yilin Liu, Dong Wang, and Yujin Wen. 2021\.  GLAD-PAW: Graph-Based Log Anomaly Detection by Position Aware Weighted Graph Attention Network. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_. Springer, 66–77.
  * Wang et al. (2021) Zhiwei Wang, Zhengzhang Chen, Jingchao Ni, Hui Liu, Haifeng Chen, and Jiliang Tang. 2021\.  Multi-scale one-class recurrent neural networks for discrete event sequence anomaly detection. In _Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining_. 3726–3734.
  * Xiao et al. (2021) Qinfeng Xiao, Jing Wang, Youfang Lin, Wenbo Gongsa, Ganghui Hu, Menggang Li, and Fang Wang. 2021.  Unsupervised anomaly detection with distillated teacher-student network ensemble.  _Entropy_ 23, 2 (2021), 201.
  * Xie et al. (2022a) Yaochen Xie, Zhao Xu, and Shuiwang Ji. 2022a.  Self-supervised representation learning via latent graph prediction. In _International Conference on Machine Learning_. PMLR, 24460–24477.
  * Xie et al. (2022b) Yongzheng Xie, Hongyu Zhang, and Muhammad Ali Babar. 2022b.  LogGD: Detecting Anomalies from System Logs by Graph Neural Networks.  _arXiv preprint arXiv:2209.07869_ (2022).
  * Xie et al. (2021) Yongzheng Xie, Hongyu Zhang, Bo Zhang, Muhammad Ali Babar, and Sha Lu. 2021.  LogDP: Combining Dependency and Proximity for Log-Based Anomaly Detection. In _International Conference on Service-Oriented Computing_. Springer, 708–716.
  * Xu et al. (2023) Hongzuo Xu, Yijie Wang, Juhui Wei, Songlei Jian, Yizhou Li, and Ning Liu. 2023\.  Fascinating Supervisory Signals and Where to Find Them: Deep Anomaly Detection with Scale Learning. In _Proceedings of the 40th International Conference on Machine Learning (Poster), ICML_.
  * Xu et al. (2009a) Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael Jordan. 2009a.  Largescale system problem detection by mining console logs.  _Proceedings of SOSP’09_ (2009).
  * Xu et al. (2009b) Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I Jordan. 2009b.  Detecting large-scale system problems by mining console logs. In _Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles_. 117–132.
  * Yang et al. (2021) Lin Yang, Junjie Chen, Zan Wang, Weijing Wang, Jiajun Jiang, Xuyuan Dong, and Wenbin Zhang. 2021.  Semi-supervised log-based anomaly detection via probabilistic label estimation. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 1448–1460.
  * Yao and Shepperd (2021) Jingxiu Yao and Martin Shepperd. 2021.  The impact of using biased performance metrics on software defect prediction research.  _Information and Software Technology_ 139 (2021), 106664.
  * Zhang et al. (2020) Bo Zhang, Hongyu Zhang, Pablo Moscato, and Aozhong Zhang. 2020\.  Anomaly detection via mining numerical workflow relations from logs. In _2020 International Symposium on Reliable Distributed Systems (SRDS)_. IEEE, 195–204.
  * Zhang et al. (2022) Shengming Zhang, Yanchi Liu, Xuchao Zhang, Wei Cheng, Haifeng Chen, and Hui Xiong. 2022\.  Cat: Beyond efficient transformer for content-aware anomaly detection in event sequences. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 4541–4550.
  * Zhang et al. (2019) Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li, et al. 2019\.  Robust log-based anomaly detection on unstable log data. In _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 807–817.
  * Zhu et al. (2023a) Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, and Michael R. Lyu. 2023a.  GitHub.  https://github.com/logpai/loghub
  * Zhu et al. (2023b) Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, and Michael R. Lyu. 2023b.  Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics. In _IEEE International Symposium on Software Reliability Engineering (ISSRE)_.
  * Zhu et al. (2019) Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R Lyu. 2019.  Tools and benchmarks for automated log parsing. In _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 121–130.