# Leveraging Log Instructions in Log-based Anomaly Detection

Jasmin Bogatinovski1, Gjorgji Madjarov3, Sasho Nedelkoski1, Jorge Cardoso2 and
Odej Kao1  1 Technical University Berlin, Berlin, Germany, Email:
<EMAIL_ADDRESS>2 Huawei Munich Research, Munich, Germany  3
University Ss Cyril and Methodius, Skopje, North Macedonia

###### Abstract

Artificial Intelligence for IT Operations (AIOps) describes the process of
maintaining and operating large IT systems using diverse AI-enabled methods
and tools for, e.g., anomaly detection and root cause analysis, to support the
remediation, optimization, and automatic initiation of self-stabilizing IT
activities. The core step of any AIOps workflow is anomaly detection,
typically performed on high-volume heterogeneous data such as log messages
(logs), metrics (e.g., CPU utilization), and distributed traces. In this
paper, we propose a method for reliable and practical anomaly detection from
system logs. It overcomes the common disadvantage of related works, i.e., the
need for a large amount of manually labeled training data, by building an
anomaly detection model with log instructions from the source code of 1000+
GitHub projects. The instructions from diverse systems contain rich and
heterogenous information about many different normal and abnormal IT events
and serve as a foundation for anomaly detection. The proposed method, named
ADLILog, combines the log instructions and the data from the system of
interest (target system) to learn a deep neural network model through a two-
phase learning procedure. The experimental results show that ADLILog
outperforms the related approaches by up to 60% on the F1 score while
satisfying core non-functional requirements for industrial deployments such as
unsupervised design, efficient model updates, and small model sizes.

###### Index Terms:

anomaly detection, log data, system dependability, AIOps, deep learning

##  I Introduction

IT infrastructures in numerous application fields consist of thousands of
networked software (microservices) and hardware (e.g., IoT, Edge) components.
The uninterrupted and correct interaction is crucial for the functionality of
the overall system and the deployed applications. However, this IT complexity
combined with the required QoS guarantees (e.g. maximal latency) increasingly
overwhelms the IT operators in charge. The current trends of agile software
development with hundreds of updates and daily deployments further exacerbate
the operational challenges. The holistic overview, operation, and maintenance
of the IT infrastructure grow even more challenging when additionally it is
affected by unforeseen factors such as failures, software errors, security
breaches, or external environmental events. Companies react to these threats
by employing additional site reliability engineers (SREs) as well as by
deploying AI-enabled methods for IT operations (AIOps) [1].

The AIOps methods collect and analyse plenty of IT system information – metric
data (e.g., CPU utilization), logs, and traces (paths of function calls) to
detect anomalies, locate their root causes and remediate them. The diverse
AIOps techniques enable fast, efficient and effective prevention of upcoming
failures, aiming to minimize their hazardous effects during the daily
operational activities [1].

In this paper, we focus on anomaly detection in the context of AIOps, as a
core step towards enhancing fault tolerance: the earlier an anomaly is
detected, the more time is available to prevent the failure and mitigate the
impact on the QoS. We focus on system log messages (logs) as semantically rich
data written by humans for humans. The logs allow a more insightful analysis
and interpretation than, e.g., metric data [2]. For example, a sharp increase
in the network packet loss (a commonly used metric for network monitoring)
only indicates a problem with the network, but it does not provide a clue why
it happens. In comparison, logs give semantically meaningful clues for the
anomaly. For example, when a switch generates the log “System is rebooting
now.”, the operator detects that the switch is failing (potentially anomalous)
and obtain a clue that the potential anomaly is caused by switch rebooting.

Logs are generated from log instructions that developers insert in the source
code (e.g., log.info(”VM took %f seconds to spawn.”, createSeconds)) to
visualise important system events and to create hints for the operators that
run the system as a black-box [3]. The log instructions are commonly composed
of static text (log template), variable parameters of the event (e.g.,
createSeconds), and log level giving information about the severity level of
the event (e.g., ”info”, ”fatal”, ”error”). The log levels come at different
granularity, conditioned on the used programming languages and logging
libraries. The lower log levels such as ”info” are usually used when
describing normal state or state transitions, e.g., ”Successful connection.”.
In contrast, higher log levels such as ”error”, ”critical”, or ”fatal”
commonly accompany events that describe abnormal states or state transitions,
e.g., ”Machine failure”. Therefore, the log levels encode rich expert
information for manual detection of anomalous events, frequently used in
today’s operational practices [3]. For example, to diagnose an anomaly,
operators commonly use manual search for logs with higher levels such as
”error”, ”critical”, or ”fatal” [4].

Owning to the ever-increasing IT system complexities, logs are constantly
generated in large volumes (e.g., up to several TB per day [5]). The emergence
of complexity makes the manual log-based anomaly detection time-consuming [3],
prompting the need for automation [6, 7]. Thereby, automatic methods for log-
based anomaly detection are increasingly researched and adopted [8, 9, 10, 11,
12, 13, 14]. Current methods are commonly grouped into two families, i.e.,
supervised and unsupervised [6]. Existing supervised methods depend on
manually labeled training data. Due to the constant evolution of the software
systems [11], the supervised methods require a repetitive, time-expensive
labeling process, which is oftentimes practically challenging and infeasible
[5]. The unsupervised methods mitigate the labeling problem by modeling with
logs from normal system states and detecting any significant deviations from
the modeled normality state as anomalies. However, the lack of explicit
information about anomalous logs during modeling leads to limited input
representation, reducing their detection performance, and questioning their
practical usability [6].

To address the two challenges, we propose ADLILog. The central idea of the
method is to use data from public code projects (e.g., GitHub) alongside the
data from the system of interest (target system) when learning the anomaly
detection model. Since the public code projects contain numerous log
instructions for diverse normal and abnormal events, we assume that they may
encode rich anomaly-related information. Following the usage of the log levels
for manual log anomaly detection, we considered grouping the instructions
based on the log levels to extract anomaly-related information. Specifically,
we created two severity level groups from the log instructions based on their
log levels – ”normal” (composed of ”info”) and ”abnormal” (composed of
”error”, ”fatal”, and ”critical”). To verify our assumption, we conducted a
study to examine the anomaly-related language properties between the two
groups (i.e., diversity in the vocabulary and the sentiment of the words). The
study results show that the two groups extract anomaly-related information
that can be used as a basis for anomaly detection. Based on this observation,
we introduce ADLILog, which uses the anomaly-related information alongside the
target system data to learn a deep learning anomaly detection model through a
two-phase learning procedure.

An important advantage of ADLILog is that, by having access to ”normal” and
especially ”abnormal” event descriptions from many different software systems,
it learns a model by supervised learning objectives, without the need for
target-system log labels (i.e., its unsupervised method). Thereby, ADLILog
eliminates the need for time-expensive labeling while preserving the advantage
of supervised modeling. The latter addresses the challenge of limited input
representation during modeling. To prove the quality of detection, we
extensively evaluate ADLILog against seven related methods on two widely used
benchmark datasets and demonstrate that our method outperforms the supervised
methods by 5-24%, and the unsupervised by 40-63% on F1 score. The
datasets111https://zenodo.org/record/6376763 and method
implementation222https://github.com/ADLILog/ADLILog are available as open-
source for fostering the research on this practically relevant problem.

The remaining of the paper is structured as follows. Section II presents our
study that examines the potential of the log instructions to aid anomaly
detection. Section III introduces ADLILog. Section IV gives the experimental
results. Section V discusses the related work. Section VI concludes the paper
and gives directions for future work.

##  II Examining the potential of log instructions for log-based anomaly
detection

In this section, we examine the potential of the log instructions to aid
anomaly detection. We start with our observation that there exist two log
instructions severity groups, based on their log levels, i.e., ”normal”
(”info”) and ”abnormal” (”fatal”, ”critical”, and ”error”). Following the
usages of the log levels for anomaly detection [4], we assume that the static
texts of the instructions have complementary properties concerning the two
severity level groups, preserving anomaly-related information. To study the
validity of the assumption, we analyze two language properties of the word
combinations (n-grams) in the log instructions static texts with respect to
the two groups. Specifically, by studying the n-gram uniqueness among the
groups, we examine the differences in the vocabulary used to describe
normal/abnormal events. By relating the n-grams with the expressed intent
(e.g., positive intent relates to normal system state), we examine the
semantic diversity between the groups, i.e., if the n-grams express positive
(normal state transition) or negative (abnormal state transition) intents. In
the following, we first describe the 1) log instruction collection procedure
and then present the 2) uniqueness and the 3) sentiment analyses of the log
instructions static texts.

###  II-A Log Instruction Collection and Processing

For the starting point of the analysis, we created a representative dataset by
collecting log instructions from the source code of more than 1000 public code
projects from GitHub. We included a wide spectrum of domains and programming
languages (Python, Java, C++), covering different log instruction types. The
heterogeneity enables us to examine the vocabulary diversity and semantic
properties used in describing normal and abnormal events across systems. That
way, we consider diverse logging styles and a wide range of events, with
complementary severity levels. To account for the reliability in the log level
assignment, we selected projects with more than a 100-stars and at least 20
contributors. The collection procedure resulted in more than 100.000 log
instructions.

Afterwards, we process the log instructions by extracting the log levels and
the static texts to represent their severity levels and the event
descriptions. The diverse programming languages use different names for the
log levels. Therefore, as a first step, we unify all the log levels. We
preprocess the static texts by applying several preprocessing techniques,
similar to related works [15], including lower-case word transformation,
splitting the static texts on whitespace, removing placeholders, removing
ASCII special characters and stopwords from the Spacy English dictionary [16].
We refer to this data as Severity Level (SL) data. It is a set of tuples from
two elements – (1) the static text of log instruction, and (2) the severity
group based on the aforenamed log level to severity group mapping (e.g.,
(”machine error”, ”abnormal”)). We used the SL data to conduct the log
instruction examination study. Similar to related log instruction analysis
studies [17], we extracted the n-grams from the static text by varying the
value for the n parameter in the range $n=\\{3,4,5\\}$. An n-grams analysis
shows that many n-grams appear once. To eliminate the impact of the rare
n-grams on the analysis, we considered the n-grams that appear more than three
times [17].

###  II-B Log Instructions Static Texts Uniqueness Analysis

Intuitively, when describing abnormal events, the static text typically
contains n-grams like ”failure” or ”error connection”, as opposed to normal
events, where n-grams like ”successful” and ”accepted” are more likely to
appear. Therefore, we assume that the log instructions static texts of the two
severity level groups share different, partially overlapping vocabularies. To
verify this, we considered an approach from information theory that defines
the amount of information uncertainty in a message [18]. In our case, we
analyze the relation of the n-grams with the two severity groups. At first,
given an n-gram (e.g., ”machine failure”), there is high uncertainty for the
assigned severity group. As we receive more information for the n-gram (e.g.,
new logs with the n-gram ”machine failure”), its uncertainty concerning the
associated severity group is reduced. For example, if the n-gram ”machine
failure” is associated five times with the ”abnormal” and one time with the
”normal” severity group, we have low uncertainty. In contrast, if another
n-gram, e.g., ”verifying connection” is associated three times with the
”abnormal” and three times with the ”normal” group, the n-gram uncertainty is
high. To measure the uncertainty, we used Normalized Shanon’s entropy [18]. We
calculated the entropy for each n-gram and reported the key statistics of the
n-grams entropy distribution.

TABLE I: Log Instructions Static Texts Uniqueness Analysis Results

| Min | 1st Qu. | Median | 3rd Qu. | Max
---|---|---|---|---|---
| Average Entropy
---
0.00 | 0.00 | 0.00 | 0.27 | 0.51

TABLE I summarizes the key properties of the n-gram entropy distribution. It
is seen that the median of the distribution is 0. This means that the majority
of the n-grams are associated with only one of the two severity groups.
Thereby, the two severity groups are characterized with a rather unique
vocabulary. While this analysis gives information about the uniqueness of the
vocabularies, it does not account for the type of intent expressed with the
n-grams. To investigate the expressed event intent, we made an n-gram
sentiment analysis (where the sentiment is used to quantify the intent type,
i.e., positive or negative), given in the following.

###  II-C Log Instructions Static Texts Sentiment Analysis

To evaluate the n-gram sentiment concerning the two severity groups, we
considered a pretrained sentiment analysis model from Spacy [16]. We justify
the applicability of the sentiment model by pointing to the observed
similarities between general language and logs (as short texts) [17]. Since
the sentiment model is trained on diverse language texts, it has learned
notions of positive, neutral or negative intent. We run the n-grams through
the model to obtain the sentiment score. We used the sentiment score to
categorize the n-grams into three categories, i.e., positive, negative and
neutral. We relate the events from the ”normal” severity group with positive
intent because they describe a successful state or state transition.
Similarly, we relate the ”abnormal” group with a negative intent because it
describes unsuccessful system state or state transition. The third category
contains n-grams with neutral intent, i.e., events without strongly expressed
intent.

TABLE II: Log Instructions Static Texts Sentiment Analysis Results

Sentiment | Positive | Negative | Neutral
---|---|---|---
Severity Group | Normal | Abnormal | Shared | Normal | Abnormal | Shared | Normal | Abnormal | Shared
N-gram Coverage [%] | 66.94% | 28.13% | 4.93% | 23.13% | 69.75% | 7.12% | 46.98% | 43.43% | 9.59%

TABLE II summarize the results of the n-gram sentiment analysis. For each of
the three sentiment categories, we show the percentages of the n-grams
concerning the two severity groups. In the positive intent category 66.94% of
the n-grams are associated with the normal severity group, and 28.13% are
related to the abnormal severity group. In contrast, from the n-grams
associated with negative intent, 69.75% are associated with the abnormal
group, 23.13% are associated with the normal severity group, and 7.12% are
shared between the two. These two observations show that there exists a
relationship between the normal group and positive intent, and the abnormal
group and the negative intent. Therefore, the proposed severity log level
grouping aligns with human intuition when expressing positive and negative
sentiments. Structuring the static text of the log instructions by their log
levels in the proposed way extracts anomaly-related information. Combining
this observation with the uniqueness in the vocabularies between the two
severity groups demonstrates that SL data has rich anomaly-related properties,
which can serve as a foundation for anomaly detection.

##  III ADLILog: Log-based Anomaly Detection by Log Instructions

Following the affirmative observations about anomaly-related information
encoded in the SL data from the examination study, in this section, we
introduce ADLILog as an unsupervised log-based anomaly detection method.
Figure 1 illustrates the overview of the approach. Logically, it is composed
of (1) log preprocessing, (2) deep learning framework and (3) anomaly
detector. The role of the log preprocessing is to process the raw logs by
carefully selecting preprocessing transformations that expose rich information
for the deep learning framework. The deep learning framework’s goal is to
learn and output useful log representations for the target-system logs. It
does so by training a deep neural network model with a sequential two-phase
learning process (pretraining and finetuning), during which data from the
target-system logs and the SL data are used. The anomaly detector detects if
the input target-system logs are normal or anomalous. In the following, we
describe the three components of ADLILog.

###  III-A Log Preprocessing

The raw target-system logs are characterized by high noise due to the
parameter values generated during system runtime (e.g., IP address, endpoints,
numerical parameters). The log noise can significantly affect the anomaly
detection performance [6]. Therefore, the log preprocessing aims to reduce the
noise by applying a set of preprocessing steps. To that end, we start by
removing all path endpoints (e.g., /home/spelce1/HPCCIBM/bin/) and split the
static text using whitespaces into singleton items we call tokens. The tokens
with numeric values most often denote variable parameters that are not
relevant for the semantics of the logs. We consider them as noise and remove
them. Similar to the preprocessing for the SL data, we apply Spacy and remove
all ASCII special characters (e.g., $\$$), the stopwords (e.g., is and the)
[16] and transformed each character into a lower case, following related work
[15]. Notably, as previously described, the SL data is already preprocessed by
a similar set of operations making the preprocessing uniform. In addition,
each log is prepended with a dedicated Log Message Embedding ([LME]) token.
The [LME] token is an important design detail because we use it to extract a
numerical representation of the log from the neural network, further given as
input to the anomaly detector. An important advantage of our method over the
related work is that ADLILog does not depend on log parsing (a preprocessing
procedure that extracts event templates from raw logs) [19]. Since the
existent log parsers are imperfect, the incorrect parsing adds additional
noise and can degrade the anomaly detection performance [11]. By directly
learning features from the raw logs, we eliminate this source of errors.
Finally, different logs can have a variable number of tokens while the neural
network requires fixed-size input. Therefore, we specify a hyperparameter
$max\\_len$ to unify the lengths. The shorter logs are appended with a special
pad token ([PD]), while the longer ones are truncated at $max\\_len$ size.

Figure 1: ADLILog: Detailed design of the log anomaly detection method

###  III-B Deep Learning Framework

The deep learning framework consists of three components: 1) embedding layer,
2) encoder network from Transformer architecture [20] and 3) classification
layers. Given the preprocessed and tokenized logs at the input, the embedding
layer transforms the input tokens into numerical vector representations, which
we refer to as vector token embeddings. The token embeddings are numerical
features represented in a suitable format for the neural network. We then use
the encoder network to learn relationships between the vector embeddings from
the embedding layer and the appropriate target. The output from the encoder
layer is the vector embedding of the input log/(static text), i.e., the [LME]
vector. Depending on the training phase (pretraining or finetuning), the [LME]
vector proceeds towards one of the two classification layers. The output from
the classification layers is used as input in the appropriate loss function.
After finetuning, the output from the second set of classification layers is
the final vector embedding of the input log, which proceeds towards the
anomaly detector.

####  III-B1 Embedding Layer

The embedding layer receives the preprocessed logs as input. It serves as an
interface between the textual and numerical token representation format.
Specifically, each token is assigned a single index corresponding to a token
embedding vector. The embeddings are learned during pretraining and are
adjusted to learn the properties of the normal and abnormal events. The
embeddings are learned jointly with the parameters of the neural network.
Notably, the embedding layer is updated just during the pretraining phase.
Updating the embedding layer during the two phases is challenged by the
appearance of unseen words during finetuning. For example, there may be some
operations in the target system that are not covered by the SL data, which
leads to the appearance of missing tokens. The joint training with the new
tokens requires an update of the internal structure of the neural network and
learning new parameters every time we encounter new words. Therefore, the
effective transfer of the parameters between the two learning phases is
challenged. To address this issue, we introduce a special token referred to as
an unknown ([UNK]) token. Notably, during pretraining (first training phase),
we randomly sample 15% of the SL data and in each sample, we replace 20% of
the tokens with [UNK] (a similar strategy is used in related works from
general language [21]). Therefore, the pretrained model learns contexts with
missing tokens. During finetuning, whenever we encounter a new token from the
target-system data, we replace it with the [UNK], effectively handling the new
tokens.

####  III-B2 Log Message Encoder

As a suitable architecture for the log message encoder, we identified the
encoder of the Transformer [20] architecture. This architecture provides
state-of-the-art results in many NLP tasks (e.g., sentiment analysis,
translation) [21]. By pointing to the similarities between the log static
texts and natural language [17], we justify our design of choice. The encoder
implements a multi-head self-attention mechanism that exploits the relations
between tokens within the log instructions static texts. This property enables
learning discriminative features between the words and the different contexts
they appear in (e.g., diverse vocabularies, intent). The embedding vectors and
the encoder parameters are updated via the backpropagation algorithm during
pretraining. At the output of the encoder, we provide the vector embedding of
the [LME] token. Due to the architectural design, the vector of the [LME]
token attends over all the other token vectors during training. We considered
this implementation architectural design detail because it allows learning the
most relevant information from the input concerning the normal and abnormal
events. The model size, number of heads in the encoder, and the number of
encoder layers are three hyperparameters of the log message encoder.

####  III-B3 Classification Layers

The classification layers as input receive [LME] tokens from the encoder. It
is composed of two sets of linear neural layers. As depicted in Figure 1, the
first layer set (Set 1) has two linear layers, with parameters
$\theta^{{}^{\prime}}$. It is trained jointly with the log message encoder
during the pretraining procedure. The size of the first linear layer (from the
first set of linear layers) is equal to the model size of the encoder layer,
while the second layer (from the first set of linear layers) has two neurons
that correspond to the ”normal” and ”abnormal” severity groups from the SL
data. The output of the first set of classification layers is given towards
the binary cross-entropy as a loss function during pretraining.

The second set of classification layers, with parameters $\theta^{"}$, has two
linear layers (Set 2 in Figure 1). The two layers have the same number of
neurons equal to the model size. The output of the second set of linear layers
is given as input for the loss function during finetuning. Additionally, the
output of this layer is used as the final log representation, proceeded as
input to the anomaly detector.

####  III-B4 Learning Process

The learning process is split into two sequential phases: pretraining and
finetuning. During the pretraining phase, we update the parameters of the
embedding layer, the log message encoder and the first set of classification
layers. We perform the pretraining with the SL data, using the binary cross-
entropy as a commonly used loss for binary classification [22]. The
pretraining is terminated when a lack of loss improvement is observed for five
consecutive epochs on a separate validation SL set. After pretraining, the
parameters of the encoder and the embedding layer (as pretrained model) are
frozen and no longer updated. Thereby, they preserve anomaly-related
information. The pretrained model is used to extract the initial log
representation in the finetuning phase.

For the finetuning phase, we pair the pretrained model with the second set of
linear layers. Notably, the training data in this phase consists of the
target-system data and the ”abnormal” severity group from the SL data (as its
subset). Since by definition of the anomaly detection task, the majority of
the target-system data is assumed to describe normal system behaviour (i.e.,
class 0) by considering the ”abnormal” class of the SL data as anomalous
(i.e., class 1), the finetuning can be addressed as a binary classification
problem. The ”abnormal” class of the SL data is always available, thereby,
ADLILog does not need manually labeled target-system data, i.e., its
unsupervised method. Consequently, ADLILog addresses the challenge of time-
expensive labeling. In the finetuning phase, we update just the parameters of
the second set of linear layers ($\theta^{"}$), while the pretrained model is
used to extract the log embeddings of the input data. The finetuning enables
learning the specifics of the target-system data while relying on the anomaly-
related information from the SL data. In addition, since the normal target-
system data and the normal events from the SL data can differ, the finetuning
adjusts the log representation embeddings to these differences.

Another important aspect of the finetuning phase is the choice of the
finetuning loss. It determines the form of the final learned log vector
embeddings. Since the finetuning is defined as a binary classification
problem, multiple loss choices are possible (e.g., binary cross-entropy [22],
or hyperspherical loss [23]). The binary-cross entropy is a formidable choice
if the anomalous labels originate from target-system data because it allows
learning of the exact discriminative properties between the classes. However,
in the case of logs, the expensive labeling process makes this assumption
hard. In contrast, hyperspherical loss concentrates the normal class around a
single point, e.g., the centre of the hypersphere. At the same time, it is
scattering the anomalous logs further apart. This is known as the
concentration property [24]. The literature on anomaly detection [24] suggests
that preserving this property when learning representations often improves
performance. Consequently, the hyperspherical loss has more desirable
properties for anomaly detection, and we use it as finetuning loss. Eq. 1
gives its definition for a single log $l_{i}$:

$L_{ad}^{i}=(1-y_{i})||g(\mathbf{x_{i}};\theta,\theta^{"})||^{2}-y_{i}log(1-exp(-||g(\mathbf{x_{i}};\theta,\theta^{"})||^{2}))$

(1)

where $\mathbf{x_{i}}$ is the log representation as output from the second
classification layers set, $y_{i}\in\\{0,1\\}$ is a label for the normal
target-system data or the ”abnormal” SL severity class, $\theta$ and
$\theta^{"}$ are parameters of the encoder and the second set of linear
layers, and $g(\mathbf{x_{i}};\theta,\theta^{"})$ is the function learned by
the network.

###  III-C Anomaly Detector

The goal of the anomaly detector is to highlight the anomalous target-system
logs represented as log vector embeddings ($\mathbf{x_{i}}$). It has two
components, i.e., 1) an assumed target-system normality function
$\tilde{p}_{ad}^{+}$, and 2) anomaly decision rule. The normality function is
an assumed model of the normal target-system logs. It is a positive function,
having small values for the anomalous and large values for the normal target-
system logs [24]. The form of the function depends on the type of finetuning
loss. Since the chosen hyperspherical loss learns a model that places the
normal logs (class 0) close to the centre of the hypersphere, the smaller
distances correspond to normal system behaviour. Following the definition of
the normality function, we use the reciprocal value of the Euclidean distance
between the learned log representation $\mathbf{x_{i}}$ and the hypersphere
centre (set to the origin), given by Eq. 2. The large distances of the vector
representation from the centre of the hypersphere will result in small values
for the normality score (denoting anomalies) and vice versa (as seen in Figure
1).

$\tilde{p}^{+}_{ad}(\mathbf{x_{i}})=\frac{1}{||\mathbf{x_{i}}-\mathbf{c}||^{2}},\quad\quad\mathbf{c}=\mathbf{0}$
(2)

Finally, to detect anomalies, we apply a decision rule on top of the normality
function score values of the input logs. The decision rule involves setting a
decision threshold $\tilde{a}$ over the scores, such that the logs with lower
normality scores are reported as anomalous. We calculate the threshold
$\tilde{a}$ on a separate validation set. The validation set is composed of
the normal target-system data and the ”abnormal” SL class. The threshold is
set to the score that maximizes the chosen performance criteria (e.g., F1
score) on the validation set.

##  IV Experimental Design and Evaluation

In this section, we present the experimental evaluation of ADLILog in
comparison to three state-of-the-art and four traditional log-based anomaly
detection methods. We set the focus on evaluating the detection performance,
as the precise detection of anomalies is a key quality indicator for real-
world deployment. The performance evaluation is made on HDFS and BGL [7] (as
commonly used benchmark datasets), using three performance evaluation metrics.
To estimate the ADLILog’s deployment complexity, we further analyze the
quality and quantity of the training data and two hyperparameters of the
method and the learning procedure. These experiments evaluate the practical
value of ADLILog.

###  IV-A Experimental Design

We evaluate the anomaly detection performance in two separate evaluation
scenarios (1) single log line and (2) sequential log anomaly detection. The
advantage of the single line anomaly detection resides in the potential to
fast diagnose anomalies because the method directly points to the potentially
anomalous log. However, the large volume of logs in short time intervals can
lead to bursts of reported anomalies which in certain situations can be
overwhelming. To that end, we evaluate the method’s performance on event
sequences as well.

####  IV-A1 Datasets

BGL and HDFS are two benchmark datasets for log-based anomaly detection that
are mostly used by the research community [11, 7, 6]. TABLE III shows the key
datasets properties. To find the unique log events, we used Drain [25], a
state-of-the-art log parsing method. Drain’s hyperparameters were set as
recommended by Zhu et al. [19] resulting in an output of 29 and 360 unique
events for HDFS and BGL, respectively. Following He, et al. [6] we split the
dataset into 80-20% train-test split. The first, chronologically ordered 80%
were used for training (and model tuning), while the remaining 20% were used
for performance evaluation for the two datasets accordingly.

TABLE III: Dataset Properties

Dataset | Time Span | # Logs | # Anomalies
---|---|---|---
HDFS | 38.7 hours | 11,175,629 | 16,838
BGL | 7 months | 4,747,963 | 348,460

HDFS contains 11,175,629 logs generated from a map-reduce tasks on more than
200 Amazon’s EC2 nodes [10]. Each log has a unique identifier (block_id) for
each operation such as allocation, writing, replication and deletion. After
parsing, there are 29 unique events, from which ten describe anomalous events
and appear just when the block_id is anomalous. Therefore, they are indicators
of an anomaly. We used this observation to create two datasets, which we refer
to as HDFS-sin and HDFS-seq. HDFS-sin is composed of time-ordered logs with a
label for each event if it is anomalous or not. This data allows the
evaluation of methods performance in absence of external identifiers, i.e.,
single log anomaly detection. HDFS-seq uses the block_id as a natural
identifier to construct sequences of events, and it is used in sequential
anomaly detection evaluation.

BGL contains 4,474,963 logs collected from a BlueGene/L supercomputer at
Livermore Lab [26]. BGL has two important characteristics. The first BGL
characteristic is the availability of labels for individual log events given
by the system administrators. We use these labels as ground truth information
for single log line anomaly detection. We refer to this data as BGL-sin (with
348,460 anomalous logs). The second BGL characteristic is the absence of
identifiers for task sequences. To create log sequences, similar to related
work [6], we use a time window of size $T$. Following, He et al. [6] we set
$T=$ 6h, as optimal for BGL. This resulted in a total of 828 log sequences. To
obtain sequence labels, similar to related work [6], if there is a single
anomalous log in the window, the sequence is labeled as anomalous. We refer to
this data as BGL-seq, and we use it for the sequential evaluation.

####  IV-A2 Competing Methods

We compare ADLILog (for both, the single line and the sequential evaluation)
with three state-of-the-art deep learning-based methods; two supervised
(LogRobust and CNN) methods and one unsupervised (DeepLog) [7]. According to
the log-based anomaly detection survey by Chen et al. [7], these three methods
show the best detection performance on the two benchmark datasets. We used the
public implementations of the methods available open-source in a GitHub
repository 333https://github.com/logpai/deep-loglizer. The three methods were
evaluated with the suggested values for their hyperparameters. Since the three
methods require fixed input, similar to Chen et al. [7], we use
$window\\_size$ of 10 events to create fixed-size sequences and predict the
next log (i.e., the stride is one).

For the sequential evaluation, alongside the three state-of-the-art methods,
we further considered four traditional log-based anomaly detection methods:
two supervised, i.e., Logistic Regression (LR) [27] and Decision Tree (DT)
[9], and two unsupervised methods, i.e., Principle Component Analysis [10] and
LogCluster (LC) [4]. While deep learning-based methods are directly applicable
for single log line [8], we are not aware of related work that directly
applies the four traditional methods on single log lines. Therefore, we do not
use them in this evaluation type. To implement the baselines we used
logilizer444https://github.com/logpai/loglizer, an open-source library for
log-based anomaly detection. We set the hyperparameters of these four methods
as recommended by He et al. [6].

####  IV-A3 Performance Evaluation Metrics

Following related work, we use three evaluation metrics (precision, recall and
F1) to estimate the detection performance of the compared methods [7].
Precision shows the fraction of the correctly reported anomalies
$(Precision=\frac{\\#DetectedAnomalies}{\\#ReportedAnomalies})$. Recall shows
the correctly detected anomalies that are true anomalies
$(Recall=\frac{\\#DetectedAnomalies}{AllAnomalies})$. For anomaly detection in
logs, on one side, it is important not to miss anomalies (missing an anomaly
can lead to severe outages). On the other side, reporting many false positives
overwhelms the operators, leading to alarm fatigue [3], making the method’s
practical usability questionable. Therefore, a natural trade-off between
precision and recall emerges. In this regard, we considered $F_{1}$ (a
harmonic mean between precision and recall $F_{1}=\frac{2\times
Precision\times Recall}{Precision+Recall})$ as the primary evaluation metric.

####  IV-A4 ADLILog Experimental Setup

We have performed the experiments using three different values for the model
size $\\{16,64,256\\}$. Using the model size of 16, we have obtained the best
predictive performance. The $max\\_len$ parameter was set to 32 because this
length covers the majority of the log lengths. To prevent overfitting, we used
the dropout regularization technique with a probability rate of $0.05$. In the
pretraining phase, we used Adam [28] optimizer with a learning rate $10^{-4}$
and values for $\beta_{1}$ and $\beta_{2}$ set to 0.9 and 0.99. Also, we
explored four different values for the batch size $\\{32,64,256,512\\}$. The
batch size of 512 showed the best average results. The finetuning was
performed for five epochs with the same values for the optimizer. The
experiments were conducted on a machine using Ubuntu 18.04, with CPU Intel(R)
i5-9600K, RAM 128 GB, and GPU RTX 2080.

TABLE IV: Single Line Log Anomaly Detection Comparison

| Single Log Line
---
window size: 10
stride: 1
BGL-sin | HDFS-sin
Method | F1 | Prec. | Recall | F1 | Prec. | Recall
ADLILog | 0.61 | 0.55 | 0.70 | 0.98 | 1.00 | 0.96
DeepLog | 0.21 | 0.12 | 0.82 | 0.35 | 0.62 | 0.24
LogRobust | 0.37 | 0.63 | 0.26 | 0.89 | 1.00 | 0.79
CNN | 0.56 | 0.47 | 0.68 | 0.88 | 1.00 | 0.78

###  IV-B Experimental Results and Discussion

The performance evaluation of the proposed approach is made using two
independent experimental scenarios: single line and sequential log anomaly
detection. For the single log line anomaly detection methods comparison, we
compared ADLILog to the three state-of-the-art deep learning-based approaches.
TABLE IV presents the results. ADLILog significantly outperforms the
supervised methods on almost all evaluation metrics. In particular, ADLILog
showed the best predictive performance in terms of recall on both datasets
(BGL-sin, and HDFS-sin). While DeepLog is being slightly better on recall on
BGL-sin, it has significantly worsened performance on precision. On the
$F_{1}$ as a primary evaluation metric, our method outperforms the supervised
methods between 5-24% and the unsupervised one by 40-63%. The improvements of
ADLILog are predominantly due to the rich set of abnormal events from the many
diverse log instructions that help to discriminate the anomalous logs. ADLILog
has a significant practical advantage in comparison to the competing
supervised approaches because does not require labeled target-system
anomalies. Therefore, it can be directly applied to a target system while
obtaining detection performance similar or even better than the supervised
approaches (which will still require expensive manual labeling). Therefore,
the good performance of ADLILog comes at a smaller practical cost.

Further, we noted that the predictive performances of all the methods for the
BGL-sin dataset are significantly lower compared to the HDFS-sin. For example,
the F1 score of LogRobust from 0.89 on HDFS-sin falls to 0.37 on BGL-sin. A
potential explanation for this observation is that the logs from BGL-sin
originate from many different simultaneously running tasks. Therefore, there
is a large difference between the local log neighbourhood (nearby logs) of the
log subject to analysis, a phenomenon in log analysis literature known as
unstable sequences [11]. In BGL-sin there are 26.94% new log events in the
test data compared to the training data, which additionally diversifies the
local neighbourhoods of the events. Although some of the new events are
normal, the methods that exploit the local context miss-detect them as
anomalous (e.g., as seen by the drop in precision for DeepLog on BGL-sin).
DeepLog, as the state-of-the-art unsupervised method, leverages the local
context (window size of 10 events) to detect anomalies and it is significantly
affected by the unstable sequences, resulting in the lowest performance.
LogRobust and CNN leverage supervised information about the events, which
helps to improve the performance. In contrast, the HDFS-sin dataset is
characterized by high regularity in the sequences due to the data generation
procedure. The repetitiveness of task operations (e.g., deletion, allocation),
the smaller number of events and the low context change lead to higher
regularity in the local contexts, which increases the detection performance.
ADLILog is not affected by the local contexts differences because it examines
each log independently.

TABLE V: Sequential Log Anomaly Detection Comparison

| Log Sequences
---
window size=10,
stride=1
| BGL-seq
---
(time window = 6h)
| HDFS-seq
---
(block ids)
Method | F1 | Prec. | Recall | F1 | Prec. | Recall
ADLILog | 0.86 |  0.84 |  0.88 | 0.93 | 0.92 | 0.94
DeepLog | 0.63 | 0.46 | 1.00 | 0.94 | 0.96 | 0.93
LC | 0.57 | 0.42 | 0.87 | 0.80 | 0.87 | 0.74
PCA | 0.55 | 0.50 | 0.61 | 0.79 | 0.98 | 0.67
LogRobust | 0.83 | 0.71 | 1.00 | 0.96 | 0.93 | 0.98
CNN | 0.82 | 0.69 | 1.00 | 0.97 | 0.94 | 0.99
LR | 0.71 | 0.95 | 0.57 | 0.98 | 0.95 | 0.99
DT | 0.72 | 0.95 | 0.57 | 1.00 | 1.00 | 1.00

ADLILog detects anomalies into single log lines. To be able to detect
anomalies in log sequences, ADLILog aggregates the predictions of the
individual logs from the sequence. If there is at least a single detected
anomalous log, the whole sequence is detected as anomalous. For BGL-seq, we
aggregated ADLILog’s logline predictions into the fixed time intervals (same
to BGL-seq data generation procedure), while for HDFS-seq, we aggregate the
predictions based on the block_ids. This way, ADLILog additionally can be
evaluated on sequential log anomaly detection.

TABLE V gives the results of the sequential log anomaly evaluation. We discuss
them for each dataset individually, starting with BGL-seq. For BGL-seq,
ADLILog achieves the best performance among the deep-learning state-of-the-art
and the traditional approaches on the $F_{1}$ evaluation metric. Considering
precision, ADLILog outperforms the three unsupervised (DeepLog, LC and PCA)
and the two deep-learning supervised methods, while being outperformed by DT
and LR. In contrast, when considering recall, ADLILog outperforms all
traditional approaches. Since the logs from the BGL-seq dataset resemble a
real-world behaviour of a supercomputer, the logs originate from many
different independent tasks and are intertwined. This leads to low
repetitiveness of the same log sequences. Therefore, the state-of-the-art
methods (LogRobust, DeepLog and CNN), which directly model the sequences face
the challenge of unstable sequences. Line of works [7, 11] also shows that the
unstable sequences significantly affects the performance of log anomaly
detection methods. In contrast, ADLILog focuses on the discriminative
properties of the individual events, ignoring the sequential features of the
anomalous patterns. These experimental results show that in the case of lower
repetitiveness in the log sequences, leveraging solely the differences in the
language used to describe normal and anomalous events can lead to better
anomaly detection performance.

The results on HDFS-seq show that the methods exploiting sequential properties
achieve better results. The key characteristic of HDFS-seq is the high
sequence regularity which explains the good performance of the sequential
methods. In addition, not all anomalies are logged into a single log. For
example, around 70% of the anomalies in the test dataset can be identified by
shortened log sequences, however, the majority of them have at least one
anomalous logline. Since ADLILog does not model the sequential properties
directly, it does not learn the anomalous sequence properties (e.g., long
sequences caused by delays where no anomalous events are recorded).
Nevertheless, it outperforms the traditional unsupervised baseline methods by
simply relying on the language properties of the logs. The supervised methods
are showing stronger performance in comparison to the unsupervised. However,
in comparison to ADLILog (which does not require labeled data), the supervised
methods require expensive labeling. ADLILog has strong anomaly detection
performance for the two log datasets with weak and strong sequential
dependencies. Notably, a significant practical improvement of ADLILog over the
related methods is that it does not require labels from the target system to
learn a model. Therefore, ADLILog achieves competitive detection performance
at a lower practical cost.

Figure 2: Quantitative and qualitative evaluation of the ”Abnormal” logs

The acceptance of AI-enabled methods in production settings depends strongly
on the necessary effort for creating training data. Therefore, we examine the
quantitative and qualitative properties of the training data ADLILog needs to
learn a good model. Specifically for the quantitative property, we varied the
ratio of normal versus anomalous logs in the training data when finetuning.
This experiment examines the relative ratio between normal target-system and
”abnormal” logs needed for finetuning. For the qualitative property, we varied
the origin of the anomalous class (class 1), i.e., if it comes from human-
provided labels from other software systems (external system labels) or the
”abnormal” class of the SL data (GitHub Labels). The external system labels
are obtained from publicly available datasets (e.g., TBIRD and SPIRIT [26] –
two supercomputer datasets with labeled anomalies). Similarly, as the adoption
of the ”abnormal” class in finetuning, the external-system labels denote
anomalous concepts from related systems (thus eliminating the requirement for
labeling), and can be used for model finetuning (in place of the ”abnormal”
GitHub class). Figure 2 depicts the experimental results for quantitative
evaluation when incrementally changing the ratio between the anomalous and
normal logs in the training data for BGL-sin in the ranges
$\\{1\%,5\%,10\%,20\%\\}$. They show that with a small ratio of 1 %, ADLILog
achieves $>95\%$ of the optimal performance on the $F_{1}$ metric. The
improvement is consistent with further increasing the ratio. Concerning the
qualitative evaluation, by varying the two sets of labels, the results show
that the GitHub labels provide robust and better performance because of the
greater semantic variety in the ”abnormal” events over the external system
labels. This experiment demonstrates that a small number of labels from the
”abnormal” class of the SL data have significant practical usage for log
anomaly detection.

Figure 3: Sensitivity analysis of the influence of batch and model size

Finally, the correct parameter setting influences the needed effort for fast
configuration and the quality of the detection. To evaluate the impact of the
hyperparameters over the detection performance and efficiency, we examined two
hyperparameters, model and batch sizes, influencing the model performance and
update time. We considered the BGL-sin dataset. The experimental results when
varying the model size in the range $\\{16,64,256\\}$ and batch size in the
range $\\{32,64,256,512\\}$, reported in Figure 3, show that the larger batch
size and smaller model size provide better detection performances while being
faster for updating. The prediction time per batch size of 512 is 17 ms
($\sim$30000 logs per second). Together with the small model size, these
experiments imply that ADLILog has desirable practical properties.

##  V Related Work

There exist multiple methods for the automation of log-based anomaly detection
[27, 11, 9, 8]. From the perspective of target-system log labels availability,
the methods are categorized into supervised and unsupervised. The supervised
methods assume the existence of labels from the target system when learning a
model. In one of the earliest applications of these methods, Bodik et al. [27]
applied Logistic Regression (LR) to successfully detect anomalies in data
centres, by treating the problem as a binary classification. Decision Trees
(DT) [9] were utilized in the detection of anomalous web requests from access
logs. These two methods start by log parsing to extract events and then use
count vectors in a fixed time interval as input samples. The recent advances
in deep learning resulted in the appearance of several supervised deep-
learning-based methods, e.g., LogRobust [11], and CNN [12]. LogRobust uses the
LSTM architecture, augmented with attention. These two are popular deep
learning architectures frequently combined for sequence modeling. LogRobust,
as input, receives a sequence of events, and as output, it predicts if the
observed sequence is anomalous or not. By careful sequence construction, i.e.,
by incremental sliding over the log sequences by one element, it can be used
to predict single log lines [8]. An additional feature of LogRobust is using
vector embeddings from general-purpose languages to represent the logs. Lu et
al. [12] use Convolutions Neural Networks (CNN), another type of deep learning
architecture, to learn normal and abnormal sequences from template indices.
Similar to LogRobust, CNN can be used to detect anomalies from a single log.
While having strong detection performance, the large frequency of the software
updates and the large volume of the produced logs make the labeling process
expensive. Therefore, supervised methods are frequently considered impractical
[6].

In contrast, the unsupervised methods do not assume the existence of labeled
data. This has an important practical implication because it eliminates the
need for expensive labeling. Therefore, the unsupervised anomaly detection
methods are easier to adopt. In one of the earliest works on unsupervised
anomaly detection, Xu et al. apply PCA [10] to learn the normal state of the
event counts by projecting them as points in a vector space. In the test
phase, the test sample is projected in the constructed vector space and
reported as an anomaly if the projection significantly deviates from the
learned normal state. Lin et al. [4] introduce LogCluster which uses the TFIDF
algorithm for sequence representation. It first constructs a knowledge base of
normal/anomalous sequence clusters by agglomerative clustering and human-based
cluster labeling (normal or anomalous). A test sample is detected as an
anomaly if it is clustered into anomalous clusters. DeepLog [8] and LogAnomaly
[29] are two popular unsupervised deep learning-based methods. The innovative
feature of the two is the introduction of an auxiliary task called ”next event
prediction” (NEP). NEP is a supervised task that given a sequence of events,
forecasts the most probable next event. Notably, the labels originate from the
input data itself, i.e., no labeling is performed, which makes the methods
unsupervised. The test sequences with an incorrect prediction for the next
event are considered anomalous. As stated by the authors, DeepLog can be
applied for sequential and single logs given as inputs. To learn the normal
state, an LSTM architecture is trained on the NEP task. LogAnomaly has two
additional features: 1) it uses log semantics and 2) event counts in joint
training a DeepLog-like model. The empirical results show that the
improvements over DeepLog are not significant [7, 29]. Unsupervised methods
are often criticized for their lower performance in comparison to the
supervised ones [6] leading to alarm fatigue and discouraging their wide
applicability.

In addition, there are other methods for log-based anomaly detection in both
industry and research [30, 14, 13]. However, those solutions are part of
production systems [13] or have specific implementation challenges while do
not provide public implementations [14, 31, 30]. Due to the inability of
transparent comparison, we do not discuss them in detail.

##  VI Conclusion

This paper addresses the problem of automating log-based anomaly detection as
a crucial maintenance task in enhancing the reliability of IT systems. It
introduces a novel unsupervised method for log anomaly detection, named
ADLILog. The key idea of ADLILog is to use the large unstructured information
from the logging instructions of 1000+ GitHub public code projects to improve
the target-system log representations, which directly improves anomaly
detection. We first conducted a study to examine the language properties of
the log instructions, and we show that they encode rich anomaly-related
information. ADLILog combines the anomaly-related information and the target-
system data to learn a deep neural network model by a sequential two-phase
learning procedure. The extensive experimental results on the two most
commonly used benchmark datasets show that ADLILog outperforms the related
methods: the supervised by 5-24%, and the unsupervised by 40-63% on the F1
score. Further experiments demonstrate that ADLILog has desirable practical
properties concerning the time-efficient model updates and small model sizes.
This study signifies the benefit of using large unstructured information in
aiding the automation of IT operations. Regarding future work, the paper opens
additional questions on how to apply the SL data for automating higher-order
IT operational tasks, like failure identification and root-cause analysis.

## References

  * [1] P. Notaro, J. Cardoso, and M. Gerndt, “A survey of aiops methods for failure management,” _ACM Trans. Intell. Syst. Technol._ , vol. 12, 2021.
  * [2] W. Meng, Y. Liu, S. Zhang, F. Zaiter, Y. Zhang, Y. Huang, Z. Yu, Y. Zhang, L. Song, M. Zhang, and D. Pei, “Logclass: Anomalous log identification and classification with partial labels,” _IEEE Trans. Netw._ , vol. 18, pp. 1870–1884, 2021.
  * [3] H. Li, W. Shang, B. Adams, M. Sayagh, and A. E. Hassan, “A qualitative study of the benefits and costs of logging from developers’ perspectives,” _IEEE Transactions on Software Engineering_ , pp. 1–17, 2020\.
  * [4] Q. Lin, H. Zhang, J.-G. Lou, Y. Zhang, and X. Chen, “Log clustering based problem identification for online service systems,” in _Proceedings of the 38th ICSE_. New York, NY, USA: Association for Computing Machinery, 2016, p. 102–111.
  * [5] S. He, P. He, Z. Chen, T. Yang, Y. Su, and M. R. Lyu, “A survey on automated log analysis for reliability engineering,” _ACM Comput. Surv._ , vol. 54, 2021.
  * [6] S. He, J. Zhu, P. He, and M. R. Lyu, “Experience report: System log analysis for anomaly detection,” in _2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)_. New York, USA: IEEE, 2016, pp. 207–218.
  * [7] Z. Chen, J. Liu, W. Gu, Y. Su, and M. R. Lyu, “Experience report: Deep learning-based system log analysis for anomaly detection,” _CoRR_ , vol. 2107.05908, 2021.
  * [8] M. Du, F. Li, G. Zheng, and V. Srikumar, “Deeplog: Anomaly detection and diagnosis from system logs through deep learning,” in _Proceedings of the 2017 ACM SIGSAC_. New York, NY, USA: Association for Computing Machinery, 2017, p. 1285–1298.
  * [9] M. Chen, A. Zheng, J. Lloyd, M. Jordan, and E. Brewer, “Failure diagnosis using decision trees,” in _International Conference on Autonomic Computing, 2004. Proceedings._ , 2004, pp. 36–43.
  * [10] W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan, “Detecting large-scale system problems by mining console logs,” in _Proceedings of the ACM 22nd SOSP_. New York, NY, USA: Association for Computing Machinery, 2009, p. 117–132.
  * [11] X. Zhang and et. al., “Robust log-based anomaly detection on unstable log data,” in _Proceedings of the 2019 27th ACM Joint Meeting on ESEC/FSE_. New York, NY, USA: Association for Computing Machinery, 2019, p. 807–817.
  * [12] S. Lu, X. Wei, Y. Li, and L. Wang, “Detecting anomaly in big data system logs using convolutional neural network,” in _IEEE 16th Conf on Dependable, Autonomic and Secure Computing_ , 2018, pp. 151–158.
  * [13] T. Li, Y. Jiang, C. Zeng, B. Xia, Z. Liu, W. Zhou, X. Zhu, W. Wang, L. Zhang, J. Wu, L. Xue, and D. Bao, “Flap: An end-to-end event log analysis platform for system management,” in _Proceedings of the 23rd ACM SIGKDD International Conference on KDD_. New York, NY, USA: Association for Computing Machinery, 2017, p. 1547–1556.
  * [14] X. Li, P. Chen, L. Jing, Z. He, and G. Yu, “Swisslog: Robust and unified deep learning based log anomaly detection for diverse faults,” in _2020 IEEE 31st ISSRE_ , 2020, pp. 92–103.
  * [15] T.-H. Chen, S. W. Thomas, and A. E. Hassan, “A survey on the use of topic models when mining software repositories,” _Empirical Softw. Engg._ , vol. 21, p. 1843–1919, 2016.
  * [16] M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd. (2020) spaCy: Industrial-strength Natural Language Processing in Python. Explosion.ai. [Online]. Available: https://doi.org/10.5281/zenodo.1212303
  * [17] P. He, Z. Chen, S. He, and M. R. Lyu, “Characterizing the natural language descriptions in software logging statements,” in _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. New York, NY, USA: Association for Computing Machinery, 2018, p. 178–189.
  * [18] T. M. Cover and J. A. Thomas, _Elements of Information Theory_. USA: Wiley-Interscience, 2006.
  * [19] J. Zhu, S. He, J. Liu, P. He, Q. Xie, Z. Zheng, and M. R. Lyu, “Tools and benchmarks for automated log parsing,” in _Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice_. NY, USA: IEEE Press, 2019, p. 121–130.
  * [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_. Red Hook, NY, USA: Curran Associates, 2017, p. 6000–6010.
  * [21] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_. Minneapolis, Minnesota: Association for Computational Linguistics, 2019, pp. 4171–4186.
  * [22] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. MIT Press, 2016. [Online]. Available: http://www.deeplearningbook.org
  * [23] W. Liu, Y.-M. Zhang, X. Li, Z. Yu, B. Dai, T. Zhao, and L. Song, “Deep hyperspherical learning,” in _Proceedings of the 31st International Conference on NeurIPS_. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 3953–3963.
  * [24] L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller, “A unifying review of deep and shallow anomaly detection,” _Proceedings of the IEEE_ , vol. 109, no. 5, pp. 756–795, 2021.
  * [25] P. He, J. Zhu, Z. Zheng, and M. R. Lyu, “Drain: An online log parsing approach with fixed depth tree,” in _2017 IEEE International Conference on Web Services_. NY, USA: Curran Associates, 2017, pp. 33–40.
  * [26] A. Oliner and J. Stearley, “What supercomputers say: A study of five system logs,” in _37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks_. Los Alamitos, CA, USA: IEEE Computer Society, 2007, pp. 575–584.
  * [27] P. Bodik, M. Goldszmidt, A. Fox, D. B. Woodard, and H. Andersen, “Fingerprinting the datacenter: Automated classification of performance crises,” in _Proceedings of the 5th European Conference on Computer Systems_. New York, NY, USA: ACM, 2010, p. 111–124.
  * [28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations_ , 2015.
  * [29] W. Meng and et al., “Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs,” in _Proceedings of the 28, IJCAI-19_. IJCAI, 2019, pp. 4739–4745.
  * [30] S. Nedelkoski, J. Bogatinovski, A. Acker, J. Cardoso, and O. Kao, “Self-attentive classification-based anomaly detection in unstructured logs,” _CoRR_ , 2020. [Online]. Available: https://arxiv.org/abs/2008.09340
  * [31] Y. Zhu, W. Meng, Y. Liu, S. Zhang, T. Han, S. Tao, and D. Pei, “Unilog: Deploy one model and specialize it for all log analysis tasks,” 2021.


# Near-IR and Visual High Resolution
Polarimetric Imaging with AO Systems

Hans M. Schmid  Institute for Particle Physics and Astrophysics, ETH Zurich,
Wolfgang-Pauli-Str. 27, 8093 Zurich, Switzerland

###### Abstract

Many spectacular polarimetric images have been obtained in recent years with
adaptive optics (AO) instruments at large telescopes because they profit
significantly from the high spatial resolution. This paper summarizes some
basic principles for AO polarimetry, discusses challenges and limitations of
these systems, and describes results which illustrate the performance of AO
polarimeters for the investigation of circumstellar disks, of dusty winds from
evolved stars, and for the search of reflecting extra-solar planets.

###### keywords:

polarimetry, AO systems, high resolution, high contrast, scattering,
circumstellar disks, stellar winds, reflecting planets

††volume: 360††journal: Astronomical Polarimetry 2020: New Era of
Multiwavelength
Polarimetry††editors: H.Shinnaga, B-G Andersson, A.M.Magalhães & E.Falgarone,
eds.

##  1 Science drivers for AO polarimetry

Pushing observations to higher spatial resolution is a key for progress for
the investigation of the geometry and the physics of extended objects in
astronomy. The diffraction limited angular resolution of a telescope is given
by the ratio between wavelength and the telescope diameter
${\cal{R}}\approx\lambda/D$. High resolution ${\cal{R}}\;\hbox to0.0pt{\lower
2.5pt\hbox{$\sim$}\hss}\raise 1.5pt\hbox{$<$}\;0.1^{\prime\prime}$ imaging
polarimetry in the visual to near-IR range requires therefore large telescopes
on the ground equipped with adaptive optics (AO) systems for the correction of
the atmospheric seeing or observations at $\lambda<1~{}\mu$m with the Hubble
Space Telescope (HST). This review focuses on the recent, significant
technical progress achieved with imaging polarimeters using AO systems at
large ground based telescopes.

The goal of AO systems is the correction of the wavefront deformations
introduced by the atmospheric turbulence with a control loop (Fig. 1), which
measures the deviations with a wavefront sensor (WFS), calculates the
corrections with a real time computer (RTC), and applies them to a deformable
mirror (DM). This correction must be faster than the turbulence time scale of
a few milli-seconds and this requires a bright photon source close to the
science target. The central stars of a circumstellar disk or shell, or a
planetary system are ideal wave front probes for good AO observations.

Figure 1: Light path and schematic block diagram for a telescope with AO
system, coronagraph, and polarimetric imager. Possible locations of a half
wave plate (HWP) switch and a derotator are indicated (DM: deformable mirror,
WFS: wave front sensor, RTC: real time computer).

The AO system converts the seeing disk of a star into a point spread function
(PSF) with a diffraction limited core with a width of $\approx{\cal{R}}$ and a
residual halo with about the size of the initial seeing disk (e.g. [Davies &
Kasper 2012]). For an 8 m telescope ${\cal{R}}$ is $\approx 17$ mas in the
R-band ($\lambda=0.65~{}\mu$m) and $\approx 42$ mas in the H-band
($1.6~{}\mu$m).

The performance of an AO system is described by the Strehl ratio $S$, which is
roughly the fraction of light concentrated in the PSF core. $S$ depends for a
given system strongly on $\lambda$, for example VLT-SPHERE reaches for a
bright guide star under excellent atmospheric conditions up to $S\approx 0.9$
in the H-band but only $S\approx 0.6$ for the R-band ([Fusco et al. 2016]) and
therefore the PSF halo is stronger at shorter wavelengths. The AO performance
depends also on the seeing and already average conditions can reduce $S$ by
30–50 %. The Strehl ratio is also reduced for faint guide stars (e.g.
$>12^{\rm m}$) because of the lack of photons for accurate wave front
measurements ([Milli et al. 2017]).

The PSF core is typically so bright, that sensitive observations of the
stellar surroundings would heavily saturate the detector. Many observations
require therefore a stellar coronagraph (Fig. 1) to block the light of the PSF
core for $r\;\hbox to0.0pt{\lower 2.5pt\hbox{$\sim$}\hss}\raise
1.5pt\hbox{$<$}\;0.1^{\prime\prime}$. For extreme AO systems, like Gemini-GPI
and VLT-SPHERE, the coronagraphs suppress also the diffraction pattern of the
PSF at separations $r\approx 0.1^{\prime\prime}$ to $0.4^{\prime\prime}$
([Sivaramakrishnan et al. 2001]).

The residual PSF halo is variable and typically stronger than the signal of a
faint companion, a disk, or the dusty stellar wind. Differential techniques
are required to disentangle the light of a target from the halo of the central
star (e.g. [Racine et al. 1999]) and polarimetry is an ideal differential
technique to distinguish between the polarized scattered light of a faint
circumstellar target and the direct (and usually unpolarized) light of the
bright central star.

The main science drivers for AO polarimetry are:

  * •

Bright stars with circumstellar scattering are ideal for a very good AO
performance and often very interesting targets for polarimetric
investigations.

  * •

Separating and resolving circumstellar scattering regions from the star
enhances significantly the measurable polarization signal.

  * •

Polarimetry is an ideal differential technique for high contrast observations
of polarized sources.

  * •

Often it is much easier to detect and image a target with differential
polarimetry, while measurements of the intensity requires a follow-up in a
second step.

##  2 Technical challenges for polarimetry with AO systems

###  2.1 Historical developments

The first AO systems with polarimetric modes were the Cassegrain instrument
Subaru-CIAO ([Murakawa et al. 2004]) and the Nasmyth instrument VLT-NACO
([Lenzen et al. 2003]), and they delivered very promising polarimetric result
for disks and circumstellar shells ([Apai et al. 2004, Murakawa et al. 2005]).
They also showed that simultaneous measurements of opposite polarization modes
$I_{\perp}$ and $I_{\parallel}$, and the polarimetric self-calibration with a
half wave plate (HWP) switch are essential for good results, and that the
calibration of systematic polarization effects for the Nasmyth instrument NACO
is difficult ([Witzel et al. 2011]). This triggered efforts for better
correction procedures for the polarization of the Nasmyth instruments VLT-NACO
and Subaru-HiCIAO and disk images with much higher quality were achieved (e.g.
[Quanz et al. 2011, Hashimoto et al. 2011, Muto et al. 2012]). In addition,
the lessons learned from the “early” AO polarimeters were considered for the
designs of the subsequent extreme AO instruments Gemini-GPI ([Macintosh et al.
2014]) and VLT-SPHERE ([Beuzit et al. 2019]) which provide currently the best
performance in AO polarimetry.

Table 1 lists key system parameters and corresponding references of widely
used systems, which produced a large fraction of the recent scientific result
in AO polarimetry. The table illustrates the progress in AO performance from
the older Subaru-HiCIAO system (the parameters of VLT-NACO are similar), to
the newer extreme AO systems Gemini-GPI and VLT-SPHERE. Also of interest are
the very different polarimetric designs of the instruments, which are
described below.

Table 1: Rough parameters for widely used AO polarimeters, which are available
or were available until recently at 8m class telescopes.

|  Subaru | Gemini-South |  VLT-SPHERE1
---|---|---|---
|  HiCIAO2 | GPI3 | IRDIS4 | ZIMPOL5
AO system |  AO1886 | (GPI) |  SAXO7
in operation | 2009-2015 |  2013-2020a | since 2015
DM actuators | 188 | $43\times 43$ |  $41\times 41$
Strehl ratiob |  $\approx 0.4$, H-band6 |  $\approx 0.9$, H-band3 |  $\approx 0.9$, H-band7,8 |  $\approx 0.6$, R-band7,8
guide star limitc | $I<11^{\rm mag}$ | $I<10^{\rm mag}$ | $R<14^{\rm mag}$ | $R<11^{\rm mag}$
coronagraphy | X | X | X | X
polarimetric imager |  |  |  |
polarimetry type | double beam |  integral field pol.9 |  double beam4 |  fast modulation5
wavelength range |  $0.85-2.5~{}\mu$m |  $0.9-2.3~{}\mu$m |  $1.0-2.3~{}\mu$m |  $0.52-0.9~{}\mu$m
spatial resolution |  $40-60$ mas |  $30-60$ mas |  $30-60$ mas |  $20-25$ mas
field of view | $10^{\prime\prime}\times 20^{\prime\prime}$ | $2.7^{\prime\prime}\times 2.7^{\prime\prime}$ | $11^{\prime\prime}\times 11^{\prime\prime}$ | $3.6^{\prime\prime}\times 3.6^{\prime\prime}$
telescope and instrument polarization |  |  |
focal station | Nasmyth | Cassegrain | Nasmyth
HWP switch | after AO | after AO | before derotator and AO
telescope polarization | moderate |  small10 |  moderate11 |  large/compensated5,12
derotator cross talks | large | none |  large12 |  large/controlled12

Notes: aat Gemini-South, will be upgraded and mounted at Gemini-North in
future, bfor a bright guide star and excellent atmospheric conditions, cStrehl
ratio degraded by a factor of $\approx 2$ because of guide star “faintness”
References: 1[Beuzit et al. 2019], 2[Hodapp et al. 2008], 3[Macintosh et al.
2014], 4[Dohlen et al. 2008], 5[Schmid et al. 2018], 6[Suzuki et al. 2010],
7[Fusco et al. 2016], 8[Milli et al. 2017], 9[Perrin et al. 2015],
10[Wiktorowicz et al. 2014], 11[de Boer et al. 2020], 12[Bazzon et al. 2012].

###  2.2 Polarization imagers

Double beam polarimeters for linear polarization measurements are based on a
polarization beam splitter, often a Wollaston prism, and a rotating HWP. This
provides two images side by side on the detector with “opposite” polarization
intensities $I_{\perp}$ and $I_{\parallel}$ from which one can determine the
differential signal $I_{\perp}$–$I_{\parallel}$. The two images are taken
simultaneously and this is essential for the suppression of the strongly
variable speckle noise present in AO observations. The HWP is used for the
selection of the Stokes $Q$=$I_{0}$–$I_{90}$ and $U$=$I_{45}$–$I_{135}$
parameters and as switch for the compensation of the instrument polarization
$q_{\rm inst}I$ and $u_{\rm inst}I$. A rotation of the HWP by $45^{\circ}$
switches the sign of the incoming $Q_{\rm in}$-signal (or $U_{\rm in}$) for
two consecutive measurements $Q^{+}$ and $Q^{-}$, while the instrument
polarization remains unchanged. The difference

$Q^{+}-Q^{-}=(Q_{\rm in}+q_{\rm inst}I)-(-Q_{\rm in}+q_{\rm inst}I)=2\,Q_{\rm
in}$

compensates then the instrumental polarization. The use of a HWP switch is
essential for all AO polarimeters. The basic double beam set-up was used by
the “early” AO polarimeters Subaru-CIAO and VLT-NACO, but also for Subaru-
HiCIAO. For SPHERE-IRDIS the principle is the same, but it uses a “grey” beam
splitter plate and polarizers to avoid the differential aberrations of a
Wollaston prism between the two beams ([Dohlen et al. 2008]). Double beam
polarimeters provide a high efficiency and typically a large field of view.

Integral field polarimeter: This is an innovative concept used as polarimetric
mode in combination with the integral field spectrograph (IFS) of Gemini-GPI
which proved to work very successfully ([Perrin et al. 2015]). The IFS uses a
2D-array of micro-lenses in the focal plane, producing spots which are then
dispersed with a grism producing an array of about 36 000 low resolution
spectra on the detector. In polarimetric mode, the grism is replaced by a
Wollaston prism producing an array of double spots with “opposite”
polarizations $I_{\perp}$ and $I_{\parallel}$ instead of the low resolution
spectra. The field of view is given by the focal plane sampling of the micro-
lens array which is smaller (for a given detector size) than for the
“standard” double beam polarimeters.

Fast modulation polarimeter: SPHERE-ZIMPOL is based on a fast polarization
modulation concept using a liquid crystal device and a polarizer, which
converts the polarization modulation into an intensity modulation between
$I_{\perp}$ and $I_{\parallel}$. A CCD-detector is used for the signal
demodulation with the main advantage that $I_{\perp}$ and $I_{\parallel}$ are
registered with the same detector pixels so that flat-fielding effects are
minimized. The modulation is faster than the speckle variations and
$I_{\perp}$ and $I_{\parallel}$ are registered essentially simultaneously
([Schmid et al. 2018]). The system is optimized in various ways for
polarimetry, achieves a high dynamic range and a very high sensitivity of
$10^{-5}$ for the search of reflecting planets around bright nearby stars in
the visual range. Less good, when compared to double beam imagers, are the
parameters for the polarimetric efficiency (70–90 %), the small field of view,
and the bright guide star limit for the AO system, because ZIMPOL shares the
light with the visual WFS.

###  2.3 Telescope and instrument polarization effects

“Smart” polarimeters have a simple design which minimizes telescope and
instrument polarization effects with a Cassegrain instrument and a straight
through beam. Subaru-CIAO ([Murakawa et al. 2004]) and Gemini-GPI ([Perrin et
al. 2015]) followed as far as possible this principle and therefore the
polarimetric calibration of the data is relatively simple (e.g. [Wiktorowicz
et al. 2014]).

All other systems in Table 1 are “not smart”, but complex, large, and located
at the Nasmyth focus because this is favorable for many AO aspects. Therefore,
the telescope and instrument polarization effects are complicated and require
a good concept for the measurements of the polarization without compromising
the performance of the AO system and the coronagraph. A few typical problems
are:

  * •

the polarization effects of the inclined telescope mirror M3 for Nasmyth
systems,

  * •

“rotating” instrument polarization effects introduced by an image derotator,

  * •

static instrument polarization effects from inclined optical components.

There are many ways how these problems can be solved and each system must be
considered individually. The HWP switch discussed for the polarization imagers
can be placed early in the beam (Fig. 1) to compensate for the instrument
polarization of many components. However, this approach does not correct for
polarization cross-talks, in particular the substantial conversion from linear
to circular polarization by the inclined mirrors of an image derotator.
Therefore, a derotator can strongly reduce the measurable linear polarization
depending on its orientation ([de Boer et al. 2020]), which must be corrected
with detailed hardware calibrations. Also the Nasmyth telescope mirror M3
requires polarimetric corrections (e.g. [Joos et al. 2008]), which must be
derived or at least verified with standard star data. Extremely helpful for
the polarimetric calibration of many data sets are objects with an unpolarized
central star which serves as zero polarization reference source.

For SPHERE-IRDIS there are moderate telescope and strong derotator
polarization effects which can be corrected with a data reduction package
based on a detailed polarization model of the whole instrument ([van Holstein
et al. 2020]). SPHERE-ZIMPOL uses three additional HWPs and one polarization
compensator plate, which actively correct for the strong telescope
polarization and derotator effects ([Bazzon et al. 2012]).

Figure 2: Images for (a) $Q$, (b) $U$, (c) $Q_{\phi}$, and (d) $p\times I$ for
the pole-on disk HD 169142 with a bright inner disk wall at
$r=0.17^{\prime\prime}$ and an outer low surface brightness disk. Panel (e)
gives azimuthally averaged radial profiles for $Q_{\phi}$, $p\times I$, the
stellar PSF $I_{\rm star}$, and a model for the intrinsic $Q_{\phi}^{\rm mod}$
([Tschudi & Schmid 2021]), which fits after convolution the observed
$Q_{\phi}$ curve.

###  2.4 Quantitative polarimetric measurements

Often only differential polarization images $Q_{\rm cs}(\alpha,\delta)$ and
$U_{\rm cs}(\alpha,\delta)$ of a circumstellar scattering source are available
because the intensity $I_{\rm cs}$ cannot be disentangled from the bright and
highly variable PSF of the central star. The quantitative analysis of $Q_{\rm
cs}$ and $U_{\rm cs}$ must take noise bias effects and the polarimetric
cancellation into account.

If the $Q_{\rm cs}$ and $U_{\rm cs}$ images are noisy then the determination
of the polarized flux according to $p_{\rm cs}\times I_{\rm cs}=(Q_{\rm
cs}^{2}+U_{\rm cs}^{2})^{1/2}$ is strongly biased upward ([Simmons & Stewart
1985]) as can be seen in Fig. 2 for the outer disk. This problem can be
avoided with the use of the azimuthal polarization parameters
$Q_{\phi},U_{\phi}$. For circumstellar scattering the polarization is
essentially in azimuthal direction described by $Q_{\phi}$ and the
$U_{\phi}$-signal is very small. Therefore, the $Q_{\phi}$-signal is a very
good approximation of the polarized flux $Q_{\phi}\approx p_{\rm cs}\times
I_{\rm cs}$ and this parameter avoids the noise bias ([Schmid et al. 2006a]).

The measurable polarization $Q_{\rm cs},U_{\rm cs}$ is also affected by the
limited observational resolution, which introduces smearing and for
polarimetry also a cancellation between image regions with $+Q$ and $-Q$ or
$+U$ and $-U$ signals ([Schmid et al. 2006a]). This effect can be very
substantial for AO polarimetry of compact sources as shown in Fig. 2(e) for
the inner ring of HD 169142. There, the measurable signal $Q_{\phi}$ is about
4 times smaller than the model for the intrinsic signal $Q_{\phi}^{\rm mod}$
before convolution with the PSF. For the more extended disk the cancellation
effect is much reduced. This effect depends strongly on the atmospheric
conditions, but can be modeled and corrected well, if the PSF is known
accurately ([Tschudi & Schmid 2021]).

##  3 Scientific results

###  3.1 Protoplanetary disks

Protoplanetary disk are the “easy targets” for polarimetric imaging with AO
systems especially the so-called transition disks. They have a large central
cavity and the strongly illuminated inner disk wall can be separated by
$d>10~{}$AU from the star or $>0.1^{\prime\prime}$ for a disk in a nearby
($d\approx 100$ pc) star forming region. The disk integrated polarized flux is
up to $Q_{\phi}/I_{\rm star}\approx 1~{}\%$.

The images show the scattering polarization of the dust in the surface layer
of optically thick disks and they reveal a surprising diversity of
hydrodynamical structures: central disk cavities ([Quanz et al. 2011,
Hashimoto et al. 2012]), large scale spirals ([Garufi et al. 2013, Benisty et
al. 2015], Fig. 3), circular gaps ([Quanz et al. 2013], Fig. 2), shadows from
tilted disks near the star ([Marino et al. 2015, Pinilla et al. 2018]), and
much more. Some structures are explained by the presence of newly formed
planets and two young planets were indeed found in the large disk cavity of
PDS 70 ([Keppler et al. 2018]). These images and corresponding ALMA maps are
of much interest for the understanding of the planet formation processes in
disks.

Well calibrated disk data provide the spectral dependence of the polarized
reflectivity $Q_{\rm disk}(\lambda)/I_{\rm star}(\lambda)$ and the few
existing measurements indicate a “reddish” color (e.g. [Hunziker et al.
2021]). The fractional polarization $p_{\rm disk}$ of the reflected light was
first measured with HST for AB Aur ([Perrin et al. 2009]) but this can now
also be achieved with AO systems. Reported values for a scattering angle of
$\approx 90^{\circ}$ range from $p_{\rm disk}\approx 20-30~{}\%$ for visual
wavelengths, to higher values $p_{\rm disk}\approx 30-60~{}\%$ in the near-IR
([Monnier et al. 2019, Hunziker et al. 2021, Tschudi & Schmid 2021]). More
such quantitative measurements will become available and they will help to
constrain the properties of the scattering dust.

###  3.2 Debris disks

Dust debris disk are signposts for the presence of colliding planetesimals
around a star. Typically, we see the scattered light from dust rings at
separations of $\approx 10-100$ AU, sometimes surrounded by scattering halos
from small grains blown out by radiation pressure. Debris disk are optically
thin, with a low polarization contrast $Q_{\phi}/I_{\star}\;\hbox
to0.0pt{\lower 2.5pt\hbox{$\sim$}\hss}\raise 1.5pt\hbox{$<$}\;0.05~{}\%$
([Engler et al. 2017]) requiring deep observations with extreme AO systems
like GPI or SPHERE. Edge-on systems are easier to detect because the large
grains are strongly forward scattering ([Esposito et al. 2020]).

Debris disks with high inclination, like the bright system HR 4796A (Fig. 3),
are ideal for the determination of the dust scattering phase function for the
intensity and the polarization because the photons undergo only one scattering
and the scattering angles are well defined ([Perrin et al. 2015, Milli et al.
2019, Chen et al. 2020, Arriaga et al. 2020]). Such data provide important
information for model calculation of the light scattering by complex dust
particles.

Figure 3: $Q_{\phi}$-images of (a) the protoplanetary spiral disk HD 135334B
and (b) the bright debris disk HR 4796A. The data are described in Stolker et
al. (2016) and Milli et al. (2019).

###  3.3 Dust formation in the stellar wind of red giants

Important processes for the mass loss from red giants are the formation of
dust and the wind acceleration by the radiation pressure acting on dust
grains. The light scattered by the dust can be directly observed with non-
coronagraphic AO polarimetry. The key technical requirement is a high spatial
resolution because the stellar diameters are only about 50 mas for the most
extended red giants (e.g. $\alpha$ Ori, Mira, W Hya, R Dor, $\alpha$ Sco).
This favors strongly observations at small $\lambda$ with SPHERE-ZIMPOL (Tab.
1), which achieves a spatial resolution of (${\cal{R}}\approx 20$ mas).
Interesting monitoring results are also obtained with an “AO free” system
based on differential speckle polarimetry ([Safonov et al. 2020]) and very
promising are the prospects for polarimetry using sparse aperture masking
interferometry, which will achieve a spatial resolution of up to 10 mas with
the new Subaru-SCeXAO-Vampires instrument ([Norris et al. 2015]).

The polarimetric observations show clumpy and variable dust structures, and
the derived radial distribution defines the locations for dust formation and
wind acceleration. From the color of the scattered light one can estimate the
dust particle sizes (e.g. [Ohnaka et al. 2016, Khouri et al. 2020]).

###  3.4 Reflecting extrasolar planets

Light reflecting from planets is polarized and this offers the possibility for
detecting extra-solar planets with high resolution polarimetry. The SPHERE-
ZIMPOL instrument was optimized for this science case ([Schmid et al. 2006b])
and a search for extra-solar planets around the nearest bright stars $\alpha$
Cen A and B, Sirius, $\epsilon$ Eri, Altair and $\tau$ Cet has been carried
out by Hunziker et al. (2020). The requirements are extreme, for example a
Jupiter-sized planet at a separation of 1 AU would produce only a polarimetric
contrast of about $C_{\rm pol}=p_{\rm planet}\times I_{\rm planet}/I_{\rm
star}\approx 2\cdot 10^{-8}$ (e.g. [Buenzli & Schmid 2009]). Such detection
limit could be reached with an exposure time of $t_{\rm exp}=3.4^{h}$ for
$\alpha$ Cen A but no successful detection has been achieved. However, it is
shown that the ZIMPOL polarimetry is for $r>0.5^{\prime\prime}$ photon noise
limited and the contrast limit should just improve with $t_{\rm exp}$ like
$C_{\rm limit}\propto(t_{\rm exp})^{-1/2}$. Deeper searches for promising
targets are therefore ongoing for $\epsilon$ Eri b or the recently reported
planet candidate around $\alpha$ Cen A ([Wagner et al. 2021]).

##  4 Conclusions

High resolution polarimetric imaging with AO systems made in the last decade a
major step forward from a special technique used by a few experts to a main
stream observing mode offered by leading observatories to the astronomical
community. Many astronomers use now this technique and they produced a
continuous string of first class science results on circumstellar disks, dusty
shells around evolved stars, and several other science topics not covered in
this review.

AO polarimetry is not a simple observing technique, because of instrument
polarization effects and the variable AO performance which depends on the
atmospheric turbulence. In addition, many of the most interesting targets are
very faint and very close to the bright central star. Despite these extreme
challenges the results obtained with different systems agree very well, giving
much credibility to this technique and proving its maturity.

This success will lead to more progress in the near and more distant future,
like deeper observations for fainter targets, improved observing and
calibration procedures for quantitative measurements, but also upgrades on
existing instruments and hopefully the development of much more powerful AO
polarimeters for the coming generation of extremely large telescopes.

## References

  * [Apai et al. 2004] Apai, D., Pascucci, I., Brandner, W., et al. 2004, A&A, 415, 671
  * [Arriaga et al. 2020] Arriaga, P., Fitzgerald, M. P., Duchêne, G., et al. 2020, AJ, 160, 79
  * [Bazzon et al. 2012] Bazzon, A., Gisler, D., Roelfsema, R., et al. 2012, SPIE Proc., 8446, 844693
  * [Buenzli & Schmid 2009] Buenzli, E. & Schmid, H. M. 2009, A&A, 504, 259
  * [Benisty et al. 2015] Benisty, M., Juhasz, A., Boccaletti, A., et al. 2015, A&A, 578, L6
  * [Beuzit et al. 2019] Beuzit, J.-L., Vigan, A., Mouillet, D., et al. 2019, A&A, 631, 155
  * [Chen et al. 2020] Chen, C., Mazoyer, J., Poteet, C. A., et al. 2020, ApJ, 898, 55
  * [Davies & Kasper 2012] Davies, R. & Kasper, M. 2012, Ann.Rev.A&A, 50, 305
  * [de Boer et al. 2020] de Boer, J., Langlois, M., van Holstein, R. G., et al. 2020, A&A, 633
  * [Dohlen et al. 2008] Dohlen, K., Langlois, M., Saisse, M., et al. 2008, SPIE Proc., 7014, 70143L
  * [Engler et al. 2017] Engler, N., Schmid, H. M., Thalmann, C., et al. 2017, A&A, 607, 90
  * [Esposito et al. 2020] Esposito, T. M., Kalas, P., Fitzgerald, M. P., et al. 2020, AJ, 160, 24
  * [Fusco et al. 2016] Fusco, T., Sauvage, J.-F., Mouillet, D., et al. 2016, SPIE Proc., 9909, 99090U
  * [Garufi et al. 2013] Garufi, A., Quanz, S. P., Avenhaus, H., et al. 2013, A&A, 560, 105
  * [Hashimoto et al. 2011] Hashimoto, J., Tamura, M., Muto, T., et al. 2011, ApJ (Letters), 729, L17
  * [Hashimoto et al. 2012] Hashimoto, J., Dong, R., Kudo, T., et al. 2012, ApJ (Letters), 758, L19
  * [Hodapp et al. 2008] Hodapp, K. W., Suzuki, R., Tamura, M., et al. 2008, SPIE Proc., 7014, 701419
  * [Hunziker et al. 2020] Hunziker, S., Schmid, H. M., Mouillet, D., et al. 2020, A&A, 634, 69
  * [Hunziker et al. 2021] Hunziker, S., Schmid, H. M., Ma, J., et al. 2021, A&A, 648, 110
  * [Joos et al. 2008] Joos, F., Buenzli, E., Schmid, H. M., et al. 2008, SPIE Proc., 7016, 70161I
  * [Keppler et al. 2018] Keppler, M., Benisty, M., Müller, A., et al. 2018, A&A, 617, A44
  * [Khouri et al. 2020] Khouri, T., Vlemmings, W. H. T., Paladini, C., et al. 2020, A&A, 635, 200.
  * [Lenzen et al. 2003] Lenzen, R., Hartung, M., Brandner, W., et al. 2003, SPIE Proc., 4841, 944
  * [Macintosh et al. 2014] Macintosh, B., Graham, J. R., Ingraham, P., et al. 2014, Proc.Nat.Acad.Sci., 111, 12661.
  * [Marino et al. 2015] Marino, S., Perez, S., & Casassus, S. 2015, ApJ (Letters), 798, L44
  * [Milli et al. 2017] Milli, J., Mouillet, D., Fusco, T., et al. 2017, arXiv:1710.05417
  * [Milli et al. 2019] Milli, J., Engler, N., Schmid, H. M., et al. 2019, A&A, 626, A54
  * [Monnier et al. 2019] Monnier, J. D., Harries, T. J., Bae, J., et al. 2019, ApJ, 872, 122
  * [Murakawa et al. 2004] Murakawa, K., Suto, H., Tamura, M., et al. 2004, PASJ, 56, 509
  * [Murakawa et al. 2005] Murakawa, K., Suto, H., Oya, S., et al. 2005, A&A, 436, 601
  * [Muto et al. 2012] Muto, T., Grady, C. A., Hashimoto, J., et al. 2012, ApJ (Letters), 748, L22
  * [Norris et al. 2015] Norris, B., Schworer, G., Tuthill, P., et al. 2015, MNRAS, 447, 2894
  * [Ohnaka et al. 2016] Ohnaka, K., Weigelt, G., & Hofmann, K.-H. 2016, A&A, 589, 91.
  * [Perrin et al. 2009] Perrin, M. D., Schneider, G., Duchene, G., et al. 2009, ApJ (Letters), 707, L132
  * [Perrin et al. 2015] Perrin, M. D., Duchene, G., Millar-Blanchaer, M., et al. 2015, ApJ, 799, 182
  * [Pinilla et al. 2018] Pinilla, P., Benisty, M., de Boer, J., et al. 2018, ApJ, 868, 85
  * [Quanz et al. 2011] Quanz, S. P., Schmid, H. M., Geissler, K., et al. 2011, ApJ, 738, 23
  * [Quanz et al. 2013] Quanz, S. P., Avenhaus, H., Buenzli, E., et al. 2013, ApJ (Letters), 766, L2
  * [Racine et al. 1999] Racine, R., Walker, G. A. H., Nadeau, D., et al. 1999, PASP, 111, 587
  * [Safonov et al. 2020] Safonov, B., Dodin, A., Burlak, M., et al. 2020, arXiv:2005.05215
  * [Schmid et al. 2006a] Schmid, H. M., Joos, F., & Tschan, D. 2006, A&A, 452, 657
  * [Schmid et al. 2006b] Schmid, H. M., Beuzit, J.-L., Feldt, M., et al. 2006, IAU Colloq. 200, p. 165.
  * [Schmid et al. 2018] Schmid, H. M., Bazzon, A., Roelfsema, R., et al. 2018, A&A, 619, 9
  * [Simmons & Stewart 1985] Simmons, J. F. L. & Stewart, B. G. 1985, A&A, 142, 100
  * [Sivaramakrishnan et al. 2001] Sivaramakrishnan, A., Koresko, C. D., Makidon, R. B., et al. 2001, ApJ, 552, 397
  * [Stolker et al. 2016] Stolker, T., Dominik, C., Avenhaus, H., et al. 2016, A&A, 595, A113
  * [Suzuki et al. 2010] Suzuki, R., Kudo, T., Hashimoto, J., et al. 2010, SPIE Proc., 7735, 773530
  * [Tschudi & Schmid 2021] Tschudi, C., Schmid, H. M. 2021, A&A (submitted)
  * [van Holstein et al. 2020] van Holstein, R. G., Girard, J. H., de Boer, J., et al. 2020, A&A, 633, A64
  * [Wagner et al. 2021] Wagner, K., Boehle, A., Pathak, P., et al. 2021, Nat. Comm., 12, 922.
  * [Wiktorowicz et al. 2014] Wiktorowicz, S. J., Millar-Blanchaer, M., Perrin, M. D., et al. 2014, SPIE Proc., 9147, 914783
  * [Witzel et al. 2011] Witzel, G., Eckart, A., Buchholz, R. M., et al. 2011, A&A, 525, A130