# MaskRenderer: 3D-Infused Multi-Mask Realistic Face Reenactment

Tina Behrouzi∗
Department of Computer Science
University of Toronto
<EMAIL_ADDRESS>
& Atefeh Shahroudnejad∗
Amii (Alberta Machine Intelligence Institute)
<EMAIL_ADDRESS>
& Payam Mousavi
Amii (Alberta Machine Intelligence Institute)
<EMAIL_ADDRESS>

###### Abstract

We present a novel end-to-end identity-agnostic face reenactment system,
MaskRenderer, that can generate realistic, high fidelity frames in real-time.
Although recent face reenactment works have shown promising results, there are
still significant challenges such as identity leakage and imitating mouth
movements, especially for large pose changes and occluded faces. MaskRenderer
tackles these problems by using (i) a 3DMM to model 3D face structure to
better handle pose changes, occlusion, and mouth movements compared to 2D
representations; (ii) a triplet loss function to embed the cross-reenactment
during training for better identity preservation; and (iii) multi-scale
occlusion, improving inpainting and restoring missing areas. Comprehensive
quantitative and qualitative experiments conducted on the VoxCeleb1 test set,
demonstrate that MaskRenderer outperforms state-of-the-art models on unseen
faces, especially when the Source and Driving identities are very different.

***These authors contributed equally.

_Keywords_ Face Reenactment $\cdot$ Deepfake $\cdot$ Generative Models $\cdot$
3D Morphable Model $\cdot$ Multi-scale Occlusion Masks $\cdot$ Triplet Loss

##  1 Introduction

Identity-agnostic face reenactment is the process of generating sequential
face images of a target person (i.e., Source), where the talking (i.e.,
Driving) person’s pose and facial expression are transferred to the target
person’s image. In other words, the talking person controls the Source image
and its pose and expression like a puppet (See Fig. 1).

This well-established computer vision problem has a wide range of application
areas, such as the film industry, teleconferencing, and virtual reality [1].
However, high-fidelity and identity-agnostic face reenactment still remains
challenging, particularly when the model should (i) preserve the Source’s
identity, (ii) generate a photo-realistic face, and (iii) ideally require only
a single image of the Source. More precisely, the algorithm is required to
preserve the face texture and shape of the Source, even in the presence of
occlusion or any changes in the expression or pose of the Driving. We assume
that only one photo is available from the Source, and the model is suitable
for real-time applications.

###  1.1 Background

In contrast to identity-agnostic face reenactment for which a single Source
image is used, two other face reenactment categories use multiple frames
(i.e., few-shot) or video footage (i.e., identity-specific). However, few-shot
methods generally only work well when all Source images are extracted from the
same video because photos of the same person at different times typically have
different backgrounds, lighting conditions, and facial details (e.g., hair).
That alteration in images can make identity preservation harder and worsen the
cross-reenactment quality, compared to the single-image methods. Moreover,
some few-shot methods such as Vid2Vid [2] and MarioNETte [3] are not real-time
and have high computational costs.

Figure 1: This example face reenactment result is generated by MaskRenderer.

Most face reenactment methods use one or more facial representations to
accurately understand and capture face structure (i.e., identity), pose, and
expression. Facial landmarks, a common representation, have been used to
determine the pose and expression changes in Driving videos [4, 5]. Sparse
landmarks, trained in a supervised manner, are fixed points that cannot fully
capture facial structure and features. Moreover, a heavy dependence on
supervised landmarks makes the model more sensitive to occlusion and
inaccurate prediction. Recent works [6, 7, 8] have shown that unsupervised
facial features can better represent face motion for reenactment. Moreover,
unsupervised facial features better depict pose changes than supervised key
points.

The vast majority of existing works [8, 9] find facial representation and
motion based only on a person’s 2D image. However, these models do not capture
3D geometry from other views with different poses and expressions. To tackle
this issue, a 3D Morphable Model (3DMM) [10] can be used as a 3D
representation to enhance the robustness of face reconstruction and improve
mouth movements, particularly in the presence of occlusion and large pose
changes. Furthermore, 3DMM models extract person-specific disentangled
parameters such as pose, lighting, and identity. The original 3DMM employs
Principal Component Analysis (PCA) to find statistical properties of face
shape. More recent 3DMMs [11, 12] have included a light Neural Network or a
GAN model to generate a full 3D representation of a face.

Most 3DMM models [11] are trained on 3D face scans that are extremely
expensive to collect and, due to annotation costs, they have less variety in
expression. To tackle this problem, some recent works, such as DECA [12], can
estimate accurate 3D faces based on images in the wild. However, these models
usually require multiple views of the face to construct a detailed texture map
that is robust to occlusion and pose changes. Although 3DMM models have shown
impressive results in generating a 3D face, they still suffer in recovering
facial components, such as eyes, neck, teeth, facial detail (e.g., beard and
makeup), and image background. To address these limitations, there are image
inpainting methods, such as DE-GAN [13], that can help in recovering a face
image.

###  1.2 Contributions

This article introduces the MaskRenderer framework that reenacts the face of
an unseen Source based on a sequence of Driving frames while preserving the
Source identity. The proposed model can generate images in real-time. Our main
contributions are as follows:

  * •

Inspired by the recent advance in 3D vision models, we incorporated Deng 3DMM
[10] into our model. Combining 3D head parameters and 2D motion transformation
resulted in more accurate pose changes and iris and mouth movements.

  * •

Current face reenactment models do not perform cross-reenactment during
training. Not considering different identities for Source and Driving during
training causes under-performance at inference time when the Source and
Driving images are from two different people. We designed a triplet loss to
incorporate cross-reenactment loss during training, which improves identity
preservation where the Source and Driving facial shapes are completely
different.

  * •

We used multi-scale occlusion in the generator for better inpainting of
unwanted or missing areas.

  * •

MaskRenderer outperformed state-of-the-art methods on the VoxCeleb1 benchmark
in terms of identity preservation, transferring pose and expression, and mouth
movements.

The remaineder of the article is as follows: In Section 2, recent related
works on face reenactment are discussed. The framework of our proposed method
and the detailed structure are explained in Section 3. Section 4 summarizes
the datasets, inference requirements, evaluation metrics, and training
paradigm. In section 5, MaskRenderer is compared with baselines and other
state-of-the-art methods. Section 6 contains empirical ablation studies to
evaluate and justify our algorithmic design decisions. Finally, Section 7
concludes and discusses the future directions.

##  2 Related Works

In this section, we review existing face reenactment methods and categorize
them based on the three types of face representation.

###  2.1 3D-based models

3D-based models leverage 3D facial geometry to improve upon 2D techniques,
better controlling different expressions and poses. The notable challenges in
3D-based models are (i) achieving high-fidelity identity preservation and
looking realistic instead of like a 3D avatar, (ii) some tasks, such as hair
generation, is more problematic in 3D, and (iii) the quality of 3D animatable
head reconstruction typically depends on the extracted geometry, texture
rendering, model generalization, and disentanglement between identity, pose,
and expression. Earlier works [14, 15, 16] transfer 3D facial parameters from
Driving to Source and generate the reenactment video by rendering the input.
The primary limitation of these works is that they all need to be trained on a
multi-view video of a specific person. Many recent agnostic works [6, 17, 18,
19, 20, 21] incorporate a pre-trained 3DMM into their structure to model the
shape, texture, expression, and pose of faces. PIRenderer [17] first maps 3DMM
parameters to a latent space and then uses it to guide the motion network and
the generator. Yao et al. [22] predicts 3DMM parameters to reconstruct 3D face
meshes of Source and Driving. Then, they apply graph Convolutional operations
on these meshes to extract motion features to guide the motion estimation
network.

###  2.2 Landmark-based models

In this class of model, facial landmarks are used as a prior condition to
guide the generator by representing pose and expression. The main challenge in
landmark-based models is preserving identity due to the person-specific
identity information in facial landmarks. Zakharov et al. [23] proposed
NeuralHead as a few-shot face reenactment that generates reenactment directly
from landmarks by adopting a meta-learning strategy and fine-tuning for unseen
identities. A personalized pose-invariant embedding vector is created based on
different sets of frames and is used to predict AdaIN parameters [24] in the
generator for appearance transfer. NeuralHead was later extended [25] for
better cross-identity reenactment by using latent pose vectors instead of
directly employing landmarks.

Another few-shot work to address the identity preservation problem is
MarioNETte [3]. It introduces a landmark transformer to adapt facial points
between Source and Driving and uses an image attention block for better
appearance style transfer. The idea of a landmark transformer was later used
in multiple works [4, 26, 27]. To improve the visual quality of results,
others [28, 29] consider facial components landmarks separately, in addition
to the facial landmarks in the reenactment process. Hsu et al. [5] employ a 3D
landmark detector, rather than 2D, to cope with large pose changes.

###  2.3 Motion-based models

Motion-based models are composed of two sub-networks: (i) a motion estimation
network to predict a dense motion field from Source to Driving that is used
for warping the Source and (ii) a generative network to generate the final
reenactment result using the warped Source. Siarohin et al. [30] first
proposed MonkeyNet, which encodes motion by learning self-supervised
keypoints. Then in their follow-up work, named First-Order Motion Model (FOMM)
[8], they significantly improved the results’ appearance, particularly for
more complex motions, by considering local affine transformations instead of
rigid motions. Many subsequent works, including ours, have drawn inspiration
from the FOMM’s performance and structure. Zhao et al. [9] reformulated FOMM
for better transferring the motion when there is a large pose difference
between Source and Driving. They employ Thin-Plate Spline transformations
along with the affine transformation for the dense motion estimation in the
training path.

Another work tackling the large pose changes between Source and Driving is
proposed by Liu et al. [31]. They adopted Neural-ODE to differentially refine
the initial keypoints and model the dynamics of motion deformation. Tripathy
et al. [32] use a keypoint transformer to stabilize keypoints in FOMM for
better cross-identity reenactment. To address the limitations of 2D
representations in motion changes and cross-identity issues, Hong et al. [33]
proposed estimating facial depth maps from Source and Driving and using them
as 3D facial geometries to guide the keypoint estimator in FOMM. These depth
maps are also used to refine the motion field for more detailed and accurate
reenactment. Wang et al. [34] also extended FOMM by predicting canonical 3D
keypoints instead of 2D to support large pose changes.

In a recent work, LIA [7] uses StyleGAN2 [35] and sparse coding to estimate
the motion from Source to Driving without keypoints prediction. Therefore, the
quality of reenactment results is not dependent on the initial Driving frame.
Other methods [36, 37] extract regions or masks instead of keypoints to
separate the foreground from the background for improving shapes capturing.
Hence, these methods are more suitable for articulated objects like human
bodies. Our proposed model comprises both motion-based and 3D-based models
resulting in high-quality, high-fidelity, and photo-realistic face
reenactment.

##  3 MaskRenderer: Methodology and Approach

In this section, we explain the detailed structure of the MaskRenderer. Fig. 2
shows a high-level schematic. The network is divided into four parts: (i) the
3DMM module generates disentangled Source’s facial shape and Driving’s pose
and expression. (ii) the Facial Features Detector (FFD) finds representative
facial characteristics. Then, both the FFD features and 3DMM parameters are
passed to (iii), the Dense Motion network, to find the optical flow between
the Source and Driving images. In (iv), the Multi-scale Occlusion Masks are
constructed to represent the image’s missing parts. The generator takes the
occlusion masks and dense motion matrix to create the reenacted result.
MaskRenderer is a GAN-based model that uses a Generative Adversarial Network
(GAN) architecture to generate realistic reenacted images from a given source
image. The final part of this section is devoted to discussing our novel use
of triplet loss among other loss terms.

Figure 2: The architecture of the proposed model

###  3.1 3DMM

Given the Source image $I_{S}$, the 3DMM objective is to reproduce the 3D
shape of $I_{S}$, which is consistent with various image viewpoints. The
original 3DMM is defined as Eq. (1), where $S$ represents the 3D face shape.
$\mathbf{\bar{\text{$S$}}}$ is the mean face shape value. $\alpha_{id}$ and
$\alpha_{exp}$ are identity and expression coefficients, and finally $A_{id}$
and $A_{exp}$ are the identity and expression PCA bases, respectively. Further
explanation can be found in the supplementary material and the 3DMM paper
[38].

$S=\mathbf{\bar{\text{$S$}}}+A_{id}\alpha_{id}+A_{exp}\alpha_{exp}$ (1)
$T=\mathbf{\bar{\text{$T$}}}+A_{tex}\alpha_{tex}$ (2)

The skin texture $T$ is combined with the face shape $S$, defined in Eq. (2),
where the texture coefficient $\alpha_{tex}$ adjusts the mean texture value
$\mathbf{\bar{\text{$T$}}}$ and texture PCA base $A_{tex}$. The input image
only changes the coefficients ($\alpha_{id}\in\mathbb{R}^{80}$,
$\alpha_{exp}\in\mathbb{R}^{64}$, $\alpha_{tex}\in\mathbb{R}^{80}$). The PCA
bases and mean values ($A_{id}$, $A_{exp}$, $A_{tex}$,
$\mathbf{\bar{\text{$S$}}}$, $\mathbf{\bar{\text{$T$}}}$) are set to be
consistent with prior work [38].

We employ the pre-trained Deep3DFace model [10] as the 3DMM model as this
model finds an explicit texture’s mesh structure from a single-view image and
can capture major changes in the head and mouth area. Since the 3DMM’s
features are disentangled, the pose and expression can be taken from the
Driving frame $I_{D}$, while other parameters are derived from the Source
image $I_{S}$. The 3D reenacted face and motion field between the Source and
Driving face vertexes are passed to the Dense Motion network.

Although the reenacted 3D face shape is stable to different poses,
expressions, and lighting, the final texture is not detailed and does not
include regions such as teeth, hair, and inner eye parts. We address this
issue in the later steps, bottom right of Fig. 2, where the skin texture and
facial details are reconstructed in 2D space.

###  3.2 Facial Feature Detection

The Facial Feature Detection (FFD) structure is inspired by FOMM’s keypoints
detector network [8]. The FFD structure and the corresponding equivariance
losses are designed to ensure that facial features111In this article, we use
the term “feature” instead of “keypoint” because the unsupervised features do
not directly represent facial landmarks. are presentable on an input face. As
shown in Fig. 3, the input image $I\in\mathbb{R}^{256\times 256\times 3}$
(from either the Source or the Driver) is passed through the hourglass network
to create a heatmap with the shape of $(K\times 64\times 64)$, where $K$ is
the spatial feature shape. The heatmap is passed via two parallel convolution
layers to create the feature locations in 2D coordinate
$F\in\mathbb{R}^{K\times 2}$ and motion matrix $J\in\mathbb{R}^{K\times
2\times 2}$.

The facial features are used to identify the sparse motion and dense motion,
subsection 3.3. The sparse motion matrix $M_{k}(z)$ between Source and Driving
images is defined as:

$M_{k}(z)=F_{S_{k}}+(J_{S_{k}}J_{D_{k}}^{-1})(z-F_{D_{k}}),$ (3)

where $M_{k}(z)$ is calculated at coordinate positions $z$ near each feature
point for each feature row $k\in\\{1,\dotsc,K\\}$. The subscripts $S$ and $D$
denote whether features belong to the Source or Driving image. The learnable
$J_{S}J_{D}^{-1}$ matrix control the impact of changes in $F_{D}$ on $F_{S}$.

Figure 3: In the FFD module, the attention module is applied to the skip
connections’ features

We have included the attention mechanism in the hourglass model by adding
Convolutional Block Attention Module (CBAM) [39] blocks. The new hourglass can
capture the gradual changes between $F_{S}$ and $F_{D}$ using channel and
spatial attention. The attention blocks also help FFD to be robust against
occlusion, face rotation, and changes in illumination by considering the
positional relationship between features and only influential information.

###  3.3 Dense Motion Network

The objective of the Dense Motion network is to find the optical flow between
$I_{S}$ and $I_{D}$, transferring the Driving’s pose and expression into the
Source image. However, deriving dense motion between $I_{S}$ and $I_{D}$ is
very challenging due to the small $K$ number of unsupervised features $F$.
Hence, we incorporated 3DMM-based parameters to better address motions such as
eye blinking, head movement, and mouth opening. The sparse motion $M_{k}(z)$
is concatenated with the motion field $M_{3D}(z)$ derived from the 3DMM module
to better represent the flow changes from the Driving to the Source’s face.

To estimate the final dense motion $\hat{M}(z)$, $K+2$ dense heatmaps’ masks
$H$ are required to combine static points, 3D motions, and the $K$ local
features (Eq. 4). To generate $H_{0}$, $H_{3D}$, and $H_{k}$ matrices, another
attention hourglass network is applied respectively to the sparse
transformation of the Source image, the reenacted 3D face, and to the
features.

$\hat{M}(z)=H_{0}z+H_{3D}M_{3D}(z)+\sum_{k=1}^{K}H_{k}M_{k}(z)$ (4)

###  3.4 Multi-Scale Occlusion Masks & Generator

Occlusion masks address the pixel locations that are missing or occluded in
the Source image by inpainting them during generation. However, different
feature maps in the Dense Motion module’s decoder have different relative
importance. The high-level feature map represents facial details such as face
and color, while the low-level feature map focuses on coarse features such as
pose and face shape. Accordingly, we define an occlusion mask $O_{i}\in[0,1]$
for the feature map in layer $i$, as shown bottom right part of the Fig. 2.
The $O_{i}$ is estimated using a parallel convolutional layer with a sigmoid
activation function on the output of the attention hourglass’s decoder layer
$i$ in the Dense Motion module.

To generate a photo-realistic reenacted 2D image, a generator with an encoder-
decoder structure is designed. The encoder extracts Source features for
different scales, $F_{Enc_{S,i}}$, where $i$ is the $i^{th}$ down-sampled
layer. The decoder employs per-pixel mapping derived from the dense motion
module to warp the Source features and up-sample $F_{Enc_{S}}$ to the original
image shape.

After warping $F_{Enc_{S,i}}$ based on the Driving’s pose and expression, the
missing pixels should be inpainted. For this complex inpainting, we apply the
occlusion mask $O_{i}$ to the warped features,
$\mathcal{T}(F_{Enc_{S,i}},\hat{M}(z))$, to locate the hidden regions that
require reconstruction. The decoder’s output at layer $i$ is concatenated with
$O_{i}\times\mathcal{T}(F_{Enc_{S,i}},\hat{M}(z))$ and passed to the next up-
sample layer to generate the final reenacted image $I_{F}$.

###  3.5 Training Losses

The model is trained in a self-supervised fashion. All the training losses,
except the triplet loss, are defined based on self-reenactment, where $I_{S}$
and $I_{D}$ are two frames of the same person in a video. The triplet loss is
based on cross-reenactment and uses identity soft labels derived from the
VGGFace2 [40]. MaskRenderer is trained end-to-end, and all the losses listed
below are summed and minimized.

Triplet loss $L_{T}$: To encourage the network to generate identity-consistent
images robust to changes in pose and expression, we employed the triplet loss.
The triplet loss, Eq. (5), forces the network to preserve the identity of the
Source image.

Figure 4: Triplet loss illustration.

$L_{T}=\sum_{a,p,n}{[D_{x_{a},x_{p}}-D_{x_{a},x_{n}}+m]_{+}}\text{;
\thickspace\thickspace\thickspace}y_{a}=y_{p}\neq y_{n}$ (5)

Where $x_{a}$ is an anchor identity feature and $x_{p}$ and $x_{n}$ are
positive and negative features, respectively. The input and reenacted images
are passed through the pre-trained VGGFace2 [40] to extract identity features
$x$. Pair $(x_{a},x_{p})$ belongs to the same identity class $y_{a}$, whereas
negative samples have different identities $y_{n}$. The objective of triplet
loss is to bring the reconstructed image’s identity closer to the Source image
while increasing its distance $D$ from a negative sample with the margin $m$.
We expanded the triplet loss as Eq. (6) to represent the cross-reenactment
loss illustrated in Fig. 4. The second part of the equation captures difficult
positive cases where Source and Driving have different facial shapes and
poses.

$L_{T}=\frac{1}{|B|}\sum_{i=1}^{|B|}{[\alpha||x_{a}-x_{p_{1}}||_{2}+(1-\alpha)||x_{a}-x_{p_{2}}||_{2}-||x_{a}-x_{n}||_{2}+m]_{+}}$
(6)

A drawback of adding a triplet loss term to the network is that it requires
additional computation. Moreover, triplet loss can cause negative bias and
force the model to compromise the pose and expression to preserve identity. We
address these issues by setting triplet loss to zero at initial epochs and
applying it in the later epochs. Moreover, the weight between the first and
second terms in the Eq. (6) is tuned to reduce negative bias.

Warping loss $L_{W}$: The warping loss attempts to bring the warped Source
features closer to the Driving features $F_{Enc_{D}}$ to improve the warping
accuracy and dense flow estimation. The warping loss is defined as:

$L_{W}=\sum_{i}mean({|\mathcal{T}(F_{Enc_{S,i}},\hat{S}(z))-F_{Enc_{D,i}}|}),$
(7)

where the index $i$ represents the $i^{th}$ down-sampled layer in the
generator. The next losses are defined as FOMM’s [8] loss functions.

Equivariance losses $L_{E}$ & $L_{J}$: By adding the equivariance losses, we
address the feature $F$ consistency issue to non-linear Thin Plate Spline
(TPS) warping $\mathcal{T}_{TPS}$ [8]. The $L_{E}$ and $L_{J}$ losses
encourage the warped Driving features $\mathcal{T}_{TPS}(F_{D})$ and motion
$\mathcal{T}_{TPS}(J_{D})$ to become closer to their corresponding values from
the warped Driving frame $\mathcal{T}_{TPS}(I_{D})$.

Perceptual loss $L_{P}$: The perceptual loss attempts to bring the reenacted
image $I_{F}$ and Driving frame closer to each others using the pre-trained
VGG_19 network’s features. In the following equations, the index $i$
represents the feature scale.

$L_{P}=\sum_{i}mean({|VGG_{i}(I_{D})-VGG_{i}(I_{F})|})$ (8)

GAN loss $L_{G}$: The MaskRenderer is trained similarly to a GAN, by
alternating between the training of the generator and the discriminator. The
discriminator consists of four layers for four feature maps, $F_{M}$, and a
final discriminator map prediction, $D_{M}$. In the generator, the GAN loss
$L_{G}$, defined as Eq. (9) attempts to make generated images look like real
images.

$L_{G}=\sum_{i}mean({(1-D_{M_{i}}(I_{F}))^{2}})$ (9)

The Feature Matching loss $L_{F}$, Eq. (10), brings the Driving and reenacted
frames’ discriminator feature maps closer to each other.

$L_{F}=\sum_{i}\sum_{j}mean({|F_{M_{i},j}(I_{D})-F_{M_{i},j}(I_{F})|})$ (10)

The index $j$ indicates the $F_{M}$ layer. At each iteration, first, the
mentioned losses are combined as shown in Eq. (11) and passed to the
optimizer.

$L_{generator}=\alpha L_{T}+\beta(L_{W}+L_{E}+L_{J}+L_{P}+L_{G}+L_{F})$ (11)

Then, the discriminator is trained to discriminate between the real $I_{D}$
and generated $I_{F}$ photos to make the model robust to parameters that help
distinguish between real and fake images. The discriminator loss $L_{D}$ is
defined as follows:

$L_{D}=\sum_{i}mean[{{(1-D_{M_{i}}(I_{D}))}^{2}+{D_{M_{i}}(I_{F})}^{2}}]$ (12)

##  4 Experimental Setup

In this section, we discuss how we will train and evaluate MaskRenderer: the
dataset, testing setup, and evaluation metrics. We also include implementation
details for reproducability.

###  4.1 Dataset

We used VoxCeleb1
dataset222https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html [41] to
train and evaluate our model. It contains 22,496 videos from 1,251 identities
that have been extracted from interview videos on YouTube. After preprocessing
and cleaning the data (based on prior work [8]), we used 17,765 videos for
training and 464 videos for evaluation. There is no overlap between the
identities in the training and test sets.

###  4.2 Inference stage

Self-reenactment evaluations were performed on all 464 testing videos, where
the first frame is set as the Source image, and the rest are the Driving
images. On the other hand, in cross-reenactment evaluation, we randomly
selected 61 test videos of 12 individuals. For each video, the first frame is
assigned as the initial driving frame. This helps to consider the relative
pose and expression distance between the initial frame and any other frame
used as a driving image instead of the absolute pose value. Due to the
difference in facial structures between Source and Driving, considering the
relative changes in the Driving’s face coordinates supports preserving the
Source’s identity. Additionally, The Source image is randomly selected from
videos with different identities.

###  4.3 Evaluation Metrics

In the self-reenactment scenario, the final generated frame must replicate the
Driving frame. Therefore, the Driving frame is the ground truth for pose,
expression, and identity of the corresponding generated image. We considered
standard metrics for comparing the SOTA models’ self-reenactment performance
[3, 6, 8], as follows. L1 distance calculates the mean absolute difference
between Driving the generated frame. Absolute Keypoint Difference (AKD)
measures the keypoints’ contrast between Driving and the generated frame.
Absolute Identity Difference (AID) measures the similarity between source and
generated images’ identity vector, estimated as in prior work [12]. Moreover,
we computed the Euclidean Pose Distance (EPD) to compare the Driving and the
generated frame’s pose and expression using the pose embedding vectors [10].

For evaluating cross-reenactment performance, since there is no ground truth
available when the Source and Driving people are different, we calculated
cosine similarity between two features instead of calculating absolute
distance. The cross-reenactment metrics are as follows. Identity Similarity
(ISIM) determines how well the generated frame has preserved the Source’s
identity and facial structure. Pose Similarity (PSIM) and Keypoint Similarity
(KSIM) quantify how close the poses and positional facial landmarks are,
respectively, between the Driving and generated frames.

The Fréchet Inception Distance (FID) estimates the difference between the
normal distribution of the fake and real images’ features by looking at the
distance between the generated (fake) and Source (real) images, estimating how
photo-realistic it is. We used the pre-trained InceptionV3 for extracting
images’ features compared using FID as described in [6].

###  4.4 Implementation Details

Our MaskRenderer was trained on the training data with 421 different
identities. Both Driving and Source frames are randomly selected from a random
patch of each person’s videos. To cover various combinations of each
identity’s frames, the process is repeated 150 times with a batch size of 8
for 100 epochs. Both generator and discriminator learning rates are set to
$2\times 10^{-4}$ with $\beta_{1}=0.5$ and $\beta_{2}=0.99$. We used three
NVIDIA 3090 GPUs with 24GB memory to train the model. The loss hyper-
parameters $\alpha$ and $\beta$ in the Eq. 11 are set to 10, and the triplet
loss margin $m$ is set to 1. The facial features’ number $K$ is set to 10
after performing experiments on the set $K=\\{5,10,15,32,64\\}$. Fewer points
could not fully capture the face and body movements, while using more points
resulted in the system becoming overly complicated (the model failed to
accurately map and detect changes in the facial features). After testing the
range of $[0.4,1]$ for $alpha$ in the Eq. (6), the higher weight,
$\alpha=0.8$, was selected to emphasize the first self-reenactment term and
reduce the likelihood of any negative bias. We replaced the nvdiffrast
renderer in Deep3DFace [10] with the Pytorch3D
renderer333https://pytorch3d.readthedocs.io/en/latest/modules/renderer/index.html,
as it is computationally more efficient.

##  5 Results

In this section, we compare the MaskRenderer with state-of-the-art (SOTA)
methods: FOMM [8], SAFA [6], DaGAN [33], TPS [9], and DualGen [5]. The metrics
for SOTA are calculated for the released pre-trained models. All evaluations
are performed on the same device. The inference time of MaskRenderer for
cross-reenactment is less than 40 fps.

###  5.1 Cross-Reenactment

The main objective of this work is to improve the cross-reenactment
performance when Driving and Source images have different facial structures
and identities.

####  5.1.1 Quantitative Results

Table 1 shows that MaskRenderer has the highest ISIM and KSIM scores. Higher
identity and landmark similarity indicate that MaskRenderer better preserves
the Source identity during pose and expression changes compared to the SOTA
models. The visual qualitative comparison verifies this performance,
especially for faces with very different structures. Moreover, MaskRenderer
has the lowest FID score, confirming that our model generates the most photo-
realistic images.

Table 1: Cross-reenactment quantitative results. Bold values correspond to the
best result among SOTA and our methods.

Model | ISIM$\uparrow$ | KSIM$\uparrow$ | PSIM$\uparrow$ | FID$\downarrow$
---|---|---|---|---
FOMM[8] | 0.873 | 0.899 | 0.512 | 59.522
SAFA [6] | 0.856 | 0.902 | 0.575 | 55.456
DaGAN [33] | 0.872 | 0.91 | 0.507 | 61.643
TPS [9] | 0.851 | 0.91 | 0.571 | 69.144
MaskRenderer | 0.891 | 0.914 | 0.527 | 49.469

Although the pose similarity score of the SAFA and TPS is higher than
MaskRenderer, these methods perform much worse in preserving the identity of
Source image. The decrease in PSIM is because our method tries to make smooth
transitions between frames, and in some cases, the generated result does not
fully follow the Driving’s pose in the area of the iris. However, this pose
adjustment does not affect indicative features such as mouth movements, and
the final result is highly realistic. All in all, MaskRenderer is smoother and
more consistent in generating photo-realistic identity-preserved reenacted
results than SOTA models.

####  5.1.2 Qualitative Results

Source

Driving

FOMM

SAFA

DaGAN

DualGen

MaskRenderer

Figure 5: Qualitative comparison of our method with the state of the art for
cross-reenactment

Fig. 5 shows the qualitative comparison of our MaskRenderer with the baseline
(FOMM) and other SOTA models. The first and second columns show sample Source
images and Driving frames, respectively. The rest of the columns represent the
face reenactment results of each model corresponding to the Source and Driving
images (first two columns). As shown in the last column, our method generates
high-quality, high-fidelity, and photo-realistic face reenactment results.
MaskRenderer visually outperforms SOTA methods in several aspects. First,
MaskRenderer preserves the Source identity even when it is very different from
the Driving face, while other models show signs of identity leakage more or
less (as most apparent in the first row). Second, MaskRenderer handles the
hand occlusion better than other models (see the third row). Third,
MaskRenderer has the capability of generating reenactment in different head
pose angles. Fourth, our method can generate fine-grained details, such as
wrinkles, facial hair, and teeth. Fifth, it can also produce reasonable mouth
movements, which is critical when the Driving person is speaking.

###  5.2 Self-Reenactment

Table 2: Quantitative comparison of the proposed method with state-of-the-art
methods for self-reenactment.

Model | AKD | AID | $L_{1}$ | EPD
---|---|---|---|---
FOMM [8] | 1.378 | 0.12 | 0.045 | 0.917
SAFA [6] | 1.21 | 0.118 | 0.041 | 0.831
DaGAN [33] | 1.27 | 0.123 | 0.042 | 0.88
TPS [9] | 1.322 | 0.145 | 0.039 | 0.93
MaskRenderer | 1.395 | 0.103 | 0.044 | 1.023

We also compared MaskRenderer with SOTA methods in self-reenactment cases
where the Source and Driving frames are from the same person in a video. Table
2 shows that our proposed method has the lowest AID, and the identity error
has improved at least by 13% compared to SOTA methods. This increase indicates
that our method elevates identity preservation in self-reenactment scenarios
and accurately reconstructs the facial details even for large pose changes,
more than 45∘ head rotation. However, the AKD, EPD, and $L_{1}$ losses of
MaskRenderer are higher than the previous methods. Adding the triplet loss
helps reduce the bias of the same-identity reenactment during training. This
modification comes with the trade-off of decreasing the self-reenactment’s
pose and landmarks accuracy, relative to SOTA methods. However, we believe the
slight decrease in AKD and EPD are justified by the improved cross-identity
image generation.

##  6 Ablation Studies

In this section, we conduct ablation studies to test the impact of adding the
main components of our network to the baseline FOMM [8] in cross-reenactment
scenarios. Table 3 quantitatively shows that adding the 3D parameters to the
baseline (second row) helps mouth and body movements and pose changes;
however, it does poorly in preserving the identity with high quality. On the
other hand, including an attention unit to the hourglass network (third row)
improves the ISIM and FID scores but leads to less accurate pose and
expression generation. The forth row results validate the role of triplet loss
in maintaining the Source identity while facing changes in pose and
expression. This indicates the importance of considering identity change
during training. The multi-scale occlusion masks and warping loss (fifth row)
help with image background blending and inpainting unseen facial details.

Table 3: Ablation on adding the main components of our network to the baseline
for cross-reenactment

Method | ISIM$\uparrow$ | KSIM$\uparrow$ | PSIM$\uparrow$ | FID$\downarrow$
---|---|---|---|---
FOMM (baseline) [8] | 0.873 | 0.899 | 0.512 | 59.522
w/ 3DMM | 0.871 | 0.902 | 0.559 | 52.339
w/ 3DMM+attention | 0.926 | 0.9 | 0.505 | 57.164
w/ 3DMM+Masks+$L_{W}$ | 0.873 | 0.912 | 0.533 | 50.919
w/ 3DMM+$L_{T}$ | 0.875 | 0.905 | 0.508 | 50.181
MaskRenderer | 0.891 | 0.914 | 0.527 | 49.469

From these quantitative results, we conclude that all the components in
MaskRenderer contribute to the generation of highly realistic reenacted faces
with high identity preservation. The final model has a high performance
regarding identity, pose, and landmark, with the best FID score as compared to
other models. Furthermore, MaskRenderer replicates face landmarks changes
accurately and has the highest KSIM. Fig. 6 also qualitatively supports all
the aforementioned advantages of the proposed components.

Source

Driving

Baseline

+3DMM

+3DMM+ Attention

+3DMM+ Masks+$L_{W}$

+3DMM+ $L_{T}$

MaskRenderer

Figure 6: Qualitative ablation.

##  7 Conclusion

We have introduced MaskRenderer, a real-time identity-agnostic face
reenactment framework that is robust to occlusion and large mismatches between
Source and Driving facial structures. We incorporate the 3DMM model into the
network to find a reliable 3D facial pose. The 3D and 2D motions are combined
to precisely follow changes in the eyes and lips. By adopting the triplet
loss, the impact of identity alteration is considered during training thereby
reducing the bias to self-reenactment.

We further designed multi-scale occlusion masks and the corresponding warping
loss to improve the separation of foreground and background of a face and
blending with the border. Ablation studies demonstrate the importance of each
network’s component in maintaining the facial structure and inpainting missing
sections.

Both qualitative and quantitative results illustrate that Maskrenderer
improves the identity preservation of a Source image even for large poses
compared to the SOTA models. The generated cross-reenacted results of the
proposed method are photo-realistic, and the FID score is 10% higher than the
SOTA models. In the future, we plan to improve on the feature fusion and
normalization in the generator to enhance the inpainting of the hair and teeth
even further.

##  8 Acknowledgment

We wish to thank Mara Cairo, Deborah Akaniru, and Talat Iqbal Syed for their
valuable expertise and support during this project. We would also like to
thank Dr. Matt Taylor for his great advise and constructive feedback, which
helped us to improve the quality of this work.

## References

  * [1] Ruben Tolosana, Ruben Vera-Rodriguez, Julian Fierrez, Aythami Morales, and Javier Ortega-Garcia.  Deepfakes and beyond: A survey of face manipulation and fake detection.  Information Fusion, 64:131–148, 2020.
  * [2] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro.  Few-shot video-to-video synthesis.  In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 5013–5024, 2019.
  * [3] Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim.  MarioNETte: Few-shot face reenactment preserving identity of unseen targets.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 10893–10900, 2020.
  * [4] Jiangning Zhang, Xianfang Zeng, Mengmeng Wang, Yusu Pan, Liang Liu, Yong Liu, Yu Ding, and Changjie Fan.  FReeNet: Multi-identity face reenactment.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5326–5335, 2020.
  * [5] Gee-Sern Hsu, Chun-Hung Tsai, and Hung-Yi Wu.  Dual-generator face reenactment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 642–650, 2022.
  * [6] Qiulin Wang, Lu Zhang, and Bo Li.  SAFA: Structure Aware Face Animation.  In 2021 International Conference on 3D Vision (3DV), pages 679–688. IEEE, 2021.
  * [7] Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva.  Latent image animator: Learning to animate images via latent space navigation.  In International Conference on Learning Representations, 2022.
  * [8] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  First order motion model for image animation.  Advances in Neural Information Processing Systems, 32, 2019.
  * [9] Jian Zhao and Hui Zhang.  Thin-plate spline motion model for image animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3657–3666, 2022.
  * [10] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [11] Zhenfeng Fan, Xiyuan Hu, Chen Chen, Xiaolian Wang, and Silong Peng.  A landmark-free approach for automatic, dense and robust correspondence of 3D faces.  Pattern Recognition, 133:108971, 2023.
  * [12] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart.  Learning an animatable detailed 3D face model from in-the-wild images.  ACM Transactions on Graphics (ToG), 40(4):1–13, 2021.
  * [13] Xian Zhang, Xin Wang, Canghong Shi, Zhe Yan, Xiaojie Li, Bin Kong, Siwei Lyu, Bin Zhu, Jiancheng Lv, Youbing Yin, et al.  DE-GAN: Domain embedded GAN for high quality face image inpainting.  Pattern Recognition, 124:108415, 2022.
  * [14] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner.  Face2Face: Real-time face capture and reenactment of RGB videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2387–2395, 2016.
  * [15] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt.  Deep video portraits.  ACM Transactions on Graphics (TOG), 37(4):1–14, 2018.
  * [16] Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, and Matthias Nießner.  HeadOn: Real-time reenactment of human portrait videos.  ACM Transactions on Graphics (TOG), 37(4):1–13, 2018.
  * [17] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu.  Pirenderer: Controllable portrait image generation via semantic neural rendering.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13759–13768, 2021.
  * [18] Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang.  Styleheat: One-shot high-resolution editable talking face generation via pretrained StyleGAN.  arXiv preprint arXiv:2203.04036, 2022.
  * [19] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska.  HeadGAN: One-shot neural head synthesis and editing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14398–14407, 2021.
  * [20] Connor Z Lin, David B Lindell, Eric R Chan, and Gordon Wetzstein.  3D GAN inversion for controllable portrait image animation.  arXiv preprint arXiv:2203.13441, 2022.
  * [21] Kewei Yang, Kang Chen, Daoliang Guo, Song-Hai Zhang, Yuan-Chen Guo, and Weidong Zhang.  Face2Face $\rho$: Real-time high-resolution one-shot face reenactment.  In European Conference on Computer Vision, pages 55–71. Springer, 2022.
  * [22] Guangming Yao, Yi Yuan, Tianjia Shao, and Kun Zhou.  Mesh guided one-shot face reenactment using graph convolutional networks.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 1773–1781, 2020.
  * [23] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9459–9468, 2019.
  * [24] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In Proceedings of the IEEE international conference on computer vision, pages 1501–1510, 2017.
  * [25] Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky.  Neural head reenactment with latent pose descriptors.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13786–13795, 2020.
  * [26] Jin Liu, Peng Chen, Tao Liang, Zhaoxing Li, Cai Yu, Shuqiao Zou, Jiao Dai, and Jizhong Han.  Li-Net: Large-pose identity-preserving face reenactment network.  In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2021.
  * [27] Pu Sun, Yuezun Li, Honggang Qi, and Siwei Lyu.  LandmarkGAN: Synthesizing faces from landmarks.  Pattern Recognition Letters, 161:90–98, 2022.
  * [28] Ruiqi Zhao, Tianyi Wu, and Guodong Guo.  Sparse to dense motion transfer for face image animation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1991–2000, 2021.
  * [29] Guangming Yao, Yi Yuan, Tianjia Shao, Shuang Li, Shanqi Liu, Yong Liu, Mengmeng Wang, and Kun Zhou.  One-shot face reenactment using appearance adaptive normalization.  arXiv preprint arXiv:2102.03984, 2021.
  * [30] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  Animating arbitrary objects via deep motion transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2377–2386, 2019.
  * [31] Peirong Liu, Rui Wang, Xuefei Cao, Yipin Zhou, Ashish Shah, Maxime Oquab, Camille Couprie, and Ser-Nam Lim.  Self-appearance-aided differential evolution for motion transfer.  arXiv preprint arXiv:2110.04658, 2021.
  * [32] Soumya Tripathy, Juho Kannala, and Esa Rahtu.  Single source one shot reenactment using weighted motion from paired feature points.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2715–2724, 2022.
  * [33] Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu.  Depth-aware generative adversarial network for talking head video generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3397–3406, 2022.
  * [34] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu.  One-shot free-view neural talking-head synthesis for video conferencing.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10039–10049, 2021.
  * [35] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  * [36] Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov.  Motion representations for articulated animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13653–13662, 2021.
  * [37] Yoav Shalev and Lior Wolf.  Image animation with perturbed masks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3647–3656, 2022.
  * [38] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou.  FaceWarehouse: A 3D facial expression database for visual computing.  IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013.
  * [39] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  CBAM: Convolutional Block Attention Module.  In European Conference on Computer Vision, pages 3–19. Springer, 2018.
  * [40] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman.  VGGFace2: A dataset for recognising faces across pose and age.  In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 67–74. IEEE, 2018.
  * [41] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.  VoxCeleb: a large-scale speaker identification dataset.  arXiv preprint arXiv:1706.08612, 2017.


# Forecast-based Multi-aspect Framework for Multivariate Time-series Anomaly
Detection

Lan Wang, Yusan Lin, Yuhang Wu, Huiyuan Chen, Fei Wang, Hao Yang  Visa
Research, Palo Alto, CA, USA, 94306  {lwang4, yusalin, yuhawu, hchen, feiwang,
<EMAIL_ADDRESS>
###### Abstract

Today’s cyber-world is vastly multivariate. Metrics collected at extreme
varieties demand multivariate algorithms to properly detect anomalies.
However, forecast-based algorithms, as widely proven approaches, often perform
sub-optimally or inconsistently across datasets. A key common issue is they
strive to be one-size-fits-all but anomalies are distinctive in nature. We
propose a method that tailors to such distinction. Presenting FMUAD - a
Forecast-based, Multi-aspect, Unsupervised Anomaly Detection framework. FMUAD
explicitly and separately captures the signature traits of anomaly types -
spatial change, temporal change and correlation change - with independent
modules. The modules then jointly learn an optimal feature representation,
which is highly flexible and intuitive, unlike most other models in the
category. Extensive experiments show our FMUAD framework consistently
outperforms other state-of-the-art forecast-based anomaly detectors.

###### Index Terms:

Anomaly Detection, Multivariate Time Series, Unsupervised Learning.

##  I Introduction

Anomaly detection for multivariate time series is of great interest in many
real-word applications, such as road traffic surveillance [1], financial fraud
detection [2], software monitoring [3], web log analysis [4] and network
analysis [5], etc. Detecting unexpected behavior that deviates from the normal
behavior accurately and efficiently can avoid further damage of core systems
and save huge amount of revenue for business.

Among the various challenges to the said anomaly detection application, the
most important one lies in the nature of the multi-variate time series
themselves. For example, anomalous events are the mutual effect of multiple
time series, such that looking at each one separately can result in poor
quality detection [6]. Intra-series patterns are far from trivial as well. The
temporal dynamics, such as shift in frequency and abrupt change-of-trend [7],
need specific modeling to make accurate forecasting based on historical data
[8]. In addition, value changes themselves may be subtle enough to disguise as
normal data, but looking at the same time series in different granularities or
scales tells completely different stories. All of these peculiarities need to
be accounted for designing an effective detector.

(a) Frequency change

(b) Correlation change

(c) Value abrupt change

(d) Value subtle change

Figure 1: Examples of anomaly patterns: (a) temporal change; (b) correlation
change; (c, d) spatial change where (c) corresponds to a dramatic change and
(d) more subtle change in which anomalies accumulates over a time period. The
red region represents where the anomalies happen while data points located
elsewhere are normal.

Most forecast-based models in the past do not explicitly address the above-
mentioned patterns, let alone separately [3, 9, 10, 11, 12], and therefore are
subject to each of the above deficiencies. However, we argue that there is a
way to fix such deficiencies and forecast-based models in general have room
for improvement. In this paper, we propose a modular approach whereby we
create different modules to best characterize each of the three anomalous
traits: inter-series correlation dynamics, intra-series temporal dynamics and
multi-scale spatial dynamics. These modules are trained jointly to provide a
unified inference.

Another key concern in our proposal is the lack of labels for anomalous events
[13]. As anomalies are commonly outnumbered by normal data points, it is
crucial to have a label-agnostic model. Consequently, a noticeable trend in
the domain is pivoting towards unsupervised models, i.e. ones that train on
normal data exclusively, and detect anomalies with the intuition that they
deviate from the patterns of normal events. Our proposal follows suit with
this trend.

The main contributions of this paper are summarized as follows:

  * •

We introduce a Forecast-based Multi-aspect Unsupervised Anomaly Detection
framework (FMUAD) to capture anomalies with different patterns. The model
outperforms other state-of-the-art forecast-based methods by achieving over
17% F1-score improvement on average cross multiple public datasets.

  * •

We design a modular architecture that explicitly addresses different patterns
of anomalies, which makes our framework highly intuitive and explainable. By
learning different modules jointly, our approach not only captures each
individual anomaly pattern but also detects complicated real-world anomalies
with mixed patterns.

  * •

We define a novel loss function that involves the forecasting error as well as
its compactness (or variance). Compactness has proved to be a desired quality
of a feature to control the variance within one class [9]. By controlling the
compactness of normal class, we aim to learn a tighter representation that is
more sensitive to outliers.

##  II Related Work

As Chandola et al.[14] put in his nice words, “Anomalies are patterns in data
that do not conform to a well defined notion of normal behavior.” And such is
the fundamental principle of unsupervised anomaly detectors (UADs) - Instead
of learning to tell apart normal and abnormal events, UADs simply discover
data that deviates from normal. Technically speaking, UADs leverage the
following major characteristics of anomalies: (1) The distribution of
anomalies differs remarkably from the normal data points; (2) Anomalies happen
infrequently and thus form a small set of data points compared with the normal
data.

Following that, the importance of modeling normal behavior goes without
saying. In particular, we need to find an effective way to represent normal
data. Time series, a prime subject for anomaly detection, is often diverse and
complex due to temporal dependency between data points. Some researchers
attempted to leverage a graph structure to model the dynamic change [6, 8, 15,
16, 17, 18]. However, graph construction and modeling adds extra complexity to
the system, which makes it heavy and time costly, and thus are not scalable
for real-world applications. To this day, how to find an universally
satisfactory representation for time series is still under intense discussions
in the field.

Among the vast literature on time series anomaly detection over the past
decade [19, 20, 21, 7, 22, 23, 6], one can divide them into three groups: (1)
proximity-based; (2) reconstruction-based; and (3) forecast-based. Proximity-
based methods quantify object similarities based on a defined distance measure
and detect objects that are far away from the majority as anomalies.
Reconstruction-based methods use the reconstruction error as the core and
assume that anomalies lie in a different manifold from most data points, and
thus cannot be effectively reconstructed from low-dimensional space. Forecast-
based methods assume anomalies incorporate unusual patterns that are hard to
predict based on historical data, and detect anomalies based on the
forecasting error. Our work falls under this category. Below, we present a
brief background, along with some important past work on forecast-based time
series anomaly detection.

In a forecast-based anomaly detector, we prepare the model by fitting it
against the training data, which only contains normal data points. The trained
model is then put forth to make predictions on future “normal” data, to be
compared against the input. We infer the input point as anomalous if its value
deviates from the model’s prediction range. Given such fundamental mechanism,
it easily follows that forecast-based anomaly detectors mainly differentiate
by their forecasting methods. One of the most classic approaches, ARIMA [24]
is a popular statistical analysis model that learns the auto-correlations in
the time series for future value prediction. Other statistics models such as
Holt-Winters[24] and FDA [25] serve the same purpose. While these methods are
efficient, they are sensitive to datasets and model parameter choices. Fine
tuning these models often requires strong assumptions and extensive domain
knowledge about the data. Machine-learning based models attempt to overcome
such limitations. Hierarchical Temporal Memory (HTM) [26] is an unsupervised
sequence memory algorithm to detect anomalies in streaming data. Ding et
al.[27] combined the HTM model and Bayesian network into for a real-time
framework for multi-variate time series anomaly detection. More recently, Long
Short-Term Memory Recurrent Neural Network (LSTM-RNN), an effective deep
learning network that models temporal dynamics for sequential data, has been
widely used for many state-of-the-art works. Hundman et al.[28] proposed LSTM-
NDT, an unsupervised and non-parametric thresholding approach to interpret
prediction generated by an LSTM network. Zong et al.[11] proposed DAGMM to
detect anomalies by jointly optimizing the parameters of a deep neural network
and a Gaussian Mixture Model.

Although the past forecast-based models have proven the category’s potential
in anomaly detection, they were designed to be a capture-all solution for
every possible anomaly, which can be risky due to the drastically difference
between the anomaly patterns. We believe a degree of tailoring in the model to
anomaly patterns can help bring out better performance.

##  III The Proposed Method

We propose an unsupervised, forecast-based, multi-variate time-series anomaly
detection algorithm (FMUAD) to explicitly address different natures of anomaly
patterns - temporal change, spatial change and correlation change, which are
challenging to detect using forecast-based detectors of the past. In this
section, we first formalize our problem and then provide an overview of the
model methodology.

###  III-A Problem Formulation

A multivariate time series is a sequence of multi-dimensional data points

$\mathcal{X}=\\{\bm{x}_{1},\bm{x}_{2},....,\bm{x}_{T}\\},$

where $\bm{x}_{i}\in\mathbb{R}^{m}$ for $i\in\\{1,...,T\\}$, $m$ is the number
of features and $T$ is the number of time steps. The uni-variate setting is a
particular case of the multi-variate one with $m=1$.

The model takes $\mathcal{X}$ as the input training data assuming
$\mathcal{X}$ contains only normal time steps. Anomaly detection refers to
identifying a new unobserved instance $\hat{\bm{x}}_{t},t>T$ based on how
differently it behaves compared to $\mathcal{X}$. Due to the temporal
dependency within the time series and the fact that anomalies usually occur
over a contiguous time period, we do not model each time step independently
but rather consider a window segment. Suppose $W_{t}\in\mathbb{R}^{m\times k}$
is a window with length $k$ at time $t$:

$W_{t}=\\{\bm{x}_{t-k+1},\bm{x}_{t-k+2},...,\bm{x}_{t}\\}.$

We can transform the original time series $\mathcal{X}$ into a sequence of
windows $\bm{W}=\\{W_{k},...,W_{T}\\}$. For each window $W_{t}$, we train the
model to forecast its behavior using the historical time steps that were
already seen. With proper training, the model will be capable of making decent
predictions for normal data. Then given a new unseen window
$\tilde{W}_{t},t>T$, our goal for the model is to detect whether an anomaly
happens at time $t$ based on the forecast error: the larger the error is, the
more likely an anomaly happens.

Table I: Notations $m\in\mathbb{Z}$ | Number of time series (features)
---|---
$T\in\mathbb{Z}$ | Number of total time steps
$k\in\mathbb{Z}$ | Window size
$\tau\in\mathbb{Z}$ | Input instance time span
$\mathcal{X}\in\mathbb{R}^{m\times T}$ | Input time series
$I_{t}\in\mathbb{R}^{m\times\tau}$ | Individual input instance
$W_{t}\in\mathbb{R}^{m\times k}$ | True window at time $t$
$S_{t}\in\mathbb{R}^{m\times m}$ | True signature matrix (correlation) for $W_{t}$
$F_{t}\in\mathbb{R}^{m\times k/2}$ | True frequency matrix for $W_{t}$
$W^{h}_{t}\in\mathbb{R}^{m\times(\tau-k)}$ | Historical segment for $W_{t}$
$\hat{W}_{t}\in\mathbb{R}^{m\times k}$ | Predicted window at time $t$
$\hat{S}_{t}\in\mathbb{R}^{m\times m}$ | Predicted signature matrix (correlation) for $W_{t}$
$\hat{F}_{t}\in\mathbb{R}^{m\times k/2}$ | Predicted frequency matrix for $W_{t}$

###  III-B Our Method

In this section, we first present the overall architecture of the model and
then break down each module into details.

####  III-B1 Overall Architecture

We illustrate the overall architecture of our model in this part. See Figure 2
for a high-level walk-through and Figure 4 for a more detailed illustration.

To detect anomaly in a target window $W_{t}$, we concatenate the necessary
historical information needed for forecasting, or $W^{h}_{t}$, with $W_{t}$
itself, to formulate the input to our model:

$I_{t}=[W^{h}_{t};W_{t}],$

which takes the form of a $m\times\tau$ matrix:

$I_{t}=\\{\bm{x}_{t-\tau+1},\bm{x}_{t-\tau+2},...,\bm{x}_{t}\\},$

where $m$ is the number of features and $\tau$ is the number of time steps we
trace back for each prediction. In our setting, we choose $\tau=500$ and
$k=30$. The output of our framework, $\hat{y}_{t}\in\\{0,1\\}$ is a boolean
representing whether anomaly happens at time $t$.

Figure 2: High-level system architecture

To holistically characterize and capture the traits of the anomalies within
each window, we consider the time series dynamics from three aspects and
design a corresponding detector tailored to each aspect:

  * –

$D^{c}$, a detector for inter-series correlation change;

  * –

$D^{t}$, a detector for intra-series temporal pattern change;

  * –

$D^{s}$, a detector for multi-scale spatial pattern change.

Because the nature of the detectors are largely dictated by the traits they’re
assigned to capture, each detector should use a different input/output data
representation that best suits the detector. To this regard, we create two
transformed matrices $S_{t}$ and $F_{t}$, and use them along with $W_{t}$ for
the three detectors $D^{c},D^{t},D^{s}$ respectively. The transformation
process TF1: $W_{t}\mapsto S_{t}$ and TF2: $W_{t}\mapsto F_{t}$ will be
discussed in details in part 2) and 3).

  * –

$W_{t}$, the original matrix for the window at $t$;

  * –

$F_{t}$ (frequency matrix), a derived matrix that contains temporal
information for $W_{t}$ in the frequency domain;

  * –

$S_{t}$ (signature matrix), a derived matrix that contains inter-series
correlation information for $W_{t}$.

Figure 3: Concatenation of three matrices

As depicted by Figure 3, $Y_{t}$ is a concatenation of the ground truth
matrices $S_{t},F_{t}$ and $W_{t}$, i.e.,

$Y_{t}=[S_{t};F_{t};W_{t}]=[\text{TF1}(W_{t});\text{TF2}(W_{t});W_{t}].$

Our model outputs $\hat{Y}_{t}=[\hat{S}_{t};\hat{F}_{t};\hat{W}_{t}]$ as the
predicted matrix derived from the concatenation of the forecasting results
obtained from the three detectors. The error between $Y_{t}$ and $\hat{Y}_{t}$
will be served as an indicator of anomalies.

Figure 4: Detailed model architecture

####  III-B2 Inter-series correlation detector

Previous studies [19, 29] suggest the importance of correlation among time
series in the characterization of the dynamic of the multi-variate time series
system for anomaly detection. For instance, the anomaly shown in the red
region of Figure 1(b) is determined based on abnormal correlation: The two
time series have been mostly negative correlated, while in the labelled
pattern they are positively correlated.

To capture the correlation changes, inspired by [19], we define a signature
matrix $S_{t}$ to characterize the correlation between time series. The
transformer TF1 that produces $S_{t}$ is defined as follows: Given a window
$W_{t}$, $S_{t}$ is a $m\times m$ matrix whose entry at the $i$th row and
$j$th column is computed as the Cosine similarity score between the $i$th time
series $\\{x^{i}_{t-k+1},...,x^{i}_{t}\\}$ and the $j$th time series
$\\{x^{j}_{t-k+1},...,x^{j}_{t}\\}$. The reason we adopt Cosine similarity
instead of the inner product as used in [19] is that the former is less prone
to scaling effects, and allows us to focus on the correlation effects only.

Now we transform the input time series into a sequence of signature matrices.
First, we splice the input $I_{t}$ into a sequence of windows with stride $s$.
Then, we apply TF1 to each of the window to form a sequence containing
$d=\lfloor(\tau-k)/s\rfloor$ signature matrices.

${S}_{t}\leftarrow\text{TF1}(W_{t}).$ (1)

To make a forecast on the signature matrix $S_{t}$, we apply ConvLSTM with an
attention mechanism. The final forecasting signature matrix is denoted as
$\hat{S}_{t}\in\mathbb{R}^{m\times m}$. ConvLSTM [30] has been developed and
proven effective to capture the sequential temporal information, as formulated
below:

$\displaystyle i_{t}=$
$\displaystyle\sigma(W_{si}*{S}_{t}+W_{hi}*{H}_{t-1}+W_{ci}\circ{C}_{t-1}+b_{i})$
(2) $\displaystyle f_{t}=$
$\displaystyle\sigma(W_{sf}*{S}_{t}+W_{hf}*{H}_{t-1}+W_{cf}\circ{C}_{t-1}+b_{f})$
$\displaystyle{C}_{t}=$ $\displaystyle
f_{t}\circ{C}_{t-1}+i_{t}\circ\tanh{(W_{sc}*{S}_{t}+W_{hc}*{H}_{t-1}+b_{c})}$
$\displaystyle o_{t}=$
$\displaystyle\sigma(W_{so}*{S}_{t}+W_{ho}*{H}_{t-1}+W_{co}\circ{C}_{t}+b_{o})$
$\displaystyle{H}_{t}=$ $\displaystyle o_{t}\circ\tanh({C}_{t}),$

where $i_{t}$, $f_{t}$, and $o_{t}$ denote the input gate, forget gate, and
output gate, respectively. $C_{t}$ and $H_{t}$ are the cell states and hidden
states. $*$ denotes the convolution operator, $\circ$ is the Hadamard product,
and $\sigma(\cdot)$ is the sigmoid function.

In addition, since not all previous hidden states have equal effects on the
current state, we apply a temporal attention mechanism to adaptively adjust
the weight of the previous steps that are relevant to the current one and
aggregate all to obtain the overall effect [31]. More concretely, we have

${H}^{*}_{t}=\sum_{i=t-k}^{t}c_{i}{H}_{i},\quad
c_{i}=\frac{\exp{\langle\text{vec}(H_{i}),\text{vec}(H_{t})\rangle}}{\sum_{j=t-k}^{t}\exp{\langle\text{vec}(H_{j}),\text{vec}(H_{t})\rangle}},$
(3)

where $\text{vec}(\cdot)$ is the matrix flattened into a vector,
$\langle\cdot,\cdot\rangle$ denotes the inner product, and ${H}^{*}_{t}$ is
the representation for window $t$. Then with an additional convolutional
neural network with a $1\times 1$ kernel, we project the representation
${H}^{*}_{t}$ into a lower dimensional space with only one channel and then
apply a fully-connected layer to derive the prediction of $S_{t}$:

$\hat{S}_{t}\leftarrow\text{FC}(\text{Conv}_{1\times 1}({H}^{*}_{t})).$ (4)

The output serves as the final prediction $\hat{S}_{t}$ containing forecasted
correlation information for window at time $t$.

####  III-B3 Intra-series temporal pattern detector

As are inter-series correlations, the intra-series temporal patterns are also
crucial for anomaly detection. For instance, the anomaly shown in the red
region of Figure 1(a) is due to temporal pattern changes: the frequency
suddenly increases twice in the anomaly region compared to the rest of the
time series. Several recent work [8, 15] has shown the advantages of joint
learning inter-series correlations and intra-series temporal patterns for
multi-variate time series forecasting.

To capture the temporal pattern within each series, we project each uni-
variate time series to the spectral domain via DFT (discrete Fourier
transform):

$\xi_{j}=\frac{1}{k}\sum\limits_{\ell=0}^{k-1}x_{l}e^{2\pi
ij\ell/k},j=1,2,...,k,$ (5)

where $i$ is the imaginary unit. The DFT transforms $k$ discrete-time samples
$\vec{x}=\\{x_{1},x_{2},...,x_{k}\\}$ in the time domain to the frequency
domain $\vec{\xi}=\\{\xi_{1},\xi_{2},...,\xi_{k}\\}$. Due to the symmetry
property of DFT for real-valued discrete-time signals: $\xi_{j}=\xi_{k-j}$,
only half of the $\xi$’s need to be stored. As a result, the window $W_{t}$
can be transformed to a frequency matrix
$F_{t}=\\{\xi_{1},...,\xi_{k/2}\\}\in\mathbb{R}^{m\times\frac{k}{2}}$
representing the amplitude of frequencies for the $m$ uni-variate time series
in the spectral domain. Such process defines the transformer TF2.

${F}_{t}\leftarrow\text{TF2}(W_{t}).$ (6)

Similar to how the inter-series correlation was constructed in part 2), we
transform the input time series into a sequence of frequency matrices, then
apply ConvLSTM with an attention mechanism to obtain a high-dimensional hidden
representation. Lastly we pass an additional neural network with a $1\times 1$
kernel to project it to a low-dimensional space and apply a fully-connected
layer:

$\hat{F}_{t}\leftarrow\text{FC}(\text{Conv}_{1\times
1}(\text{Attn}(\text{ConvLSTM}(F_{t^{-}})))).$ (7)

Here, $\text{ConvLSTM}(\cdot)$ refers to Eq. (2), $\text{Attn}(\cdot)$ refers
to Eq. (3), and $F_{t^{-}}$ refers to the sequence of frequency matrices
before time $t$. The forecast $\hat{F}_{t}$ captures the temporal information
from each time series within window $W_{t}$.

####  III-B4 Multi-scale spatial pattern detector

Another pattern of anomaly, sometimes overlooked but extremely common in time
series is the spacial dynamic, or in layman’s term, “value change”. For
instance, the red region of Figure 1(c) shows a dramatic value change, which
is trivial to capture. However, the majority of spatial changes are gradual
and subtle, and hard to detect like the red region in Figure 1(d). The value
change at each individual time step is so small that they are hardly
discernible to the human eye, but when accumulated over a time period, the
deviation is significant. Thus, it is important for our anomaly detector to
capture both types of spacial dynamics.

The intuition is natural: Aggregating the time series in Figure 1(d) at a
larger time scale easily manifests the subtle changes as a spike, thus
reducing the problem to a trivial one. In fact, looking at time series at
different levels of aggregation can tell completely different stories. To
capture multi-scale patterns, one needs to consider relatively long-term
dynamics, which is why we take $W^{h}_{t}\in\mathbb{R}^{m\times(\tau-k)}$ to
model instead of taking individual windows as we did before. In our model, we
adopt the Dilated Convolutional Neural Network [32] for this purpose. The
major difference between dilated CNN and basic CNN is that it introduces
dilated convolution operator to capture information aggregated at multiple
scales. Suppose $F$ is a discrete function and $k$ is the convolution kernel,
we can define $\ell$ as a dilation factor and $*_{\ell}$ as a dilated
convolution operator

$(F*_{\ell}k)(p)=\sum\limits_{s+\ell t=p}F(s)k(t).$ (8)

With different dilation sizes $\ell$, multi-scale information can be obtained
without losing resolution.

In our setting specifically, we firstly apply a 3-layer Dilated-CNN with
dilating numbers $r=1,3,5$ and respective channels numbers $c=32,64,128$ to
obtain a deep hidden representation. In particular, we apply a rectangle
kernel $(3,1)$ for each layer instead of an usual square kernel $(3,3)$ to
limit the multi-scale aggregation to over the time dimension rather than the
feature dimension. Then we decode the hidden representation to forecast the
target window $W_{t}$. The final forecasting matrix is denoted as
$\hat{W}_{t}\in\mathbb{R}^{m\times k}$:

$\hat{W}_{t}\leftarrow\text{FC}(\text{Conv}_{1\times 1}(\text{Dilated-
CNN}(W^{h}_{t}))).$ (9)

Method | SMD | MSL | Total
---|---|---|---
Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1
LSTM-NDT | 0.5684 | 0.6438 | 0.6037 | 0.5934 | 0.5374 | 0.5640 | 0.5809 | 0.5906 | 0.5839
DAGMM | 0.5951 | 0.8782 | 0.7094 | 0.5412 | 0.9934 | 0.7007 | 0.5681 | 0.9358 | 0.7051
LSTM-VAE | 0.7922 | 0.7075 | 0.7842 | 0.5257 | 0.9546 | 0.6780 | 0.6590 | 0.8311 | 0.7311
FMUAD | 0.8574 | 0.8789 | 0.8680 | 0.8311 | 0.8786 | 0.8539 | 0.8443 | 0.8664 | 0.8610
Table II: Performance of FMUAD and 3 baseline approaches.

####  III-B5 Loss function

In this section, we introduce a novel loss function based on the forecasting
error as well as its variance.

Following steps 2)-4), for a given target window $W_{t}$, we have derived the
ground truth matrix $Y_{t}=[S_{t};F_{t};W_{t}]$ and the forecasted matrix
$\hat{Y}=[\hat{S}_{t};\hat{F}_{t};\hat{W}_{t}]$ as shown in Figure 4. The
forecasting error is the $L_{2}$ norm between $Y_{t}$ and $\hat{Y}_{t}$. As we
train our neural network in batches of size $b$, the batch loss is averaged
over the batch as shown in Eq. (10). The total loss is further averaged across
the batches.

$\ell_{1}=\frac{1}{b}\sum_{i=1}^{b}\left\lVert
Y_{i}-\hat{Y}_{i}\right\rVert^{2}_{2}.$ (10)

Besides forecasting error, we introduce an extra term we call the compactness
loss to capture the variance of the forecasting error within each batch, which
is defined in Eq. (11).

$\ell_{2}=\frac{1}{nb}\sum_{i=1}^{b}z_{i}^{T}z_{i},\
z_{i}=\hat{Y}_{i}-\frac{1}{b-1}\sum_{j\neq i}\hat{Y}_{j},$ (11)

where $n=m+k+\frac{k}{2}$ is the number of columns of $\hat{Y}_{i}$. Due to
the fact that we are training on a single class (normal instances), the
variance control $\ell_{2}$ helps to obtain a compact representation that is
more sensitive to anomalies [9, 33].

The final loss is computed based on $\ell_{1}$ and $\ell_{2}$ via a function
$f$ as defined below:

$\ell=f(\ell_{1},\ell_{2})=(\epsilon+\ell_{2})\cdot\ell_{1}.$ (12)

Here, we take a small positive constant $\epsilon$ (In our experiment, we set
$\epsilon=10^{-5}$) to avoid a zero trivial solution: A trivial network with
all zero parameters gives zero loss value $\ell$ if $\epsilon=0$. We adopt
this loss function instead of a weighted sum of $\ell_{1}$ and $\ell_{2}$ due
to two reasons. First, this eliminates the need to fine-tune the weights,
making the training process much more efficient. Second, the effect each term
adds on scales with its order of change. In other words, if the mean and
variance of the forecasting error have the same order of changes, then these
changes are emphasized to the same level no matter how different their scales
are. To the best of our knowledge, this is the first work that incorporates
the product of the error mean and variance in the loss function.

Our experiments show that by considering the variance factor $\ell_{2}$, the
model performance can be consistently improved. In addition, While we consider
both the forecasting error and its variance in training, we only use the
forecasting error $\ell_{1}=\|Y_{t}-\hat{Y}_{t}\|^{2}_{2}$ as the testing
loss. The testing loss serves as the anomaly score to make the judgement: the
larger the testing loss is, the more likely an anomaly has occurred.

Figure 5: Case studies - (a1, b1): anomaly detection for MSL; (a2, b2):
anomaly detection for SMD. (a1, a2) corresponds to the original time series
and (b1, b2) corresponds to the anomaly scores output by our framework.

##  IV Experiments

In this section, we perform extensive experiments to validate the
effectiveness of our framework. We first describe the datasets that the
framework tests on, then introduce three baseline models and the performance
metrics, and make a comprehensive comparison between our model the baselines.
Detailed case study and ablation study is presented at the end.

###  IV-A Datasets

To intuitively illustrate the inner workings of our model and evaluate its
performance, we adopt two highly regarded and commonly used public multi-
variate time-series datasets:
SMD111https://github.com/NetManAIOps/OmniAnomaly/tree/master/ServerMachineDataset
(Server Machine Dataset) [20] and MSL222https://s3-us-
west-2.amazonaws.com/telemanom/data.zip (Mars Science Laboratory rover) [28].
SMD is 5 weeks worth of server telemetry collected from 28 production server
nodes of a reputable Internet company. It contains 38 simultaneously collected
floating-point metrics, with anomaly events labelled in a binary format and
the anomaly ratio is 4.16%. The data is segmented among the 28 servers into 3
groups. We train and test on each server machine separately. The total
training set for SMD contains 708405 data points while the testing set
contains 708420. The MSL dataset captures telemetry data throughout the Mars
mission duration of the MSL rover, better known as Curiosity. It contains data
for 55 entities each monitored by 25 metrics, with anomalies expertly labelled
by NASA and the anomaly ratio is 10.72%. The training set size for MSL is
58317 while the testing size is 73729. Both datasets depict highly real-world
scenarios in the target applications of our model, and therefore can provide
an accurate benchmark for our model.

Dataset Name | # Features | Feature Name |  | Anomaly
---
Ratio(%)
SMD | group 1 | 38 |  CPU load, network usage, memory usage, etc. | 4.16
group 2
group 3
MSL | 55 |  | Telemetry data:
---
radiation, power,
temperature,
computational activity, etc
10.72
Table III: Dataset descriptions

###  IV-B Comparison with Baselines

We compare our model with three state-of-the-art anomaly detection models:
LSTM-NDT [28], DAGMM [11] and LSTM-VAE [34]. The first two models are
forecast-based, which falls in the same scope as our model. The last one is
also included here for comparison is due to the fact that they also adopted
LSTM as a substantial part in their design to capture temporal dynamics. To
ensure a fair comparison, we use the same set of evaluation metrics as the
baselines: precision (P), recall (R) and F1 score (F1):

$P=\frac{TP}{TP+FP},\ R=\frac{TP}{TP+FN},\ F1=2\cdot\frac{P\cdot R}{P+R},$

where the F1-score is picked as the highest one via anomaly threshold
selection. We also follow the same evaluation strategy by treating the whole
window segment as correct if any observation in this segment is correctly
detected as an anomaly [20]. Table II details the obtained performance results
for our model as well as the three baselines. As one can see, our model keeps
a great balance between precision and recall while the baselines tend to favor
one of the metrics, and thus resulting to the best F1-score consistently over
the baselines. On average, we achieve 17.8% improvement over the best state-
of-the-art performance.

###  IV-C Case Study

Besides the performance gains FMUAD brings to the stage, another significant
benefit of our model is its intuitiveness. In this section, we take a deep
dive in two particular data samples FMUAD labelled as anomalous along with the
model’s output, to illustrate the detection mechanism.

Shown in Figure 5 are two cases: the left one is taken from MSL and the right
one from SMD. In both cases, we sample a time period out of the time series
containing anomalous labels, and the top chart (a1, a2) displays a hand-picked
set of 6 metrics containing interesting and indicative patterns. Portions of
the time series labelled as anomalies are highlighted in red, and the forecast
error (anomaly score) output from the model is displayed in the lower chart
(b1, b2).

In the first case study (a1, b1, MSL dataset), our model accurately made a
capture on a temporal frequency pattern change. While the value range of the
displayed metrics did not change in the anomaly event, metric 3 shows outputs
a less-concentrated cluster of spikes, and metric 4 shows significantly more
frequent ramps and fluctuations. While these are subtle patterns to human
eyes, FMUAD correctly outputs a high anomaly score, indicating a high
confidence in the anomaly judgement which matches the provided label.

In the second case study (a2, b2, SMD dataset), we demonstrate the model’s
capabilities of capturing value changes. The sample presented in (a2) contains
two drastically different anomaly patterns: The anomaly on the left contains a
short-lived, sparing spikes on metric channel 1. It is a prime example of a
point anomaly. In contrast, the right anomaly has a longer duration, involves
more metrics and contains a more subtle and sustained value increase. While it
is easy to notice by humans, the statistical significance of the spatial
pattern is only highly pronounced when aggregated over the entire period.
Again, by design, FMUAD is able to detect both types of anomalies, evidently
through a high forecasting error output.

Through the two case studies above, one can easily appreciate the
intuitiveness of FMUAD. The model provides explainable and predictable
outputs, opening up opportunities for further improvements and targeted fine-
tuning based on the patterns of the target dataset. However, another major
strength of the model remains to be addressed - its modularity. While we now
have empirical evidence that FMUAD can pick up distinct patterns of anomaly,
one naturally wonders if and how well the modularity provides a constructive
impact, and whether each module indeed captures its target patterns.

Figure 6: Anomaly patterns captured by different detectors.

###  IV-D Ablation Study

To address the curiosity over the model’s modularity, we perform an ablation
study to analyze the effect of each detector module to the overall performance
of the framework. To recap the model architecture, we use three separate and
complementary forecast-based detector modules, each dedicated at one target
pattern by design. Specifically, $D^{c}$ captures inter-series correlation,
$D^{t}$ captures intra-series temporal patterns and $D^{s}$ captures multi-
scale intra-series spatial patterns. We run the same benchmark experiments on
all 28 machines of the SMD with only one of the three detectors enabled at a
time, and compare the detection performance when that particular detector is
present (and more importantly, when the other two are not present).

As with the case study, we sample a few time series each detector proves to
perform well on, and present their plots in Figure 6. Table IV presents the
F1-score for each of the detectors on each of the samples shown. The result
again meets the expectation. In sample (a) where $D^{c}$ excels, we can see a
clear change in the correlation: While in normal situations the two metrics
are positively correlated, the labelled anomaly event depicts a negative
correlation between the two time series. This is an instance where only
$D^{c}$ is triggered, but not $D^{s}$ or $D^{t}$. Samples (b) and (c) general
picture the same situation. In (b), a temporal change can be observed as the
metric breaks out of its usual periodicity and manifests several sudden noisy,
sudden peaks. These patterns are easily detected by $D^{t}$ but not the
others. Finally, in (c), the metric takes a sharp point-anomaly on the left,
and a more subtle but longer duration deviation on the right. Both deviations
are registered in $D^{s}$, showing its strength in multi-scale spatial change
detection.

Detector | Machine 1-1 | Machine 1-2 | Machine 2-3
---|---|---|---
$D^{c}$ | 0.6921 | 0.6976 | 0.0332
$D^{t}$ | 0.9984 | 0.5229 | 0.8968
$D^{s}$ | 0.9680 | 0.5884 | 0.9122
Table IV: Performance comparison (F1-score) among three detectors for
representative machines from SMD.

Of course, showing detection behavior on archetypal samples does not do
justice to the modules’ complementary effectiveness. As each detector works
well for some cases, one may wonder whether the performance becomes diverse
and unstable across datasets when enabling all three detectors. In the
following Table V, we present the model performance on each dataset when
individual detectors are enabled alone, and when all three are in use. It is
evident that combining the three detector modules provides a consistent best
performance. More importantly, such combination makes our model performance so
stable that only a 0.0001 variance in terms of F1-score remains cross the four
set of datasets.

| SMD | MSL | Mean | Variance
---|---|---|---|---
group 1 | group 2 | group 3
$D^{c}$ | 0.619 | 0.859 | 0.646 | 0.191 | 0.579 | 0.0588
$D^{t}$ | 0.586 | 0.866 | 0.731 | 0.851 | 0.759 | 0.0127
$D^{s}$ | 0.475 | 0.723 | 0.560 | 0.323 | 0.520 | 0.0209
FMUAD | 0.860 | 0.883 | 0.861 | 0.854 | 0.865 | 0.0001
Table V: Module VS All - the overall framework outperforms each individual
module consistently and exhibits excellent stability with only 0.0001 variance
across different datasets. Figure 7: Model performance (F-1 score): Module VS
All.

Besides the modular architecture, a novelty we would like to bring to
attention is the batch variance term $\ell_{2}$ in the loss function. We
incorporate this loss in the training time in hopes to learn a more stable and
compact representation, therefore further improving the quality of anomaly
detection. To test the extent of this improvement brought by $\ell_{2}$, we
repeat the benchmark on each dataset but without the variance term in the loss
function Eq. (12). The performance numbers are detailed in Table VI. Based on
the numbers, it can be concluded that the variance control term has resulted
in a consistent improvement in performance over when only $\ell_{1}$ is
involved in training, albeit by a little.

| SMD | MSL
---|---|---
group 1 | group 2 | group 3
| FMUAD w.o.
---
loss variance control
0.845 | 0.877 | 0.846 | 0.849
FMUAD | 0.860 | 0.883 | 0.861 | 0.854
Table VI: Model performance (F1-score): Loss VS Loss without variance control.

##  V Conclusion

In this paper, we discuss a Forecast-based Multi-aspect framework for
Unsupervised multi-variate time series Anomaly Detection, or FMUAD in short.
It is a bold attempt at this category of anomaly detection models by
identifying the traits of anomaly patterns and categorizing them for a divide-
and-conquer approach. We introduce a modular framework where the input time
series is transformed into an intermediate representation better capturing
each target pattern, and feeding it through each detector module that makes
its own forecast on the said intermediate representation. We also experiment
with a novel loss function that promotes compactness in the learned
representation during training. To wrap them all up, we apply the loss
function to compute the forecasting error, and thus produce an anomaly score.

We prove that the three detectors can indeed learn jointly for the optimal
anomaly detection by contrasting its performance against state-of-the-art
techniques on public reference datasets. The standard F-1 score from FMUAD is
indeed higher than other models of its class. We also relate the detection
mechanisms back to human intuition and find proof of each detector module
performing to its advantage by agreeing to the anomaly patterns we assign them
to detect. Furthermore, we investigate the role of modularity through an
ablative study, and find that the combination of all three modules can achieve
consistently higher and more stable performance than each one alone. We also
prove that the compactness-focused $l_{2}$ loss term results in consistent
improvement, albeit being minute compared to other improvements.

A recent trend we observe in the machine learning community is the tendency to
move towards “end-to-end” or “one-size-fits-all” models, as a defiant
departure from the old-school feature-engineering approaches. However, FMUAD
may serve as a reminder that distinct patterns existing in the input data may
still need a degree of tailored model design, all while being a self-adaptive
model that remains end-to-end trainable.

This is clearly not the end of our journey, as the flexibility of FMUAD opens
up a world of opportunities for further enhancements. For one, we can continue
to discover new types of anomaly pattern and tailor a corresponding detector
to its characteristics so those patterns are effectively captured; for two,
there may exist better implementations for detectors $D^{c}$, $D^{t}$ and
$D^{s}$ that can allow individual modules to perform even more optimally.
Finally, the three detectors in FMUAD have operated in complete isolation of
each other and the final aggregator. Enabling cross-interactions between these
modules may also result in interesting observations that are worth
investigating. All in all, we hope our model has availed additional potential
for forecast-based anomaly detection models, and it can start a discussion to
draw further work onto this general direction.

## References

  * [1] J. Wei, J. Zhao, Y. Zhao, and Z. Zhao, “Unsupervised anomaly detection for traffic surveillance based on background modeling,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 129–136, 2018.
  * [2] M. Ahmed, A. N. Mahmood, and M. R. Islam, “A survey of anomaly detection techniques in financial domain,” Future Generation Computer Systems, vol. 55, pp. 278–288, 2016.
  * [3] P. Ghuli, “Anomaly detection in log records,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 10, p. 343, Apr. 2018.
  * [4] Q. Cao, Y. Qiao, and Z. Lyu, “Machine learning to detect anomalies in web log analysis,” in 2017 3rd IEEE International Conference on Computer and Communications (ICCC), pp. 519–523, 2017.
  * [5] K. Ding, J. Li, and H. Liu, “Interactive anomaly detection on attributed networks,” in Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 357–365, 2019.
  * [6] Z. Hang, W. Yujing, D. Juanyong, H. Congrui, C. Defu, T. Yunhai, X. Bixiong, B. Jing, T. Jie, and Z. Qi, “Multivariate time-series anomaly detection via graph attention network,” 09 2020.
  * [7] J. Gao, X. Song, Q. Wen, P. Wang, L. Sun, and H. Xu, “Robusttad: Robust time series anomaly detection via decomposition and convolutional neural networks,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, Aug. 2020.
  * [8] D. Cao, Y. Wang, J. Duan, C. Zhang, X. Zhu, C. Huang, Y. Tong, B. Xu, J. Bai, J. Tong, and Q. Zhang, “Spectral temporal graph neural network for multivariate time-series forecasting,” 2021.
  * [9] P. Perera, P. Oza, and V. M. Patel, “One-class classification: A survey,” CoRR, vol. abs/2101.03064, 2021.
  * [10] W. Wu, L. He, W. Lin, Y. Su, Y. Cui, C. Maple, and S. A. Jarvis, “Developing an unsupervised real-time anomaly detection scheme for time series with multi-seasonality,” IEEE Transactions on Knowledge and Data Engineering, pp. 1–1, 2020.
  * [11] B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho, and H. Chen, “Deep autoencoding gaussian mixture model for unsupervised anomaly detection,” in International Conference on Learning Representations, 2018.
  * [12] Y. Guo and F. Farooq, “Proceedings of the 24th acm sigkdd international conference on knowledge discovery and data mining,” KDD’18, (New York, NY, USA), Association for Computing Machinery, 2018.
  * [13] R. Chalapathy and S. Chawla, “Deep learning for anomaly detection: A survey,” arXiv preprint arXiv:1901.03407, 2019.
  * [14] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection,” ACM Computing Surveys, vol. 41, pp. 1–58, July 2009.
  * [15] G. Spadon, S. Hong, B. Brandoli, S. Matwin, J. F. Rodrigues-Jr, and J. Sun, “Pay attention to evolution: Time series forecasting with deep graph-evolution learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021.
  * [16] Y. Wu, M. Gu, L. Wang, Y. Lin, F. Wang, and H. Yang, “Event2graph: Event-driven bipartite graph for multivariate time-series anomaly detection,” CoRR, vol. abs/2108.06783, 2021.
  * [17] H. Chen, L. Wang, Y. Lin, C.-C. M. Yeh, F. Wang, and H. Yang, “Structured graph convolutional networks with stochastic masks for recommender systems,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, (New York, NY, USA), p. 614–623, Association for Computing Machinery, 2021.
  * [18] H. Chen and J. Li, “Exploiting structural and temporal evolution in dynamic link prediction,” in Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 427–436, 2018.
  * [19] C. Zhang, D. Song, Y. Chen, X. Feng, C. Lumezanu, W. Cheng, J. Ni, B. Zong, H. Chen, and N. V. Chawla, “A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1409–1416, July 2019.
  * [20] Y. Su, Y. Zhao, C. Niu, R. Liu, W. Sun, and D. Pei, “Robust anomaly detection for multivariate time series through stochastic recurrent neural network,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, July 2019.
  * [21] J. Audibert, P. Michiardi, F. Guyard, S. Marti, and M. A. Zuluaga, “USAD: UnSupervised Anomaly Detection on Multivariate Time Series,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, July 2020.
  * [22] T. Zhao, B. Ni, W. Yu, and M. Jiang, “Early anomaly detection by learning and forecasting behavior,” ACM, 2021.
  * [23] M.-C. Lee, J.-C. Lin, and E. G. Gran, “Salad: Self-adaptive lightweight anomaly detection for real-time recurrent time series,” IEEE, 2021.
  * [24] E. H. M. Pena, M. V. O. de Assis, and M. L. Proença, “Anomaly detection using forecasting methods arima and hwds,” in 2013 32nd International Conference of the Chilean Computer Science Society (SCCC), pp. 63–66, 2013.
  * [25] J. M. Torres, P. G. Nieto, L. Alejano, and A. Reyes, “Detection of outliers in gas emissions from urban areas using functional data analysis,” Journal of Hazardous Materials, vol. 186, pp. 144–149, Feb. 2011.
  * [26] S. Ahmad, A. Lavin, S. Purdy, and Z. Agha, “Unsupervised real-time anomaly detection for streaming data,” Neurocomputing, vol. 262, pp. 134–147, Nov. 2017.
  * [27] N. Ding, H. Gao, H. Bu, H. Ma, and H. Si, “Multivariate-time-series-driven real-time anomaly detection based on bayesian network,” Sensors, vol. 18, p. 3367, Oct. 2018.
  * [28] K. Hundman, V. Constantinou, C. Laporte, I. Colwell, and T. Soderstrom, “Detecting spacecraft anomalies using LSTMs and nonparametric dynamic thresholding,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, July 2018.
  * [29] D. Song, N. Xia, W. Cheng, H. Chen, and D. Tao, “Deep r -th root of rank supervised joint binary embedding for multivariate time series retrieval,” KDD’18, (New York, NY, USA), p. 2229–2238, Association for Computing Machinery, 2018.
  * [30] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-k. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, (Cambridge, MA, USA), p. 802–810, MIT Press, 2015.
  * [31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017.
  * [32] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” CoRR, vol. abs/1511.07122, 2016.
  * [33] P. Perera and V. M. Patel, “Learning deep features for one-class classification,” IEEE Transactions on Image Processing, vol. 28, pp. 5450–5463, Nov. 2019.
  * [34] D. Park, Y. Hoshi, and C. C. Kemp, “A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder,” CoRR, vol. abs/1711.00614, 2017.