# Weakly Augmented Variational Autoencoder in Time Series Anomaly Detection

Zhangkai Wu, Longbing Cao The work is partially sponsored by Australian
Research Council Discovery and Future Fellowship grants (DP190101079 and
FT190100734). Zhangkai Wu, Junxian Zhou with the School of Computer Science,
the University of Technology Sydney, 15 Broadway, Ultimo 2007, NSW, Australia.
(E-mail<EMAIL_ADDRESS>junxian.zhou@student.uts.edu.au)Hui Chen and
Longbing Cao are with the DataX Research Centre and School of Computing,
Macquarie University, NSW 2109, Australia. (email:
<EMAIL_ADDRESS>longbing.cao@mq.edu.au)Qi Zhang is with the
Department of Computer Science, Tongji University, Shanghai 201804, China.
(E-mail<EMAIL_ADDRESS>
###### Abstract

Due to their unsupervised training and uncertainty estimation, deep
Variational Autoencoders (VAEs) have become powerful tools for reconstruction-
based Time Series Anomaly Detection (TSAD). Existing VAE-based TSAD methods,
either statistical or deep, tune meta-priors to estimate the likelihood
probability for effectively capturing spatiotemporal dependencies in the data.
However, these methods confront the challenge of inherent data scarcity, which
is often the case in anomaly detection tasks. Such scarcity easily leads to
latent holes, discontinuous regions in latent space, resulting in non-robust
reconstructions on these discontinuous spaces. We propose a novel generative
framework that combines VAEs with self-supervised learning (SSL) to address
this issue. Our framework augments latent representation to mitigate the
disruptions caused by anomalies in the low-dimensional space, aiming to lead
to corresponding more robust reconstructions for detection.

This framework marks a significant advancement in VAE design by integrating
SSL to refine likelihood enhancement. Our proposed VAE model, specifically
tailored for TSAD, augments latent representations via enhanced training,
increasing robustness to normal data likelihoods and improving sensitivity to
anomalies. Additionally, we present a practical implementation of this
conceptual framework called the Weakly Augmented Variational Autoencoder
(WAVAE), which directly augments input data to enrich the latent
representation. This approach synchronizes the training of both augmented and
raw models and aligns their convergence in data likelihood optimization space.
To achieve this, we maximize mutual information within the Evidence Lower
Bound (ELBO), utilizing contrastive learning for shallow learning and a
discriminator-based adversarial strategy for deep learning.

Extensive empirical experiments on five public synthetic and real datasets
validate the efficacy of our framework. These experiments provide compelling
evidence of the superior performance of our approach in TSAD, as demonstrated
by achieving higher ROC-AUC and PR-AUC scores compared to state-of-the-art
models. Furthermore, we delve into the nuances of VAE model design and time
series preprocessing, offering comprehensive ablation studies to examine the
sensitivity of various modules and hyperparameters in deep optimization.

###### Index Terms:

Variational Autoencoder, Time Series Anomaly Detection, Self Supervised
Learning, Data Augmentation, Contrast Learning, Adversarial Learning.

##  I Introduction

Deep probabilistic generative models have revolutionized unsupervised data
generation by leveraging the neural network’s universal approximation theorem.
This innovation manifests in various forms, such as encoding-decoding
mechanisms [1], diffusion-denoise processes [2], and sender-receiver in
compression [3]. Notably, VAEs have emerged as a central focus in this domain.
Deep VAEs empowered by large-scale neural networks [4] and the organization of
semantic representations [5, 6] have demonstrated exceptional capabilities in
reconstructing and generating multimodal data, including images [7, 8],
tabular [9, 10], and time series [11, 12]. Owing to their ability to learn
generative factors in continuous and smooth low-dimensional space while
fitting data likelihoods, VAEs exhibit robust representation learning
capabilities in disentanglement [13, 14], classification [15], and clustering
[16]. In particular, VAEs are designed to estimate likelihood distribution
learned from most normal samples as a detector, providing an unsupervised and
interpretable paradigm for anomaly detection. The underlying assumption is
that unknown anomaly patterns typically exhibit statistical characteristics
that deviate significantly from the normal distribution.

Recent research has shown a growing preference for VAEs in Time Series Anomaly
Detection (TSAD), particularly those that integrate meta-priors[17] into their
design. These methods have been validated to be effective and crucial in
capturing the spatiotemporal dependencies within data, thereby enhancing data
likelihood modeling. Specifically, these models often assume certain meta-
priors, e.g., latent structures, which are crafted using deep learning or
probabilistic tools. The goal is to accurately represent the likelihood of
most data points, enabling the models to detect anomalies effectively. For
instance, time-varying priors that adapt to dynamic assumptions [18, 19, 20,
21] have demonstrated effectiveness and powerful capabilities in capturing
sequence data likelihoods. Additionally, other studies [22, 23] have proposed
the creation of task-specific priors based on a factorized assumption
explicitly designed to model contextual dependence structure in latent space.
Various strategies have been employed to achieve this, such as using prototype
distribution-based representations optimized through meta-learning and
decomposing contextual representations.

Whether meta-priors are integrated implicitly or explicitly, these methods are
fundamentally rooted in model-based designs. However, they often cater to
specific scenarios and face training challenges stemming from the data
scarcity issue in deep learning or statistical estimation techniques.
Additionally, these methods overlook the effective utilization of data, which
becomes particularly problematic when modeling small sequence datasets in
real-world time-series TSAD scenarios. The issue in VAE-based models easily
leads to latent holes [24], i.e., discontinuous regions in latent space,
resulting in non-robust reconstruction [25, 26, 27, 28, 21]. The latent holes
issue occurs when the encoder in these models maps unknown anomalies into the
latent space without being adequately corrected by enough normal data. Since
these anomalies lack the spatiotemporal properties of normal time series data,
they disrupt the formation of a continuous and smooth latent space for normal
samples. Consequently, representations sampled from these latent holes fail to
accurately reconstruct the input samples, causing a discrepancy between the
representations and the reconstructed data. This mismatch significantly
impairs the anomaly detector’s performance and compromises the model’s overall
robustness. For a more intuitive understanding of this phenomenon, refer to
Fig. 1.

Figure 1:  Comparison of the latent hole phenomenon induced by anomalies in
Nonrobust VAE-based TSAD Models (upper section) with the robust representation
learning space fostered by WAVAE (lower section). The upper part of the figure
delineates the rise of the latent hole within the Nonrobust TSAD model and its
effect on model robustness. Specifically, anomalous sequences
$\boldsymbol{x}_{t}^{r}$ (depicted within the blue window in the upper
section), when encoded into the representation space, disrupt the structural
integrity of the latent space. This disruption results in latent hole
primarily because these anomalous sequences $\boldsymbol{x}_{t}^{r}$ lack the
spatiotemporal coherence inherent in the normal sequence
$\boldsymbol{x}_{1}^{r}$. Consequently, sampling from these discontinuous
regions leads to a mismatch between the representation (indicated by the blue
dot $\boldsymbol{z}_{t}$) and its generation (also shown by the blue dot in
the likelihood function), as illustrated in the upper section, a
disproportionately high likelihood function mass characterizes the
representation in the latent space. In such scenarios, the TSAD model may
erroneously classify an anomaly as normal, compromising its robustness. In
contrast, the lower section demonstrates how data augmentation via the WAVAE
model can engender a more continuous and smoothly distributed data likelihood
(as depicted in the central part of the bottom figure). In this context,
representation $\boldsymbol{z}_{t}$) encoded by anomalous sequences
$\boldsymbol{x}_{t}^{a}$ sampled from regions outside the normal latent space
are associated with a lower likelihood function mass, thereby enhancing the
robustness and efficacy of anomaly detection in TSAD tasks.

In light of these challenges, we propose to improve data utilization using
self-supervised learning (SSL) to enhance representation learning and induce
latent space robustness. SSL [29] enables models to extract more informative
representations from unlabeled data, leading to sufficient training. To
achieve this, we employ data augmentation on unlabeled datasets through SSL
strategies, facilitating the training of models through contrastive or
adversarial methods for the TSAD task. Our contributions can be summarized as:

  * •

Generative self-supervised learning framework for TSAD: We present an enhanced
generative framework using self-supervised learning. We define a likelihood
function for learning and the derivation of a surrogate error for
optimization. This novel approach sets the stage for more effective model
design in VAE-based TSAD.

  * •

Deep and shallow learning in augmented models: Building upon this framework,
we implement the Weakly Augmented Variational Autoencoder (WAVA) that
incorporates data augmentation, enabling the model to undergo thorough
training with support from augmented counterparts. We have also devised two
distinct learning approaches, deep and shallow, to integrate these two models
effectively.

  * •

State-of-the-art performance: Extensive experiments on five public datasets
demonstrate the effectiveness of our approach. We achieved superior
performance in ROC-AUC and PR-AUC, surpassing state-of-the-art models.
Additionally, we provide comprehensive ablation studies delving into the
design of the VAE model, time series preprocessing, and sensitivity analysis
on different modules and hyperparameters in deep optimization.

Figure 2: Graphical Model for Augmented Variational Autoencoders. Under the
plate notation rules, a white circle denotes a hidden (or latent) variable,
while a gray circle signifies an observed variable. The variables contained
within the square denote local variables, which are independently repeated $N$
times. Dashed arrow edges imply conditional dependence. Dotted lines represent
parameters. Referring to the plate diagram, it is evident that our methodology
encompasses the utilization of two generative models. The inference part of
models, i.e., $q_{\phi_{\mathrm{r}}}$ and $q_{\phi_{\mathrm{a}}}$, encode the
raw input, denoted as $\boldsymbol{x}_{\mathrm{r}}$, and the augmented input,
$\boldsymbol{x}_{\mathrm{a}}$, into their respective low-dimensional
representations, $\boldsymbol{z}_{\mathrm{r}}$ and
$\boldsymbol{z}_{\mathrm{a}}$. Subsequently, the generative parts of models
$p_{\theta_{\mathrm{r}}}$ and $p_{\theta_{\mathrm{a}}}$, sample the latent
space reconstruct the input samples, respectively. We employ a $\psi$
parameterized module to synchronize the learning outcomes of both models.

##  II Backgroud

###  II-A Generative model-based Time Series Anomaly Detection

####  II-A1 Implicit Data Fitting by Non-probabilistic Generative Models

Non-probabilistic generative model-based algorithms for TSAD aim to
reconstruct data robustly. Prior works have concentrated on optimizing this
reconstructing process to match the characteristics of time-series data via
the design of deep network embeddings. Specifically, [30] and [31] implemented
an Autoencoder (AE) framework, deploying symmetric encoder-decoder structures
and assembling one or multiple CNN-based encoder-decoders for the
reconstruction of sequence data. Furthermore, [32] utilized a Generative
Adversarial Network (GAN)-based reconstruction for anomaly detection,
implicitly fitting a likelihood function based on the normal data through an
adversarial mechanism.

####  II-A2 Explicit Data Fitting by Probabilistic generative Models

Unlike AE-based models that learn an encoding-decoding process for datasets,
VAE-based models excel in identifying continuous representations within a low-
dimensional space. These representations, characterized by their smooth and
continuous nature in the hidden space, are essential for preserving
probabilistic properties during sampling. Consequently, VAEs can reconstruct
samples with increased sharpness and interpretability, outperforming their
autoencoder-based counterparts. In contrast to GANs, VAEs explicitly model the
data’s likelihood distribution and provide additional constraints on the
data’s posterior distribution based on a preset prior, making them more
suitable for modeling data in dynamic areas and designing end-to-end anomaly
detection tasks. For instance, [33] utilizes a Gaussian Mixture Model (GMM)
assumption for data likelihood distribution, and [34] employs a dynamical
prior over time.

Issues in VAE based TSAD: VAEs tend to sacrifice representation [35] for data
fitting. In that case, the induced latent hole will lead to the lack of
robustness Represented by latent hole. At the same time, the model’s failure
to learn the likelihood of the sequence data exacerbates its robustness
issues. Specifically, VAE-based anomaly detection algorithms typically employ
a Convolutional Neural Network (CNN) architecture for data encoding. While
effective for image data, this approach often fails to capture the temporal
characteristics of time-series data, such as seasonality, periodicity, and
frequency domain features, through CNN encoding filters. The shallow Fully
Connected Network (FCN) networks are employed in VAEs as substitutes. As a
result, the naive structure cannot capture varying dependence, and compared to
the image data, the sequence in training is relatively small. Due to the model
and data issue, the generative modal cannot converge to the optimal.

Advances in VAE-based TSAD: To remedy this, the traditional variational
framework has been revoluted, integrating the meta-prior in generative
modeling. For instance, a Variational Recurrent Neural Network (VRNN) has been
proposed, establishing a model for the variational autoencoder’s inference,
prior, and reconstruction processes by capturing the temporal dependencies
between intermediate variable $\boldsymbol{h}$ in the deterministic model and
input variables $\boldsymbol{x}$ in the recurrent neural network. This
approach and its variants [18, 20, 21] effectively utilize the variational
autoencoder to learn and model the latent distribution of data while
maintaining the temporal dependence of the recurrent neural network. On the
other hand, the variational representation can be designed. [23] utilizes
prototype-based approaches to define latent representations for Multivariate
Time Series (MTS) and learn a robust likelihood distribution of normal data.

###  II-B Self-supervised Learning on Time Series Data in Determistic and
Generative Models

In deterministic models, augmenting time-series data or their representations,
combined with specific self-supervised algorithms, can provide sufficient
depth for training in downstream tasks. For instance, in prediction tasks,
[36] encodes time-series segments in both time and frequency domains to obtain
positive and negative sample pairs, using contrastive learning to capture the
seasonal-trend representation of time-series data. [37] constructs positive
pairs with multi-granularity time-series segments and corresponding latent
variable representations, enhancing fine-grained information for prediction by
maximizing mutual information.

In classification tasks, [38] forms pairwise representations of global and
local input series, obtaining informational gains through adversarial
learning. For anomaly detection tasks, [39] aims to acquire spatio-temporal
dependent representations suitable for downstream tasks. [40] proposes a
multi-layer representation learning framework to obtain consistent, contextual
representations of overlapping segments, designing a contrastive loss by
decomposing overlapping subsequences in both instance and temporal dimensions
to obtain positive and negative sample pairs. Additionally, [41] employs a
dual bilinear process at the encoding level to capture positive and negative
samples of time sequences, thereby capturing both long and short-term
dependencies.

In contrast, SSL based on time-series generative models typically focuses on
data and representation augmentation as a generative approach. For instance,
Autoencoder (AE) based methods, such as those presented in [42, 43, 44, 45],
leverage the AE architecture for data augmentation. Similarly, diffusion-based
approaches, as seen in [46, 47, 48], employ diffusion processes to augment
time-series data and representations.

##  III Augumentation Guided Generative Anomaly Detection

In this section, we first provide the problem definition for generative model-
based TSAD. Subsequently, we depict the structure of the augmented generative
model in the form of a plate diagram, illustrating the random variables and
their dependency structure. The augmented guided generative anomaly detection
model operates within the framework of probabilistic generative models,
employing self-supervised techniques to augment the latent variables
$\boldsymbol{z}$ during the training process of the generative model, thereby
enhancing the deep model’s fit to the data likelihood. Herein, we have
implemented a self-supervised variational autoencoder based on input data
augmentation, which preprocesses the input data $\boldsymbol{x}$ to generate
latent variables $\boldsymbol{z}_{\mathrm{a}}$ with different views. To align
the likelihood functions of the raw and augmented models, we have developed
two distinct mutual information loss functions, one grounded in depth and the
other in statistics. By maximizing the mutual information between them, we
draw the models closer to fitting the same distribution.

###  III-A Problem Definition

Time series data is succinctly represented as
$\mathcal{X}:=\\{(\boldsymbol{x}^{(i)},\boldsymbol{y}^{(i)})\\}_{i=1}^{n}$,
encompassing $n$ time-stamped observations $\boldsymbol{x}\in\mathbb{R}^{c}$
situated within a $c$-dimensional representation space, each paired with a
discrete observation $\boldsymbol{y}$. The observation $\boldsymbol{y}$ is
assigned discrete values across $l$ predefined classes, delineated as
$\boldsymbol{y}\in\\{0,1,\ldots,l-1\\}$. Here, $c$ denotes the feature
dimensionality at each time point, categorizing the dataset as a Multivariate
Time Series (MTS) when $c>1$ and as a Univariate Time Series (TS) for $c=1$.
In all figures and equations, to enhance notational conciseness, we propose
abbreviating $\mathrm{raw}$ as $\mathrm{r}$ and $\mathrm{augmentation}$ as
$\mathrm{a}$.

In generative model-based TSAD, the focus is on learning a reconstructing
model, i.e., $\mathcal{M}_{\mathrm{normal}}$, that models the mass of
loglikelihood of the majority of normal data points within the entire dataset
$\mathcal{X}=\\{\mathcal{X}_{\mathrm{normal}},\mathcal{X}_{\mathrm{abnormal}}\\}$.
Anomalies are then identified in an unsupervised, end-to-end fashion by
calculating the anomaly score, denoted as
$AS(\boldsymbol{x},\hat{\boldsymbol{x}})$, which quantifies the difference
between a given input $\boldsymbol{x}$ and its modeled copy
$\hat{\boldsymbol{x}}$ as reconstructed by $\mathcal{M}_{\mathrm{normal}}$.
This approach is feasible, assuming that the log-likelihood learned from
normal observations will diverge notably when encountering anomalous data,
yielding elevated anomaly scores.

###  III-B Data Augmentation Guided Probabilistic Generative Model

The plate diagram in Fig. 2 defines an augmented-based probabilistic
generative model. The upper part of the diagram specifies the learned joint
distribution of the original data $\boldsymbol{x}_{\mathrm{r}}$ and its latent
variable $\boldsymbol{z}_{\mathrm{r}}$, i.e.,
$p(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{r}})$, where the latent
variable $\boldsymbol{z}_{\mathrm{r}}$ is generated by an inference network
$q_{\phi_{\mathrm{r}}}(\boldsymbol{z}_{\mathrm{r}}|\boldsymbol{x}_{\mathrm{r}})$
parameterized by $\phi_{\mathrm{r}}$, and the reconstructed variable
$\hat{x}_{\mathrm{r}}$ is produced by a generative network
$p_{\theta_{\mathrm{r}}}(\boldsymbol{x}_{\mathrm{r}}|\boldsymbol{z}_{\mathrm{r}})$
parameterized by $\theta_{\mathrm{r}}$. The model optimizes an approximate
surrogate error $\mathcal{L}^{\mathrm{r}}_{\mathrm{ELBO}}$, comprising a
reconstruction loss that maximizes the likelihood distribution
$\mathcal{L}_{\mathrm{R}}^{\mathrm{r}}$ and a $D_{\mathrm{KL}}$ loss that
minimizes the discrepancy between the prior of the latent variables and their
variational posterior $\mathcal{L}_{\mathrm{I}}^{\mathrm{r}}$, i.e.

$\displaystyle\mathcal{L}^{\mathrm{r}}_{\mathrm{ELBO}}:=$
$\displaystyle\underbrace{\mathbb{E}_{q_{\phi_{\mathrm{r}}}(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})}\Big{[}\log
p_{\theta_{\mathrm{r}}}(\boldsymbol{x}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{r}})}_{\mathcal{L}_{\mathrm{R}}^{\mathrm{r}}}\Big{]}$
(1)
$\displaystyle-\beta\underbrace{D_{KL}\big{(}q_{\phi}(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})||p(\boldsymbol{z})}_{\mathcal{L}_{\mathrm{I}}^{\mathrm{r}}}\big{)}.$

The lower part of the plate diagram outlines the probabilistic model of the
joint distribution of the augmented view data $\boldsymbol{x}_{\mathrm{a}}$
and latent variables $\boldsymbol{z}_{\mathrm{a}}$, i.e.,
$p(\boldsymbol{x}_{\mathrm{a}},\boldsymbol{z}_{\mathrm{a}})$ with latent
variables $\boldsymbol{z}_{\mathrm{a}}$ derived from an augmented inference
network $q_{\phi_{\mathrm{a}}}(\boldsymbol{z}_{\mathrm{a}}|\boldsymbol{x})$,
i.e., $\boldsymbol{z}_{\mathrm{a}}\sim
q_{\phi_{\mathrm{a}}}(\boldsymbol{z}_{\mathrm{a}}|\boldsymbol{x})$. Similar to
the above, the model optimizes an augmented reconstruction loss
$\mathcal{L}_{\mathrm{R}}^{\mathrm{a}}$ and inference loss
$\mathcal{L}_{\mathrm{I}}^{\mathrm{a}}$, i.e.,

$\displaystyle\mathcal{L}^{\mathrm{a}}_{\mathrm{ELBO}}:=$
$\displaystyle\underbrace{\mathbb{E}_{q_{\phi_{\mathrm{a}}}(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{a}})}\Big{[}\log
p_{\theta_{\mathrm{a}}}(\boldsymbol{x}_{\mathrm{a}}\mid\boldsymbol{z}_{\mathrm{a}})}_{\mathcal{L}_{\mathrm{R}}^{\mathrm{a}}}\Big{]}$
(2)
$\displaystyle-\beta\underbrace{D_{\mathrm{KL}}\big{(}q_{\phi}(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{a}})||p(\boldsymbol{z})}_{\mathcal{L}_{\mathrm{I}}^{\mathrm{a}}}\big{)}.$

On the one hand, both models strive to fit their respective data distribution
likelihoods. On the other, we leverage the advantage of data augmentation by
maximizing the mutual information
$I(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$ between two
latent models, optimizing a mutual information loss parameterized by $\psi$
(in deep learning approximation), to train the models for maximal data
likelihood synergistically.

Given the variety of latent variable augmentations, this paper augments the
raw data to augment the model. In summary, we propose an augmented
probabilistic generative model to learn a joint likelihood function
$p(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}},\boldsymbol{z}_{\mathrm{a}})$
for anomaly detection while simultaneously optimizing an inference network
parameterized by $\phi_{\mathrm{r}},\phi_{\mathrm{a}}$, a generative network
parameterized by $\theta_{\mathrm{r}},\theta_{\mathrm{a}}$, and an alignment
network parameterized by $\psi$. The $\psi$ can be parameterized by neurons in
deep learning approximation and pseudo-parameters in shallow learning. The
generative process is as follows:

$p\left(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)=\int
p\left(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}},\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)\mathrm{d}\boldsymbol{z}_{\mathrm{r}}\mathrm{d}\boldsymbol{z}_{\mathrm{a}},$
(3)

where $\boldsymbol{x}_{\mathrm{r}}$ represents the raw input datapoint,
$\boldsymbol{x}_{\mathrm{a}}$ is the augmented sample based on the input,
$\boldsymbol{z}_{\mathrm{r}}$ is the raw latent variable, and
$\boldsymbol{z}_{\mathrm{a}}$ is the augmented latent variable.

The joint distribution is often too high-dimensional and sophisticated to
solve it directly. To address this, a tractable variational distribution
$q(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$ is employed as an
approximation within the framework of Variational Inference (VI). Due to the
computational convenience it offers, we typically take the logarithm of the
distribution. Consequently, as depicted in Equation 3, the likelihood of data
that encompasses latent variables can be decomposed as follows:

$\displaystyle
p\left(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)$ (4)
$\displaystyle=$
$\displaystyle\int\frac{p\left(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}},\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)q(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}{q(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}\mathrm{d}\boldsymbol{z}_{\mathrm{r}}\mathrm{d}\boldsymbol{z}_{\mathrm{a}},$

and we can get the $\log$ versions as follows:

$\displaystyle\log
p\left(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)$ (5)
$\displaystyle=$
$\displaystyle\log\int\frac{p\left(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}},\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)q(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}{q(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}\mathrm{d}\boldsymbol{z}_{\mathrm{r}}\mathrm{d}\boldsymbol{z}_{\mathrm{a}}.$

Given the $\log$ is a convex function, we can get a lower bound by Jensen’s
inequality:

$\displaystyle\log
p\left(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)$ (6)
$\displaystyle=$
$\displaystyle\log\int\frac{p\left(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}},\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)q(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}{q(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}\mathrm{d}\boldsymbol{z}_{\mathrm{r}}\mathrm{d}\boldsymbol{z}_{\mathrm{a}}$
$\displaystyle=$
$\displaystyle\log\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\left[\frac{p\left(\boldsymbol{x}_{\mathrm{r}},x_{\mathrm{a}},\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)}{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\right]$
$\displaystyle\geq$
$\displaystyle\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\log\left[\frac{p\left(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}},\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)}{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\log\left[\frac{p(\boldsymbol{x}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{r}})p\left(\boldsymbol{x}_{\mathrm{a}}\mid\boldsymbol{z}_{\mathrm{a}}\right)p\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)}{q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})q\left(\boldsymbol{z}_{\mathrm{a}}\mid
x_{\mathrm{a}}\right)}\right]$ $\displaystyle=$
$\displaystyle\underbrace{\mathbb{E}_{q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})}\log[p(\boldsymbol{x}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{r}})]+\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{a}}\right)}\log\left[p\left(\boldsymbol{x}_{\mathrm{a}}\mid\boldsymbol{z}_{\mathrm{a}}\right)\right]}_{\leavevmode\hbox
to11.75pt{\vbox to11.75pt{\pgfpicture\makeatletter\hbox{\hskip
5.87407pt\lower-5.87407pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.67407pt}{0.0pt}\pgfsys@curveto{5.67407pt}{3.13374pt}{3.13374pt}{5.67407pt}{0.0pt}{5.67407pt}\pgfsys@curveto{-3.13374pt}{5.67407pt}{-5.67407pt}{3.13374pt}{-5.67407pt}{0.0pt}\pgfsys@curveto{-5.67407pt}{-3.13374pt}{-3.13374pt}{-5.67407pt}{0.0pt}{-5.67407pt}\pgfsys@curveto{3.13374pt}{-5.67407pt}{5.67407pt}{-3.13374pt}{5.67407pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.75pt}{-2.25555pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$
$\displaystyle+\underbrace{\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\log\left[\frac{p\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)}{q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})q\left(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{a}}\right)}\right]}_{\leavevmode\hbox
to11.75pt{\vbox to11.75pt{\pgfpicture\makeatletter\hbox{\hskip
5.87407pt\lower-5.87407pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.67407pt}{0.0pt}\pgfsys@curveto{5.67407pt}{3.13374pt}{3.13374pt}{5.67407pt}{0.0pt}{5.67407pt}\pgfsys@curveto{-3.13374pt}{5.67407pt}{-5.67407pt}{3.13374pt}{-5.67407pt}{0.0pt}\pgfsys@curveto{-5.67407pt}{-3.13374pt}{-3.13374pt}{-5.67407pt}{0.0pt}{-5.67407pt}\pgfsys@curveto{3.13374pt}{-5.67407pt}{5.67407pt}{-3.13374pt}{5.67407pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.75pt}{-2.25555pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{2}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys<EMAIL_ADDRESS>
As we can see, the 1 part can be decomposed into two reconstruction losses,
i.e., $\leavevmode\hbox to14.18pt{\vbox
to14.18pt{\pgfpicture\makeatletter\hbox{\hskip 7.09111pt\lower-7.09111pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}=\mathcal{L}_{\mathrm{R}}^{\mathrm{r}}+\mathcal{L}_{\mathrm{R}}^{\mathrm{a}}$
and the 2 part in E.q. 6 can be decomposed as:

$\begin{aligned}
&\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\log\left[\frac{p\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)}{q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})q\left(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{a}}\right)}\right]\\\
=&\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\log\left[\frac{p\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)p(\boldsymbol{z}_{\mathrm{r}})p\left(\boldsymbol{z}_{\mathrm{a}}\right)}{q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})q\left(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{a}}\right)p(\boldsymbol{z}_{\mathrm{r}})p\left(\boldsymbol{z}_{\mathrm{a}}\right)}\right]\\\
=&\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\log\left[\frac{p\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)}{p(\boldsymbol{z}_{\mathrm{r}})p\left(\boldsymbol{z}_{\mathrm{a}}\right)}\right]\\\
&+\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\log\left[\frac{p(\boldsymbol{z}_{\mathrm{r}})p\left(\boldsymbol{z}_{\mathrm{a}}\right)}{q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})q\left(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{a}}\right)}\right]\\\
=&\underbrace{\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}}\right)}\log\left[\frac{p\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)}{p(\boldsymbol{z}_{\mathrm{r}})p\left(\boldsymbol{z}_{\mathrm{a}}\right)}\right]}_{\leavevmode\hbox
to13.13pt{\vbox to13.13pt{\pgfpicture\makeatletter\hbox{\hskip
6.56548pt\lower-6.56548pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.36548pt}{0.0pt}\pgfsys@curveto{6.36548pt}{3.5156pt}{3.5156pt}{6.36548pt}{0.0pt}{6.36548pt}\pgfsys@curveto{-3.5156pt}{6.36548pt}{-6.36548pt}{3.5156pt}{-6.36548pt}{0.0pt}\pgfsys@curveto{-6.36548pt}{-3.5156pt}{-3.5156pt}{-6.36548pt}{0.0pt}{-6.36548pt}\pgfsys@curveto{3.5156pt}{-6.36548pt}{6.36548pt}{-3.5156pt}{6.36548pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.625pt}{-2.39166pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{A}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\\\
&-\underbrace{D_{KL}[q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})\|p(\boldsymbol{z}_{\mathrm{r}})]}_{\leavevmode\hbox
to10.88pt{\vbox to10.88pt{\pgfpicture\makeatletter\hbox{\hskip
5.43755pt\lower-5.43755pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.23755pt}{0.0pt}\pgfsys@curveto{5.23755pt}{2.89265pt}{2.89265pt}{5.23755pt}{0.0pt}{5.23755pt}\pgfsys@curveto{-2.89265pt}{5.23755pt}{-5.23755pt}{2.89265pt}{-5.23755pt}{0.0pt}\pgfsys@curveto{-5.23755pt}{-2.89265pt}{-2.89265pt}{-5.23755pt}{0.0pt}{-5.23755pt}\pgfsys@curveto{2.89265pt}{-5.23755pt}{5.23755pt}{-2.89265pt}{5.23755pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-0.97221pt}{-2.3375pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{i}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}-\underbrace{D_{KL}\left[q\left(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{a}}\right)\|p\left(\boldsymbol{z}_{\mathrm{a}}\right)\right]}_{\leavevmode\hbox
to12.09pt{\vbox to12.09pt{\pgfpicture\makeatletter\hbox{\hskip
6.04399pt\lower-6.04399pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.844pt}{0.0pt}\pgfsys@curveto{5.844pt}{3.22758pt}{3.22758pt}{5.844pt}{0.0pt}{5.844pt}\pgfsys@curveto{-3.22758pt}{5.844pt}{-5.844pt}{3.22758pt}{-5.844pt}{0.0pt}\pgfsys@curveto{-5.844pt}{-3.22758pt}{-3.22758pt}{-5.844pt}{0.0pt}{-5.844pt}\pgfsys@curveto{3.22758pt}{-5.844pt}{5.844pt}{-3.22758pt}{5.844pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.94444pt}{-2.3375pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{ii}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}},\end{aligned}$

(7)

where the 2 part can be the combination of two inference losses and mutual
information between latent variables, i.e., $\leavevmode\hbox to14.18pt{\vbox
to14.18pt{\pgfpicture\makeatletter\hbox{\hskip 7.09111pt\lower-7.09111pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{2}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}=\mathcal{L}_{\mathrm{I}}^{\mathrm{r}}+\mathcal{L}_{\mathrm{I}}^{\mathrm{a}}+I(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$,
where we denote $\leavevmode\hbox to13.06pt{\vbox
to13.06pt{\pgfpicture\makeatletter\hbox{\hskip 6.52788pt\lower-6.52788pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.32788pt}{0.0pt}\pgfsys@curveto{6.32788pt}{3.49483pt}{3.49483pt}{6.32788pt}{0.0pt}{6.32788pt}\pgfsys@curveto{-3.49483pt}{6.32788pt}{-6.32788pt}{3.49483pt}{-6.32788pt}{0.0pt}\pgfsys@curveto{-6.32788pt}{-3.49483pt}{-3.49483pt}{-6.32788pt}{0.0pt}{-6.32788pt}\pgfsys@curveto{3.49483pt}{-6.32788pt}{6.32788pt}{-3.49483pt}{6.32788pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.38889pt}{-3.3393pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{i}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}=\mathcal{L}_{\mathrm{I}}^{\mathrm{r}}$,
$\leavevmode\hbox to14.71pt{\vbox
to14.71pt{\pgfpicture\makeatletter\hbox{\hskip 7.35625pt\lower-7.35625pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.15625pt}{0.0pt}\pgfsys@curveto{7.15625pt}{3.95233pt}{3.95233pt}{7.15625pt}{0.0pt}{7.15625pt}\pgfsys@curveto{-3.95233pt}{7.15625pt}{-7.15625pt}{3.95233pt}{-7.15625pt}{0.0pt}\pgfsys@curveto{-7.15625pt}{-3.95233pt}{-3.95233pt}{-7.15625pt}{0.0pt}{-7.15625pt}\pgfsys@curveto{3.95233pt}{-7.15625pt}{7.15625pt}{-3.95233pt}{7.15625pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.77779pt}{-3.3393pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{ii}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}=\mathcal{L}_{\mathrm{I}}^{\mathrm{a}}$,
and $\leavevmode\hbox to16.14pt{\vbox
to16.14pt{\pgfpicture\makeatletter\hbox{\hskip 8.07158pt\lower-8.07158pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.87158pt}{0.0pt}\pgfsys@curveto{7.87158pt}{4.3474pt}{4.3474pt}{7.87158pt}{0.0pt}{7.87158pt}\pgfsys@curveto{-4.3474pt}{7.87158pt}{-7.87158pt}{4.3474pt}{-7.87158pt}{0.0pt}\pgfsys@curveto{-7.87158pt}{-4.3474pt}{-4.3474pt}{-7.87158pt}{0.0pt}{-7.87158pt}\pgfsys@curveto{4.3474pt}{-7.87158pt}{7.87158pt}{-4.3474pt}{7.87158pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-3.41666pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{A}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}=I(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$.

Minimization of the term denoted by 1 leads to an increased log-likelihood for
both $p(\boldsymbol{x}_{\mathrm{r}}|\boldsymbol{z}_{\mathrm{r}})$ and
$p(\boldsymbol{x}_{\mathrm{a}}|\boldsymbol{z}_{\mathrm{a}})$, applicable to
the raw and augmented data perspectives, respectively. Reducing the inference
loss, represented as $\mathcal{L}_{\mathrm{I}}^{\mathrm{r}}$,
$\mathcal{L}_{\mathrm{I}}^{\mathrm{a}}$ within the section labeled 2,
contributes to a more coherent latent space that facilitates the
reconstruction process. Moreover, enhancing the mutual information, denoted as
$I(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$, serves to bridge
the disparities between the raw and augmented models. This process ensures a
cohesive framework for incorporating data augmentation within the generative
model. In conclusion, the proposed objective for learning is to approximate
the joint distribution
$p(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{x}_{\mathrm{a}})$ within an
augmentation-informed generative modeling context, denoted as:

$\displaystyle\mathcal{L}_{\mathrm{AVAE}}=$ $\displaystyle\leavevmode\hbox
to14.18pt{\vbox to14.18pt{\pgfpicture\makeatletter\hbox{\hskip
7.09111pt\lower-7.09111pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}+\leavevmode\hbox
to16.14pt{\vbox to16.14pt{\pgfpicture\makeatletter\hbox{\hskip
8.07158pt\lower-8.07158pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.87158pt}{0.0pt}\pgfsys@curveto{7.87158pt}{4.3474pt}{4.3474pt}{7.87158pt}{0.0pt}{7.87158pt}\pgfsys@curveto{-4.3474pt}{7.87158pt}{-7.87158pt}{4.3474pt}{-7.87158pt}{0.0pt}\pgfsys@curveto{-7.87158pt}{-4.3474pt}{-4.3474pt}{-7.87158pt}{0.0pt}{-7.87158pt}\pgfsys@curveto{4.3474pt}{-7.87158pt}{7.87158pt}{-4.3474pt}{7.87158pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-3.41666pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{A}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}-\leavevmode\hbox
to13.06pt{\vbox to13.06pt{\pgfpicture\makeatletter\hbox{\hskip
6.52788pt\lower-6.52788pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.32788pt}{0.0pt}\pgfsys@curveto{6.32788pt}{3.49483pt}{3.49483pt}{6.32788pt}{0.0pt}{6.32788pt}\pgfsys@curveto{-3.49483pt}{6.32788pt}{-6.32788pt}{3.49483pt}{-6.32788pt}{0.0pt}\pgfsys@curveto{-6.32788pt}{-3.49483pt}{-3.49483pt}{-6.32788pt}{0.0pt}{-6.32788pt}\pgfsys@curveto{3.49483pt}{-6.32788pt}{6.32788pt}{-3.49483pt}{6.32788pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.38889pt}{-3.3393pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{i}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}-\leavevmode\hbox
to14.71pt{\vbox to14.71pt{\pgfpicture\makeatletter\hbox{\hskip
7.35625pt\lower-7.35625pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.15625pt}{0.0pt}\pgfsys@curveto{7.15625pt}{3.95233pt}{3.95233pt}{7.15625pt}{0.0pt}{7.15625pt}\pgfsys@curveto{-3.95233pt}{7.15625pt}{-7.15625pt}{3.95233pt}{-7.15625pt}{0.0pt}\pgfsys@curveto{-7.15625pt}{-3.95233pt}{-3.95233pt}{-7.15625pt}{0.0pt}{-7.15625pt}\pgfsys@curveto{3.95233pt}{-7.15625pt}{7.15625pt}{-3.95233pt}{7.15625pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.77779pt}{-3.3393pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{ii}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$ (8) $\displaystyle=$
$\displaystyle\mathcal{L}^{\mathrm{r}}_{\mathrm{ELBO}}+\mathcal{L}^{\mathrm{a}}_{\mathrm{ELBO}}+I(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}),$

where $\mathcal{L}_{\mathrm{AVAE}}$ represents the augmentation based VAE
loss.

Figure 3: Illusration of adversarial learning in mutation information
approximation. In the first stage, the discriminator is frozen to update the
parameters of Encoders and decoders. In the second stage, We freeze the
parameters of both the generator and the discriminator while simultaneously
inverting the pseudo-labels of positive and negative samples to train the
discriminator.

###  III-C Deep and Shallow Learning in Mutual Information Approximation

####  III-C1 MI approximation in Shallow Learning

We employ a $\mathcal{L}_{\mathrm{infoNCE}}$ loss to approximate the lower
bound of MI. Since this method uses a non-parametric variational distribution
in variational inference, it can be considered a form of shallow learning.
When the variational distribution
$q(\boldsymbol{z}_{\mathrm{r}}|\boldsymbol{z}_{\mathrm{a}})$ is employed to
approximate the untractable posterior distribution
$p(\boldsymbol{z}_{\mathrm{r}}|\boldsymbol{z}_{\mathrm{a}})$, as in E.q. (9a),
we can derive a lower bound, as in E.q. (9b). Specifically, by using an
energy-based variational function
$q(\boldsymbol{z}_{\mathrm{r}}|\boldsymbol{z}_{\mathrm{a}})=\frac{p(\boldsymbol{z}_{\mathrm{r}})}{a(\boldsymbol{z}_{\mathrm{a}})}e^{f(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}$,
where $f(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$ is a critic
value function, and
$a(\boldsymbol{z}_{\mathrm{a}})=\mathbb{E}_{p(x)}\left[e^{f(x,y)}\right]$ for
approximation, we use the convexity of the $\log$ function to apply Jensen’s
inequality to $\mathbb{E}_{p(\boldsymbol{z}_{\mathrm{a}})}[\log
a(\boldsymbol{z}_{\mathrm{a}})]$ to further derive a lower bound, as in E.q.
(9c). By utilizing the inequality:
$\log(\boldsymbol{z})\leq\frac{\boldsymbol{z}}{\tau}+\log(\tau)-1$, we can
approximate further to obtain another lower bound, as in E.q. (9d). Using $K$
samples for an unbiased estimate, we obtain E.q. (9e), and through Monte Carlo
estimation, we can approximate it to the infoNCE loss, that is,
$\mathcal{L}_{\mathrm{infoNCE}}$ in (9f):

$\displaystyle I(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$
$\displaystyle=$
$\displaystyle\mathbb{E}_{p(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}\left[\log\frac{q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{a}})p(\boldsymbol{z}_{\mathrm{r}}|\boldsymbol{z}_{\mathrm{a}})}{p(\boldsymbol{z}_{\mathrm{r}})q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{a}})}\right]$
(9a) $\displaystyle=$
$\displaystyle\mathbb{E}_{p(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}\left[\log\frac{q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{a}})}{p(\boldsymbol{z}_{\mathrm{r}})}\right]$
$\displaystyle+\mathbb{E}_{p(\boldsymbol{z}_{\mathrm{a}})}[D_{\mathrm{KL}}(p(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{a}})\|q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{a}}))]$
$\displaystyle\geq$
$\displaystyle\mathbb{E}_{p(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}[\log
q(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{a}})]$ (9b)
$\displaystyle\geq$
$\displaystyle\mathbb{E}_{p(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}[f(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})]$
$\displaystyle-\mathbb{E}_{p(\boldsymbol{z}_{\mathrm{a}})}\left[\frac{\mathbb{E}_{p(\boldsymbol{z}_{\mathrm{r}})}\left[e^{f(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}\right]}{a(\boldsymbol{z}_{\mathrm{a}})}+\log(a(\boldsymbol{z}_{\mathrm{a}}))-1\right]$
(9c) $\displaystyle\geq$ $\displaystyle
1-\mathbb{E}_{p\left(\boldsymbol{z}_{(\mathrm{r},1:K)}\right)p(\boldsymbol{z}_{\mathrm{a}})}\left[\frac{e^{f\left(\boldsymbol{z}_{(\mathrm{r},1)},\boldsymbol{z}_{\mathrm{a}}\right)}}{a\left(\boldsymbol{z}_{\mathrm{a}};\boldsymbol{z}_{(\mathrm{r},1:K)}\right)}\right]$
$\displaystyle+\mathbb{E}_{p\left(\boldsymbol{z}_{(\mathrm{r},1:K)}\right)p\left(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{z}_{(\mathrm{r},1)}\right)}\left[\log\frac{e^{f\left(\boldsymbol{z}_{(\mathrm{r},1)},\boldsymbol{z}_{\mathrm{a}}\right)}}{a\left(\boldsymbol{z}_{\mathrm{a}},\boldsymbol{z}_{(\mathrm{r},1:K)}\right)}\right]$
(9d) $\displaystyle\geq$
$\displaystyle\mathbb{E}\left[\frac{1}{K}\sum_{i=1}^{K}\log\frac{e^{f\left(\boldsymbol{z}_{(\mathrm{r},i)},\boldsymbol{z}_{(\mathrm{a},i)}\right)}}{\frac{1}{K}\sum_{j=1}^{K}e^{f\left(\boldsymbol{z}_{(\mathrm{r},i)},\boldsymbol{z}_{(\mathrm{a},j)}\right)}}\right]$
(9e) $\displaystyle\geq$
$\displaystyle\mathbb{E}\left[\frac{1}{K}\sum_{i=1}^{K}\log\frac{p\left(\boldsymbol{z}_{(\mathrm{a},i)}\mid\boldsymbol{z}_{(\mathrm{r},i)}\right)}{\frac{1}{K}\sum_{j=1}^{K}p\left(\boldsymbol{z}_{(\mathrm{a},i)}\mid\boldsymbol{z}_{(\mathrm{r},j)}\right)}\right]\triangleq\mathcal{L}_{\mathrm{infoNCE}}.$
(9f)

Actually, we optimize an infoNCE loss scaled by the temperature coefficient
$\tau$:

$\displaystyle\mathcal{L}_{\text{InfoNCE }}$ (10) $\displaystyle=$
$\displaystyle-\log\frac{\exp\left(\boldsymbol{z}_{u}^{\mathrm{r},\top}\boldsymbol{z}_{u}^{\mathrm{a}}/\tau\right)}{\sum_{v}\exp\left(\boldsymbol{z}_{u}^{\mathrm{r},\top}\boldsymbol{z}_{v}^{\mathrm{a}}/\tau\right)+\sum_{v\neq
u}\exp\left(\boldsymbol{z}_{u}^{\mathrm{r},\top}\boldsymbol{z}_{v}/\tau\right)},$

where and the negative pairs are none, indicating that the negative keys for a
sample are the positive keys for others.

####  III-C2 MI approximation in Deep Learning

We can decompose the mutation information into two ratios in E.q. (11a) and
approximate in density ratio trick, guided by [49]. In that case, the density
ratio is approached by a parameterized neural network, and we can approximate
the MI implicitly in a deep learning scheme. Specifically, instead of modeling
two distributions directly, i.e.,
$q(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$ and
$q(\boldsymbol{z}_{\mathrm{r}})$, we can learn the ratio
$r=\frac{q(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})}{q(\boldsymbol{z}_{\mathrm{r}})}$
in an adversival manner, i.e., training a discriminator to classify whether
the label comes from the target distribution $\mathcal{P}$ or not, as shown in
E.q. (11b), where the $y$ is a preset pseudo-label. Since we use the
discriminator method to estimate the mutual information, the upper bound is
denoted as E.q. (11c).

$\displaystyle\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)}\frac{q\left(\boldsymbol{z}_{\mathrm{r}}|\boldsymbol{z}_{\mathrm{a}}\right)}{q\left(\boldsymbol{z}_{\mathrm{a}}\right)}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q\left(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}\right)}\log\frac{q\left(\boldsymbol{z}_{\mathrm{a}}|\boldsymbol{z}_{\mathrm{a}}\right)}{q(\boldsymbol{z}_{\mathrm{r}})}$
(11a) $\displaystyle\leq$
$\displaystyle\log\frac{\mathcal{P}(y=1\mid\boldsymbol{z}_{\mathrm{r}})}{\mathcal{P}(y=0\mid\boldsymbol{z}_{\mathrm{r}})}+\log\frac{\mathcal{P}(y=1\mid\boldsymbol{z}_{\mathrm{a}})}{\mathcal{P}(y=0\mid\boldsymbol{z}_{\mathrm{a}})}$
(11b) $\displaystyle\leq$
$\displaystyle\log\frac{\Psi(\boldsymbol{z}_{\mathrm{r}})}{1-\Psi(\boldsymbol{z}_{\mathrm{r}})}+\log\frac{\Psi_{\mathrm{a}}(\boldsymbol{z}_{\mathrm{a}})}{1-\Psi_{\mathrm{a}}(\boldsymbol{z}_{\mathrm{a}})}\triangleq\mathcal{L}_{\mathrm{adversial}}$
(11c)

###  III-D End-to-End Anomaly Detection Training

This section proposes an end-to-end TSAD model based on a weakly augmented
generative model.

####  III-D1 Weakly Augmentation

In augmentation-based generative models, the likelihood fitting is enhanced by
reusing training data. In VAEs, this leads to improved generative models
$p_{\theta}(\boldsymbol{x}\mid\boldsymbol{z})$ parameterized by $\theta$ via
enriched data representations in the inference network
$q_{\phi}(\boldsymbol{z}\mid\boldsymbol{x})$ parameterized by $\phi$. The
augmented latent variable, $\boldsymbol{z}_{\mathrm{a}}$, is derived as
$\boldsymbol{z}_{\mathrm{a}}\sim
q(\boldsymbol{z}_{\mathrm{a}}|\boldsymbol{x})$. During data preprocessing, we
can augment latent representations directly by manipulating the input data
augmentation, represented as $\boldsymbol{z}_{\mathrm{a}}\sim
q(\boldsymbol{z}_{\mathrm{a}}|\boldsymbol{x}_{\mathrm{a}})$. Here, the
augmented input $\boldsymbol{x}_{\mathrm{a}}$ is obtained from the raw input
$\boldsymbol{x}_{\mathrm{r}}$ using the augmentation operation $\mathcal{O}$,
formulated as
$\boldsymbol{x}_{\mathrm{a}}=\mathcal{O}(\boldsymbol{x}_{\mathrm{r}})$.

Data augmentation methods for time series data typically require an input
array of size (batch, time_steps, channel), with manipulations possible in the
batch domain (such as jittering with noise, scaling, and normalization) or in
the time domain (including window slicing and warping). Additionally,
augmentations can be applied in the frequency domain. These techniques enrich
the original dataset through various methods, effectively enhancing data
diversity. This diversification is crucial for models to comprehend better and
predict time series patterns.

Nevertheless, our findings suggest that applying weak augmentation to the
original input data may yield more favorable outcomes for likelihood fitting
in anomaly detection tasks. Specifically, weak augmentation involves subtle
modifications to the data, primarily through different normalization
techniques. These include:

  * •

Standardization:

$\boldsymbol{x}_{\mathrm{a}}=\mathcal{O}^{\mathrm{stand}}(\boldsymbol{x}_{\mathrm{r}})=\frac{\boldsymbol{x}_{\mathrm{r}}-\mu}{\sigma},$
(12)

where, $\mu$ is the mean and $\sigma$ the standard deviation of the data.

  * •

Min-Max Normalization:

$\boldsymbol{x}_{\mathrm{a}}=\mathcal{O}^{\mathrm{mm}}(\boldsymbol{x}_{\mathrm{r}})=\frac{\boldsymbol{x}_{\mathrm{r}}-\mathrm{min}(\boldsymbol{x}_{\mathrm{r}})}{\mathrm{max}(\boldsymbol{x}_{\mathrm{r}})-\mathrm{min}(\boldsymbol{x}_{\mathrm{r}})},$
(13)

this scales the data to a specified range, such as 0 to 1.

Such moderate adjustments typically preserve the fundamental characteristics
and trends of the time series. They enable the model to discern the core
attributes of the data more effectively, thereby enhancing prediction
accuracy. Furthermore, weak augmentation maintains the data’s authenticity,
mitigating the risk of over-distorting the original data structure. This is
vital for preserving both the reliability and interpretability of the model.

####  III-D2 Training

For the raw input data $\boldsymbol{x}_{\mathrm{r}}$, we first obtain its
augmented variable $\boldsymbol{x}_{\mathrm{a}}$ during the data preprocessing
phase. Then, we train the inference networks for both the raw and augmented
perspectives, namely
$q_{\phi_{\mathrm{r}}}(\boldsymbol{z}_{\mathrm{r}}\mid\boldsymbol{x}_{\mathrm{r}})$
and
$q_{\phi_{\mathrm{a}}}(\boldsymbol{z}_{\mathrm{a}}\mid\boldsymbol{x}_{\mathrm{a}})$,
as well as the generative networks
$p_{\theta_{\mathrm{r}}}(\boldsymbol{x}_{\mathrm{r}}\mid\boldsymbol{z}_{\mathrm{r}})$
and
$p_{\theta_{\mathrm{a}}}(\boldsymbol{x}_{\mathrm{a}}\mid\boldsymbol{z}_{\mathrm{a}})$.
At the same time, we optimize the inference and reconstruction losses based on
Eq. (1) and E.q. (2).

During the training phase, we employ a strategy of sharing parameters for
joint likelihood consolidation in raw one
$p(\boldsymbol{x}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{r}})$ and augmented one
$p(\boldsymbol{x}_{\mathrm{a}},\boldsymbol{z}_{\mathrm{a}})$ to align the
reconstruction effects. This means allowing both the inference and
reconstruction networks to share the same structure and parameters during
training. Such an approach reduces the number of model parameters and
increases the generalization of the model, enabling it to learn normal
patterns of different data for reconstructing normal data.

In the variational inference of the joint distribution’s posterior
distribution, we maximize the mutual information of the two latent variables
to encourage the original generator and the augmented generator to tend
towards producing similar data distributions. In our actual optimization
objectives, i.e., the surrogated loss $\mathcal{L}_{\mathrm{WAVAE}}$, we
implemented two methods to control the divergence of the two likelihood
distributions: the $\mathcal{L}_{\mathrm{WAVAE}}^{\mathrm{infoNCE}}$ loss
based on contrastive learning:

$\displaystyle\mathcal{L}_{\mathrm{WAVAE}}^{\mathrm{infoNCE}}=$
$\displaystyle\leavevmode\hbox to14.18pt{\vbox
to14.18pt{\pgfpicture\makeatletter\hbox{\hskip 7.09111pt\lower-7.09111pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}+\leavevmode\hbox
to16.14pt{\vbox to16.14pt{\pgfpicture\makeatletter\hbox{\hskip
8.07158pt\lower-8.07158pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.87158pt}{0.0pt}\pgfsys@curveto{7.87158pt}{4.3474pt}{4.3474pt}{7.87158pt}{0.0pt}{7.87158pt}\pgfsys@curveto{-4.3474pt}{7.87158pt}{-7.87158pt}{4.3474pt}{-7.87158pt}{0.0pt}\pgfsys@curveto{-7.87158pt}{-4.3474pt}{-4.3474pt}{-7.87158pt}{0.0pt}{-7.87158pt}\pgfsys@curveto{4.3474pt}{-7.87158pt}{7.87158pt}{-4.3474pt}{7.87158pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-3.41666pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{A}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}-\leavevmode\hbox
to13.06pt{\vbox to13.06pt{\pgfpicture\makeatletter\hbox{\hskip
6.52788pt\lower-6.52788pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.32788pt}{0.0pt}\pgfsys@curveto{6.32788pt}{3.49483pt}{3.49483pt}{6.32788pt}{0.0pt}{6.32788pt}\pgfsys@curveto{-3.49483pt}{6.32788pt}{-6.32788pt}{3.49483pt}{-6.32788pt}{0.0pt}\pgfsys@curveto{-6.32788pt}{-3.49483pt}{-3.49483pt}{-6.32788pt}{0.0pt}{-6.32788pt}\pgfsys@curveto{3.49483pt}{-6.32788pt}{6.32788pt}{-3.49483pt}{6.32788pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.38889pt}{-3.3393pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{i}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}-\leavevmode\hbox
to14.71pt{\vbox to14.71pt{\pgfpicture\makeatletter\hbox{\hskip
7.35625pt\lower-7.35625pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.15625pt}{0.0pt}\pgfsys@curveto{7.15625pt}{3.95233pt}{3.95233pt}{7.15625pt}{0.0pt}{7.15625pt}\pgfsys@curveto{-3.95233pt}{7.15625pt}{-7.15625pt}{3.95233pt}{-7.15625pt}{0.0pt}\pgfsys@curveto{-7.15625pt}{-3.95233pt}{-3.95233pt}{-7.15625pt}{0.0pt}{-7.15625pt}\pgfsys@curveto{3.95233pt}{-7.15625pt}{7.15625pt}{-3.95233pt}{7.15625pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.77779pt}{-3.3393pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{ii}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$ (14)
$\displaystyle=$
$\displaystyle\mathcal{L}^{\mathrm{r}}_{\mathrm{ELBO}}+\mathcal{L}^{\mathrm{a}}_{\mathrm{ELBO}}+I(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$
$\displaystyle=$
$\displaystyle\mathcal{L}^{\mathrm{r}}_{\mathrm{ELBO}}+\mathcal{L}^{\mathrm{a}}_{\mathrm{ELBO}}+\mathcal{L}_{\mathrm{infoNCE}}(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}),$

and the $\mathcal{L}_{\mathrm{WAVAE}}^{\mathrm{adversial}}$ loss based on
adversarial learning:

$\displaystyle\mathcal{L}_{\mathrm{WAVAE}}^{\mathrm{adversial}}=$
$\displaystyle\leavevmode\hbox to14.18pt{\vbox
to14.18pt{\pgfpicture\makeatletter\hbox{\hskip 7.09111pt\lower-7.09111pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}+\leavevmode\hbox
to16.14pt{\vbox to16.14pt{\pgfpicture\makeatletter\hbox{\hskip
8.07158pt\lower-8.07158pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.87158pt}{0.0pt}\pgfsys@curveto{7.87158pt}{4.3474pt}{4.3474pt}{7.87158pt}{0.0pt}{7.87158pt}\pgfsys@curveto{-4.3474pt}{7.87158pt}{-7.87158pt}{4.3474pt}{-7.87158pt}{0.0pt}\pgfsys@curveto{-7.87158pt}{-4.3474pt}{-4.3474pt}{-7.87158pt}{0.0pt}{-7.87158pt}\pgfsys@curveto{4.3474pt}{-7.87158pt}{7.87158pt}{-4.3474pt}{7.87158pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.75pt}{-3.41666pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{A}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}-\leavevmode\hbox
to13.06pt{\vbox to13.06pt{\pgfpicture\makeatletter\hbox{\hskip
6.52788pt\lower-6.52788pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.32788pt}{0.0pt}\pgfsys@curveto{6.32788pt}{3.49483pt}{3.49483pt}{6.32788pt}{0.0pt}{6.32788pt}\pgfsys@curveto{-3.49483pt}{6.32788pt}{-6.32788pt}{3.49483pt}{-6.32788pt}{0.0pt}\pgfsys@curveto{-6.32788pt}{-3.49483pt}{-3.49483pt}{-6.32788pt}{0.0pt}{-6.32788pt}\pgfsys@curveto{3.49483pt}{-6.32788pt}{6.32788pt}{-3.49483pt}{6.32788pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.38889pt}{-3.3393pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{i}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}-\leavevmode\hbox
to14.71pt{\vbox to14.71pt{\pgfpicture\makeatletter\hbox{\hskip
7.35625pt\lower-7.35625pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.15625pt}{0.0pt}\pgfsys@curveto{7.15625pt}{3.95233pt}{3.95233pt}{7.15625pt}{0.0pt}{7.15625pt}\pgfsys@curveto{-3.95233pt}{7.15625pt}{-7.15625pt}{3.95233pt}{-7.15625pt}{0.0pt}\pgfsys@curveto{-7.15625pt}{-3.95233pt}{-3.95233pt}{-7.15625pt}{0.0pt}{-7.15625pt}\pgfsys@curveto{3.95233pt}{-7.15625pt}{7.15625pt}{-3.95233pt}{7.15625pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.77779pt}{-3.3393pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{ii}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$ (15)
$\displaystyle=$
$\displaystyle\mathcal{L}^{\mathrm{r}}_{\mathrm{ELBO}}+\mathcal{L}^{\mathrm{a}}_{\mathrm{ELBO}}+I(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}})$
$\displaystyle=$
$\displaystyle\mathcal{L}^{\mathrm{r}}_{\mathrm{ELBO}}+\mathcal{L}^{\mathrm{a}}_{\mathrm{ELBO}}+\mathcal{L}_{\mathrm{adversial}}(\boldsymbol{z}_{\mathrm{r}},\boldsymbol{z}_{\mathrm{a}}),$

and the discriminator’s performance is optimized in an adversarial manner,
with the specific optimization process illustrated in Fig. 3. The first part
focuses on maximizing the encoder-decoder capabilities, and the second part
involves swapping pseudo-labels to maximize discriminator loss.

Figure 4: The overall framework of WAVAE, training begins with the raw data
$\boldsymbol{x}_{\mathrm{r}}$ undergoing an augmentation algorithm $AUG$,
resulting in augmented data $\boldsymbol{x}_{\mathrm{a}}$. Concurrently, we
train a shared-parameter VAE separately for both sets of data. However,
evaluation, i.e., Anomaly detector, is conducted solely on the original model
between raw input $\boldsymbol{x}_{\mathrm{r}}$ and its’ reconstruction
$\hat{\boldsymbol{x}}_{\mathrm{r}}$, essentially designing an end-to-end
anomaly detector.

####  III-D3 Anomaly Scores

The training process and the determination of anomalies are illustrated in
Fig. 4. Reconstruction-based anomaly detection utilizes the deviation between
the original data and the reconstructed data as an anomaly score, denoted as
$AS(\boldsymbol{x},\hat{\boldsymbol{x}})$. We determine whether the input data
is anomalous by comparing the anomaly score with a pre-set threshold $\eta$.
The specific process is shown in Algorithm 1.

Input: Dataset $\mathcal{D}=\\{B_{tr}^{i},B_{e}^{i}\\}_{i=1}^{m}$, Training
batch
$B_{\mathrm{tr}}=\\{(\boldsymbol{x}^{(j)},\boldsymbol{y}^{(j})\\}^{b}_{j=1}\in\mathbb{R}^{b\times
s\times f}$, Evaluation batch $B_{\mathrm{e}}\in\mathbb{R}^{b\times s\times
c}$. // $b,s,c$ is the size of batch, sequence length and features

Output: Parameters of encoder $f_{\phi}$, decoder $g_{\theta}$, and
discriminator $\psi$, Anomaly threshold $\eta$.

1

2for  _each $B^{i}$ in training batch $B_{tr}$_ do

3 $B^{i}_{\mathrm{a}}=\mathcal{O}(B^{i})$; // Here the operation $\mathcal{O}$
is defined in E.q. (12) and E.q. (13).

4 $B^{i}_{\mathrm{a}}\subset B_{\mathrm{a}}$;

5 end for

6

7while  _unconverged_ do

8 for  _each $B^{i},B^{i}_{\mathrm{a}}$ in $\mathcal{D}$_ do

9 Compute gradients of Eq. (14) or Eq. (15) w.r.t. $\theta$ and $\phi$;

10 Update the parameters of $f,g$;

11 end for

12

13 end while

14

15for  _each $B^{i}_{e}$ in evaluation batch $B_{e}$_ do

16 $\hat{B^{j}_{e}}=g_{\theta}\big{(}f_{\phi}(B^{j}_{e})\big{)}$; //
Reconstruct the sequence

17 for  _each
$\boldsymbol{x}_{\mathrm{r}}^{i},\boldsymbol{x}_{\mathrm{a}}^{i}$ in $B_{e}$_
do

18
$Score=AS(\boldsymbol{x}_{\mathrm{r}}^{j},\boldsymbol{x}_{\mathrm{a}}^{j})$;
//Calculate the anomaly score based on the similarity

19 if  _$Score <\eta$_ then

20 $\boldsymbol{x}_{\mathrm{r}}^{i}$ is an anomaly;

21 else

22 $\boldsymbol{x}_{\mathrm{r}}^{i}$ is not an anomaly;

23 end if

24

25 end for

26

27 end for

Algorithm 1 The training process of WAVAE

##  IV Experiments

###  IV-A Benchmarks

To validate the effectiveness of our approach, we selected 16 reconstruction-
based models for anomaly detection in time series data as benchmarks, which
included 6 generative models (GANs and AEs based), which are specifically:

  1. 1.

Transformer Autoencoder (TAE) [50]: A transformer autoencoder encodes and
decodes time-series data to capture temporal dependencies.

  2. 2.

The Multi-scale CNN-RNN based Autoencoder (MSCREA): A CNN, RNN-based
autoencoder leverages convolutional neural networks for representation
extraction across multiple scales and recurrent neural networks for capturing
temporal dependencies, tailored for enhancing anomaly detection in time-series
data.

  3. 3.

BeatGAN (BGAN) [32]: A GAN-based model for ECG anomaly detection, learning
normal heartbeats to identify irregular patterns in time-series data.

  4. 4.

RNN-based Autoencoder (RAE) [51]: A Gated Recurrent Unit (GRU) based
autoencoder designed to encode and decode time-series data for anomaly
detection by learning sequential patterns and temporal relationships.

  5. 5.

CNN-based Autoencoder [52]: A CNN-based autoencoder architecture tailored for
time-series analysis, utilizing convolutional layers to identify spatial
patterns in data, essential for detecting anomalies in sequential datasets.

  6. 6.

RandNet (RN) [30]: An ensemble of randomly structured autoencoders with
adaptive sampling recognize efficiently and robustly detect outliers in data.

and $10$ anomaly detection methods for time-series data based on probabilistic
generative models, namely Variational Autoencoders, which are specifically:

  1. 1.

The Gaussian Mixture Model Variational Autoencoder (GMMVAE) [33]: VAE with GMM
priors combines the probabilistic framework of Gaussian mixtures with the
generative capabilities of VAEs to model complex distributions in time-series
data, facilitating robust anomaly detection through learned latent
representations.

  2. 2.

Variational Autoencoder (VAE) [34]: A traditional VAE model to model the
likelihood of generative data.

  3. 3.

Recurrent Neural Network based VAE (RNNVAE) [19]: Merges Recurrent Neural
Networks (RNN) with the variational approach to autoencoding, capturing
temporal dependencies within sequential data for improved anomaly detection
through stochastic latent spaces.

  4. 4.

The Variational RNN Autoencoder (VRAE) [20]: The Variational RNN Autoencoder
combines the sequence modeling strengths of RNNs with the probabilistic latent
space of variational autoencoders, aiming to improve anomaly detection in
time-series by learning complex temporal structures.

  5. 5.

$\alpha$-VQRAE, $\beta$-VQRAE, and $\gamma$-VQRAE [21]: Extensions of VRAE,
with RNN substituted by a Quasi-Recurrent Network and $\alpha$, $\beta$,
$\gamma$-loglikelihood loss to help the VAE model achieve robust
representation.

  6. 6.

$\alpha$-biVQRAE, $\beta$-biVQRAE, and $\gamma$-biVQRAE [21]: Variants of
VQRAE, with RNN extended to bilevel to achieve time dependence on time-series
data, while the $\alpha$, $\beta$, $\gamma$-loglikelihood loss helps the VAE
based model achieve robust representation.

###  IV-B Experiment Setup

TABLE I: Datasets overview. | Sequences | Anomaly Sequences | Avg. length | Avg. anomalies | Avg. Anomaly ratio (%) | Features | Time Range
---|---|---|---|---|---|---|---
GD | 5 | 1 | 16,220 | 39 | 0.24% | 18 | in 760 ms
HSS | 4 | 2 | 19,634 | 4,517 | 23.01% | 18 | in 14 ms
ECG | 1 | 1 | 500 | 31 | 6.2% | 2 | NA
TD | 1 | 1 | 1,000 | 362 | 36.2% | 2 | NA
S5_A1 | 67 | 67 | 1,416 | 25 | 1.79% | 1 | NA
S5_A2 | 100 | 100 | 1,421 | 5 | 0.35% | 1 | NA
S5_A3 | 100 | 100 | 1,680 | 9 | 0.54% | 1 | NA
S5_A4 | 100 | 100 | 1,680 | 8 | 0.48% | 1 | NA

Datasets: To validate the effectiveness of our proposed methodology, we
executed a series of experiments on a quartet of multivariate time series
datasets: Genesis Demonstrator Data for Machine Learning, High Storage System
Data for Energy Optimization, Electrocardiogram, and Trajectory Data, along
with a single univariate time series dataset, the Yahoo S5. These datasets,
encompassing several hundred temporal sequences, are sourced from real-world
industrial systems or are synthetically generated, comprehensively evaluating
the algorithm’s performance.

The Genesis Demonstrator dataset for machine learning
(GD)111https://www.kaggle.com/datasets/inIT-OWL/genesis-demonstrator-data-for-
machine-learning comprises 5 distinct sequences, encapsulating continuous or
discrete signals recorded from portable pick-and-place robots at millisecond
intervals. We harness the sequence replete with anomalies to target anomaly
detection, specifically the Genesis_AnomalyLabels.csv, which consists of
16,220 records. Within this framework, records marked with class 0 are
designated as normal, whereas the remaining classifications, classes 1 and 2,
are delineated as anomalies.

The High Storage System Data for Energy Optimization
(HSS)222https://www.kaggle.com/datasets/inIT-OWL/high-storage-system-data-for-
energy-optimization dataset is composed of 4 sequences documenting the
readings from induction sensors situated on conveyor belts. Anomaly detection
is conducted on two labeled sequences: HRSS_anomalous_standard.csv and
HRSS_anomalous_optimized.csv, together encompassing 23,645 records. Within
each sequence, records tagged with class 0 are categorized as normal, whereas
those labeled with class 1 are identified as anomalous.

The Electrocardiogram dataset
(ECG)333https://www.cs.ucr.edu/~eamonn/time_series_data_2018/ is comprised of
a solitary time-series sequence collected from PhysioNet signals attributed to
a patient with severe congestive heart failure. For the sake of consistent
comparison, we followed the guidelines proposed by researcher [53], utilizing
ECG5000_TRAIN.tsv from the training datasets for anomaly detection. This
approach involves classifying three classes (Supraventricular ectopic beats,
PVC, and Unclassifiable events) as anomalies, while the two remaining classes
(R-on-T Premature Ventricular Contraction (PVC), Normal) are maintained as the
normative data.

The Trajectory Data
(TD)444https://www.cs.ucr.edu/~eamonn/time_series_data_2018/ dataset
encapsulates a unique time-series sequence, with each data point being two-
dimensional. These points represent the detection algorithm’s accuracy in
delineating the skeletal structure of a hand, coupled with assessments from
three human evaluators on the algorithm’s predictive accuracy. We undertake an
unsupervised anomaly detection task in this setting using the
HandOutlines_TRAIN.tsv file extracted from the training set, comprising 1000
instances. Within this dataset, instances classified as normal bear the label
of class 1, and those recognized as anomalies carry the label of class 0.

The Yahoo S5
(S5)555https://webscope.sandbox.yahoo.com/catalog.php?datatype=s&did=70
dataset encompasses an array of both authentic and synthetic time-series
sequences. The synthetic component of this collection is distinguished by
sequences that display diverse trends, degrees of noise, and seasonal
patterns. Conversely, the real segment of the dataset encapsulates sequences
that chronicle the performance metrics of assorted Yahoo services. This
compilation contains a total of 367 labeled anomalies across both real and
synthetic time series. The dataset is segmented into four distinct subsets:
the A1, A2, A3, and A4 benchmarks. Within this schema, entries marked with
class 0 are categorized as normal, whereas those annotated with class 1 are
identified as anomalous.

The comparison of each dataset concerning data volume, dimensionality,
temporal span of sequence acquisition, and proportion of anomalies is
illustrated in Table I.

Anomaly Scores: The presented code snippet delineates a section of an anomaly
detection algorithm employing a Variational Autoencoder (VAE). The VAE is
tasked with modeling the probability distribution of the input data. During
the anomaly detection phase, the input data is reconstructed, and the
reconstruction error is computed as the sum of squared differences between the
original data $\boldsymbol{x}_{\mathrm{r}}$ and the reconstructed data
$\hat{\boldsymbol{x}}$. Anomalies are flagged by setting a threshold at the
99th percentile of the error distribution, isolating the top $1\%$ of
instances with the highest errors as anomalies.

Performance metrics such as F1 score, precision, and recall are calculated to
gauge the accuracy of the test. These metrics are critical in determining the
anomaly detection mechanism’s true positive rate (precision) and sensitivity
(recall). Furthermore, the algorithm utilizes the confusion matrix
$\mathrm{C}$, the F1 score, and Cohen’s kappa score $\kappa$ to evaluate its
performance comprehensively.

The utilization of the Area Under the Receiver Operating Characteristic curve
(AUROC) and the Precision-Recall curve (PRAUC) is noteworthy:

  * •

AUROC is derived by plotting the False Positive Rate (FPR) against the True
Positive Rate (TPR) and calculating the Area Under the Curve (AUC). It is a
robust measure of the classifier’s discriminative power, particularly in class
imbalance.

  * •

PRAUC is obtained by plotting precision against the recall and computing the
AUC, providing valuable insight into the performance of the positive class,
which is often the minority in anomaly detection tasks.

The advantages of AUROC and PRAUC include their ability to provide a holistic
measure of model performance insensitive to threshold selection and their
effectiveness in conditions of class imbalance. The AUROC reflects the
likelihood that the classifier will rank a randomly chosen positive instance
more highly than a negative one. On the other hand, the PRAUC focuses on the
model’s performance in the positive class, making it particularly useful for
datasets with significant class imbalances. The calculation of these metrics
is as follows:

  * •

The function roc_curve computes the FPR and TPR for various threshold values,
while the function auc calculates the area under the ROC curve to determine
the AUROC.

  * •

The function precision_recall_curve is employed to compute the precision and
recall at different thresholds, with the auc function again used to calculate
the area under the precision-recall curve, yielding the PRAUC.

Using these metrics, the model’s performance can be evaluated not solely based
on accuracy but also on its robustness against false anomaly classifications
(precision) and its capability to identify all true anomalies (recall).

Implementation details: Our experimental setup was standardized to ensure a
level playing field and control for potential performance biases introduced by
Pytorch Versions. The versions selected for all implementations were Python
3.7.16, PyTorch 1.1.0, NumPy 1.19.2, CUDA toolkit 10.0.130, and cuDNN 7.6.5.
This approach guaranteed that the proposed and comparative algorithms were
evaluated under equivalent computational environments. Our hardware setup
included NVIDIA Quadro RTX 6000 GPUs with driver version 525.105.17 and CUDA
version 12.0. Additionally, we incorporated a randomness control module,
employing seed values to govern the stochasticity across computational units
such as GPU, Python, and PyTorch.

###  IV-C Performance

The anomaly detection performance on five public datasets can be found in
Tables II,III. In terms of PRAUC and ROCAUC metrics, our method outperformed
the baseline across all datasets, regardless of whether they are AE and GAN-
based generative models or VAE-based ones. Note that our comparative data
originates from [21]. Additionally, we observed that methods based on
adversarial mechanisms generally underperform compared to those using
contrastive loss.

TABLE II: Overall accuracy, PR-AUC. For each dataset, the three best-
performing methods are denoted using distinct markings: bold for the top
method, superscript asterisk∗ for the second-best, and underline for the
third-best.

Models/Datasets |  GD |  HSS |  ECG |  TD |  S5
---|---|---|---|---|---
TAE |  0.088 |  0.195 |  0.138 |  0.175 |  0.298
MSCREA |  0.075 |  0.161 |  0.105 |  0.148 |  N/A
BGAN |  0.109 |  0.214 |  0.103 |  0.151 |  0.434
RAE |  0.128 |  0.242 |  0.118 |  0.163 |  0.421
CAE |  0.116 |  0.207 |  0.107 |  0.177 |  0.383
RN |  0.112 |  0.146 |  0.105 |  0.168 |  0.232
GMMVAE |  0.142 |  0.216 |  0.163 |  0.364 |  0.458
VAE |  0.097 |  0.203 |  0.131 |  0.188 |  0.272
RNNVAE |  0.086 |  0.204 |  0.079 |  0.118 |  0.211
VRAE |  0.131 |  0.219 |  0.144 |  0.165 |  0.298
$\alpha$-VQRAE |  0.235 |  0.225 |  0.177 |  0.428 |  0.487
$\beta$-VQRAE |  0.242 |  0.223 |  0.177 |  0.427 |  0.525
$\gamma$-VQRAE |  0.245 |  0.222 |  0.184 |  0.423 |  0.499
$\alpha$-biVQRAE |  0.249 |  0.227∗ |  0.141 |  0.429 |  0.490
$\beta$-biVQRAE |  0.256 |  0.224 |  0.189∗ |  0.430 |  0.527∗
$\gamma$-biVQRAE |  0.258∗ |  0.222 |  0.186 |  0.432 |  0.524
WAVQRAE-Adverisal |  0.304 |  0.286 |  0.190 |  0.440 |  0.805
WAVQRAE-Contrast |  0.307 |  0.358 |  0.200 |  0.504 |  0.838

TABLE III: Overall accuracy, ROC-AUC.

Models/Datasets |  GD |  HSS |  ECG |  TD |  S5
---|---|---|---|---|---
TAE |  0.652 |  0.563∗ |  0.542 |  0.531 |  0.635
MSCREA |  0.582 |  0.509 |  0.509 |  0.519 |  N/A
BGAN |  0.673 |  0.549 |  0.547 |  0.622 |  0.677
RAE |  0.608 |  0.537 |  0.552 |  0.593 |  0.753
CAE |  0.641 |  0.560 |  0.574 |  0.583 |  0.757∗
RN |  0.731 |  0.526 |  0.524 |  0.533 |  0.575
GMMVAE |  0.763 |  0.534 |  0.533 |  0.531 |  0.815
VAE |  0.664 |  0.525 |  0.531 |  0.643 |  0.678
RNNVAE |  0.595 |  0.516 |  0.536 |  0.574 |  0.642
VRAE |  0.658 |  0.521 |  0.551 |  0.662∗ |  0.660
$\alpha$-VQRAE |  0.970 |  0.529 |  0.592 |  0.539 |  0.858
$\beta$-VQRAE |  0.968 |  0.520 |  0.583 |  0.535 |  0.849
$\gamma$-VQRAE |  0.969 |  0.524 |  0.598 |  0.547 |  0.875
$\alpha$-biVQRAE |  0.975 |  0.538 |  0.597 |  0.542 |  0.873
$\beta$-biVQRAE |  0.976 |  0.527 |  0.603∗ |  0.546 |  0.864
$\gamma$-biVQRAE |  0.978∗ |  0.526 |  0.601 |  0.549 |  0.882∗
WAVQRAE-Adverisal |  0.991 |  0.563 |  0.612 |  0.579 |  0.883
WAVQRAE-Contrast |  0.996 |  0.575 |  0.630 |  0.646 |  0.899

###  IV-D Sensitivity Analysis

To rigorously assess the sensitivity of hyperparameters in our model, we have
conducted an extensive series of ablation experiments. This comprehensive
evaluation encompasses many hyperparameter sets throughout the entire end-to-
end training process.

  * •

Specifically, we investigate variations in VAE-related hyperparameters such as
the $\beta$ in E.q. (1) and E.q. (2) to balance the inference and
reconstruction in VAE training, the dimension of $\boldsymbol{z}$, and the
reconstruction loss $\mathcal{L}_{\mathrm{R}}$.

  * •

We also scrutinize SSL-related hyperparameters like the number of
discriminator layers, the weight of infoVAE loss, and the augmentation method.

  * •

In addition, we delve into hyperparameters pertinent to time-series
processing, including the sequence length and hidden variables in embeddings.

  * •

Lastly, we explore adjustments in deep learning hyperparameters, including
batch size, learning rates, and the number of epochs.

In each set of experiments, we systematically vary a selected hyperparameter
within its feasible range while maintaining the default settings for all other
hyperparameters to isolate and understand the individual impact of each
hyperparameter adjustment on the model’s performance.

Figure 5: Sensitivity analysis of VAE related hyperparameters indicates
significant findings: (a) reveals that the dimension of $\boldsymbol{z}$
profoundly influences outcomes, with optimal performance when the dimension
ranges between 14 and 20. (b) shows that $\beta$ exerts a minimal effect on
optimization, peaking in efficacy at 0.001. (c) demonstrates the superior
performance of the MSE loss function. Figure 6: Sensitivity analysis of SSL
loss related hyperparameters. From (a), it is observed that the weight of the
infoNCE Loss has a minimal impact on the overall effectiveness. Conversely,
(b) indicates that the number of layers in the discriminator significantly
affects the results, with the best performance observed between 2 and 3
layers. (c) illustrates varying augmentation approaches, indicating that using
min-max normalization (MinMax) on both original and augmented data is the most
effective. Figure 7: Sensitivity analysis of sequence-related hyperparameters.
(a) indicates that the model’s anomaly detection performance is not affected
by the length of the series. (b) shows that the encoding network achieves the
best performance when the hidden state size is 32.  Figure 8: Sensitivity
analysis of deep learning related hyperparameters. (a) A batch size of 64
yields optimal results. (b) The learning rate has minimal impact on the model.
(c) The best performance was observed at 50 epochs.

####  IV-D1 Effect of VAEs

Ideally, VAE is adept at modeling data distributions, encapsulating the
potential to fit the likelihood of diverse data modalities through its
sophisticated encoder-decoder architecture rooted in deep neural networks.
Concurrently, it postulates a manifold-based, low-dimensional, continuous and
smooth space. However, in real-world applications, the efficacy of a VAE’s
data likelihood estimation is subject to substantial variability, influenced
by the selection of encoder-decoder architectures, the diversity of data
modalities, and the specificities of the task at hand. To isolate and assess
the effects of these factors on anomaly detection performance, we embark on a
systematic sensitivity analysis of hyperparameters spanning three pivotal
domains: weight of KL Divergence $\beta$, dimensions of latent variables
$\boldsymbol{d}$, and reconstruction loss function. Through this methodical
examination, we aim to elucidate the impact of these variables on the VAE’s
reconstruction proficiency, thereby enhancing the model’s suitability for
anomaly detection endeavors.

Dimensions of Latent variables: The dimension of the latent variable
determines the amount of information the encoder compresses to maximize the
log-likelihood under the Information Bottleneck Theory. Simultaneously, it
influences the dependency and causality of low-dimensional space
representations under the manifold assumption. The fundamental assumptions of
generative model-based time-series anomaly detection posit that anomaly data
will deviate from the likelihood of normal data. We conducted experiments with
varying dimensions of the latent variable $\boldsymbol{z}$ to develop a robust
likelihood function, specifically exploring $\\{8,10,12,14,16,18,20\\}$. The
outcomes and in-depth analysis of these experiments are detailed in Fig. 5
(a), demonstrating the optimal dimension for $\boldsymbol{z}$.

KL Divergence Weight: The KL weight controls the balance between
representation learning and reconstruction in the VAE model and the
information during the compression process, affecting the model’s robustness
during training. We use the hyperparameter $\beta$ to adjust the VAE’s
compression capability. We selected five distinct values for the KL term to
assess their impact, specifically $\\{1e-5,5e-5,1e-4,5e-4,1e-3\\}$. Detailed
results and analysis of this exploration are presented in Fig. 5 (b).

Reconstruction Loss Function: In the Eq. (1) and E.q. (2), we fit different
likelihood distributions by optimizing the specific reconstruction loss. For
discrete data, we optimize the Binary Cross Entropy (BCE) loss, i.e.,
$\mathcal{L}_{\mathrm{R}}^{\mathrm{BCE}}$ to fit the log-likelihood of a
multivariate Bernoulli distribution, denoted as:

$\displaystyle\mathcal{L}_{\mathrm{R}}^{\mathrm{BCE}}$ (16) $\displaystyle=$
$\displaystyle
E_{q_{\phi}\left(\boldsymbol{z}\mid\boldsymbol{x}\right)}\left[\log
p_{\theta}\left(\boldsymbol{x}\mid\boldsymbol{z}\right)\right]$
$\displaystyle=$ $\displaystyle
E_{q_{\phi}\left(\boldsymbol{z}\mid\boldsymbol{x}\right)}\left[\sum_{d=1}^{D}\boldsymbol{x}_{d}\log\lambda_{\theta,d}\left(\boldsymbol{z}\right)+\left(1-\boldsymbol{x}_{d}\right)\log\left(1-\lambda_{\theta,d}\left(\boldsymbol{z}\right)\right)\right],$

Where $\boldsymbol{x}\in\\{0,1\\}^{D}$ and $\lambda\in\\{0,1\\}^{D}$ are the
parameters of univariate Bernoulli distributions. For continuous data, we
optimize the Mean Square Error (MSE) $\mathcal{L}_{\mathrm{R}}^{\mathrm{MSE}}$
to fit the log-likelihood of a multivariate Gaussian distribution, denoted:

$\displaystyle\mathcal{L}_{\mathrm{R}}^{\mathrm{MSE}}=$ $\displaystyle
E_{q\left(\boldsymbol{z}\mid\boldsymbol{x}\right)}\left[\log
p\left(\boldsymbol{x}\mid\boldsymbol{z}\right)\right]$ (17) $\displaystyle=$
$\displaystyle\frac{1}{D}\sum_{d=1}^{D}||\boldsymbol{x}_{d}-\hat{\boldsymbol{x}}_{d}||^{2}.$

We also tested two robust variants [54] based on the Bernoulli likelihood
distribution:

$\displaystyle\mathcal{L}_{\mathrm{R}}^{\mathrm{robust1}}$ (18)
$\displaystyle=$
$\displaystyle\frac{\alpha_{1}+1}{\alpha_{1}}\left(\prod_{d=1}^{D}\left(\boldsymbol{x}_{d}\hat{\boldsymbol{x}}_{d}^{\alpha_{1}}+\left(1-\boldsymbol{x}_{d}\right)\left(1-\hat{\boldsymbol{x}}_{d}\right)^{\alpha_{1}}\right)-1\right),$

and Gaussian likelihood distribution:

$\displaystyle\mathcal{L}_{\mathrm{R}}^{\mathrm{robust2}}$ (19)
$\displaystyle=$
$\displaystyle\frac{\alpha_{2}+1}{\alpha_{2}}\left(\frac{1}{\left(2\pi\sigma^{2}\right)^{\alpha_{2}D/2}}\exp\left(-\frac{\alpha_{2}}{2\sigma^{2}}\sum_{d=1}^{D}\left\|\hat{\boldsymbol{x}}_{d}-\boldsymbol{x}_{d}\right\|^{2}\right)-1\right),$

where $\alpha_{1},\alpha_{1}$ are the hyperparameters and $\sigma$ is the
variance. The analysis and comparison of four types of loss functions are
illustrated in Fig. 5 (c).

####  IV-D2 Effect of SSL Loss

The SSL Loss in E.q. (14) and E.q. (15) will be biased by the approximation
methods and augmentation types.

Mutual Information Approximation: Our study investigated two loss functions
for mutual information maximization: the infoNCE Loss in contrastive learning,
with weight hyperparameters $\\{0.1,0.2,0.3,0.4,0.5\\}$, detailed in Fig. 6
(a), and the adversarial learning discriminator, varying layers [3, 4, 5, 6],
analyzed in Fig. 6 (b).

Augumentation Methods: For self-supervised methods applied to time-series
data, augmentation can be employed to mine the intrinsic characteristics of
the data, addressing the issue of insufficient data for deep models. To
validate the effectiveness of our approach, we experimented with various
strong augmentations that enhance the time dependencies and frequency domain
representations of time-series data. In parallel, we also explored several
weak augmentations, specifically normalization techniques applied to time-
series data. Our findings indicate that the domains transformed by strong
augmentations are ill-suited for generating robust likelihood functions,
leading to suboptimal results in anomaly detection. In that case, We conducted
sensitivity analysis experiments by testing two combinations of weak
augmentations. These combinations include both raw and augmented data using
MinMax (Fig. 6 (c) MM), raw data with MinMax and augmented data with Standard
(Fig. 6 (c) MS), raw data with Standard and augmented data with MinMax (Fig. 6
(c) SM), and both raw and augmented data using standardization (Fig. 6 (c)
SS). Specific experimental results and analysis are presented in Fig. 6 (c).

####  IV-D3 Effect of Time Series Processing

The time series’ inherent characteristics, such as the window size in a batch
and the memory step length in the encoding model, can impact model
performance.

Sequence Length In time-series data analysis, the window length is critical as
it sets the data truncation extent, which is essential for detecting anomalies
with periodicity or spatio-temporal continuity. Furthermore, the length of the
time series plays a significant role in identifying contextual anomalies. We
chose time series lengths of $\\{8,16,32,64,96\\}$ for our sensitivity
analysis. Detailed experimental results and analyses are illustrated in Fig. 7
(a).

Hidden Vairbales: We evaluated the impact of different hidden space sizes in
the embedding, experimenting with dimensions of
$\\{1,2,3,4,8,16,32,64,128,256\\}$. Detailed analysis and results are
presented in Fig. 7 (b).

####  IV-D4 Effect of Deep Learning

In deep models, batch size, learning rates, and epochs cooperate to guide the
model convergence to the optimal.

Batch Sizes: By modulating the batch size, we gain insights into the stability
of gradient updates and their consequent impact on model convergence. To this
end, we selected batch sizes $\\{32,64,128\\}$ to empirically ascertain their
influence on the model’s performance. Detailed experimental results and
analyses are illustrated in Fig. 8 (a).

Leanring Rates: Step size in gradient descent induced by the learning rates is
taken during optimization and can significantly influence the model’s ability
to find minima. We test learning rates of $\\{0.001,0.01,0.1\\}$ and
systematically study their effects and optimize the model’s performance. Fig.
8 (b) illustrates detailed experimental results and analyses.

Number of Epochs: In the context of unsupervised anomaly detection, rather
than focusing on model generalization, we prioritize the impact of training
duration on performance. We fix the randomness and maintain consistent
hyperparameters, testing the same model’s anomaly detection capabilities at
epochs $\\{10,20,30,40,50\\}$. Fig. 8 (c) illustrates detailed experimental
results and analyses.

##  V Conclusion

The VAE-based anomaly detection effectively captures underlying data
distributions in time series analysis. As a result, anomalies outside this
distribution show notable reconstruction errors. However, the limited amount
of data samples can affect the model’s ability to fit this distribution,
especially when considering the rare and hard-to-detect nature of anomalies in
real-world situations. To combat data scarcity, we introduce a weakly
augmented VAE for time series anomaly detection. The model can achieve a more
robust representation in the latent space through joint training on augmented
data. Meanwhile, we present two self-supervised strategies, adversarial and
contrastive learning, to enhance the performance in data fitting. Quantitative
experimental results demonstrate that our approach exhibits commendable
performance across five datasets under two distinct foundational model
architectures.

## References

  * [1] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [2] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [3] A. Graves, R. K. Srivastava, T. Atkinson, and F. Gomez, “Bayesian flow networks,” _arXiv preprint arXiv:2308.07037_ , 2023.
  * [4] A. Vahdat and J. Kautz, “Nvae: A deep hierarchical variational autoencoder,” _Advances in neural information processing systems_ , vol. 33, pp. 19 667–19 679, 2020.
  * [5] Y. Takida, T. Shibuya, W. Liao, C.-H. Lai, J. Ohmura, T. Uesaka, N. Murata, S. Takahashi, T. Kumakura, and Y. Mitsufuji, “Sq-vae: Variational bayes on discrete representation with self-annealed stochastic quantization,” _arXiv preprint arXiv:2205.07547_ , 2022.
  * [6] L. Manduchi, M. Vandenhirtz, A. Ryser, and J. Vogt, “Tree variational autoencoders,” _arXiv preprint arXiv:2306.08984_ , 2023.
  * [7] A. Razavi, A. Van den Oord, and O. Vinyals, “Generating diverse high-fidelity images with vq-vae-2,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [8] Z. Wu, L. Cao, and L. Qi, “evae: Evolutionary variational autoencoder,” _arXiv preprint arXiv:2301.00011_ , 2023.
  * [9] L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni, “Modeling tabular data using conditional gan,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [10] A. Kotelnikov, D. Baranchuk, I. Rubachev, and A. Babenko, “Tabddpm: Modelling tabular data with diffusion models,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 17 564–17 579.
  * [11] H. Zhu, C. Balsells-Rodas, and Y. Li, “Markovian gaussian process variational autoencoders,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 42 938–42 961.
  * [12] X.-B. Jin, W.-T. Gong, J.-L. Kong, Y.-T. Bai, and T.-L. Su, “Pfvae: a planar flow-based variational auto-encoder prediction model for time series data,” _Mathematics_ , vol. 10, no. 4, p. 610, 2022.
  * [13] X. Liu, J. Yuan, B. An, Y. Xu, Y. Yang, and F. Huang, “C-disentanglement: Discovering causally-independent generative factors under an inductive bias of confounder,” _arXiv preprint arXiv:2310.17325_ , 2023.
  * [14] Z. Wu and L. Cao, “C2vae: Gaussian copula-based vae differing disentangled from coupled representations with contrastive posterior,” _arXiv preprint arXiv:2309.13303_ , 2023.
  * [15] T. Z. Xiao and R. Bamler, “Trading information between latents in hierarchical variational autoencoders,” _arXiv preprint arXiv:2302.04855_ , 2023.
  * [16] S. Tonekaboni, C.-L. Li, S. O. Arik, A. Goldenberg, and T. Pfister, “Decoupling local and global representations of time series,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2022, pp. 8700–8714.
  * [17] M. Tschannen, O. Bachem, and M. Lucic, “Recent advances in autoencoder-based representation learning,” _arXiv preprint arXiv:1812.05069_ , 2018.
  * [18] J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio, “A recurrent latent variable model for sequential data,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [19] D. Park, Y. Hoshi, and C. C. Kemp, “A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 1544–1551, 2018.
  * [20] Y. Su, Y. Zhao, C. Niu, R. Liu, W. Sun, and D. Pei, “Robust anomaly detection for multivariate time series through stochastic recurrent neural network,” in _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 2019, pp. 2828–2837.
  * [21] T. Kieu, B. Yang, C. Guo, R.-G. Cirstea, Y. Zhao, Y. Song, and C. S. Jensen, “Anomaly detection in time series with robust variational quasi-recurrent autoencoders,” in _2022 IEEE 38th International Conference on Data Engineering (ICDE)_. IEEE, 2022, pp. 1342–1354.
  * [22] C.-Y. Lai, F.-K. Sun, Z. Gao, J. H. Lang, and D. S. Boning, “Nominality score conditioned time series anomaly detection by point/sequential reconstruction,” _arXiv preprint arXiv:2310.15416_ , 2023.
  * [23] Y. Li, W. Chen, B. Chen, D. Wang, L. Tian, and M. Zhou, “Prototype-oriented unsupervised anomaly detection for multivariate time series,” in _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202. PMLR, 2023, pp. 19 407–19 424. [Online]. Available: https://proceedings.mlr.press/v202/li23d.html
  * [24] A. Khan and A. Storkey, “Adversarial robustness of vaes through the lens of local geometry,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2023, pp. 8954–8967.
  * [25] C. Zhou and R. C. Paffenroth, “Anomaly detection with robust deep autoencoders,” in _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2017, pp. 665–674.
  * [26] H. Akrami, A. A. Joshi, J. Li, S. Aydore, and R. M. Leahy, “Robust variational autoencoder,” _arXiv preprint arXiv:1905.09961_ , 2019.
  * [27] S. Eduardo, A. Nazábal, C. K. Williams, and C. Sutton, “Robust variational autoencoders for outlier detection and repair of mixed-type data,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2020, pp. 4056–4066.
  * [28] S. Cao, J. Li, K. P. Nelson, and M. A. Kon, “Coupled vae: Improved accuracy and robustness of a variational autoencoder,” _Entropy_ , vol. 24, no. 3, p. 423, 2022.
  * [29] K. Zhang, Q. Wen, C. Zhang, R. Cai, M. Jin, Y. Liu, J. Zhang, Y. Liang, G. Pang, D. Song _et al._ , “Self-supervised learning for time series analysis: Taxonomy, progress, and prospects,” _arXiv preprint arXiv:2306.10125_ , 2023.
  * [30] J. Chen, S. Sathe, C. Aggarwal, and D. Turaga, “Outlier detection with autoencoder ensembles,” in _Proceedings of the 2017 SIAM international conference on data mining_. SIAM, 2017, pp. 90–98.
  * [31] T. Kieu, B. Yang, and C. S. Jensen, “Outlier detection for multidimensional time series using deep neural networks,” in _2018 19th IEEE international conference on mobile data management (MDM)_. IEEE, 2018, pp. 125–134.
  * [32] B. Zhou, S. Liu, B. Hooi, X. Cheng, and J. Ye, “Beatgan: Anomalous rhythm detection using adversarially generated time series.” in _IJCAI_ , vol. 2019, 2019, pp. 4433–4439.
  * [33] W. Liao, Y. Guo, X. Chen, and P. Li, “A unified unsupervised gaussian mixture variational autoencoder for high dimensional outlier detection,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 1208–1217.
  * [34] H. Xu, W. Chen, N. Zhao, Z. Li, J. Bu, Z. Li, Y. Liu, Y. Zhao, D. Pei, Y. Feng _et al._ , “Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications,” in _Proceedings of the 2018 world wide web conference_ , 2018, pp. 187–196.
  * [35] S. Zhao, J. Song, and S. Ermon, “Infovae: Information maximizing variational autoencoders,” _arXiv preprint arXiv:1706.02262_ , 2017.
  * [36] G. Woo, C. Liu, D. Sahoo, A. Kumar, and S. Hoi, “Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting,” _arXiv preprint arXiv:2202.01575_ , 2022.
  * [37] M. Hou, C. Xu, Z. Li, Y. Liu, W. Liu, E. Chen, and J. Bian, “Multi-granularity residual learning with confidence estimation for time series prediction,” in _Proceedings of the ACM Web Conference 2022_ , 2022, pp. 112–121.
  * [38] H. Lee, E. Seong, and D.-K. Chae, “Self-supervised learning with attention-based latent signal augmentation for sleep staging with limited labeled data,” in _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, LD Raedt, Ed. International Joint Conferences on Artificial Intelligence Organization_ , vol. 7, 2022, pp. 3868–3876.
  * [39] J. Xu, H. Wu, J. Wang, and M. Long, “Anomaly transformer: Time series anomaly detection with association discrepancy,” _arXiv preprint arXiv:2110.02642_ , 2021.
  * [40] Z. Yue, Y. Wang, J. Duan, T. Yang, C. Huang, Y. Tong, and B. Xu, “Ts2vec: Towards universal representation of time series,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 8, 2022, pp. 8980–8987.
  * [41] L. Yang and S. Hong, “Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 25 038–25 054.
  * [42] Z. Wang, X. Xu, W. Zhang, G. Trajcevski, T. Zhong, and F. Zhou, “Learning latent seasonal-trend representations for time series forecasting,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 38 775–38 787, 2022.
  * [43] W. Chen, L. Tian, B. Chen, L. Dai, Z. Duan, and M. Zhou, “Deep variational graph convolutional recurrent network for multivariate time series anomaly detection,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 3621–3633.
  * [44] S. N. Shukla and B. M. Marlin, “Heteroscedastic temporal variational autoencoder for irregularly sampled time series,” _arXiv preprint arXiv:2107.11350_ , 2021.
  * [45] W. Zhang, C. Zhang, and F. Tsung, “Grelen: Multivariate time series anomaly detection from the perspective of graph relational learning,” in _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , vol. 7, 2022, pp. 2390–2397.
  * [46] Y. Li, X. Lu, Y. Wang, and D. Dou, “Generative time series forecasting with diffusion, denoise, and disentanglement,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 009–23 022, 2022.
  * [47] J. M. L. Alcaraz and N. Strodthoff, “Diffusion-based time series imputation and forecasting with structured state space models,” _arXiv preprint arXiv:2208.09399_ , 2022.
  * [48] H. Wen, Y. Lin, Y. Xia, H. Wan, R. Zimmermann, and Y. Liang, “Diffstg: Probabilistic spatio-temporal graph forecasting with denoising diffusion models,” _arXiv preprint arXiv:2301.13629_ , 2023.
  * [49] P. Cheng, W. Hao, S. Dai, J. Liu, Z. Gan, and L. Carin, “Club: A contrastive log-ratio upper bound of mutual information,” in _International conference on machine learning_. PMLR, 2020, pp. 1779–1788.
  * [50] H. Meng, Y. Zhang, Y. Li, and H. Zhao, “Spacecraft anomaly detection via transformer reconstruction error,” in _Proceedings of the International Conference on Aerospace System Science and Engineering 2019_. Springer, 2020, pp. 351–362.
  * [51] P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, and G. Shroff, “Lstm-based encoder-decoder for multi-sensor anomaly detection,” _arXiv preprint arXiv:1607.00148_ , 2016.
  * [52] C. Zhang, D. Song, Y. Chen, X. Feng, C. Lumezanu, W. Cheng, J. Ni, B. Zong, H. Chen, and N. V. Chawla, “A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 33, no. 01, 2019, pp. 1409–1416.
  * [53] Y. Chen, Y. Hao, T. Rakthanmanon, J. Zakaria, B. Hu, and E. n. Keogh, “A general framework for never-ending learning from time series streams,” _Data mining and knowledge discovery_ , vol. 29, pp. 1622–1664, 2015.
  * [54] F. Futami, I. Sato, and M. Sugiyama, “Variational inference based on robust divergences,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2018, pp. 813–822.