2021ine

.

1]University of Rochester, NY, USA

2]University of Connecticut, CT, USA

3]NEC American Lab, NJ, USA

# Deep Federated Anomaly Detection for Multivariate Time Series Data

Wei Zhu  Dongjin Song  Yuncong Chen  Wei Cheng  Bo Zong  Takehiko Mizoguchi
Cristian Lumezanu  Haifeng Chen  Jiebo Luo  [  [  [

###### Abstract

Despite the fact that many anomaly detection approaches have been developed
for multivariate time series data, limited effort has been made on federated
settings in which multivariate time series data are heterogeneously
distributed among different edge devices while data sharing is prohibited. In
this paper, we investigate the problem of federated unsupervised anomaly
detection and present a Federated Exemplar-based Deep Neural Network (Fed-
ExDNN) to conduct anomaly detection for multivariate time series data on
different edge devices. Specifically, we first design an Exemplar-based Deep
Neural network (ExDNN) to learn local time series representations based on
their compatibility with an exemplar module which consists of hidden
parameters learned to capture varieties of normal patterns on each edge
device. Next, a constrained clustering mechanism (FedCC) is employed on the
centralized server to align and aggregate the parameters of different local
exemplar modules to obtain a unified global exemplar module. Finally, the
global exemplar module is deployed together with a shared feature encoder to
each edge device and anomaly detection is conducted by examining the
compatibility of testing data to the exemplar module. Fed-ExDNN captures local
normal time series patterns with ExDNN and aggregates these patterns by FedCC,
and thus can handle the heterogeneous data distributed over different edge
devices simultaneously. Thoroughly empirical studies on six public datasets
show that ExDNN and Fed-ExDNN can outperform state-of-the-art anomaly
detection algorithms and federated learning techniques.

###### keywords:

Federated Learning, Unsupervised Anomaly Detection, Representation Learning

##  1 Introduction

Anomaly detection in multivariate time series refers to identifying abnormal
status in certain time steps of the time series data kim2012robust ;
zong2018deep . Building an effective unsupervised anomaly detection algorithm,
however, is challenging since it requires collecting and profiling as much as
(normal) multivariate time series data so as to reduce potential false
positives ruff2018deep ; malhotra2015long ; zhang2019deep . With the rapid
development of 5G networks, multivariate time series data are increasingly
collected in various types of Internet of Things (IoT) edge devices, e.g.,
mobile phones, healthcare, wearable devices, etc. However, due to privacy
concerns geyer2017differentially , regulations yang2019federated , and
transmission load bonawitz2019towards , directly transferring data from edge
devices to a centralized server in order to train a unified anomaly detection
model is usually infeasible or prohibited yang2019federated ;
mcmahan2017communication . Consequently, there is a huge demand to develop an
anomaly detection algorithm that can collaboratively handle the multivariate
time series data distributed on different edge devices while preserving
privacy.

For this purpose, we resort to federated learning and aim to conduct privacy-
preserving anomaly detection. Specifically, assuming the normal status of
multivariate time series data consists of $K$ different modes which are
heterogeneously distributed over $L$ different edge devices, we aim to learn a
unified model that can not only preserve data privacy on each edge device but
also identify all anomalous situations accurately. A notable challenge is that
due to the environmental conditions or other external factors, the time series
data collected on each edge device may only partially (i.e., less than $K$
modes) cover all different modes of normal status. In this case, simply
training an anomaly detection model may involve many false positives. Taking
wearable devices as an example, a mode can be “walking”, “sitting”, “running”,
“bicycling”, or “standing”, while an anomaly can represent “falling down”
which is unusual to happen. A notable issue with this setting is that due to
environmental conditions or other external factors, the time series data
collected on each edge device may only partially cover the entire normal
space. In other words, they are often heterogeneously distributed among
different edge devices. For instance, an elder person tends to have activities
of “walking”, “sitting”, and “standing”, while a young person may prefer
activities of “walking”, “running”, and “bicycling”. In this case, if we only
train an anomaly detection model based on the data collected from the elder
person’s edge device, we may falsely detect “running” and “bicycling” as
anomalies (i.e., “falling down”) since they are not considered in the
training.

Directly applying existing federated learning approaches to perform
unsupervised anomaly detection in multivariate time series data often leads to
inferior performance. This is because most existing anomaly detection methods
assume that the entire normal space is covered by the training data
pang2020deep  which may not be true with the current setting; another reason
is that existing federated learning algorithms, e.g., Federated Averaging
(FedAvg) mcmahan2017communication , are originally designed for supervised
learning and may not be able to properly handle the unsupervised tasks with
heterogeneous data distributed on edge devices. Simply combining federated
learning and unsupervised anomaly detection algorithms could lead to a series
of problems. One issue is that each locally trained model may only partially
cover the entire normal space, the federated learning algorithms, e.g.,
federated averaging operation, could produce a global model which may collapse
to certain normal modes and even cover a certain part of the abnormal space
wang2020federated . As a result, many anomalies could be mistreated as normal
status and vice versa. Another issue is that existing unsupervised anomaly
detection methods often rely on the encoder-decoder framework malhotra2015long
; zhang2019deep  and Generative Adversarial Network (GAN)
goodfellow2014generative ; schlegl2017unsupervised  to extract semantic
representation, and the extra parameters brought by the decoder and generator
may result in a heavy communication cost.

To address the aforementioned issues, in this paper, we present a Federated
Exemplar-based Deep Neural Network (Fed-ExDNN) to perform federated anomaly
detection with multivariate time series data. On the edge device side, we
specifically designed an Exemplar-based Deep Neural Network (ExDNN) to perform
anomaly detection. ExDNN can simultaneously learn local time series
representations based on their compatibility with an exemplar module which
consists of hidden parameters learned to capture varieties of normal patterns
in the hidden feature space. On the server side, to cope with the
heterogeneity of time series data on different edge devices, Fed-ExDNN employs
a Federated Constrained Clustering (FedCC) technique to align and aggregate
parameters of different local exemplar modules. Eventually, the updated global
exemplar module together with a shared feature encoder will be sent back to
edge devices and the anomaly detection is conducted by measuring the
compatibility of test data (with extracted features) to the global exemplar
module.

The main contributions of this paper are summarized as follows:

  * •

We formally investigate the problem of federated unsupervised anomaly
detection (FedUAD) for multivariate time series data and develop Fed-ExDNN
which consists of ExDNN for local anomaly detection and FedCC for model
aggregation to handle FedUAD.

  * •

On the edge device side, Exemplar-based Deep Neural Network (ExDNN) can
simultaneously learn local time series representations based on their
compatibility with an exemplar module which is developed to capture potential
normal patterns in the hidden feature space. On the server side, FedCC could
align and aggregate different local exemplar modules. ExDNN and FedCC work
jointly to address the heterogeneous distribution among edge devices.

  * •

Our empirical studies on six public multivariate time series datasets
demonstrate the effectiveness of the proposed ExDNN and Fed-ExDNN.

##  2 Related Work

The proposed Fed-ExDNN is closely related to unsupervised anomaly detection
and federated learning.

###  2.1 Unsupervised Anomaly Detection

Recently, deep learning-based anomaly detection methods has shown fruitful
progress compared with traditional methods including one class SVM
scholkopf2000support , Isolated Forest liu2008isolation , etc.. In general,
they can be categorized into four different types, i.e., One-class
classification based methods, reconstruction based approaches, contrastive
learning based techniques, and clustering based methods.

For one-class classification based methods, Deep Support Vector Data
Descriptor (SVDD) replaces the kernel used in OCSVM scholkopf2000support  with
a deep neural network ruff2018deep . Context Vector Data Description (CVDD)
generates multi semantic contexts by multi-head attention mechanism
ruff2019self . Temporal Hierarchical One-Class (THOC) conducts multi-scale one
class learning in a hierarchical manner shen2020timeseries . Reconstruction-
based approaches mainly rely on autoencoder framework to reconstruct the input
and employ the reconstruction error to detect anomalies. For instance, LSTM
Autoencoder (AE) malhotra2015long  adopts LSTM to encode and decode
multivariate time series (MTS) data. To better model inter-correlation between
different time series, Multiscale Convolutional Recurrent Encoder-Decoder
(MSCRED) is proposed to reconstruct system signature matrices by an attention-
based convLSTM zhang2019deep . Memory Augmented Autoencoder (MemAE) augments
autoencoder with an external memory gong2019memorizing . BeatGAN
zhou2019beatgan  regularizes the reconstructed data by a generative
adversarial network goodfellow2014generative . OmniAnomaly learns robust
representation with stochastic variable connection and planar normalizing flow
su2019robust . Unsupervised Anomaly Detection (USAD) imposes additional
constraints on reconstruction by an additional decoder audibert2020usad . Li
et al. conduct anomaly detection with hierarchical VAE and low dimensional
embedding for anomaly detection li2021multivariate . More recently,
contrastive learning based techniques are becoming popular for time series
anomaly detection. For instance, self-supervised contrastive predictive coding
is proposed to handle anomaly points deldari2021time . Cho et al. propose a
masked contrastive method by using class-wise scale factor cho2021masked . A
unified contrastive anomaly detection framework is proposed by sehwag2021ssd .
Carmona et al. perform time series anomaly detection by generating abnormal
series with expertise knowledge carmona2021neural . Qiu et al. propose
deterministic contrastive loss to enable the anomaly score to be consistent
with training loss qiu2021neural . Clustering-based deep neural networks are
also applied to anomaly detection. For instance, Zong et al. proposes Data
Encoder Gaussian Mixture Model (DAGMM) for anomaly detection and they conduct
GMM on the feature space composed of reconstruction score and encoding of the
autoencoder. In this paper, we introduce a novel task as Federated
Unsupervised Anomaly Detection (FedUAD). The heterogeneous data distribution
on edge devices and data-free server-side model aggregation brings challenges
for existing anomaly detection methods. We specially design Fed-ExDNN to
overcome these problems.

###  2.2 Federated Learning

Federated learning receives increasing attention recently yang2019federated ;
geyer2017differentially ; konevcny2016federated ; liu2021fedlearn . One of the
major concerns of federated learning is the heterogeneity problem
sahu2018convergence ; li2019convergence ; karimireddy2020scaffold ;
reisizadeh2020robust ; dinh2020personalized ; zhu2021data . Federated
Averaging (FedAvg) is the most widely used algorithm for federated tasks
li2019convergence . Federated Proximal (FedProx) is proposed to alleviate the
heterogeneity challenge sahu2018convergence . Sattler et al.
(sattler2019clustered ) propose to hierarchically cluster the locally learned
models. Xie et al. (xie2020multi ) propose to maintain multi global models and
assign user’s gradient to different global models. Liang et al.
(liang2020think ) propose to learn local representations on each device and an
overall global model across devices . Federated Matching Average (FedMA)
learns to cluster the local models before averaging the weights
wang2020federated . Yu et al. (yu2020federated ) propose FedAwS where there is
only one class on each device. Fedfast is proposed to handle federated
recommendation system muhammad2020fedfast . FedDF conducts data-free knowledge
distillation on server side to aggregate local modelslin2020ensemble . Fallah
et. al. (fallah2020personalized ) applies meta learning finn2017model  on
client updates. However, these methods are all proposed to handle supervised
federated learning tasks. The unsupervised setting of UAD makes the
heterogeneity problem much more challenging, and we find that the proposed
Fed-ExDNN has clear advantages over conventional federated learning methods
for FedUAD.

More recently, federated anomaly detection draws increasing attention
singh2021anomaly ; nguyen2019diot ; zhao2019multi ; sater2021federated ;
wang2021towards . Nguyen et al. propose to apply federated anomaly detection
for IoT devices nguyen2019diot . Federated anomaly detection is also used to
address the IoT security attacks mothukuri2021federated . Liu et al. propose
an on-device method for industrial IoT anomaly detection. Zhao et al. propose
a multi-task network for federated anomaly detection zhao2019multi . Compared
to existing works, our proposed Fed-ExDNN focuses on unsupervised anomaly
detection and employs an effective method to address the heterogeneity
problem.

##  3 Federated Exemplar-based Deep Neural Network

In this section, we present a Federated Exemplar-based Deep Neural Network
(Fed-ExDNN) to perform federated unsupervised anomaly detection. Fed-ExDNN
consists of ExDNN for local anomaly detection and Federated Constrained
Clustering (FedCC) for model aggregation.

We find that simply combining existing federated learning and anomaly
detection approaches to handle the task often leads to inferior performance.
This should be attributed to the fact that most existing anomaly detection
methods are developed based on the assumption that the entire normal space is
covered by the training data pang2020deep , and this assumption does not hold
when we conduct training on edge devices with heterogeneously distributed
data, i.e., not all normal patterns can be accessed during the training stage
on each edge device. On the other hand, existing federated learning
algorithms, e.g., Federated Averaging (FedAvg) mcmahan2017communication , are
originally designed for supervised classification tasks and they may not
properly aggregate the unsupervised anomaly detection models trained on edge
devices. To this end, we present Fed-ExDNN, which consists of a clustering-
based anomaly detection method ExDNN and a federated aggregation method FedCC
to align and aggregate different local exemplar modules.

Figure 1: Block diagram of the proposed Fed-ExDNN which contains five steps
for each communication round. (1) We train the device-specific model,
including the embedding network and learnable local exemplars (shown as
stars), on each edge device by the proposed ExDNN. The models on all devices
are initialized with a global model fetched from the central server. (2) Edge
devices send device-specific embedding networks and learned local exemplars to
the server. (3) Compute global exemplars from the local exemplars using the
proposed Federated Constrained Clustering (FCC). (4) Aggregate the local
models to obtain the global model using FedAvg. (5) Updated global model is
sent back to edge devices. The procedure will be repeated until convergence is
achieved.

We brief the basic training procedure as follows: assuming there are $L$ edge
devices, the $l$-th local device learns a device-specific model, which
includes an embedding network $f^{l}(\cdot\mathchar 24635\relax\;\theta^{l})$
for feature encoding and an exemplar module in which a set of $K$ local
exemplars
$\mathbf{C}^{l}=\\{\mathbf{c}_{1}^{l},\cdots,\mathbf{c}_{K}^{l}\\}\in\mathbb{R}^{d\times
K}$ is learned to capture the potential normal patterns in the hidden feature
space. The local model is trained based on the time series data collected on
the $l$-th local device for unsupervised anomaly detection. The central server
aggregates local models from different devices to construct a global model.
The embedding network (for feature encoding) of the global model
$g(\cdot\mathchar 24635\relax\;\bar{\theta})$ is obtained by Federated
Averaging, and the global exemplar module with $K$ learnable exemplars
$\mathbf{U}=\\{\mathbf{u}_{1},\cdots,\mathbf{u}_{K}\\}$ is obtained by
aggregating and align all local exemplar modules with the proposed FedCC.
Finally, the server sends the global model to different edge devices to update
their local models. Fed-ExDNN jointly adopts FedCC and ExDNN to enable the
global model to capture entire normal patterns even when the data are
heterogeneously distributed among edge devices. Please refer to Fig. 1 for
detailed illustration.

###  3.1 Local Device: ExDNN

Exemplar-based Deep Neural Network (ExDNN), as a clustering-based anomaly
detection method, is naturally suitable for handling heterogeneous data.
Compared with existing clustering-based methods liao2018unified ; zong2018deep
that adopt Gaussian Mixture Model (GMM) as the clustering objective, the ExDNN
is developed based on an advanced clustering algorithm hu2017learning ;
xie2016unsupervised . Moreover, we propose Deep Relation Preserving (DRP) to
learn the representation of multivariate time series data in an unsupervised
manner. We’d like to emphasize that ExDNN conducts clustering and
representation learning simultaneously to mutually boost their performance for
better anomaly detection. The details of ExDNN are shown in Fig. 2. In the
following, the algorithm described is for one particular local device, so we
omit the superscript $l$ for brevity.

For a specific local device with $n$ multivariate time series segments
$\\{\mathbf{X}^{i}\\}_{i=1}^{n}\in\mathbb{R}^{m\times t}$, where $m$ denotes
the number of time series and $t$ is the length of a segment, our method
learns the optimal embedding network $f(\cdot\mathchar 24635\relax\;\theta)$
and an exemplar module with a learnable parameter
$\mathbf{C}\in\mathbb{R}^{d\times K}$. In this paper, we use LSTMs to encode
temporal dynamics in the multivariate time series. The exemplar module
$\mathbf{C}$ is implemented by a fully connected layer, and is jointly trained
with the embedding network parameter $\theta$ in a end-to-end manner.

Specifically, our proposed method is motivated by Deep Embedding Clustering
xie2016unsupervised  with the following objective:

$\begin{split}\min_{\theta,\mathbf{C}}\frac{1}{n}\sum_{i=1}^{n}KL(\mathbf{p}_{i}\|\mathbf{q}_{i})\end{split}$
(1)

where $KL(\cdot\|\cdot)$ is the Kullback–Leibler divergence and Eq. (1)
encourages the exemplars to be close to the training samples on the embedding
space, and each learned exemplar could then capture a specific pattern of the
normal data just like a clustering center in K-means. Specifically,
$\mathbf{q}_{i}\in R^{K}$ is the cluster indicator vector for the $i$-th
segment where $q_{ij}$ is the probability of assigning the $i$-th data to the
$j$-th exemplar. This probability is computed as

$\begin{split}q_{ij}=\frac{\exp(\gamma_{1}s(f(\mathbf{X}^{i}),\mathbf{c}_{j}))}{\sum_{k=1}^{K}\exp(\gamma_{1}s(f(\mathbf{X}^{i}),\mathbf{c}_{k}))},\end{split}$
(2)

where $s$ is the cosine similarity
$s(\mathbf{a},\mathbf{b})=\frac{\mathbf{a}^{\mathsf{T}}\mathbf{b}}{\|\mathbf{a}\|_{2}\|\mathbf{b}\|_{2}}$,
$\gamma_{1}$ is a learnable scale factor, and
$\mathbf{C}=\\{\mathbf{c}_{1},\cdots,\mathbf{c}_{K}\\}\in\mathbb{R}^{d\times
K}$ denote local exemplars. We should highlight that the local exemplars are
the same as the weights of a fully connected layer and their privacy concern
could be addressed by well-studied approaches, e.g., differential privacy
geyer2017differentially . Following xie2016unsupervised , we raise
$\mathbf{q}_{i}$ to the second power and normalize it by the size of clusters
to obtain $\mathbf{p}_{i}$ as

$p_{ij}=\frac{q_{ij}^{2}/\sum_{i^{\prime}=1}^{n}q_{i^{\prime}j}}{\sum_{j^{\prime}}(q_{ij^{\prime}}^{2}/\sum_{i^{\prime}=1}^{n}q_{i^{\prime}j^{\prime}})}$
(3)

Figure 2: The time series segment $X^{i}$ is processed by a 4-layer LSTM
followed by a fully connected embedding layer. The exemplars module are
updated by gradient descent and is implemented with a fully connected layer.

However, exemplar learned by optimizing Eq. (1) may converge to patterns with
few samples and even noisy data, and we then adopt the balanced loss to
alleviate the problem as

$\begin{split}\min_{\theta,\mathbf{C}}-\bm{\alpha}^{\mathsf{T}}\log\left(\frac{1}{n}\sum_{i=1}^{n}\mathbf{q}_{i}\right)\end{split}$
(4)

$\bm{\alpha}\in\mathbb{R}^{K}$ is a prior distribution over the exemplars, and
this term encourages the cluster sizes on edge devices to match the prior
hu2017learning . We set $\bm{\alpha}=\frac{1}{K}\textbf{1}$, i.e., uniform
distribution. The effectiveness of the balancing term on handling contaminated
data is shown in our ablation studies.

Representation learning is critical for deep neural network training, and DEC
is often trained with auto-encoder initialization xie2016unsupervised , which
may lead to privacy concern and computational cost especially in federated
learning settings. Furthermore, we propose Deep Relative Preserving (DRP) that
encourages the latent space to preserve the local similarity induced by the
original featureroweis2000nonlinear ; chui2018deep :

$\begin{split}M(\mathbf{X}^{i})=\min_{\theta}\log\left(1+\sum_{\begin{subarray}{c}p\in\mathcal{P}_{i}\\\
n\notin\mathcal{P}_{i}\end{subarray}}\exp(\gamma_{2}(s_{in}-s_{ip}))\right)\end{split}$
(5)

where $\mathcal{P}_{i}$ is the set of nearest neighbors of the $i$-th example.
$\gamma_{2}$ is a learnable scale factor. $s_{ij}$ is shorthand for
$s(f(\mathbf{X}^{i}),f(\mathbf{X}^{j}))$. Eq. (5) encourages the similarity of
positive pairs to be larger than that of negative pairs sun2020circle .
Moreover, to avoid computation and storage cost of the KNN graph, we
approximate KNN by the samples within each minibatch.

We could perform anomaly detection by jointly optimizing Eq. (1), Eq. (4), and
Eq. (5) as

$\begin{split}\min_{\theta,\mathbf{C}}\frac{1}{n}\sum_{i=1}^{n}\left(KL(\mathbf{p}_{i}\|\mathbf{q}_{i})+M(\mathbf{X}^{i})\right)-\bm{\alpha}^{\mathsf{T}}\log\left(\frac{1}{n}\sum_{i=1}^{n}\mathbf{q}_{i}\right),\end{split}$
(6)

and the anomaly score for sample $\mathbf{X}$ is calculated by the negative
cosine similarity between the samples and its nearest exemplars as

$\begin{split}\text{Score}(\mathbf{x})=-\max_{j}\;s(f(\mathbf{X}),\mathbf{c}_{j}).\end{split}$
(7)

However, Eq. (6) only forces the relative similarity between the sample and
its nearest exemplar to be larger than the similarity between the sample and
other exemplars xie2016unsupervised . After training, we actually have little
guarantee on the numeric value of anomaly score. Therefore, although Eq. (6)
may be effective for clustering, it leads to sub-optimal performance for
anomaly detection in our experiments. To alleviate the problem, we introduce
an absolute term to directly optimize the numeric value of anomaly score, and
our final objective for anomaly detection on a local device becomes:

$\begin{split}\min_{\theta,\mathbf{C}}\frac{1}{n}\sum_{i=1}^{n}\left(KL(\mathbf{p}_{i}\|\mathbf{q}_{i})+M(\mathbf{X}^{i})\right)-\bm{\alpha}^{\mathsf{T}}\log\left(\frac{1}{n}\sum_{i=1}^{n}\mathbf{q}_{i}\right)\\\
+\frac{1}{n}\sum_{i=1}^{n}\log\left(1+\exp(-\gamma_{3}(s(f(\mathbf{X}^{i}),\bar{\mathbf{c}}_{i})-m)\right).\end{split}$
(8)

The last term of Eq. (8) 111Here we slightly abuse the symbol of cosine
similarity and $s(f(\mathbf{X}^{i}),\bar{\mathbf{c}}_{i})$ is calculated by
the dot product between $f(\mathbf{X}^{i})^{T}$ and $\bar{\mathbf{c}}$ with
normalized $f(\mathbf{X}^{i})$ and $\mathbf{c}_{j}$. is the proposed absolute
score loss. Since the max operation in Eq. (7) is non-differentiable, our
absolute score term maximizes the cosine similarity between the $i$-th segment
and a soft approximation of the nearest exemplar
$\bar{\mathbf{c}}_{i}=\sum_{j=1}^{K}{q}_{ij}\mathbf{c}_{j}$, where
$\mathbf{q}_{i}$ is the soft indicator vector for the $i$-th sample.
$\gamma_{3}$ is a learnable scale factor and $m>0$ is the margin.We adopt
softplus operation to make the proposed term have similar scale as other
terms. Since the distribution of pairwise cosine similarity of two random
high-dimensional unit vectors approaches a zero mean Gaussian, it is thus
necessary to have $s(f(\mathbf{X}^{i}),\bar{\mathbf{c}}_{i})$ larger than a
positive margin $m$. We highlight that although the ExDNN in Eq. (8) contains
several hyper-parameters, most of them are fixed during the empirical studies
in this paper besides the number of exemplars $K$.

After certain rounds of training on the $l$-th local device, the parameters of
the embedding network $\theta^{l}$ and the set of exemplars
$\\{\mathbf{c}_{1}^{l},\cdots,\mathbf{c}_{K}^{l}\\}$ are all uploaded to a
central server.

###  3.2 Central Server: FedCC for Exemplars Aggregation

The central server aggregates the local models uploaded from edge devices to
obtain a global feature encoding network $g$ and a global exemplar module
$\mathbf{U}=\\{\mathbf{u}_{1},\cdots,\mathbf{u}_{K}\\}$ that capture the
heterogeneous data distribution on all edge devices. The feature encoding
network could be aggregated by existing federated learning methods, e.g.,
federated averaging (FedAvg) mcmahan2017communication  and Federated Proximal
(FedProx) sahu2018convergence . However, due to heterogeneous data on edge
devices, the local exemplar module, even with the same initialization, may
significantly deviate from each other to better fit the local data. Even
worse, due to the cost of transmission, the local training is desired to be
longer to reduce the communication load between server and clients. As a
result, the alignment between the updated local exemplar module and the
previous global exemplar module may not hold as the training on local devices
proceeds.

Most of existing federated learning methods, e.g., FedAvg and FedProx,
element-wisely averages the local exemplars based on the assumption that all
local exemplar modules and the global exemplar module are still well-aligned
during the training. For supervised tasks, the alignment could be regularized
by the ground truth label. For the proposed Federated Unsupervised Anomaly
Detection (FedUAD) task, however, no label is available. Therefore, vanilla
federated learning methods will lead to suboptimial performance in practice.
Aggregating the exemplar modules by K-means seems to be a reasonable choice
wang2020federated . However, K-means may also results in misalignment since
its objective and the representation is decoupled which make it impossible to
adjust the representation so as to mitigate the issue. In this paper, we
propose an innovative approach, namely Federated Constrained Clustering
(FedCC) to address the heterogeneity problem. The basic idea is to first learn
a projection function $h$ that could align the local exemplars to discover and
enhance the alignment, and then conduct clustering on the learned embedding
space to obtain the global exemplar module. The effectiveness of FedCC is
verified in the experiment section.

From $L$ edge devices each with exemplar module consisting of $K$ learnable
exemplars, the central server receives a total of $N=LK$ local exemplars which
is denoted as
$\\{\mathbf{c}_{1}^{1},\cdots,\mathbf{c}_{K}^{1},\mathbf{c}_{1}^{2},\cdots,$
$\mathbf{c}_{K}^{2},\cdots,\mathbf{c}_{1}^{L}\cdots\mathbf{c}_{K}^{L}\\}$. The
proposed FedCC is formulated as:

$\begin{split}\min_{\phi,\\{\mathbf{v}_{1},\cdots,\mathbf{v}_{K}\\}}-\frac{1}{N}\sum_{i=1}^{K}\sum_{l=1}^{L}\mathbf{p}_{il}^{\mathsf{T}}\log\mathbf{q}_{il}\\\
-\textbf{1}^{\mathsf{T}}\log\left(\frac{1}{N}\sum_{i=1}^{K}\sum_{l=1}^{L}\mathbf{p}_{il}\right)+\frac{1}{N}\sum_{i=1}^{K}\sum_{l=1}^{L}R(\bm{c}_{i}^{l}),\end{split}$
(9)

where $\phi$ denotes the parameters of the projection network $h$, and
$\\{\mathbf{v}_{1},\cdots,\mathbf{v}_{K}\\}$ are the latent cluster centers in
the output space of $h$. The first two terms in Eq. (9) are for clustering,
similar to the first and third terms in Eq. (8). $\mathbf{q}_{il}$ are defined
similarly to Eq. (2):

$\begin{split}q_{il,j}=\frac{\exp(\gamma_{4}\cdot
s(h(\mathbf{c}_{i}^{l}),\mathbf{v}_{j}))}{\sum_{k=1}^{K}\exp(\gamma_{4}\cdot
s(h(\mathbf{c}_{i}^{l}),\mathbf{v}_{k}))},\end{split}$ (10)

and $\mathbf{p}_{il}$ is defined based on $\mathbf{q}_{il}$ in the same way as
Eq. (3), and $\gamma_{4}$ is the scale factor. We further introduce the
constraints $R$ to encourage the learned projection $h$ to give similar
embedding for exemplars that have the same initialization (stars with same
color in Fig. 1).

$\begin{split}R(\mathbf{c}_{i}^{l})=\log\left(1+\sum_{m=1}^{L}\sum_{j=1}^{K}\exp(\gamma_{5}\mathbf{e}_{ij}s(h(\mathbf{c}_{i}^{l}),h(\mathbf{c}_{j}^{m})))\right),\end{split}$
(11)

where $\mathbf{e}_{ij}=1$ if $i=j$ and $c_{i}^{l}$ is K nearest neighbor of
$c_{j}^{m}$, and $\mathbf{e}_{ij}=-1$ otherwise. $\gamma_{5}$ is a learnable
scale factor. Finally, the global exemplar module
$\\{\mathbf{u}_{1},\cdots,\mathbf{u}_{K}\\}$ can be obtained based on the
clustering indicator matrix:

$\mathbf{u}_{z}=\frac{1}{\sum_{i=1}^{K}\sum_{l=1}^{L}q_{il,z}}\sum_{i=1}^{K}\sum_{l=1}^{L}q_{il,z}\mathbf{c}_{i}^{l}.$
(12)

After obtaining the global exemplar module and the averaged embedding network
(for feature encoding), they will be send back to each edge devices for the
next round of learning.

##  4 Experiments

In this section, we verify the superiority of ExDNN and Fed-ExDNN for anomaly
detection.

###  4.1 Datasets and Evaluation Metrics

We conduct experiments on six publicly available multivariate time series
datasets, including 2D Gesture keogh2005hot , ECG5000 goldberger2000physiobank
, SWaT mathur2016swat , HAR Laying anguita2013public , UWave liu2009uwave ,
and ArabicDigits hammami2009tree . The details of these datasets and
train/validation/test partitions are summarized as follows and in Table 1.

  * •

2D Gesture records the X-Y coordinates of a hand gesture in a video
keogh2005hot ;

  * •

SWaT contains the data of a water treatment plant for water filtering. The
data are composed of 11 days of operations with 7 days normal data and 4 days
attack data mathur2016swat ;

  * •

ECG5000 records ECG series of a patient who has sever congestive heart failure
goldberger2000physiobank ;

  * •

HAR Laying is a human activity recognition dataset that contains 6 different
activities performed by 30 subjects including walking, walking upstairs,
walking downstairs, sitting, standing, and laying. The dataset has 30 subjects
with 21 as training and 9 as testing. The laying samples are regarded as
anomaly cases. We select 5 subjects from training data for validation
anguita2013public .

  * •

AerobicDigit hammami2010improved  contains Mel-Frequency Cepstrum Coefficient
(MFCC) data of spoken arabic digits, and each digit is alternatively regarded
as anomaly following the setting in ruff2018deep .

  * •

UWave liu2009uwave  consists of x, y, z coordinates of eight gestures recorded
by accelerometers of Wii remotes, and each gesture is alternatively regarded
as anomaly following the setting in ruff2018deep .

Table 1: The detailed statistics of six multivariate time series datasets.

Dataset | # train - val - test | # dim ($m$) | # Length ($t$)
---|---|---|---
2D Gesture | 8171 - 876 - 2044 | 2 | 80
SWaT | 47420 - 11198 - 33594 | 51 | 100
ECG5000 | 292 - 1125 - 3375 | 1 | 140
HAR Laying | 4559 - 1676 - 2947 | 9 | 128
AerobicDigits | 6600 - n/a - 2200 | 13 | 20
UWave | 1600 - n/a - 2879 | 3 | 40

For 2D Gesture and SWaT, we use sliding window with stride 1 to partition
them. Moreover, since their training sets only have normal data, we select a
portion of data from original testing data to construct the validation set for
hyper-parameter tuning. We downsample the time series of SWaT by 10 following
shen2020timeseries . The length of segments of AerobicDigit and Uwave varies
and we resize their segments to 20 and 40 respectively. For 2D Gesture,
ECG5000, SWaT, and HAR Laying, we set hyperparameters based on grid search
over the validation set. We report the AUC, F1, Precision and Recall linked to
the best F1 score on validation set following audibert2020usad . For
AerobicDigit and UWave, following the setting similar to ruff2018deep , we do
not construct validation set, and only report average AUC by iteratively
treating each class as the anomaly case. All experiments are ran three times
and mean and standard deviations of all metrics are provided.

###  4.2 ExDNN for Anomaly Detection

We first conduct experiments to show the effectiveness of the proposed ExDNN
for unsupervised anomaly detection with multivariate time series (MTS) data.

####  4.2.1 Comparison Methods and Experimental Settings

We compare the proposed ExDNN with seven deep learning methods, including
LSTM-AutoEncoder (LSTM-AE) malhotra2015long , BeatGAN zhou2019beatgan , Memory
Augmented AutoEncoder (MemAE) malhotra2015long , USAD audibert2020usad , Deep
SVDD ruff2018deep , Contextual SVDD (CVDD) ruff2019self , and Deep
Autoencoding Gaussian Mixture Model (DAGMM) zong2018deep . Among them, LSTM
AE, BeatGAN, MemAE, and USAD are reconstruction-based methods, while Deep SVDD
and CVDD are one class-based methods. DAGMM jointly uses auto-encoder
reconstruction and clustering for anomaly detection.

All deep learning methods are implemented with the same feature encoder shown
in Fig. 2, which is composed of a 4-layer LSTM with the hidden dimension set
as $8$. The decoder for reconstruction-based methods is with the same
structure as the encoder. We fix the batch size as 128 and fix the learning
rate set as 0.005 for all methods. We conduct an exhaustive grid search for
the deep learning methods to find the optimal parameters for each dataset. For
ExDNN, we fix the hyperparameter as $\gamma_{1}=2$, $\gamma_{2}=10$,
$\gamma_{3}=10$, $m=0.5$, and the number of nearest neighbors in DRP as 10,
unless otherwise stated. We search the number of clusters from
$\\{8,16,32,64,128\\}$. Since AerobicDigit and UWave do not have a validation
set, we set the number of exemplars as 32 and 64 respectively. All experiments
are conducted on a server with 4 Nvidia GTX 2080 Ti graphics cards. We
summarize all methods as follows: The detail settings of compared methods are
shown as follows:

  * •

LSTM-AutoEncoder (LSTM-AE) is implemented with 4-layer BiLSTM as encoder and
decoder malhotra2015long .

  * •

Deep SVDD is implemented also with a 4-layer BiLSTM as feature extractor. We
first pretrain the network with LSTM-AE, and then train Deep SVDD by searching
the $\lambda$ from $\\{10^{-5},10^{-4},10^{-3},10^{-2},10^{-1}\\}$
ruff2018deep .

  * •

Memory Augmented AutoEncoder (MemAE) adopts similar network structure as LSTM-
AE. We search the memory size from $\\{32,64,128,256\\}$ and $\alpha$ from
$\\{10^{-3},10^{-2},10^{-1},1,10,100\\}$ gong2019memorizing .

  * •

Contextual SVDD (CVDD) is an extension of SVDD by adopting multi context
projection. We search the number of contexts from $\\{3,5,10,32\\}$ and scale
factor $\alpha$ from $\\{10^{-5},10^{-4},10^{-3},10^{-2},10^{-1}\\}$
ruff2019self .

  * •

Deep Autoencoding Gaussian Mixture Model (DAGMM) is also built on the LSTM-AE
and adopts a gaussian mixture model on the features composed of reconstruction
loss and latent embedding of autoencoder. We search $\lambda_{1}$ from
$\\{10^{-3},10^{-2},10^{-1},1,100,1000\\}$ and $\lambda_{2}$ from
$\\{10^{-5},10^{-4},10^{-3},10^{-2},10^{-1},1\\}$ zong2018deep .

  * •

USAD audibert2020usad  is based on autoencoder framework. We search the hyper-
paramter $\alpha$ from $\\{0,0.1,0.2,0.5,0.7,0.9\\}$.

  * •

BeatGAN zhou2019beatgan  applies GAN for auto-encoder. We varies the $\Lambda$
from $\\{10^{-3},10^{-2},10^{-1},1,100,1000\\}$.

  * •

ExDNN is proposed in this paper. We fix the hyperparameter as $\gamma_{1}=2$,
$\gamma_{2}=10$, $\gamma_{3}=10$, $m=0.5$, and the number of nearest neighbors
in DRP as 10, unless otherwise stated. We search the number of clusters from
$\\{8,16,32,64,128\\}$.

####  4.2.2 Effectiveness of ExDNN

The anomaly detection results on MTS datasets are shown in Table 2. According
to the results, several interesting points are summarized as follows. First,
either reconstruction and one-class based methods cannot handle all different
scenarios. By imposing additional restrictions on the latent space learned by
auto-encoder, BeatGAN, MemAE, and USAD could potentially boost the performance
of LSTM AE on homogeneous datasets especially on HAR Laying, but are of little
use and even also degrade the performance on more heterogeneous datasets,
e.g., SWaT and AerobicDigits. By contrast, Deep SVDD works much better for
homogeneous datasets, e.g. HAR Laying, but also suffers from the under-fitting
problem with a lower recall score. CVDD could largely alleviate the under-
fitting problem of SVDD by the multi-context learning. Second, the proposed
ExDNN, although is designed for anomaly detection with heterogeneous normal
samples, could handle the over-fitting and under-fitting problem by tuning the
number of exemplars and generally achieves superior performance on all
datasets. ExDNN outperforms compared methods, especially on datasets with
heterogeneous normal cases. For example, ExDNN achieves 4.58% improvements
(regarding averaged AUC) on AerobicDigit compared with the second best method
CVDD. This is because ExDNN not only extracts superior representations by DRP
but also conducts effective deep clustering to generate representative
exemplars for anomaly detection.

Table 2: Anomaly detection performance on six MTS datasets. Best methods are
highlighted in bold.

Dataset | Metric | LSTM AE | BeatGAN | MemAE | USAD | Deep SVDD | CVDD | DAGMM | ExDNN(Ours)
---|---|---|---|---|---|---|---|---|---
2D Gesture | AUC | 80.19$\pm$4.32 | 81.87$\pm$3.84 | 80.88$\pm$0.40 | 80.15$\pm$3.99 | 73.58$\pm$2.73 |  82.64$\pm$0.05 | 74.74$\pm$8.51 |  88.37$\pm$0.82
F1 | 58.57$\pm$3.59 | 60.81$\pm$4.05 |  62.77$\pm$1.14 | 61.58$\pm$3.36 | 54.42$\pm$3.83 | 62.54$\pm$2.47 | 56.06$\pm$9.12 |  71.13$\pm$2.31
Prec | 50.28$\pm$2.05 | 58.10$\pm$8.94 | 57.84$\pm$3.85 | 55.84$\pm$0.73 | 49.98$\pm$8.83 | 62.88$\pm$5.16 | 47.54$\pm$8.69 | 66.96$\pm$5.74
Rec | 70.66$\pm$8.15 | 65.83$\pm$7.05 | 69.05$\pm$2.86 | 68.79$\pm$7.37 | 63.38$\pm$9.15 | 62.48$\pm$1.25 | 71.82$\pm$1.47 | 76.40$\pm$2.39
SWaT | AUC |  91.87$\pm$0.85 | 91.39$\pm$0.30 | 91.29$\pm$0.47 | 90.69$\pm$0.33 | 89.51$\pm$8.02 |  94.70$\pm$0.94 | 90.02$\pm$0.99 | 91.61$\pm$0.78
F1 | 81.89$\pm$1.99 | 80.65$\pm$1.70 | 81.52$\pm$0.52 | 80.25$\pm$0.48 | 79.86$\pm$5.55 |  83.34$\pm$1.63 | 80.07$\pm$3.50 |  87.34$\pm$0.61
Prec | 72.56$\pm$4.63 | 70.15$\pm$4.25 | 71.35$\pm$1.42 | 69.35$\pm$1.28 | 69.19$\pm$4.78 | 71.71$\pm$2.39 | 70.15$\pm$5.71 | 85.10$\pm$0.72
Rec | 94.44$\pm$3.00 | 95.31$\pm$3.16 | 95.19$\pm$2.40 | 95.26$\pm$1.05 | 94.46$\pm$6.95 | 99.51$\pm$0.15 | 93.28$\pm$1.64 | 89.71$\pm$0.60
ECG5000 | AUC | 95.04$\pm$0.63 | 94.85$\pm$0.61 | 94.88$\pm$1.24 | 93.73$\pm$0.82 | 90.35$\pm$5.40 |  97.38$\pm$0.25 | 89.14$\pm$0.99 |  98.45$\pm$1.05
F1 | 92.09$\pm$1.57 |  92.31$\pm$0.16 | 92.16$\pm$0.99 | 91.69$\pm$1.48 | 85.91$\pm$7.74 | 91.20$\pm$0.26 | 88.00$\pm$1.43 |  92.86$\pm$2.64
Prec | 88.36$\pm$3.32 | 87.95$\pm$0.51 | 88.05$\pm$2.28 | 86.53$\pm$2.66 | 89.58$\pm$6.12 | 90.69$\pm$0.92 | 79.55$\pm$2.20 | 91.29$\pm$3.64
Rec | 96.24$\pm$0.85 | 97.12$\pm$0.88 | 96.74$\pm$0.85 | 97.57$\pm$1.07 | 82.57$\pm$5.43 | 91.76$\pm$1.31 | 98.45$\pm$0.50 | 94.55$\pm$1.80
HAR Laying | AUC | 59.73$\pm$4.76 | 95.30$\pm$2.97 | 95.20$\pm$2.38 | 69.98$\pm$1.94 |  100$\pm$0 | 94.60$\pm$5.90 | 62.82$\pm$1.92 |  99.99$\pm$0
F1 | 42.89$\pm$0.24 | 81.00$\pm$0.13 | 83.73$\pm$2.95 | 50.19$\pm$3.31 |  99.14$\pm$1.15 | 72.06$\pm$4.54 | 33.69$\pm$3.31 |  99.35$\pm$0.20
Prec | 27.30$\pm$0.19 | 73.97$\pm$0.10 | 77.39$\pm$5.84 | 34.70$\pm$1.62 | 100$\pm$0 | 64.51$\pm$6.73 | 25.47$\pm$1.62 | 98.71$\pm$0.39
Rec | 100$\pm$0 | 91.56$\pm$0.10 | 91.81$\pm$5.16 | 97.95$\pm$6.96 | 98.32$\pm$2.24 | 84.67$\pm$5.92 | 53.32$\pm$6.96 | 100$\pm$0
Aerobic Digit | Avg AUC | 79.70 | 77.27 | 77.67 | 77.80 | 74.21 | 83.12 | 81.52 | 87.70
UWave | Avg AUC | 84.33 | 86.95 | 85.35 | 86.61 | 70.32 | 81.11 | 73.20 | 88.72

Table 3: Performance of ExDNN variants. We fix the number of exemplars to 32.
Best results are highlighted in bold.

Methods | 2D Gesture | AerobicDigit
---|---|---
| AUC | F1 | Avg AUC
ExDNN w/ AE | 68.83$\pm$3.17 | 61.35$\pm$4.36 | 83.57
ExDNN w/o clus | 85.96$\pm$1.69 | 67.11$\pm$2.04 | 68.57
ExDNN w/o bal | 85.01$\pm$1.02 | 64.46$\pm$1.05 | 83.67
ExDNN w/o abs | 83.26$\pm$1.34 | 63.93$\pm$2.43 | 85.94
ExDNN |  86.31$\pm$1.47 |  67.34$\pm$1.98 | 87.70

Table 4: Performance of ExDNN with different numbers of exemplars. Best
results are highlighted in bold.

K | 2D Gesture | AerobicDigit
---|---|---
| AUC | F1 | Avg AUC
8 | 82.34$\pm$1.60 | 62.30$\pm$3.47 | 75.51
16 | 84.94$\pm$2.88 | 66.15$\pm$3.63 | 84.02
32 | 86.31$\pm$3.55 | 67.34$\pm$4.67 | 87.70
64 | 87.34$\pm$0.72 | 68.41$\pm$1.35 | 88.62
128 |  88.37$\pm$0.82 |  71.13$\pm$2.31 | 88.61

####  4.2.3 Ablation Studies

In this section, we conduct ablation studies on 2D Gesture and AerobicDigit.
For AerobicDigit, only averaged AUC is reported. We first study the
effectiveness of different components of the proposed ExDNN. The results are
shown in Table 3. We denote ExDNN without the cluster term in Eq. (8) as ExDNN
w/o clus, ExDNN without the balanced term in Eq. (8) as ExDNN w/o bal, and
ExDNN without the absolute term in Eq. (8) as ExDNN w/o abs. Moreover, to
validate the effectiveness of the proposed DRP for representation, we replace
DRP with a pretrained network by auto-encoder as ExDNN w/ AE. The experiments
are conducted with 32 exemplars for 2D Gesture and AerobicDigit. Based on the
results, we could conclude that the cluster term brought from DEC
xie2016unsupervised  is essential for the success of our method for
AerobicDigit whose normal cases are heterogeneous. The proposed absolute term
in Eq. (8) could consistently boost the anomaly detection performance on
different datasets. The balanced term could also improve the performance and
stabilize the training process. Comparing ExDNN with ExDNN w/ AE, DRP is a
better choice than the autoencoder for ExDNN.

####  4.2.4 Influence of the number of exemplars

We study the influence of the number of exemplars, and we detail the results
in Table 4. The results indicate that increasing the number of exemplars could
improve the performance of ExDNN for heterogeneous normal cases in general.

####  4.2.5 Contamincation Study

ExDNN is developed based on the assumption that the training set only contains
normal data. To study the performance of ExDNN on contaminated data, we inject
different percentages of abnormal data into the training set. We vary this
percentage from 1% to 5% and conduct the experiments on HAR Laying. To
validate the effectiveness of the balanced term for handling noisy samples, we
vary the weight of the balanced term from $\\{0,1,5\\}$, and these variants of
ExDNN are denoted as ExDNN w/o bal, ExDNN w/ bal 1, and ExDNN w/ bal 5
respectively. According to the experiments shown in Fig. 3, the balanced term
can significantly boost the robustness of ExDNN. Although it is better to set
a large weight for the balanced term, the default setting of ExDNN (ExDNN w/
bal 1) is already robust to abnormal sample contamination, and is sufficient
for most real-life applications.

####  4.2.6 Training Time

We compare the training time of reconstruction-based (LSTM AE), one class-
based (Deep SVDD), and clustering-based (ExDNN) deep anomaly detection
methods. There is no need to conduct experiments with other deep learning
methods as they are developed based on either LSTM AE or Deep SVDD. The
results are shown in Table 5. LSTM AE is more computational expensive than
ExDNN and Deep SVDD for the decoder process, and the auto-encoder pertaining
required by Deep SVDD will also significantly slow down its training process.
ExDNN replaces the auto-encoder with DRP for representation learning which is
effective and brings negligible additional computation cost.

Table 5: Running time (second) per epoch of three different types of anomaly
detection methods. Deep SVDD requires extra epochs for auto-encoder
pretraining.

Methods | 2D Gesture | SWAT | ECG5000 | HAR Laying
---|---|---|---|---
ExDNN | 3.61 | 10.87 | 0.12 | 2.05
Deep SVDD | 3.41 | 10.11 | 0.10 | 1.87
LSTM AE | 5.53 | 17.26 | 0.19 | 3.24

(a) AUC

(b) F1

Figure 3: Anomaly detection performance on the contaminated HAR Laying
dataset. Table 6: Federated anomaly detection performance on six MTS datasets.
Best performance is highlighted in bold.

Dataset |  | FedAvgAE | FedProxAE | FedAvgSVDD | FedProxSVDD | FedAvgEx | FedProxEx | FedKmsEx | Fed-ExDNN
---|---|---|---|---|---|---|---|---|---
2D Gesture | AUC | 80.64$\pm$0.89 | 81.29$\pm$1.14 | 70.22$\pm$4.61 | 77.64$\pm$4.45 | 82.81$\pm$1.37 | 82.79$\pm$0.27 | 83.57$\pm$2.70 |  85.24$\pm$1.20
F1 | 57.92$\pm$1.32 | 58.13$\pm$1.28 | 51.86$\pm$5.74 | 56.58$\pm$2.60 | 63.12$\pm$2.24 | 63.97$\pm$0.93 | 62.56$\pm$1.47 |  64.75$\pm$2.41
Prec | 50.55$\pm$2.63 | 50.17$\pm$1.65 | 45.00$\pm$8.12 | 53.85$\pm$2.67 | 57.39$\pm$2.65 | 58.42$\pm$1.79 | 59.29$\pm$0.54 | 59.88$\pm$1.34
Rec | 65.44$\pm$1.69 | 66.15$\pm$0.84 | 69.44$\pm$9.54 | 59.64$\pm$2.99 | 72.99$\pm$3.25 | 71.08$\pm$4.93 | 66.25$\pm$2.61 | 74.82$\pm$3.11
SWaT | AUC | 90.19$\pm$0.17 | 90.31$\pm$0.46 |  91.57$\pm$2.49 | 90.53$\pm$5.25 | 86.20$\pm$2.96 | 88.47$\pm$1.37 | 84.41$\pm$1.23 | 89.27$\pm$1.91
F1 | 76.94$\pm$0.08 | 76.87$\pm$0.04 | 79.19$\pm$2.32 | 79.76$\pm$4.27 | 80.38$\pm$0.35 | 80.95$\pm$0.81 | 80.58$\pm$0.08 |  83.61$\pm$0.58
Prec | 62.52$\pm$1.10 | 62.44$\pm$0.05 | 65.79$\pm$3.33 | 66.75$\pm$6.08 | 70.68$\pm$1.86 | 72.17$\pm$1.88 | 70.63$\pm$0.09 | 72.57$\pm$0.25
Rec | 100$\pm$0 | 100$\pm$0 | 99.63$\pm$0.27 | 99.54$\pm$0.11 | 93.35$\pm$2.89 | 92.22$\pm$0.97 | 93.81$\pm$0.39 | 94.90$\pm$0.17
ECG5000 | AUC | 94.01$\pm$0.62 | 95.88$\pm$1.87 | 94.81$\pm$4.71 | 95.47$\pm$4.01 | 97.39$\pm$0.79 | 97.36$\pm$0.80 | 96.50$\pm$2.03 |  98.17$\pm$0.53
F1 | 89.30$\pm$0.20 | 90.66$\pm$2.24 | 91.83$\pm$4.95 | 92.67$\pm$4.47 | 92.61$\pm$1.49 | 92.61$\pm$1.60 | 91.08$\pm$4.35 |  93.86$\pm$0.48
Prec | 82.34$\pm$0.62 | 84.15$\pm$4.27 | 87.47$\pm$9.55 | 88.33$\pm$9.02 | 93.83$\pm$2.39 | 93.94$\pm$2.39 | 92.31$\pm$2.99 | 93.49$\pm$0.78
Rec | 97.83$\pm$1.02 | 98.43$\pm$0.50 | 97.68$\pm$2.11 | 98.14$\pm$1.21 | 91.45$\pm$1.48 | 91.36$\pm$1.69 | 89.93$\pm$5.64 | 94.00$\pm$0.46
HAR Laying | AUC | 89.74$\pm$1.31 | 88.74$\pm$0.21 | 99.66$\pm$0.09 | 99.55$\pm$0.06 | 99.43$\pm$0.56 | 99.71$\pm$0.22 | 99.99$\pm$0.01 |  99.99$\pm$0.01
F1 | 69.52$\pm$2.15 | 67.63$\pm$0.56 | 98.35$\pm$0.54 | 97.38$\pm$1.33 | 97.39$\pm$0.16 | 96.95$\pm$0.47 | 97.87$\pm$0.15 |  98.99$\pm$0.08
Prec | 55.13$\pm$2.70 | 52.80$\pm$0.57 | 96.77$\pm$1.05 | 95.09$\pm$2.37 | 97.51$\pm$2.30 | 99.41$\pm$0.59 | 97.30$\pm$1.75 | 98.08$\pm$0.27
Rec | 94.23$\pm$0.19 | 94.04$\pm$0.37 | 100$\pm$0 | 99.81$\pm$0.19 | 97.39$\pm$2.61 | 94.60$\pm$0.37 | 98.51$\pm$1.49 | 99.91$\pm$0.09
AerobicDigit | Avg AUC | 63.76 | 65.57 | 66.97 | 68.52 | 67.84 | 70.18 | 72.86 | 79.53
UWave | Avg AUC | 81.74 | 84.06 | 80.27 | 81.43 | 81.17 | 81.59 | 79.65 | 86.77

###  4.3 Fed-ExDNN for Fedrated Anomaly Detection

After demonstrating the effectiveness of ExDNN, we conduct experiments to
validate the superiority of Fed-ExDNN for FedUAD.

####  4.3.1 Comparison Methods and Experimental Settings

The experiments are conducted on all six MTS datasets. To simulate federated
settings, for 2D Gesture, SWaT, and ECG5000, we sequentially partition the
data into $L$ different parts and assign them to $L$ different edge devices.
For HAR Laying, we assign the samples from each subject to an edge device and
discard samples of two random activities for each subject. For AerobicDigit
and UWave, the training set on each edge device is constructed by randomly
select 900 and 300 samples from 3 different classes.

We implement several federated anomaly detection baselines. We aggregate the
local models trained by LSTM AE (Deep SVDD) by Federated Average (FedAvg) and
Federated Proximal (FedProx) as FedAvgAE (FedAvgSVDD) and FedProxAE
(FedProxSVDD), respectively. Moreover, to justify the motivation of FedCC, we
propose several variants of ExDNN as follows: we apply FedAvg and FedProx on
the proposed ExDNN as FedAvgEx and FedProxEx, respectively; we also adopt
Kmeans to aggregate the exemplars as a direct counterpart for FedCC termed as
FedKmeans. For FedCC and FedKmeans, we adopt FedAvg to aggregate the feature
encoder network. The hyperparameters of ExDNN for local training are described
in the previous section and we search $\gamma_{4}$ and $\gamma_{5}$ from
$\\{1,5,10\\}$ for FedCC. For FedProx, we search hyperparameters from
$\\{10^{-4},10^{-3},$ $10^{-2},10^{-1}\\}$. We conduct federated learning for
5 communication rounds which is sufficient for the performance of all
federated anomaly detection methods to converge. We implement the network for
FedCC with a three layer multi-layer perceptron with ReLU as activation
function. The batchsize for FedCC is set as 256 and the learning rate is
0.005. We initialize the global exemplars with kmeans++ and train FedCC for
500 steps. All methods are implemented in Pysyft he2020fedml  and Pytorch
paszke2019pytorch

####  4.3.2 Results

The results of FedUAD are shown in Table 6. According to the results, overall,
the variants of Fed-ExDNN outperform federated anomaly detection baselines.
This should be attributed to the fact that ExDNN explicitly takes the
heterogeneous data on edge devices into consideration. Moreover, FedKmsEx and
Fed-ExDNN outperform FedAvgEx and FedProxEx, since FedKmsEx and Fed-ExDNN
could handle the deviation of exemplars. Finally, the proposed Fed-ExDNN
performs better than other variants of Federated ExDNN since FedCC can
simultaneously learn to align and aggregate local exemplars. Fig. 4 shows the
federated learning results for each communication round, and we could conclude
that the proposed Fed-ExDNN consistently outperforms its counterparts.

(a) AerobicDigit 3

(b) AerobicDigit 6

(c) Uwave 4

(d) Uwave 5

Figure 4: We show the federated learning results on AerobicDigit and Uwave
datasets with different communication round. The X-axis denotes the local
training epoch and each point denotes a communication round. We conduct
experiments on subsets of AerobicDigit and Uwave for simplicity.

(a) FedAvgEx

(b) FedKmsEx

(c) FedProxEx

(d) Fed-ExDNN

Figure 5: t-SNE visualization of global exemplar modules aggregated by
different methods on AerobicDigit 1. The small circles are local exemplars and
the stars are generated global exemplars. Local exemplars with same color
means they are at same position slot of different devices, and will be
aggregated by FedAvg for a global exemplar with same color.

####  4.3.3 Interpretation

We provide an intuitive visualization of different global exemplars (stars)
learned by different aggregation methods in Fig. 5. We could see that the
global exemplars generated by FedAvgEx are collapsed. FedProxEx and FedKmsEx
neglect the alignment between local exemplars. On the contrary, Fed-ExDNN
could produce diverse global exemplars while preserving the alignment to a
certain degree. The visualization shows Fed-ExDNN works better than its
counterparts.

##  5 Conclusions

In this paper, we developed the Federated Exemplar-based Deep Neural Network
(Fed-ExDNN) to perform federated anomaly detection with multivariate time
series data. We first investigated the problem of federated unsupervised
anomaly detection with multivariate time series data. Then, we developed an
Exemplar-based Deep Neural Network (ExDNN) to learn local time series
representations based on their compatibility with an exemplar module that can
capture varieties of normal patterns. Meanwhile, we also introduced a
constrained clustering mechanism to align and aggregate the parameters of
local exemplar modules to obtain a unified global exemplar module. Finally,
the updated embedding network (for feature encoding) along with the global
exemplar module are sent back to edge devices and the anomaly detection is
conducted by comparing to those learned global exemplars. Our thoroughly
empirical studies on six public datasets have validated the effectiveness of
the proposed ExDNN and Fed-ExDNN.

##  Appendix A Additional Results

We provide more detailed results for AerobicDigit and UWave. For these two
datasets, we only report AUC score. Table 8 show the influence of different
terms on AerobicDigit. Table 8 show how the number of exemplars influences the
performance. Table 9 and Table 10 show the local and federated learning
performance.

Table 7: AUC of variants of ExDNN on AerobicDigit.

Method | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Avg
---|---|---|---|---|---|---|---|---|---|---|---
ExDNN32 w/o indvi | 67.66 | 92.66 | 47.57 | 97.14 | 57.34 | 94.95 | 57.03 | 25.63 | 83.78 | 61.97 | 68.57
ExDNN32 w/o abs | 74.48 | 98.77 | 73.37 | 90.73 | 88.29 | 94.74 | 73.58 | 88.61 | 96.88 | 79.96 | 85.94
ExDNN32 | 73.18 | 98.80 | 79.07 | 96.38 | 91.42 | 95.66 | 75.51 | 87.22 | 97.98 | 81.75 | 87.70

Table 8: AUC with different number of exemplars on AerobicDigit.

# exemplars | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Avg
---|---|---|---|---|---|---|---|---|---|---|---
8 | 71.44 | 98.14 | 77.99 | 97.37 | 79.49 | 94.26 | 65.09 | 90.16 | 96.50 | 62.95 | 83.34
16 | 70.39 | 98.61 | 64.20 | 96.51 | 86.49 | 96.20 | 67.31 | 86.64 | 97.34 | 67.51 | 84.02
32 | 73.61 | 99.04 | 78.99 | 97.19 | 91.65 | 95.77 | 77.56 | 88.87 | 98.40 | 78.41 | 87.95
64 | 74.19 | 99.30 | 80.56 | 97.05 | 90.78 | 96.28 | 79.58 | 91.59 | 98.73 | 78.17 | 88.62
128 | 72.06 | 99.34 | 78.11 | 97.27 | 92.68 | 97.19 | 79.87 | 91.39 | 98.86 | 79.35 | 88.61

Table 9: Unsupervised Anomaly Detection performance (AUC) on AerobicDigit and
UWave. Best AUC is highlighted in bold.

Class | LSTM AE | BeatGAN | MemAE | USAD | Deep SVDD | CVDD | DAGMM | Ours
---|---|---|---|---|---|---|---|---
AerobicDigit
0 | 70.19$\pm$2.28 | 65.79$\pm$5.23 | 64.84$\pm$2.72 | 65.25$\pm$1.86 | 70.25$\pm$1.33 | 71.39$\pm$2.20 |  74.27$\pm$5.93 | 73.18$\pm$0.38
1 | 97.03$\pm$0.79 | 94.86$\pm$0.79 | 98.35$\pm$0.05 | 97.12$\pm$0.53 | 94.45$\pm$1.08 | 97.17$\pm$2.50 | 87.00$\pm$1.62 |  98.80$\pm$0.26
2 | 74.59$\pm$4.51 | 73.12$\pm$6.00 | 63.02$\pm$6.65 | 69.62$\pm$5.18 | 69.28$\pm$7.68 | 76.94$\pm$6.35 |  79.70$\pm$1.51 | 79.07$\pm$0.78
3 | 92.51$\pm$3.14 | 92.29$\pm$1.29 | 93.97$\pm$0.45 | 93.64$\pm$2.32 | 71.05$\pm$1.80 | 96.53$\pm$0.61 | 72.97$\pm$7.40 |  96.38$\pm$0.80
4 | 89.18$\pm$1.01 | 85.19$\pm$0.58 |  92.54$\pm$2.71 | 84.72$\pm$2.68 | 68.34$\pm$8.43 | 85.52$\pm$3.21 | 74.91$\pm$4.79 | 91.42$\pm$1.49
5 | 91.75$\pm$2.03 | 85.17$\pm$5.36 | 87.17$\pm$3.14 | 84.18$\pm$0.65 | 82.49$\pm$7.53 | 85.25$\pm$4.32 | 83.22$\pm$4.04 |  95.66$\pm$0.56
6 | 56.78$\pm$3.21 | 54.65$\pm$3.95 | 55.87$\pm$7.60 | 50.06$\pm$1.06 | 72.09$\pm$11.30 | 75.61$\pm$4.55 |  84.59$\pm$5.47 | 75.51$\pm$2.78
7 | 72.60$\pm$0.96 | 64.56$\pm$2.93 | 72.86$\pm$1.63 | 72.33$\pm$5.02 | 72.41$\pm$6.68 | 82.46$\pm$3.23 | 86.08$\pm$0.80 |  87.22$\pm$1.28
8 |  99.20$\pm$0.08 | 98.81$\pm$0.45 | 98.76$\pm$0.17 | 98.73$\pm$0.36 | 82.83$\pm$1.30 | 87.48$\pm$5.87 | 84.14$\pm$2.73 | 97.98$\pm$0.21
9 | 53.21$\pm$1.34 | 58.21$\pm$2.36 | 49.40$\pm$4.13 | 62.36$\pm$1.25 | 58.99$\pm$11.51 | 72.86$\pm$5.68 |  88.30$\pm$4.04 | 81.75$\pm$0.08
UWave
0 | 87.63$\pm$0.29 | 85.73$\pm$8.72 | 78.18$\pm$2.38 | 80.83$\pm$5.56 | 64.48$\pm$7.14 | 72.54$\pm$5.62 | 60.35$\pm$9.58 |  88.88$\pm$2.60
1 |  95.81$\pm$2.36 | 95.79$\pm$1.59 | 94.36$\pm$1.17 | 95.58$\pm$1.32 | 60.53$\pm$0.96 | 82.76$\pm$4.57 | 50.87$\pm$9.27 | 88.48$\pm$4.06
2 | 94.91$\pm$1.04 | 94.34$\pm$0.65 |  95.98$\pm$0.83 | 95.23$\pm$0.55 | 77.57$\pm$4.76 | 76.54$\pm$4.63 | 88.21$\pm$2.05 | 92.92$\pm$1.83
3 | 82.40$\pm$1.04 | 85.78$\pm$0.47 | 85.88$\pm$1.05 | 86.50$\pm$2.56 | 78.75$\pm$0.83 | 81.25$\pm$4.32 | 86.52$\pm$2.70 |  89.21$\pm$1.92
4 | 53.93$\pm$10.06 | 73.03$\pm$8.66 | 79.03$\pm$0.96 | 69.15$\pm$6.11 |  86.21$\pm$2.96 | 83.26$\pm$8.54 | 80.23$\pm$4.40 | 72.66$\pm$1.47
5 | 80.42$\pm$0.77 | 77.50$\pm$8.51 | 72.86$\pm$1.04 | 80.43$\pm$3.78 | 80.51$\pm$3.98 | 81.24$\pm$3.51 | 78.40$\pm$8.64 |  84.12$\pm$1.81
6 | 86.74$\pm$5.56 | 87.29$\pm$1.16 | 79.63$\pm$6.50 | 88.19$\pm$6.81 | 66.89$\pm$17.77 | 89.16$\pm$5.31 | 64.25$\pm$5.42 |  95.85$\pm$2.07
7 | 92.81$\pm$4.40 | 96.14$\pm$0.19 | 96.88$\pm$0.53 | 96.96$\pm$0.73 | 47.64$\pm$9.78 | 82.15$\pm$6.31 | 76.83$\pm$7.91 |  97.64$\pm$0.41

Table 10: Federated Anomaly Detection on AerobicDigit and UWave. Best Results
are highlighted in bold.

Class | FedAvgAE | FedProxAE | FedAvgSVDD | FedProxSVDD | FedAvgEx | FedProxEx | FedKmeans | Fed-ExDNN
---|---|---|---|---|---|---|---|---
AerobicDigit
0 | 54.79$\pm$1.02 | 55.03$\pm$1.56 | 53.63$\pm$2.13 | 56.98$\pm$4.68 | 63.45$\pm$1.95 | 63.89$\pm$1.14 | 56.87$\pm$1.58 |  67.67$\pm$8.28
1 | 92.52$\pm$1.03 | 92.12$\pm$1.37 | 77.74$\pm$1.48 | 80.96$\pm$2.28 | 93.17$\pm$0.51 | 90.33$\pm$2.83 | 91.11$\pm$1.72 |  96.34$\pm$0.55
2 | 37.18$\pm$4.54 | 40.78$\pm$4.34 | 60.94$\pm$5.14 | 63.20$\pm$11.80 | 60.49$\pm$6.20 | 65.28$\pm$2.92 | 59.83$\pm$0.80 |  79.87$\pm$4.97
3 | 76.75$\pm$0.39 | 81.34$\pm$0.22 | 72.38$\pm$2.72 | 73.16$\pm$5.42 | 75.97$\pm$5.94 | 78.41$\pm$5.71 | 84.47$\pm$0.32 |  82.19$\pm$0.31
4 | 81.42$\pm$3.87 | 79.37$\pm$2.05 | 69.49$\pm$3.80 | 71.20$\pm$1.86 | 73.40$\pm$3.69 | 73.16$\pm$2.20 | 65.53$\pm$9.33 |  82.41$\pm$5.30
5 | 71.22$\pm$3.00 | 78.04$\pm$0.49 | 55.97$\pm$2.81 | 58.91$\pm$1.90 | 71.29$\pm$6.60 | 71.88$\pm$3.11 | 79.40$\pm$3.16 |  80.18$\pm$2.66
6 | 40.58$\pm$0.10 | 42.05$\pm$4.08 | 72.87$\pm$4.63 | 75.07$\pm$2.25 | 61.54$\pm$3.19 | 63.66$\pm$2.12 | 73.10$\pm$3.57 |  81.31$\pm$10.03
7 | 59.86$\pm$1.26 | 60.81$\pm$2.29 | 70.78$\pm$6.60 | 72.38$\pm$6.08 | 68.92$\pm$0.44 | 75.82$\pm$2.42 | 73.62$\pm$3.03 |  78.62$\pm$1.70
8 | 91.22$\pm$0.95 |  92.67$\pm$0.59 | 72.14$\pm$5.38 | 69.06$\pm$6.06 | 69.27$\pm$8.07 | 77.95$\pm$6.35 | 88.96$\pm$2.89 | 88.44$\pm$0.54
9 | 32.02$\pm$6.54 | 33.44$\pm$6.12 | 63.82$\pm$3.08 |  64.29$\pm$3.46 | 40.94$\pm$3.54 | 41.48$\pm$8.53 | 55.71$\pm$8.65 | 58.31$\pm$4.81
UWave
0 | 82.08$\pm$2.40 | 80.83$\pm$0.63 | 63.91$\pm$6.07 | 70.72$\pm$4.26 | 79.51$\pm$5.38 | 74.36$\pm$1.79 | 72.05$\pm$8.66 |  86.76$\pm$4.34
1 | 89.75$\pm$4.58 | 87.26$\pm$2.82 | 93.44$\pm$5.31 | 89.46$\pm$3.61 | 84.59$\pm$1.14 | 85.73$\pm$2.66 | 87.73$\pm$2.79 |  92.09$\pm$1.52
2 | 90.32$\pm$2.68 |  91.18$\pm$2.63 | 88.40$\pm$1.44 | 84.71$\pm$6.47 | 84.49$\pm$2.34 | 81.71$\pm$0.29 | 84.18$\pm$4.12 | 87.59$\pm$1.57
3 | 78.56$\pm$4.55 | 84.97$\pm$4.38 | 84.50$\pm$1.78 |  86.80$\pm$2.40 | 77.17$\pm$1.58 | 81.73$\pm$2.27 | 80.34$\pm$2.51 | 85.85$\pm$4.97
4 | 62.43$\pm$0.84 | 76.59$\pm$2.03 | 74.21$\pm$2.40 | 76.91$\pm$1.24 | 76.92$\pm$3.46 | 78.08$\pm$0.66 | 65.96$\pm$3.90 |  81.54$\pm$2.48
5 | 79.74$\pm$3.13 | 78.95$\pm$2.07 | 69.28$\pm$2.51 | 73.30$\pm$0.60 | 67.66$\pm$4.87 | 70.51$\pm$1.28 | 73.05$\pm$3.73 |  82.20$\pm$2.92
6 | 79.71$\pm$0.20 | 78.40$\pm$3.54 | 75.77$\pm$4.40 | 78.27$\pm$0.98 | 85.90$\pm$2.47 | 86.36$\pm$0.30 | 82.76$\pm$3.38 |  89.57$\pm$4.05
7 | 91.39$\pm$3.76 |  94.30$\pm$2.54 | 92.66$\pm$3.13 | 91.34$\pm$0.45 | 93.14$\pm$1.28 | 94.25$\pm$0.79 | 91.15$\pm$2.22 | 91.58$\pm$5.19

## References

  *   * (1) Kim, J., Scott, C.D.: Robust kernel density estimation. The Journal of Machine Learning Research 13(1), 2529–2565 (2012)
  * (2) Zong, B., Song, Q., Min, M.R., Cheng, W., Lumezanu, C., Cho, D., Chen, H.: Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: International Conference on Learning Representations (2018)
  * (3) Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S.A., Binder, A., Müller, E., Kloft, M.: Deep one-class classification. In: International Conference on Machine Learning, pp. 4393–4402 (2018)
  * (4) Malhotra, P., Vig, L., Shroff, G., Agarwal, P.: Long short term memory networks for anomaly detection in time series. In: Proceedings, vol. 89, pp. 89–94 (2015). Presses universitaires de Louvain
  * (5) Zhang, C., Song, D., Chen, Y., Feng, X., Lumezanu, C., Cheng, W., Ni, J., Zong, B., Chen, H., Chawla, N.V.: A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1409–1416 (2019)
  * (6) Geyer, R.C., Klein, T., Nabi, M.: Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557 (2017)
  * (7) Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST) 10(2), 1–19 (2019)
  * (8) Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečnỳ, J., Mazzocchi, S., McMahan, H.B., et al.: Towards federated learning at scale: System design. arXiv preprint arXiv:1902.01046 (2019)
  * (9) McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282 (2017). PMLR
  * (10) Pang, G., Shen, C., Cao, L., Hengel, A.v.d.: Deep learning for anomaly detection: A review. arXiv preprint arXiv:2007.02500 (2020)
  * (11) Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., Khazaeni, Y.: Federated learning with matched averaging. arXiv preprint arXiv:2002.06440 (2020)
  * (12) Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014)
  * (13) Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: International Conference on Information Processing in Medical Imaging, pp. 146–157 (2017). Springer
  * (14) Schölkopf, B., Williamson, R.C., Smola, A.J., Shawe-Taylor, J., Platt, J.C.: Support vector method for novelty detection. In: Advances in Neural Information Processing Systems, pp. 582–588 (2000)
  * (15) Liu, F.T., Ting, K.M., Zhou, Z.-H.: Isolation forest. In: 2008 Eighth Ieee International Conference on Data Mining, pp. 413–422 (2008). IEEE
  * (16) Ruff, L., Zemlyanskiy, Y., Vandermeulen, R., Schnake, T., Kloft, M.: Self-attentive, multi-context one-class classification for unsupervised anomaly detection on text. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4061–4071 (2019)
  * (17) Shen, L., Li, Z., Kwok, J.: Timeseries anomaly detection using temporal hierarchical one-class network. Advances in Neural Information Processing Systems 33 (2020)
  * (18) Gong, D., Liu, L., Le, V., Saha, B., Mansour, M.R., Venkatesh, S., Hengel, A.v.d.: Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1705–1714 (2019)
  * (19) Zhou, B., Liu, S., Hooi, B., Cheng, X., Ye, J.: Beatgan: Anomalous rhythm detection using adversarially generated time series. In: IJCAI, pp. 4433–4439 (2019)
  * (20) Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., Pei, D.: Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2828–2837 (2019)
  * (21) Audibert, J., Michiardi, P., Guyard, F., Marti, S., Zuluaga, M.A.: Usad: Unsupervised anomaly detection on multivariate time series. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3395–3404 (2020)
  * (22) Li, Z., Zhao, Y., Han, J., Su, Y., Jiao, R., Wen, X., Pei, D.: Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3220–3230 (2021)
  * (23) Deldari, S., Smith, D.V., Xue, H., Salim, F.D.: Time series change point detection with self-supervised contrastive predictive coding. In: Proceedings of the Web Conference 2021, pp. 3124–3135 (2021)
  * (24) Cho, H., Seol, J., Lee, S.-g.: Masked contrastive learning for anomaly detection. arXiv preprint arXiv:2105.08793 (2021)
  * (25) Sehwag, V., Chiang, M., Mittal, P.: Ssd: A unified framework for self-supervised outlier detection. arXiv preprint arXiv:2103.12051 (2021)
  * (26) Carmona, C.U., Aubet, F.-X., Flunkert, V., Gasthaus, J.: Neural contextual anomaly detection for time series. arXiv preprint arXiv:2107.07702 (2021)
  * (27) Qiu, C., Pfrommer, T., Kloft, M., Mandt, S., Rudolph, M.: Neural transformation learning for deep anomaly detection beyond images. arXiv preprint arXiv:2103.16440 (2021)
  * (28) Konečnỳ, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., Bacon, D.: Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016)
  * (29) Liu, B., Tan, C., Wang, J., Zeng, T., Shan, H., Yao, H., Heng, H., Dai, P., Bo, L., Chen, Y.: Fedlearn-algo: A flexible open-source privacy-preserving machine learning platform. arXiv preprint arXiv:2107.04129 (2021)
  * (30) Sahu, A.K., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A., Smith, V.: On the convergence of federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06127 3 (2018)
  * (31) Li, X., Huang, K., Yang, W., Wang, S., Zhang, Z.: On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 (2019)
  * (32) Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., Suresh, A.T.: Scaffold: Stochastic controlled averaging for federated learning. In: International Conference on Machine Learning, pp. 5132–5143 (2020). PMLR
  * (33) Reisizadeh, A., Farnia, F., Pedarsani, R., Jadbabaie, A.: Robust federated learning: The case of affine distribution shifts. arXiv preprint arXiv:2006.08907 (2020)
  * (34) Dinh, C.T., Tran, N.H., Nguyen, T.D.: Personalized federated learning with moreau envelopes. arXiv preprint arXiv:2006.08848 (2020)
  * (35) Zhu, Z., Hong, J., Zhou, J.: Data-free knowledge distillation for heterogeneous federated learning. arXiv preprint arXiv:2105.10056 (2021)
  * (36) Sattler, F., Müller, K.-R., Samek, W.: Clustered federated learning: Model-agnostic distributed multi-task optimization under privacy constraints. arXiv preprint arXiv:1910.01991 (2019)
  * (37) Xie, M., Long, G., Shen, T., Zhou, T., Wang, X., Jiang, J.: Multi-center federated learning. arXiv preprint arXiv:2005.01026 (2020)
  * (38) Liang, P.P., Liu, T., Ziyin, L., Salakhutdinov, R., Morency, L.-P.: Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523 (2020)
  * (39) Yu, F.X., Rawat, A.S., Menon, A.K., Kumar, S.: Federated learning with only positive labels. arXiv:2004.10342 (2020)
  * (40) Muhammad, K., Wang, Q., O’Reilly-Morgan, D., Tragos, E., Smyth, B., Hurley, N., Geraci, J., Lawlor, A.: Fedfast: Going beyond average for faster training of federated recommender systems. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1234–1242 (2020)
  * (41) Lin, T., Kong, L., Stich, S.U., Jaggi, M.: Ensemble distillation for robust model fusion in federated learning. arXiv preprint arXiv:2006.07242 (2020)
  * (42) Fallah, A., Mokhtari, A., Ozdaglar, A.: Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems 33 (2020)
  * (43) Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning, pp. 1126–1135 (2017). PMLR
  * (44) Singh, S., Bhardwaj, S., Pandey, H., Beniwal, G.: Anomaly detection using federated learning. In: Proceedings of International Conference on Artificial Intelligence and Applications, pp. 141–148 (2021). Springer
  * (45) Nguyen, T.D., Marchal, S., Miettinen, M., Fereidooni, H., Asokan, N., Sadeghi, A.-R.: Dïot: A federated self-learning anomaly detection system for iot. In: 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pp. 756–767 (2019). IEEE
  * (46) Zhao, Y., Chen, J., Wu, D., Teng, J., Yu, S.: Multi-task network anomaly detection using federated learning. In: Proceedings of the Tenth International Symposium on Information and Communication Technology, pp. 273–279 (2019)
  * (47) Sater, R.A., Hamza, A.B.: A federated learning approach to anomaly detection in smart buildings. ACM Transactions on Internet of Things 2(4), 1–23 (2021)
  * (48) Wang, X., Garg, S., Lin, H., Hu, J., Kaddoum, G., Piran, M.J., Hossain, M.S.: Towards accurate anomaly detection in industrial internet-of-things using hierarchical federated learning. IEEE Internet of Things Journal (2021)
  * (49) Mothukuri, V., Khare, P., Parizi, R.M., Pouriyeh, S., Dehghantanha, A., Srivastava, G.: Federated learning-based anomaly detection for iot security attacks. IEEE Internet of Things Journal (2021)
  * (50) Liao, W., Guo, Y., Chen, X., Li, P.: A unified unsupervised gaussian mixture variational autoencoder for high dimensional outlier detection. In: 2018 IEEE International Conference on Big Data (Big Data), pp. 1208–1217 (2018). IEEE
  * (51) Hu, W., Miyato, T., Tokui, S., Matsumoto, E., Sugiyama, M.: Learning discrete representations via information maximizing self-augmented training. arXiv preprint arXiv:1702.08720 (2017)
  * (52) Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: International Conference on Machine Learning, pp. 478–487 (2016)
  * (53) Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. science 290(5500), 2323–2326 (2000)
  * (54) Chui, C.K., Mhaskar, H.N.: Deep nets for local manifold learning. Frontiers in Applied Mathematics and Statistics 4, 12 (2018)
  * (55) Sun, Y., Cheng, C., Zhang, Y., Zhang, C., Zheng, L., Wang, Z., Wei, Y.: Circle loss: A unified perspective of pair similarity optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6398–6407 (2020)
  * (56) Keogh, E., Lin, J., Fu, A.: Hot sax: Efficiently finding the most unusual time series subsequence. In: Fifth IEEE International Conference on Data Mining (ICDM’05), p. 8 (2005). Ieee
  * (57) Goldberger, A.L., Amaral, L.A., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.-K., Stanley, H.E.: Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation 101(23), 215–220 (2000)
  * (58) Mathur, A.P., Tippenhauer, N.O.: Swat: A water treatment testbed for research and training on ics security. In: 2016 International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater), pp. 31–36 (2016). IEEE
  * (59) Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J.L., et al.: A public domain dataset for human activity recognition using smartphones. In: Esann, vol. 3, p. 3 (2013)
  * (60) Liu, J., Zhong, L., Wickramasuriya, J., Vasudevan, V.: uwave: Accelerometer-based personalized gesture recognition and its applications. Pervasive and Mobile Computing 5(6), 657–675 (2009)
  * (61) Hammami, N., Sellam, M.: Tree distribution classifier for automatic spoken arabic digit recognition. In: 2009 International Conference for Internet Technology and Secured Transactions,(ICITST), pp. 1–4 (2009). IEEE
  * (62) Hammami, N., Bedda, M.: Improved tree model for arabic speech recognition. In: 2010 3rd International Conference on Computer Science and Information Technology, vol. 5, pp. 521–526 (2010). IEEE
  * (63) He, C., Li, S., So, J., Zeng, X., Zhang, M., Wang, H., Wang, X., Vepakomma, P., Singh, A., Qiu, H., et al.: Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13518 (2020)
  * (64) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32, 8026–8037 (2019)