# A Semi-Supervised Approach for Abnormal Event Prediction on Large
Operational Network Time-Series Data

Yijun Lin and Yao-Yi Chiang
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Department of Computer Science and Engineering
University of Minnesota, Twin Cities

###### Abstract

Large network logs, recording multivariate time series generated from
heterogeneous devices and sensors in a network, can often reveal important
information about abnormal activities, such as network intrusions and device
malfunctions. Existing machine learning methods for anomaly detection on
multivariate time series typically assume that 1) normal sequences would have
consistent behavior for training unsupervised models, or 2) require a large
set of labeled normal and abnormal sequences for supervised models. However,
in practice, normal network activities can demonstrate significantly varying
sequence patterns (e.g., before and after rerouting partial network traffic).
Also, the recorded abnormal events can be sparse. This paper presents a novel
semi-supervised method that efficiently captures dependencies between network
time series and across time points to generate meaningful representations of
network activities for predicting abnormal events. The method can use the
limited labeled data to explicitly learn separable embedding space for normal
and abnormal samples and effectively leverage unlabeled data to handle
training data scarcity. The experiments demonstrate that our approach
significantly outperformed state-of-the-art approaches for event detection on
a large real-world network log.

Keywords: semi-supervised learning, multivariate time series, network event
prediction

##  1 Introduction

Nowadays, many network management companies (e.g., Cisco, NTT Global Networks)
collect large volumes of data logs to measure and monitor the status (e.g.,
healthy, overloaded, power outage) of devices (e.g., routers, switches) or
interfaces (e.g., local area networks (LAN), wide area networks (WAN)) in a
network. Figure 1 shows an example network. A network usually consists of
multiple devices, each with some interfaces, e.g., an SDWAN device has LAN and
WAN interfaces. Each interface is measured by several metrics or attributes,
i.e., the number of in/out octets and in/out ucast packets, that describe
network operations or activities as multivariate time series.

Figure 1: An example network contains two SDWAN devices, and the devices have
some LAN and WAN interfaces, each measured by four attributes.

Efficiently and accurately predicting abnormal events from network data logs
is important, which helps continuously monitor network states and raise alerts
for potential incidents on time for network engineers to investigate and
resolve. Traditional monitoring systems typically trigger a real-time
notification when the network is experiencing some abnormal event (e.g.,
packet loss, intrusion) using pre-defined rules but cannot provide an early
warning on a potential incident. Also, a network typically contain hundreds if
not thousands of heterogeneous devices from different vendors, which can have
diverse criteria (e.g., some rules and thresholds) indicating varying levels
of abnormal status, making it challenging to build an early warning system and
enable the system handling all kinds of networks.

Advanced data-driven methods on analyzing multiple multivariate time series
(e.g., machine learning approaches) can automatically learn and extract
patterns from network operations that lead to a potential abnormal event,
which has the potential to handle heterogeneous networks, devices, and
interfaces. One challenge is how to effectively capture temporal dependencies
across time and relationships between time series that can jointly represent
network activities for predicting an abnormal event. Some existing approaches
overlook the relationships between time series, e.g., univariate time-series
anomaly detection algorithms [16], fusing multiple time series directly as
features [12, 10]. In terms of modeling temporal dependencies, many approaches
use recurrent neural networks (RNNs) [12, 8] yet they are often time-consuming
and require large memory for computation. Transformers, oriented from natural
language processing, become popular in time-series analysis problems [1], but
they have similar issues.

In addition, some labeled data might be available, but the labeling criteria
could vary significantly among devices and interfaces, e.g., an abnormal event
can last for several minutes, and sometimes only the beginning time point is
marked as anomalous. Unsupervised approaches, including reconstruction-based
methods [14, 21], forecasting-based methods [9, 8], and hybrid methods [29],
identify infrequent behaviors in multivariate time series as an anomaly.
However, this assumption is not always true, e.g., long-term peak usage of
memory might be rare but normal. Supervised models [11, 19] can leverage
labeled data to guide the process of learning the time-series patterns leading
to an abnormal event, but they usually require sufficient labeled samples.
Also, the number of labeled abnormal samples is usually much smaller compared
to the normal samples. Therefore, limited and imbalanced labels make it
challenging to capture the patterns leading to abnormal activities.

This paper proposes a semi-supervised learning approach that learns
representative features for network activities and use the features to predict
abnormal events (e.g., packet loss, intrusion). Specifically, our approach
first takes a given network and constructs a graph representation where each
node contains some attributes of an interface (e.g., packet volume) at a time
point. (There could be some dependencies (e.g., data transmission) between
interfaces (nodes) via the linkage.) Considering a sequence of graphs (i.e.,
network data logs over a time period) leading to some abnormal event, our
approach aims to generate an embedding to represent network behaviors and
capture the unique pattern of this graph sequence. The embedding should
contain effective temporal information by capturing temporal dependencies
across time points (i.e., temporal patterns). The embedding also includes
relevant information from neighboring nodes, capturing data communication
patterns. Therefore, our approach jointly captures temporal dependencies
across time points and spatial dependencies between nodes for generating an
embedding of the input. Then these learned embeddings are refined to form two
separable clusters representing normal and abnormal network behaviors. Our
approach uses labeled samples to explicitly learn the cluster centers in a
contrastive way. However, due to imbalanced and limited labeled samples, the
learnt centers might not be applicable to all samples and hence poor embedding
quality. Our approach updates the embeddings of all samples towards a high
confidence clustering distribution to enforce cluster cleanness with a semi-
supervised learning strategy. The goal is to improve separability between
normal and abnormal network behaviors. Finally, our approach utilizes a
support vector classifier on these embeddings to predict if the given graph
sequence of network operations will lead to an abnormal event.

In sum, the main contribution of this paper is a semi-supervised learning
approach that learns separable embeddings that jointly captures temporal
dependencies between time points and relationships between time series for
predicting abnormal events. The proposed approach can efficiently capture
complex dependencies in multiple multivariate time-series data and learn
effective features from imbalanced and limited labels, enabling accurate
prediction of abnormal events. We demonstrate the proposed network
architecture for abnormal event prediction on large real-world network
operation time-series data.

##  2 Methodology

###  2.1 Problem Definition

Given a network containing multiple interfaces with unknown connectivity, our
system first constructs a fully-connected graph,
$\boldsymbol{G}=(\boldsymbol{V},\boldsymbol{E})$, where each node
$i\in\boldsymbol{V}$ represents an interface ($|\boldsymbol{V}|=n$) and the
connection between any pair of nodes is initialized as one, meaning there can
be data communication between interfaces. Let
$\boldsymbol{G}^{(t)}\in\mathbb{R}^{n\times F}$ be the graph for network
operation measurements at time $t$, where $n$ is the number of nodes and $F$
is the number of attributes describing each node. Suppose the input is a
sequence of graphs of length $T$, i.e., graphs from $t-T+1$ to $t$, our
approach aims to learn a function $f$ that maps $T$ historical input signals
to a binary signal $y\in\\{0,1\\}$ indicating if there will be an abnormal
event in the network at $t+1$.

$[\boldsymbol{G}^{(t-T+1)},\cdots,\boldsymbol{G}^{(t)}]\xrightarrow{\emph{f}}y$

Figure 2 shows the proposed network architecture for learning feature
embeddings from the graph input. Specifically, our approach first builds an
GatedConv-NGAT module to generate an initial feature embedding by jointly
capturing dependencies among graphs (across time points in the sequence) and
between nodes (between individual time series in a graph) (Section 2.2). Then
our approach jointly refines the embeddings and cluster centers representing
normal and abnormal activities by: (1) leveraging labeled normal and abnormal
samples to form clusters in a contrastive learning way; (2) improving the
separability of clusters using both labeled and unlabeled samples in a semi-
supervised manner (Section 2.3). Finally, our approach trains a support vector
classifier with these learned embeddings for the prediction task (Section
2.4).

Figure 2: The training process of our system to generate feature
representation for predicting abnormal events

###  2.2 GatedConv-NGAT Module.

This section introduces the ST-Conv-NGAT module that generates embeddings from
the input graph sequence (Figure 3). In general, the module contains temporal
gated convolutional layers to capture temporal dependencies across time and a
normalized graph attention layer to capture dependencies between nodes. The
details of each sub-module are described as follows.

Figure 3: GatedConv-NGAT Module

GatedConv layer for extracting temporal relationships. Recurrent neural
networks (RNNs), e.g., long short-term memory (LSTM), have been widely used
for capturing temporal dependencies in time-series data, but these models
typically suffer from complex gating mechanisms, time-consuming computation,
and large memory requirement since every step relies on the previous steps. In
comparison, convolutional neural networks (CNNs) have the advantages of fast
training and parallel computation [7] that can process large network data logs
more efficiently. Also, CNNs explicitly capture local features within a
temporal neighborhood, and one or more local features might cause an abnormal
event, instead of global features from the entire sequence.

Our approach applies gated-conv layers [7] with multiple kernel sizes
consisting of 1D convolution operations on the time axis to extract local
temporal features for each node. The input to the 1D convolution operation is
a sequence of length $T$ with a channel size $F$ at each node
$\in\mathbb{R}^{T\times F}$. With a kernel size $K$ and a hidden size $C$, the
convolution kernel maps the input to the output $[\mathbf{m}\
\mathbf{n}]\in\mathbb{R}^{(T-K+1)\times(2C)}$, where $\mathbf{m}$,
$\mathbf{n}$ split the output into halves with the same hidden size $h$. The
gated-conv layer leverages gated linear units (GLU) [3] as a non-linear gating
mechanism over $\mathbf{m}$ and $\mathbf{n}$:

(2.1) $\text{GatedConv}(\mathbf{v})=\mathbf{m}\odot\sigma(\mathbf{n})\
\in\mathbb{R}^{(T-K+1)\times(C)}$

where $\mathbf{m}$ and $\mathbf{n}$ are the input to the gates of GLU, $\odot$
denotes the element-wise product, $\sigma$ is the sigmoid function, and
$\sigma(\mathbf{n})$ controls which information in $\mathbf{m}$ are relevant
for generating temporal features.

Normalized graph attention layer for extracting relationships between nodes.
After the first Gated-Conv layer, each node $i$ is represented by $(T-K+1)$
feature vectors of size $C$, where each vector contains the extracted temporal
features for the corresponding sub-sequence. In addition to temporal
dependencies, combining information from other nodes is also essential for
determining an abnormal event. Thus, our approach introduces a normalized
graph attention layer to capture relationships between nodes.

Since the initial graph topology is unknown, many graph convolution
strategies, e.g., chebyshev spectral graph convolutional operator [4], might
not be suitable because they require the graph connectivity as the input. Our
approach leverages a graph attention (GAT) layer [23] to model the spatial
relationships between nodes. A GAT layer computes the representation for each
node:

(2.2)
$\text{GAT}(i)=\sigma\Big{(}\sum_{k\in\mathcal{N}_{i}}\alpha_{ik}\mathbf{v_{k}}\Big{)}$

where node $i$ is the target node, $\mathbf{v_{i}}\in\mathbb{R}^{C}$ is the
hidden feature vector of node $i$, $\mathcal{N}_{i}$ is the neighborhood set
of node $i$, $\sigma$ is the sigmoid function, $\alpha_{ik}$ is the attention
score for node $k$ to node $i$. The attention score $\alpha_{ik}$ is defined
by:

(2.3) $\alpha_{ik}=\frac{\text{exp}(\sigma(\mathbf{w}^{T}\cdot(\mathbf{v_{i}}\
\|\
\mathbf{v_{k}}))}{\sum_{j\in\mathcal{N}_{i}}\text{exp}(\sigma(\mathbf{w}^{T}\cdot(\mathbf{v_{i}}\
\|\ \mathbf{v_{j}}))))}$

where the operation $\|$ is to concatenate two vectors,
$\mathbf{w}\in\mathbb{R}^{2C}$ is a learnable weight vector, $\mathcal{N}_{i}$
is the neighborhood set of node $i$, $\sigma$ is the LeakyReLU function.

Our approach then adapts graph normalization (GraphNorm) [2] to the output of
the GAT layer, which makes the optimization more efficient. GraphNorm
normalizes every feature dimension for individual graphs with a learnable
shift, which significantly boosts the training upon InstanceNorm [22] (also
does graph-targeted normalization):

(2.4)
$\text{GraphNorm}(h_{i,j})=\gamma_{j}\cdot\frac{h_{i,j}-\alpha_{j}\mu_{j}}{\sigma_{j}}+\beta_{j}$
$\mu_{j}=\frac{\sum_{k=1}^{n}h_{k,j}}{n},\
\sigma_{j}=\frac{\sum_{k=1}^{n}(h_{k,j}-\alpha_{j}\mu_{j})}{n}$

where $h_{i,j}$ is the value of the $j$th feature dimension of $\text{GAT}(i)$
(the output of GAT layer for node $i$), $n$ is the number of nodes in the
graph, $\alpha_{j}$ is the learnable parameter for the feature dimension $j$,
which controls how much information to keep in the mean. $\gamma_{j}$ and
$\beta_{j}$ are the affine parameters.

Putting everything together to learn embeddings. Inspired by [28], our
approach adds another temporal gated-conv layer after the NGAT layer to fuse
the features of node interactions on the time axis. This “sandwich” structure
(i.e., a spatial layer in the middle of two temporal layers) can achieve fast
spatial-state propagation through temporal convolutions. After the second
temporal gated-conv layer, our approach concatenates the output to generate
the node embeddings. Then our approach uses a global pooling layer to add node
embeddings across the node dimension to generate the graph embedding. In this
way, given
[$\boldsymbol{G}^{(t-T+1)},\cdots,\boldsymbol{G}^{(t)}]\in\mathbb{R}^{T\times
n\times F}$, the GatedConv-NGAT module generates the embedding
$\boldsymbol{E}^{(t)}$ of length ${(T-2(K-1))\times C}$, that jointly captures
the temporal dependencies and relationship between nodes.

###  2.3 Learning clustering structure.

Our approach assumes two clusters in the embedding space: normal and abnormal
events. The goal is to learn a representation space to form these two
separable clusters.

Learning clustering structure on labeled samples using contrastive center
loss. The labeled samples can explicitly guide the process of learning
clustering centers by enforcing the embeddings of normal samples to be close
to the normal center and far away from the abnormal center, and vice versa.
Our approach applies the contrastive-center loss [15] that considers intra-
class compactness and inter-class separability simultaneously. This loss
function aims to penalize the contrastive values between (1) the distances of
samples to their corresponding class center and (2) the distances of samples
to their non-corresponding class centers,

(2.5)
$\mathcal{L}_{cc}=\frac{1}{2}\sum^{m}_{i=1}\frac{\lVert\mathbf{E_{i}}-\mathbf{c_{y_{i}}}\rVert^{2}_{2}}{\lVert\mathbf{E_{i}}-\mathbf{c_{j}}\rVert^{2}_{2}+\epsilon}\
(j\neq y_{i})$

where $L_{cc}$ denotes the contrastive-center loss; $m$ is the number of
labeled samples in a min-batch; $\mathbf{E_{i}}$ is the embedding of the
$i$-th training sample; $\mathbf{c_{y_{i}}}$ is the corresponding center of
$\mathbf{E_{i}}$. Since there are two clusters, when $y_{i}$ is 0, $j$ is 1,
and vice versa ($j\neq y_{i}$). $\epsilon$ is a constant for preventing the
denominator equal to 0 and set to ${1}\mathrm{e}{-6}$ empirically.

Learning clustering structure on all samples using a semi-supervised learning
strategy. Using the contrastive learning strategy improves the clustering
cleanness for labeled samples; however, due to limited and imbalanced labeled
samples, the learned centers are easily guided by the given labeled samples,
which might fail in inferencing. To overcome the scarcity of labeled training
samples and ensure the clusters are applicable to all samples, our approach
takes the idea from [27] to minimize the distance between the cluster
assignment and some target cluster distribution. For labeled samples, the
target distribution can be directly derived from the label, while for the
unlabeled samples, the target distribution is derived from high confidence
predictions of the current cluster distribution from all samples. Formally,
our approach calculates the probability of assigning the embedding
$\mathbf{E_{i}}$ to the cluster $j$ with a center $\mu_{j}$ as

(2.6)
$q_{ij}=\frac{(1+\rVert\mathbf{E_{i}}-\mu_{j}\rVert^{2})^{-1}}{\sum_{j^{\prime}}(1+\rVert\mathbf{E_{i}}-\mu_{j^{\prime}}\rVert^{2})^{-1})}$

Following the work in [27], our approach derives an auxiliary distribution
made up of the high confidence assignments of the current distribution
(referred as $Q$). The main idea is to emphasize data points assigned with
high confidence to ensure cluster purity. Equation 2.7 defines the computation
of auxiliary distribution (referred as $P$), where the second power of
$q_{ij}$ places more weight on the instances near the center and the division
of $\sum_{i}q_{ij}$ normalizes based on the cluster size, making the model
robust to biased classes. Note that for labeled samples, our approach replaces
the true distribution with the auxiliary distribution and hence enables the
training process in a semi-supervised manner.

(2.7)
$p_{ij}=\frac{q_{ij}^{2}/\sum_{i^{\prime}}q_{i^{\prime}j}}{\sum_{j^{\prime}}(q_{ij^{\prime}}^{2}/\sum_{i^{\prime}}q_{i^{\prime}j^{\prime}})}$

To measure the distance between the distributions $P$ and $Q$, our approach
computes the KL divergence to measure their distance. The clustering oriented
loss is defined as,

(2.8)
$\mathcal{L_{KL}}=KL(P||Q)=\sum_{i}\sum_{j}p_{ij}\log\frac{p_{ij}}{q_{ij}}$

Therefore, the total loss is the sum of contrastive center loss and KL
divergence loss with a hyper-parameter $\lambda$,

(2.9) $\mathcal{L}=\mathcal{L}_{cc}+\lambda\times\mathcal{L}_{KL}$

###  2.4 Predicting abnormal events from learned embeddings using SVC.

Using the methods described so far in this section, our model generates
separable embeddings (normal vs. abnormal). Then we take the embeddings and
labels to train a support vector classifier to predict normal and abnormal
events. Our approach does not train in an end-to-end manner, e.g., using a
fully-connected layer for classification in the model because support vector
classifier usually works better if the input are already separable in some
space. Figure 4 shows the prediction process. Given a sequence of graphs
representing network operations, the GatedConv-NGAT module generates the
embedding and the support vector classifier takes the embedding to predict if
the input would lead to an abnormal event.

Figure 4: Event prediction

##  3 Experiments and Results

We implemented the proposed architecture with Python 3.8, PyTorch 1.8, and
PyTorch Geometric framework [6]. We conducted the experiments in a Docker
container deployed on a GPU server and trained the model with one physical
core and 16GB memory.

###  3.1 Datasets.

We used the network operation data from NTT-GN for the experiments. The
dataset contains more than 150 million data logs from network interfaces
(e.g., WAN routers and LAN switches) that NTT-GN collects, monitors, and
manages. The data logs we used cover the time range from 2020-05-12 to
2020-06-12, with a time interval of 5 minutes. Figure 1 shows an example of
the data structure. In this work, we assume a practical case that there could
be dependencies between any interface pairs in a network (e.g., data
transmissions) (Figure 5(a)), and the network topology in not available. The
graph representation can be modified if prior knowledge of the network
topology is available.

In addition to the network operation data, NTT-GN also provides the packet
loss event dataset that reports the ticket opening time at the device level.
We assume that events are caused by some abnormal network activities in the
network and can be captured by looking at the dynamics of the entire network.
Figure 5(b) shows an example of assigning a label to a period of network
activities (e.g., 6 hours, a sequence of 72 graphs). If there is an event
reported at $t^{\prime}$ and the closest graph before that is
$\mathbf{G}^{(t)}$, we assigned label 1 (color blue) to the graph sequence
$\mathbf{G}^{(t-71)}:\mathbf{G}^{(t)}$, denoting that these graphs contains
some abnormal network activities directly leading to an event. Since we do not
know how long the event lasts, the graph sequences overlapped with an event
within 6 hours before and after (e.g., color yellow) were marked as “unknown”
(i.e., unlabeled data). These graph sequences also help learn feature
embeddings in our semi-supervised setting. The rest graph sequences that do
not overlap with any event within 6 hours were labeled as 0, representing
normal network activities.

(a) The graph representing the example network

(b) An example of graph label construction

Figure 5: Examples of sample construction

###  3.2 Experiment Settings.

To evaluate the proposed approach, we selected the available networks with at
least 50 events. We split the samples into training, validation, and testing
sets, ensuring each set contains both label 0 and label 1. Table 1 shows the
summary of the dataset. We observe that the number of samples labeled with 0
is much larger than label 1. Also, the ratio of labeled samples to unlabeled
samples is around 0.2. We repeated the random splitting process three times
for cross-validation and took the average as the results.

We set the length of the input graph sequence in a sample as 72. The batch
size was 32, each containing both labeled and unlabeled samples. We set the
temporal gated convolutional layers as 32 hidden states and the kernel size of
1D Conv as 12. The size of the output feature embeddings was 256. The initial
learning rate was 0.001, with early stopping on the validation dataset. The
hyper-parameters were chosen by grid search on the validation set, and we
reported the results with the set of hyper-parameters yielding the best
average performance, that is $\lambda=0.1$.

Table 1: Summary of the datasets Network ID | #Nodes | #Events | # 0: # 1
---|---|---|---
1NTT18927 | 53 | 125 | 0.96:0.04
1NTT19535 | 84 | 142 | 0.89:0.11
1NTT67246 | 36 | 134 | 0.91:0.09
1NTT19478 | 120 | 147 | 0.88:0.12
143XB8CT | 97 | 78 | 0.98:0.02
1NTT30990 | 20 | 71 | 0.98:0.02
1NTT48337 | 89 | 83 | 0.95:0.05
1NTT19213 | 119 | 78 | 0.96:0.04
1NTT22885 | 91 | 91 | 0.95:0.05
1NTT18635 | 143 | 189 | 0.73:0.27

Metrics. We reported the precision, recall, and F1 score for label 1 since our
goal is to predict events. The higher the values the better the performance.
We put the detailed functions in the supplementary file.

Baselines. We compared our proposed model to the following state-of-the-art
baselines:

  * •

Support Vector Classifier (SVC): We convert the problem to a binary
classification task for SVC. We flatten an input graph sequence in time and
node dimensions to be a large vector for predicting event or non-event.

  * •

Graph Attention Encoder (GAT): We concatenate features along the temporal
dimension for each node and replace the encoder with only the graph attention
module to capture interactions between time series. This method aims to
demonstrate the necessity of capturing temporal dependencies.

  * •

Graph Attention LSTM (GAT-LSTM) [25]: Instead of using the ST-Conv module, we
apply LSTM in the encoder to capture interactions across time points. The rest
architecture remains the same as our proposed approach.

  * •

MTAD-GAT [29]: This is an unsupervised anomaly detection approach that
explicitly captures relationships between multiple features and across time
points. We modify the approach so that the node value has more than one
dimension.

  * •

DeepSAD [18]: This is a semi-supervised learning approach for anomaly
detection that aims to learn a clustering center to represent normal cases.
The original domain is computer vision. To extend the model on our dataset, we
apply the ST-Conv-NGAT module as the encoder.

  * •

We select the following variations of the proposed model for ablation study:
1. ST-Conv-NGAT + only contrastive center loss (only-CC), which would be
trained only with labeled samples; 2. ST-Conv-NGAT + only KL divergence loss
(only-KL); 3. Constructing a correlation graph as the input [5], which
computes pairwise Pearson correlation between nodes and assumes two nodes are
connected only if the correlation value is above the mean plus one std.

###  3.3 Model Performance.

In general, our approach outperformed baseline methods in all networks (see F1
scores in Figure 6 and precision & recall in the supplementary file). The F1
scores in most networks are above 0.7. Even when the ratio between the number
of label 0 and label 1 is high (e.g., “143XB8CT” and “1NTT30990”), our model
could still achieve far better results than the baseline models. Specifically,
our approach achieved more than 50% improvement in F1 scores compared to SVC.
This is because SVC simply takes the input graph sequence as a flattened
vector and does not explicitly capture relationships between time series or
across time points.

Compared to using other encoding methods (GAT and GAT-LSTM) to generate
representations for prediction, our approach with the ST-Conv-NGAT module as
the encoder showed excellent advantage. First, GAT does not explicitly
consider temporal dependencies but fuses features on the time dimension
together and expects the model to extract useful information from these values
for prediction directly. However, the temporal order of these values matters
for event prediction, e.g., the position of a sudden increment might result in
different network behaviors. Our approach leverages the 1D-CNN to summarize
local temporal features, aiming to extract effective temporal patterns leading
to an event. Second, although GAT-LSTM is commonly used for capturing
spatiotemporal dependencies in multi-time-series data [25], its performance is
poorer than our approach, which might be because GAT-LSTM is typically
generating representations for the entire time series. However, in practice,
only some subsequence(s) from the input graph sequence would cause an event.
Some effective patterns (or features) at the early stage of the input sequence
might be easily weakened or neglected during the sequential training of LSTM.
Since the impacting subsequence(s) on the event are unknown (i.e., the leading
time to an event could be varying), the sequence length (i.e., six hours in
the experiment) was chosen to cover as many time points as possible. In
comparison, the 1D-CNN in our approach focuses on manipulating temporal
interactions on every subsequence, which empirically fulfills our assumption
and achieves better performance on real datasets. Also, training GAT-LSTM for
one epoch took more than 600 seconds while our approach required around 170
seconds under the same settings.

Furthermore, compared to existing work on anomaly detection, our approach
achieved better performance than the unsupervised learning approach (MTAD-
GAT), which demonstrates that assuming rare samples as anomalies might not be
applicable in practice. Adding some labeled samples during training can guide
the learning process to generate representative embeddings. Additionally, our
approach outperformed the semi-supervised approach, DeepSAD. DeepSAD has been
demonstrated to outperform some unsupervised approaches in the original paper.
The loss function in DeepSAD enforces labeled normal samples to be close to
the predefined center and abnormal samples to be as far as possible. For the
unlabeled samples, DeepSAD has a strong assumption that the majority of data
samples are normal and treats all unlabeled samples as normal in the loss.
However, such an assumption might not hold in practice since one event can
last for some period (i.e., the unlabeled data might contain abnormal network
activities), which might confuse the model to learn effective representations
for unlabeled samples and result in poor performance.

Figure 6: The F1 scores for baselines and our method (red). See precision and
recall in the supplementary file.

###  3.4 Ablation Study.

(1) Effect of Loss Functions. Table 2 shows that our approach achieved 12%
improvement in F1 scores on average compared to only using the contrastive
center (CC) loss and 14% improvement compared to only using the KL loss. The
model with only the CC loss was trained with limited labeled samples, which
worked well for the networks with large abnormal samples (e.g., more than 120
events in “1NTT67246” and “1NTT18635”). However, the performance dropped in
the cases with only a few abnormal samples. Our proposed approach trained on
both labeled and unlabeled samples to force the learnt clustering structure
was applicable to all data samples, resulting in better performance, e.g., 48%
improvement in F1 score for “1NTT30990” and 35% for “1NTT19213”. In the other
case, the model with only KL loss was trained with all samples. Without the CC
loss to explicitly optimize the centers, finding good centers was challenging
even when the true clustering distribution could be estimated from labeled
samples.

(2) Effect of graph construction. We compared the result of the model taking
input graphs constructed based on the correlations between nodes. However, the
performance is generally worse than taking the fully-connected graph as the
input, which shows that the node correlations could not effectively describe
the interactions between interfaces. Since the network connectivity is
unknown, the proposed approach assumes that data communication (direct and
indirect) could happen between any interface pairs and uses the graph
attention module to automatically capture their relationships. The entire
results are presented as Table 2 in the supplementary file.

Table 2: F1 scores between using different loss functions (see the full table in the supplementary file) Network ID |  only-CC |  only-KL |  Ours
---|---|---|---
1NTT18927 |  0.809 |  0.717 |  0.823
1NTT19535 |  0.808 |  0.773 |  0.877
1NTT67246 |  0.955 |  0.889 |  0.946
1NTT19478 |  0.786 |  0.782 |  0.808
143XB8CT |  0.462 |  0.466 |  0.502
1NTT30990 |  0.267 |  0.288 |  0.397
1NTT48337 |  0.757 |  0.672 |  0.754
1NTT19213 |  0.422 |  0.413 |  0.573
1NTT22885 |  0.800 |  0.850 |  0.897
1NTT18635 |  0.935 |  0.962 |  0.980

##  4 Related Work

There exists abundant literature on anomaly detection with time-series data.
Deep learning approaches have shown promising results in capturing complex
relationships between time series and across time points for extracting
effective patterns that enable accurate anomaly prediction, including
unsupervised [13, 5], supervised [11, 19], and semi-supervised approaches
[24]. For example, [8] propose to use Long-Short-Term Memory (LSTM) to predict
spacecraft telemetry and detect outliers in an unsupervised manner within each
variable of a single multivariate time-series. However, the approach does not
consider the dependencies between variables. Network operation data are
typically multiple multivariate time-series data, which require careful
considerations to to capture the interactions between variables and time
series for event detection. RNNs and Transformers [1] have been widely used in
modeling temporal dependencies in time-series data, but they typically suffer
from time-consuming training process and large memory requirement [28]. [29]
propose an unsupervised anomaly detection approach on multivariate time-series
data, with each variate/time series represented as one node in a graph. Their
approach utilizes a feature attention layer and a temporal attention layer to
explicitly capture the relationships between features and times respectively.
Then the approach jointly optimizes a forecasting-based model and a
reconstruction-based model to obtain better time-series representations.
However, such unsupervised learning approaches usually make a strong
assumption that infrequent behavior is anomalous and usually requires some
scores for inference (e.g., see [17]). Although there are some techniques,
e.g., Peak Over Threshold (POT) [20], that automatically determine the
threshold during inference, the performance is still not competitive to
supervised learning methods when labels are available (e.g., see [11, 19]).
Supervised approaches can explicitly learn normal or abnormal patterns from
labels yet requires a prohibitive large number of labeled examples (e.g., see
[blazquez2021review]).

Semi-supervised learning approaches can overcome these limitations. For
example, [18] demonstrates that adding a few labeled samples in a semi-
supervised manner would guide the model to achieve better performance than
unsupervised learning methods for anomaly detection. [26] propose a semi-
supervised approach that uses an LSTM auto-encoder on both unlabeled data and
labeled data with a reconstruction loss and adds prediction loss on the
labeled data to guide the learning process. In comparison, our approach
explicitly learns a separable embedding space for normal and abnormal samples
instead of using a simple prediction loss (e.g., cross-entropy loss or focal
loss for the imbalanced label situation). [24] propose a semi-supervised
learning approach that first uses an unsupervised model, a clustering-based
approach, to define typical normal behavior and identify anomalies. Then the
approach actively adds labels from domain experts to guide the clustering
approach in a semi-supervised manner. However, the performance would highly
rely on the initialized clustering result. Our approach leverages the
unlabeled samples to learn cluster centers in a self-learning way.

##  5 Conclusion

The paper presented a semi-supervised learning approach for predicting
abnormal events from multiple multivariate time-series data. The proposed
approach learns separable embeddings by leveraging labeled samples to learn
cluster centers in a contrastive way and enforces unlabeled samples to follow
similar clustering distribution to the labeled samples in a semi-supervised
manner. We plan to explore methods for further detecting the event root cause,
for example, determining the devices or interfaces and their activities
leading to an event. Another direction is to take advantage of other networks
in a transfer learning way when the labeled data is scarce.

##  6 Acknowledgement

This material is based upon work supported in part by the NTT Global Networks
and NVIDIA Corporation.

## References

  * [1] Ling Cai et al.  “Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting”  In _Transactions in GIS_ 24.3, 2020, pp. 736–755
  * [2] Tianle Cai et al.  “Graphnorm: A principled approach to accelerating graph neural network training”  In _ICML_ , 2021, pp. 1204–1215
  * [3] Yann N Dauphin et al.  “Language modeling with gated convolutional networks”  In _ICML_ , 2017, pp. 933–941
  * [4] Michaël Defferrard, Xavier Bresson and Pierre Vandergheynst  “Convolutional neural networks on graphs with fast localized spectral filtering”  In _NeurIPS_ 29, 2016, pp. 3844–3852
  * [5] Ailin Deng and Bryan Hooi  “Graph neural network-based anomaly detection in multivariate time series”  In _AAAI_ 35.5, 2021, pp. 4027–4035
  * [6] Matthias Fey and Jan E. Lenssen  “Fast Graph Representation Learning with PyTorch Geometric”  In _ICLR Workshop on Representation Learning on Graphs and Manifolds_ , 2019
  * [7] Jonas Gehring et al.  “Convolutional sequence to sequence learning”  In _ICML_ , 2017, pp. 1243–1252
  * [8] Kyle Hundman et al.  “Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding”  In _ACM SIGKDD_ , 2018, pp. 387–395
  * [9] Nikolay Laptev, Saeed Amizadeh and Ian Flint  “Generic and scalable framework for automated time-series anomaly detection”  In _ACM SIGKDD_ , 2015, pp. 1939–1947
  * [10] Dan Li et al.  “MAD-GAN: Multivariate anomaly detection for time series data with generative adversarial networks”  In _ICANN_ , 2019, pp. 703–716
  * [11] Dapeng Liu et al.  “Opprentice: Towards practical and automatic anomaly detection through machine learning”  In _IMC_ , 2015, pp. 211–224
  * [12] Pankaj Malhotra et al.  “LSTM-based encoder-decoder for multi-sensor anomaly detection”  In _arXiv preprint arXiv:1607.00148_ , 2016
  * [13] Mohsin Munir et al.  “DeepAnT: A deep learning approach for unsupervised anomaly detection in time series”  In _IEEE Access_ 7 IEEE, 2018, pp. 1991–2005
  * [14] Daehyung Park, Yuuna Hoshi and Charles C Kemp  “A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder”  In _IEEE Robotics and Automation Letters_ 3.3, 2018, pp. 1544–1551
  * [15] Ce Qi and Fei Su  “Contrastive-center loss for deep neural networks”  In _ICIP_ , 2017, pp. 2851–2855
  * [16] Hansheng Ren et al.  “Time-series anomaly detection service at microsoft”  In _ACM SIGKDD_ , 2019, pp. 3009–3017
  * [17] Lukas Ruff et al.  “Deep one-class classification”  In _ICML_ , 2018, pp. 4393–4402
  * [18] Lukas Ruff et al.  “Deep semi-supervised anomaly detection”  In _arXiv preprint arXiv:1906.02694_ , 2019
  * [19] Dominique T Shipmon et al.  “Time series anomaly detection; detection of anomalous drops with limited features and sparse examples in noisy highly periodic data”  In _arXiv preprint arXiv:1708.03665_ , 2017
  * [20] Alban Siffer et al.  “Anomaly detection in streams with extreme value theory”  In _ACM SIGKDD_ , 2017, pp. 1067–1075
  * [21] Ya Su et al.  “Robust anomaly detection for multivariate time series through stochastic recurrent neural network”  In _ACM SIGKDD_ , 2019, pp. 2828–2837
  * [22] Dmitry Ulyanov, Andrea Vedaldi and Victor Lempitsky  “Instance normalization: The missing ingredient for fast stylization”  In _arXiv preprint arXiv:1607.08022_ , 2016
  * [23] Petar Veličković et al.  “Graph attention networks”  In _arXiv preprint arXiv:1710.10903_ , 2017
  * [24] Vincent Vercruyssen et al.  “Semi-supervised anomaly detection with an application to water analytics”  In _ICDM_ 2018, 2018, pp. 527–536
  * [25] Tianlong Wu, Feng Chen and Yun Wan  “Graph attention LSTM network: A new model for traffic flow forecasting”  In _ICISCE_ , 2018, pp. 241–245
  * [26] Hui Xiao et al.  “Semi-supervised Time Series Anomaly Detection Model Based on LSTM Autoencoder”  In _ICBDS_ , 2020, pp. 41–53
  * [27] Junyuan Xie, Ross Girshick and Ali Farhadi  “Unsupervised deep embedding for clustering analysis”  In _ICML_ , 2016, pp. 478–487
  * [28] Bing Yu, Haoteng Yin and Zhanxing Zhu  “Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting”  In _arXiv preprint arXiv:1709.04875_ , 2017
  * [29] Hang Zhao et al.  “Multivariate time-series anomaly detection via graph attention network”  In _ICDM_ , 2020, pp. 841–850

blazquez2021review