# Channel-Spatial-Based Few-Shot Bird Sound Event Detection

Lingwen Liu  Yuxuan Feng  Haitao Fu  Yajie Yang  Xin Pan  and Chenlei Jin
Jilin Agricultural University  Changchun  Jilin  China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a model for bird sound event detection that focuses
on a small number of training samples within the everyday long-tail
distribution. As a result, we investigate bird sound detection using the few-
shot learning paradigm. By integrating channel and spatial attention
mechanisms, improved feature representations can be learned from few-shot
training datasets. We develop a Metric Channel-Spatial Network model by
incorporating a Channel Spatial Squeeze-Excitation block into the prototype
network, combining it with these attention mechanisms. We evaluate the Metric
Channel Spatial Network model on the DCASE 2022 Take5 dataset benchmark,
achieving an F-measure of $66.84\%$ and a PSDS of $58.98\%$. Our experiment
demonstrates that the combination of channel and spatial attention mechanisms
effectively enhances the performance of bird sound classification and
detection.

##  1 Introduction

Birdsong, or the sounds produced by birds, serves as a means of communication
that enables birds to interact with one another. Through their calls, birds
can recognize each other’s species, gender, age, territory, and mate
information. Moreover, these sounds can function as warnings or indicate the
location of food sources. Each bird species has its own unique calls,
distinguishable by their frequency, pitch, duration, and rhythm. Some bird
sounds are exceptionally beautiful, even considered a form of music. Studying
bird sounds allows researchers to gain a better understanding of bird behavior
[1]. Many scientists and bird enthusiasts utilize bird sounds to identify
species, track migration patterns, and examine the ecological environment [2].

Birds are prevalent in various ecosystems, and detecting the status of their
sounds can offer insights into the diversity and abundance of bird species in
a given area. Shifts in the frequency and pitch of bird sounds may signal
threats or harm to the ecosystem, such as pollution, habitat loss, or climate
change. Consequently, bird calls can serve as indicators of ecosystem health.
By monitoring and analyzing bird sounds, we can evaluate the health of an
ecosystem and take appropriate measures to protect and restore it [3]. Birds
help maintain the ecological balance in the natural world, enhance the
environment’s naturalness, and serve as valuable resources for teaching and
research. Monitoring the natural environment through bird sound detection can
directly benefit humans, underscoring the importance of bird sound detection
in ecological monitoring.

Presently, various methods are employed in bird sound event detection tasks
[4, 5], including template matching [6], multiple-instance learning [7, 8, 9],
and prototype networks [10, 11, 12]. Template matching is an algorithm that
utilizes mathematical models such as Gaussian Mixture Models [13] or Hidden
Markov Models [14]. In bird sound event detection tasks, this method employs
fast and normalized cross-correlation to identify templates within
spectrograms. Subsequently, the template matching technique calculates the
cross-correlation between each event in the dataset and the remaining audio
data, setting a distinct threshold for each file based on the maximum value of
the cross-correlation results.

Template matching was the earliest method used for bird sound event detection,
but because it can only use translational matching during matching, the
feature matching of the template matching method is not complete enough. In
addition, multiple-instance learning is also used, assuming that a piece of
audio is a package, and each event in the audio is treated as an instance
without labels. The positive or negative of the event is judged based on the
labels of these packages. If a package is marked as a positive package, there
is at least one positive instance in the package. Conversely, if a package is
marked as a negative package, all instances are negative. The purpose of
multiple-instance learning is to make accurate judgments about new packages by
learning from packages, which can improve the training results of weakly
labeled datasets under the condition of using a large amount of data for
training. However, the natural conditions of the bird sound dataset affect the
dataset size [6, 15], and using multiple-instance learning methods to process
bird sounds-related issues cannot achieve high performance.

On the other hand, a prototype network is a simple and efficient neural
network for few-shot learning [16, 17, 18]. Its core idea is to create a
prototype representation for each category and then classify them through
distance calculation, which can achieve higher generalization performance on
the same dataset. In terms of distance calculation, we chose metric learning
[19]. In metric learning, different data points are mapped to a feature space,
and the distance or similarity between these data points is defined by the
metric function. Common metric learning methods include Euclidean distance,
Cosine similarity, Mahalanobis distance, Manifold, etc. [20]. Metric learning
has wide applications in fields such as image processing, Natural Language
Processing, and Recommendation Systems.

To effectively detect bird sounds in small datasets, we propose a novel
prototype network-based approach [21, 22] and introduce the Channel Spatial SE
module [23] to enhance the dependency between channels and spatial dimensions.
Our design, the Metric Channel-Spatial Network (MCS-Net) model, emphasizes
channel and spatial features during the feature extraction process. We
evaluate the MCS-Net model on the DCASE 2022 Take5 benchmark dataset,
achieving an F-measure of $66.84\%$ and a PSDS of $58.98\%$, which surpasses
previous works. This result demonstrates the MCS-Net model’s effectiveness in
improving bird sound detection performance.

The remaining parts of this paper are organized as follows. In Section 2, we
introduced the system overview and Channel-Spatial SE. In Section 3, we
conducted experiments to demonstrate the performances of the Metric Channel-
Spatial Network (MCS-Net) model. In Section 4, we summarize this paper.

##  2 Method

###  2.1 Input

This paper performs bird sounds detection in the frequency domain using the
audio data from the DCASE 2022 Take5 dataset. Since the length of these audio
files varies, we first need to crop them to a fixed length. Then, We first use
the Short-time Fourier Transform (STFT) to convert the audio from the time
domain signal to the frequency domain signal and then convert it into Mel
Frequency Cepstral Coefficients (MFCC) [24], which is a commonly used feature
in the field of speech and audio processing and is more suitable for
simulating the perceptual characteristics of the human ear. To compress the
dynamic range of MFCC and perform automatic gain control, we further
preprocess MFCC using Per-Channel Energy Normalization (PCEN) [25], which
reduces the difference in foreground loudness and suppresses background noise.
The inputs of our model are processed by MFCC ($F_{\text{MFCC}}(\cdot)$) and
PCEN ($F_{\text{PCEN}}(\cdot)$) respectively, and then concatenated together,
which is represented as follows:

Figure 1:  The model structure consists of PCEN+MFCC module, bottleneck layer,
BC Residual module, and Channel Spatial module. where orange represents the
PCEN+MFCC overlay module, pink represents the prototype network module, green
represents the residuals module, and purple represents the Channel Spatial
module.

$F_{\text{ALL}}(x)=F_{\text{MFCC}}(x)+F_{\text{PCEN}}(x)$ (1)

where $x$ represents the amplitude spectrogram of the audio data to be
detected. We treat $F_{\text{ALL}}$ as the input of the bottleneck layer
($BL$), which is a convolution-based feature extraction block. After
processing $F_{\text{ALL}}$ through three $BL$ blocks, we obtain the initial
feature map $F_{\text{initial}}$ for each audio segment. To further enhance
feature extraction and improve performance, we added an improved Channel
Spatial SE block, which adopts the idea of Squeeze-and-Excitation [26] and
performs “squeeze” and “excitation” from both channel and spatial
perspectives, as shown in the purple shaded area in Figure 1. Before
squeezing, we first calculate a location value for each channel using the
following equation 2, where the value is the statistical number of channel
ranges generated by shrinking u from the spatial dimension

$G_{\text{K}}=\frac{1}{H\times
W}\Sigma_{i}^{H}\Sigma_{j}^{W}{O_{K}}^{\left(i,j\right)}$ (2)
$\widehat{G}=W_{1}\left(\sigma\left({W_{2}}^{Z}\right)\right),W_{1}\in
R^{C\times\frac{C}{P}},W_{2}\in R^{C\times\frac{C}{P}}$ (3)

In Equation 3, $p$ represents the parameter rate, $W_{1}$ and $W_{2}$
represent the weights of the two fully connected layers, and $\widehat{G}$
represents the weighted calculation result of the $i_{th}$ channel. When we
input $F_{\text{initial}}$ into Channel Spatial SE, $F_{\text{initial}}$
enters both the channel and spatial branches. When we input
$F_{\text{initial}}$ from the channel perspective, first, we obtain a unique
feature value $O$ for each channel by global average pooling. Then, we use two
fully connected layers with different weights and activate them with the Leaky
ReLU activation function. Finally, we normalize the values through the sigmoid
layer.

$\displaystyle{\widehat{O}_{C}}^{SE}$
$\displaystyle={\widehat{F}_{C}}^{SE}\left(O\right)$ (4)
$\displaystyle=\left[\sigma\left(\widehat{C}_{1}\right)O_{1},\cdots,\sigma\left(\widehat{C}_{2}\right)O_{2},\cdots,\sigma\left(\widehat{C}_{c}\right)O_{c}\right]$

When we input $F_{\text{initial}}$ from the spatial perspective, we first use
a $1\times 1\times 1$ convolutional kernel to reduce the dimension, then
perform nonlinear operations with the sigmoid function to obtain the output in
the range of [0-1], and finally multiply the output with the spatial dimension
of the initial feature map to obtain $\widehat{O}$. The specific calculation
equation is as follows:

$\displaystyle\widehat{O}$ $\displaystyle={{F_{S}}^{SE}}\left(O\right)$ (5)
$\displaystyle=\left[\sigma\left(q_{1,1}\right)O^{1,1},\cdots,\sigma\left(q_{i,j},\right)O^{i,j},\cdots,\sigma\left(q_{H,W}\right)H,W\right]$

Finally, we add the results of the channel and spatial processing together as
the final output:

${{\widehat{O}_{SC}}^{SE}}={{\widehat{O}_{S}}^{SE}}+{{\widehat{O}_{C}}^{SE}}$
(6)

We optimize the model using the prototype network and metric learning method.
We first calculate a class prototype by embedding several samples into the
latent space, and then use the metric learning equation to calculate the
distance between the final output and the prototype to determine which class
it belongs to. The equation is as follows:

$d_{2j}^{i}=\sqrt{\sum{(x_{i}^{q}-x_{j}^{s})^{2}}},d_{2j+1}^{i}=\sqrt{\sum{(x_{i}^{q}-\widetilde{x}_{j}^{s})^{2}}}$
(7)

Table 1: Comparison with Byte Dance in DCASE 2022 Take5. Pre stands for accuracy, Rec stands for recall, F-measure stands for the balance value of accuracy and recall, and PSDS stands for Polyphonic Sound Detection Score, where Templane Matching and ProtoNet do not have PSDS values, and all metrics in the table are reported as percentages. Method |  Precision (%) |  Recall (%) | F-measure(%) | PSDS(%)
---|---|---|---|---
Templane Matching | 2.42 | 18.32 | 4.28 | N/A
ProtoNet | 36.34 | 24.96 | 29.59 | N/A
DCASE-ByteDance | 69.3 | 57.3 | 62.73 | 57.52
MCS-Net | 68.5 | 66.5 | 66.84 | 58.98

###  2.2 Learning Objective

As this paper is based on a CNN network [27, 28], traditional CNNs treat each
convolutional layer equally, which ignores the importance of channels and
spatial dimensions. Therefore, to increase the attention to channels and
spatial dimensions, we added the Channel Spatial SE module. The core idea of
the Channel Spatial SE module is to compress the feature map of each channel
into a single number (i.e. ”squeeze”), and then use a subnetwork called the
”excitation” module to learn the weight of each channel. This idea is applied
from two different perspectives to dynamically adjust the weight of each
channel and improve the representation ability of the feature map, while also
increasing the dependency between channels and spatial dimensions, thereby
improving the performance of the CNN.

###  2.3 Model Structure

Fig. 1 shows the structure of the MCS-Net model, which consists of 3 BL
modules, 1 PL module, and 2 Channel Spatial SE modules. The BL module consists
of a 2D convolutional layer, batch normalization, and a residual block, using
the Leaky ReLU activation function in the residual block. The PL module
consists of a 1D convolutional layer, batch normalization, and the Leaky ReLU
activation function, with an additional average pooling layer at the end. The
Channel Spatial SE module consists of two branches, one for channels and one
for spatial dimensions. Both branches have the same structure, first using a
$1\times 1\times 1$ convolutional kernel to ”squeeze” the feature
representation into a single number, then weighting the channel and spatial
features, and finally outputting through convolution and merging the features
from both perspectives. Our experiments have demonstrated the importance of
features in both channels and spatial dimensions. We added the Channel Spatial
SE module to the CNN network to enhance the model’s dependency on channels and
spatial dimensions, and improved the accuracy of the model through
recalibration of the feature map during the learning process.

##  3 Experiment

###  3.1 Experimental Data

The experiments in this paper used the public dataset of task 5 of DCASE 2022.
The dataset is pre-split into training and validation sets. The training set
consists of five sub-folders (BV, HV, JD, MT, and WMW) deriving from a
different source each. The number of audio recordings in the training set is
174, corresponding to 21 hours. Along with the audio files, multi-class
annotations are provided for each. The validation set consists of four sub-
folders (HV, PB, ME, and ML) deriving from a different source each, with a
single-class (class of interest) annotation file provided for each audio file.
The number of audio recordings in the validation set is 18 correspoding to
about 6 hours. The training set data $T=(S_{I},Y_{I})|_{i=1}^{\text{Ntrain}}$
contains $S^{i}=\\{S_{i}|Y_{i}=1\\}\cup\\{\widetilde{S_{l}}|Y_{i}=0\\}$ and
$Y_{i}=\\{Y_{i}|Y_{i}\in\\{0,1\\}\\}$, where $\\{S_{i}\\}$ and
$\\{\widetilde{S_{i}}\\}$ are sets of positive and negative segments of class
$i$, respectively. Ntrain is the number of the total training segments. The
evaluation set
$E=(S_{i}^{{}^{\prime}},Y_{i}^{{}^{\prime}})|_{i=1}^{\text{Neval}}$ contains
$S_{i}^{{}^{\prime}}=\\{S_{i}^{{}^{\prime}}\\}$ and the label set
$Y_{i}^{{}^{\prime}}=\\{Y_{i}^{{}^{\prime}}\\}$, where Neval is the number of
classes in the evaluation set. There is no overlap between the training set
and validation set classes, which makes the dataset more challenging.

###  3.2 Implementation Details

The sampling rate of all audio is 22.5kHz. We first use the Short-Time Fourier
transform with a window length of $1024$ and a hop size of $256$ to obtain a
frequency-domain spectrogram. Then, by using Mel-scale filtering with 128 Mel
bins, we have a 128-dimensional mel spectrogram. After that, we continue to
extract the PCEN and MFCC features based on the mel spectrogram. During
training, the model input was a stack of PCEN and MFCC features. If the length
was less than $0.2$ seconds, it was padded with zeros. The initial learning
rate was set to $0.001$, decayed by a factor of $0.65$ every $10$ epochs.
Since there were only three types of bird sounds in the validation set, we
used a 3-way 5-shot method for validation. If the validation accuracy did not
improve for $10$ consecutive epochs, the model would stop training. The model
with the best validation accuracy was used for evaluation. To better utilize
the training data, we added a dynamic data loader to generate training data
with random start times. In addition, we designed a post-processing strategy
for sound categories based on the maximum length of positive events
$t_{max}=\max[t_{1},\cdots,t_{k}]$. If the length of a detected positive event
was less than $\alpha*t_{max}$ or greater than $\beta*t_{max}$, we deleted it.
During evaluation, we used $\beta=2.0$, $\alpha=[0.1,0.2,\cdots,0.9]$, and
threshold values $h=[0.0,0.05,\cdots,0.95]$. We calculated the PSD-ROC curve
and PSDS using different combinations of $\beta$, $\alpha$, and $h$. We
selected the best F-measure among all combinations of $\beta$, $\alpha$, and
$h$ as the final F-measure.

###  3.3 Evaluation Criteria

This paper uses two evaluation metrics. One is the F-measure, which is a
balance between precision and recall. We use the F-measure as the main
evaluation metric. The other metric used is the Polyphonic Sound Detection
Score (PSDS) [29], which evaluates the algorithm’s performance based on
acoustic events. PSDS considers the occurrence, duration, and label of each
acoustic event. Calculating the PSDS metric requires the use of forced
alignment technology to align the events detected by the algorithm with the
true events.

The specific steps for using the PSDS evaluation metric are as follows: First,
preprocess the data, including data preprocessing and feature extraction.
Second, segment the data to obtain information about the start and duration of
each acoustic event. Then, evaluate the algorithm’s detection performance,
which includes event detection, event duration detection, and event label
detection. For each event, if the algorithm detects the event, it is set to
$1$, otherwise $0$. Evaluate whether the algorithm detects the duration of
each acoustic event correctly. For each event, calculate the duration score
based on the algorithm’s detected duration and the true duration. Evaluate
whether the algorithm correctly predicts the label. For each event, calculate
the label score based on the algorithm’s predicted label and the true label.
For each acoustic event, calculate the weighted average of its occurrence
score, duration score, and label score to obtain its PSDS score. Average the
PSDS scores of all acoustic events to obtain the algorithm’s average PSDS
score. Using the PSDS evaluation metric can objectively evaluate the
performance of bioacoustic event detection algorithms and adjust the
evaluation for different types of acoustic events to improve the accuracy and
reliability of the evaluation. In PSDS, the detection tolerance (DTC) is set
to $0.5$, the ground truth (GTC) is set to $0.5$, and the maximum effective
false alarm rate is $100$. Since we do not perform multi-channel detection, we
do not use the cross-triggering tolerance criteria (CTTC).

Figure 2: Polyphonic Sound Detection Score (PSDS) on the dataset of task 5 of
DCASE 2022, where the green color, blue color, and cyan color represent the
results on the PB type, HB type, and ME type of bird sounds, respectively. The
red color corresponds to $\text{mean}_{\text{TPR}}$. The solid line
corresponds to the results of the proposed MCS-Net, and the dashed line
corresponds to the result of the Byte Dance team. TPR and TPR are the true
positive rate and the false positive rate, respectively. The PSDS values are
between zero and one, where better-performing systems achieve higher scores.
Figure 3: MCS-Net model and Byte Dance model validation comparison chart,
where the horizontal axis is the training step and the vertical axis is the
accuracy, the cyan line in the chart represents the MCS-Net model and the
purple line represents the Byte Dance model, the higher the accuracy the
better, from the chart you can clearly see that the cyan line is higher than
the purple line.

###  3.4 Results

The performance of the proposed MCS-Net model on the evaluation set is
reported in Table 1. In the table, we compare our results with previous works,
including the results of the Template Matching method from the DCASE 2021
competition, the ProtoNet baseline model, and the Byte Dance team, who won the
DCASE 2022 Take5 competition with an F-measure score of $62.73\%$ and PSDS of
$57.52\%$. Our proposed MCS-Net model achieved an F-measure score of $66.84\%$
and PSDS of $58.98\%$, outperforming the previous methods.

As shown in Fig. 2, the ROC curve shows that the PB class is the most
difficult to detect, the HB class is the easiest to detect, and the ME class
achieves average performance on the evaluation set. By comparing the two sides
of the curve, it can be seen that the MCS-Net model has significantly improved
the detection of the most difficult PB class.

As shown in Fig. 3, the purple line represents the Byte Dance model, and the
blue line represents our proposed MCS-Net model. The improved MCS-Net model
completed training in 1 hour 19 minutes and 27 seconds with a precision of
97%, which is faster and more accurate than the Byte Dance model with fewer
steps.

##  4 Conclusions

In this paper, we introduce the MCS-Net network model, which employs prototype
network and metric learning techniques in the context of limited data samples.
By incorporating residual blocks and Channel Spatial SE blocks into the
prototype network, the MCS-Net model enhances network stability and focuses on
channels and spatial dimensions, thus improving feature representation
capabilities. The MCS-Net model demonstrates strong learning and
generalization abilities even with small sample data. On the publicly
available DCASE2022-Take5 dataset, the MCS-Net model achieved an F-measure
score of $66.84\%$ and a PSDS of $58.98\%$, representing absolute improvements
of $4.11\%$ and $1.46\%$ in F-measure and PSDS, respectively, compared to
previous works. These results showcase the effectiveness of the MCS-Net model
in enhancing bird sound classification and detection performance.

## References

  * [1] H. Brumm and M. Naguib, “Environmental acoustics and the evolution of bird song,” _Advances in the Study of Behavior_ , vol. 40, pp. 1–33, 2009.
  * [2] J. Bateman and A. Uzal, “The relationship between the acoustic complexity index and avian species richness and diversity: a review,” _Bioacoustics_ , vol. 31, no. 5, pp. 614–627, 2022.
  * [3] D. Stowell, M. Wood, Y. Stylianou, and H. Glotin, “Bird detection in audio: a survey and a challenge,” in _2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP)_. IEEE, 2016, pp. 1–6.
  * [4] A. Mesaros, T. Heittola, T. Virtanen, and M. D. Plumbley, “Sound event detection: A tutorial,” _IEEE Signal Processing Magazine_ , vol. 38, no. 5, pp. 67–83, 2021.
  * [5] D. Yang, H. Wang, Y. Zou, Z. Ye, and W. Wang, “A mutual learning framework for few-shot sound event detection,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 811–815.
  * [6] V. Morfi, I. Nolasco, V. Lostanlen, S. Singh, A. Strandburg-Peshkin, L. F. Gill, H. Pamula, D. Benvent, and D. Stowell, “Few-shot bioacoustic event detection: A new task at the dcase 2021 challenge.” in _DCASE_ , 2021, pp. 145–149.
  * [7] O. Maron and T. Lozano-Pérez, “A framework for multiple-instance learning,” _Advances in neural information processing systems_ , vol. 10, 1997.
  * [8] M.-A. Carbonneau, V. Cheplygina, E. Granger, and G. Gagnon, “Multiple instance learning: A survey of problem characteristics and applications,” _Pattern Recognition_ , vol. 77, pp. 329–353, 2018.
  * [9] F. Briggs, X. Z. Fern, and R. Raich, “Rank-loss support instance machines for miml instance annotation,” in _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2012, pp. 534–542.
  * [10] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [11] Y. Zhang, J. Wang, D. Zhang, and F. Deng, “Few-shot bioacoustic event detection using prototypical network with background classs,” DCASE2021 Challenge, Tech. Rep, Tech. Rep., 2021.
  * [12] M. Anderson and N. Harte, “Bioacoustic event detection with prototypical networks and data augmentation,” _arXiv preprint arXiv:2112.09006_ , 2021\.
  * [13] M.-H. Yang and N. Ahuja, “Gaussian mixture model for human skin color and its applications in image and video databases,” in _Storage and retrieval for image and video databases VII_ , vol. 3656. SPIE, 1998, pp. 458–466.
  * [14] A. Khademi, “Hidden markov models for time series: An introduction using r,” _Journal of Statistical Software_ , vol. 80, pp. 1–4, 2017.
  * [15] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 776–780.
  * [16] S. Ravi and H. Larochelle, “Optimization as a model for few-shot learning,” in _International conference on learning representations_ , 2017.
  * [17] Y. Wang, J. Salamon, N. J. Bryan, and J. P. Bello, “Few-shot sound event detection,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 81–85.
  * [18] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” _ACM computing surveys (csur)_ , vol. 53, no. 3, pp. 1–34, 2020.
  * [19] B. Kulis, “Metric learning: A survey. foundations and trends® in machine learning 5 (4), 287–364 (2013).”
  * [20] P. Rodríguez, I. Laradji, A. Drouin, and A. Lacoste, “Embedding propagation: Smoother manifold for few-shot classification,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16_. Springer, 2020, pp. 121–138.
  * [21] H. Liu, X. Liu, X. Mei, Q. Kong, W. Wang, and M. D. Plumbley, “Segment-level metric learning for few-shot bioacoustic event detection,” _arXiv preprint arXiv:2207.07773_ , 2022.
  * [22] T. Tang, Y. Liang, and Y. Long, “Two improved architectures based on prototype network for few-shot bioacoustic event detection,” _DCASE Challenge_ , 2021\.
  * [23] C. Lan, L. Zhang, Y. Zhang, L. Fu, C. Sun, Y. Han, and M. Zhang, “Attention mechanism combined with residual recurrent neural network for sound event detection and localization,” _EURASIP Journal on Audio, Speech, and Music Processing_ , vol. 2022, no. 1, p. 29, 2022.
  * [24] M. A. Hossan, S. Memon, and M. A. Gregory, “A novel approach for mfcc feature extraction,” in _2010 4th International Conference on Signal Processing and Communication Systems_. IEEE, 2010, pp. 1–5.
  * [25] Y. Wang, P. Getreuer, T. Hughes, R. F. Lyon, and R. A. Saurous, “Trainable frontend for robust and far-field keyword spotting,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 5670–5674.
  * [26] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [27] Y. Gong, Y.-A. Chung, and J. Glass, “Ast: Audio spectrogram transformer,” _arXiv preprint arXiv:2104.01778_ , 2021.
  * [28] X. Yao, X. Wang, S.-H. Wang, and Y.-D. Zhang, “A comprehensive survey on convolutional neural network in medical image analysis,” _Multimedia Tools and Applications_ , pp. 1–45, 2020.
  * [29] Ç. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta, and S. Krstulović, “A framework for the robust evaluation of sound event detection,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 61–65.


# Weakly Supervised Temporal Anomaly Segmentation with Dynamic Time Warping

Dongha Lee1 , Sehun Yu2, Hyunjun Ju2, Hwanjo Yu2
1University of Illinois at Urbana-Champaign (UIUC), Urbana, IL, United States
2Pohang University of Science and Technology (POSTECH), Pohang, South Korea
<EMAIL_ADDRESS><EMAIL_ADDRESS>This work was
done when the author was at POSTECH.Corresponding author.

###### Abstract

Most recent studies on detecting and localizing temporal anomalies have mainly
employed deep neural networks to learn the normal patterns of temporal data in
an unsupervised manner. Unlike them, the goal of our work is to fully utilize
instance-level (or weak) anomaly labels, which only indicate whether any
anomalous events occurred or not in each instance of temporal data. In this
paper, we present WETAS, a novel framework that effectively identifies
anomalous temporal segments (i.e., consecutive time points) in an input
instance. WETAS learns discriminative features from the instance-level labels
so that it infers the sequential order of normal and anomalous segments within
each instance, which can be used as a rough segmentation mask. Based on the
dynamic time warping (DTW) alignment between the input instance and its
segmentation mask, WETAS obtains the result of temporal segmentation, and
simultaneously, it further enhances itself by using the mask as additional
supervision. Our experiments show that WETAS considerably outperforms other
baselines in terms of the localization of temporal anomalies, and also it
provides more informative results than point-level detection methods.

##  1 Introduction

Anomaly detection, which refers to the task of identifying anomalous (or
unusual) patterns in data, has been extensively researched in a wide range of
domains, such as fraud detection [1], network intrusion detection [19], and
medical diagnosis [39]. In particular, detecting the anomaly from temporal
data (e.g., multivariate time-series and videos) has gained much attention in
many real-world applications, for finding out anomalous events that resulted
in unexpected changes of a temporal pattern or context.

Recently, several studies on anomaly detection started to localize and segment
the anomalies within an input instance based on deep neural networks [4, 5,
14], unlike conventional methods which simply classify each input instance as
positive (i.e., anomalous) or negative (i.e., normal). In this work, we aim to
precisely localize the anomalies in temporal data by detecting anomalous
temporal segments, defined as the group of consecutive time points relevant to
anomalous events. Note that labeling every anomalous time point is neither
practical nor precise, similarly to other segmentation problems [7, 22, 32].
In this sense, the main challenge of temporal anomaly segmentation is to
distinguish anomalous time points from normal time points without using point-
level anomaly labels for model training.

Figure 1: Two different strategies for localizing temporal anomalies. Given an
input instance of length $T$, the point-level and segment-level anomaly
detection produce the series of binary labels of length $T$ and length $L$,
respectively.

The most dominant approach to the localization of the temporal anomalies is
point-level anomaly detection based on the reconstruction of input instances
[3, 12, 15, 26, 27, 31, 36, 41]. To learn the point-level (or local) anomaly
score in an unsupervised manner, they focus on modeling the normal patterns
(i.e., temporal context of each time point, usually given in a form of its
past inputs [6]) from normal data, while considering all unlabeled data as
normal. Specifically, they mainly employ variational autoencoder (VAE) to
learn normal latent vectors of temporal inputs, or train sequence models to
predict the next input by using RNNs or CNNs. Then, they compute the anomaly
score for each time point based on the magnitude of error between an actual
input and the reconstructed (or predicted) one.

However, such unsupervised learning methods show limited performance due to
the absence of information about the target anomalies. They are not good at
discovering specific patterns caused by an anomalous event, especially in the
case that such patterns reside in training data. As a solution, we argue that
instance-level labels are often easy to acquire by simply indicating the
occurrence of any anomalous events, whereas acquiring point-level labels is
hard in practice. For example, given a time-series instance collected for a
fixed period of time, a human annotator can easily figure out whether any
anomalous events occurred or not during the period, then generate a binary
label for the instance.

In this paper, we propose a novel deep learning framework, named as WETAS,
which leverages WEak supervision for Temporal Anomaly Segmentation. WETAS
learns from labeled “instances” of temporal data in order to detect anomalous
“segments” within the instance. The segment-level anomaly detection is more
realistic than the point-level detection (Figure 1), because anomalous events
usually result in variable-length anomalous segments in temporal data. The
point-level detection cannot tell whether two nearby points detected as the
anomaly come from a single anomalous event or not, and thus the continuity of
detected points should be investigated for further interpretation of the
results. Our problem setting is similar to the multiple-instance learning
(MIL) [37] which learns from bag-level labels,111The term “multiple-instance”
used in MIL refers to same-length temporal segments that compose a single bag.
Thus, “instance” and “bag” in MIL respectively correspond to “segment” and
“instance” in our approach. but differs in that MIL makes predictions for
static segments of the same length. Table 1 summarizes recent anomaly
detection methods for temporal data.

Methods | Anomaly Prediction | Instance Label
---|---|---
Park et al. [31] | point-level | ✗
Su et al. [36] | point-level | ✗
Xu et al. [41] | point-level | ✗
Sultani et al. [37] |  (static) segment-level | ✓
WETAS (ours) |  (dynamic) segment-level | ✓

Table 1: Comparison of anomaly detection methods that are able to localize the
temporal anomalies in each instance.

WETAS fully exploits the instance-level (or weak) labels for training its
model and also for inferring the sequential anomaly labels (i.e., pseudo-
labeling) that can be used as rough segmentation masks. Based on dynamic time
warping (DTW) alignment [35] between an input instance and its sequential
pseudo-label, WETAS effectively finds variable-length anomalous segments. To
be specific, WETAS optimizes the model to accurately classify an input
instance as its instance-level label, and simultaneously to best align the
instance with its sequential pseudo-label based on the DTW. As the training
progresses, the model generates more accurate pseudo-labels, and this
eventually improves the model itself by the guidance for better alignment
between inputs and their pseudo-labels.

Our extensive experiments on real-world datasets, including multivariate time-
series and surveillance videos, demonstrate that WETAS successfully learns the
normal and anomalous patterns under the weak supervision. Among various types
of baselines, WETAS achieves the best performance in detecting anomalous
points. Furthermore, the qualitative comparison of the detection results shows
that point-level detection methods identify non-consecutive anomalous points
even for a single event, whereas our framework obtains the smoothed detection
results which specify the starting and end points of each event.

##  2 Related Work

###  2.1 Anomaly Detection for Temporal Data

Most recent studies on anomaly detection for temporal data (e.g., multivariate
time-series) utilize deep neural networks to learn the temporal normality (or
regularity) from training data [3, 15, 27]. In particular, VAE-based models
[12, 26, 31, 36, 41] have gained much attention because of their capability to
capture the normal patterns in an unsupervised manner. Their high-level idea
is to compute the point-level anomaly scores by measuring how well a temporal
context for each time point (i.e., a sliding window-based temporal input) can
be reconstructed using the VAE. They are able to localize the temporal
anomalies within an input instance to some extent by the help of this point-
level detection (or alert) approach. However, they do not utilize information
about the anomaly (e.g., anomaly labels) for model training, which makes it
difficult to accurately identify the anomalies. In general, as the anomalies
are not limited to simple outliers or extreme values, modeling the abnormality
as well is helpful to learn useful features for discrimination between normal
and anomalous time points [11, 29].

Several studies make use of anomaly labels by adopting U-Net architectures
[40, 44] which are known to be effective for spatial segmentation [25, 33].
However, their models need to be trained with full supervision, which means
that the model training is guided by anomaly labels for every time point. It
makes the detector impractical because labeling every point in each instance
or obtaining such point-level labels is infeasible or costs too much in
practice.

###  2.2 Weakly Supervised Temporal Segmentation

To address the limitation of fully supervised temporal segmentation, weakly
supervised approaches have been actively researched for video action
segmentation (or detection) tasks [7, 32, 42, 43]. They aim to learn weakly
annotated data whose labels are not given for every time point (frame) in an
instance (video). To this end, dynamic program-based approaches are employed
to search for the best results of segmentation over the time axis under the
weak supervision, including the Viterbi algorithm [32] and dynamic time
warping [7, 42, 43]. Nevertheless, they cannot be directly applied to anomaly
segmentation, because they require instance-level sequential labels being used
as rough segmentation masks, or they impose strong constraints on the
occurrence of target events (actions).222They assume that each action occurs
following a specific transition diagram [42] or occurs at most once in a
single instance [43].

Without the help of sequential labels and constraints, the multiple-instance
learning (MIL) approach [37] showed promising results in detecting anomalous
events by leveraging weakly labeled videos. It considers each video as a bag
after dividing the video into a fixed number of same-length segments, then
trains an anomaly detection model by using both positive (i.e., anomalous) and
negative (i.e., normal) bags. The trained model is able to predict the label
of each video segment as well as a bag, thus the results can be utilized for
anomaly segmentation. However, due to its statically-segmented inputs, there
exists a trade-off between accurate anomaly detection and precise segmentation
with respect to the number of segments in a single bag. The more (and shorter)
segments make it harder to capture the long-term contexts within the segments,
whereas the fewer (and longer) segments generate coarse-grained segmentation
results which could much differ from the actual observation.

##  3 Temporal Anomaly Segmentation

###  3.1 Problem Formulation

The goal of temporal anomaly segmentation is to specify variable-length
anomalous segments (i.e., the starting and end points) within an input
instance of temporal data. Formally, given a $D$-dimensional input instance
$\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{T}]\in\mathbb{R}^{D\times
T}$ of temporal length $T$, we aim to produce segment-level anomaly
predictions
$\hat{\mathbf{y}}=[\hat{y}_{t_{0}+1:t_{1}},\hat{y}_{t_{1}+1:t_{2}},\ldots,\hat{y}_{t_{L-1}+1:t_{L}}]\in\\{0,1\\}^{L}$
where $t_{l}$ denotes the end point of the $l$-th segment (i.e., $t_{0}=0$ and
$t_{L}=T$). This task can be treated similarly to the point-level anomaly
detection from the perspective of its final output $\hat{\mathbf{y}}$, but
differs in that it guarantees a one-to-one mapping between identified segments
and anomalous events.

In this work, we focus on weakly supervised learning, in order to bypass the
challenge of obtaining densely labeled temporal data for model training.
Specifically, a training set consists of $N$ instances333Each instance is
collected for the fixed period of time $T$, or obtained by splitting data
streams into fixed-length temporal data. with their instance-level binary
labels,
$\\{(\mathbf{X}^{(1)},y^{(1)}),(\mathbf{X}^{(2)},y^{(2)}),\ldots,(\mathbf{X}^{(N)},y^{(N)})\\}$.
In this context, the weak supervision means that each label simply indicates
whether any anomalous events are observed or not in the instance. Only with
the weakly annotated dataset, we train the model for temporal anomaly
segmentation, where temporal data are given without any labels at test time.

###  3.2 Dynamic Time Warping (DTW) Alignment for Temporal Anomaly
Segmentation

We first present how to obtain the segmentation result of an input instance by
using a rough segmentation mask. To this end, we define an additional type of
anomaly labels which can serve as the segmentation mask, referred to as
sequential anomaly label.444The details about synthesizing this label are
discussed in Section 3.4. This label indicates the existence (and order) of
normal and anomalous events for each input instance;
$\mathbf{z}=[z_{1},z_{2},\ldots,z_{L}]\in\\{0,1\\}^{L}$ where $L$ is the
length of the sequential label. For example, $\mathbf{z}=[0,1,0,1,0]$ in case
of $L=5$ means that two anomalous events are observed within the instance.
Note that the sequential label $\mathbf{z}$ itself cannot be the final output
of our framework, because it does not contain the information about the
starting and end points of each segment.

For temporal anomaly segmentation, we utilize dynamic time warping (DTW) [35]
which outputs the optimal alignment between a target instance and the
sequential anomaly label. The goal of DTW is to find the optimal alignment
(i.e., point-to-point matching) between $\mathbf{z}$ and $\mathbf{X}$, which
minimizes their total alignment cost with the time consistency among the
aligned point pairs. To be precise, the total alignment cost is defined by the
inner product of the cost matrix
$\Delta(\mathbf{z},\mathbf{X})\in\mathbb{R}^{L\times T}$ and the binary
alignment matrix $A\in\\{0,1\\}^{L\times T}$. Each entry of the cost matrix,
$[\Delta(\mathbf{z},\mathbf{X})]_{lt}:=\delta(z_{l},\mathbf{x}_{t})$, encodes
the cost for aligning $\mathbf{x}_{t}$ with $z_{l}$ (i.e., the penalty for
labeling $\mathbf{x}_{t}$ as $z_{l}$), and similarly, the entry of $A$
indicates the alignment between $\mathbf{z}$ and $\mathbf{X}$; that is,
$A_{lt}=1$ if $\mathbf{x}_{t}$ is aligned with $z_{l}$ and $A_{lt}=0$
otherwise. The optimal alignment matrix $A^{*}$ is obtained by

$\small A^{*}=\operatorname*{argmin}_{A\in\mathcal{A}}\ \langle
A,\Delta(\mathbf{z},\mathbf{X})\rangle,$ (1)

where $\mathcal{A}$ is the set of possible binary alignment matrices.

Figure 2: The overall framework of WETAS, optimized by both the classification
loss $\mathcal{L}_{c}$ and alignment loss $\mathcal{L}_{a}$. The red arrows in
the cost matrix $\Delta(\widetilde{\mathbf{z}},\mathbf{X})$ are the desired
alignment between a temporal input (column) and its sequential label (row)
with time consistencies. Based on the optimal alignment, WETAS can produce the
segment-level anomaly predictions.

In order to align the two series ($\mathbf{z}$ and $\mathbf{X}$) in a temporal
order, each alignment matrix is enforced to represent a single path on a
$L\times T$ matrix that connects the upper-left $(1,1)$-th entry to the lower-
right $(L,T)$-th entry using $\rightarrow,\searrow$ moves. Based on the
obtained alignment $A^{*}$, the starting and end points of the $l$-th temporal
segment and its anomaly label are determined by
$\hat{y}_{t_{l-1}+1:t_{l}}=z_{l}$ such that $A^{*}_{lt}=1$ for all
$t_{l-1}<t\leq t_{l}$. For example, in Figure 2 (Left), the red arrows
represent non-zero entries in the optimal alignment matrix, and we can easily
identify the segments whose segment-label is 1 (i.e., $z_{l}=1$) as the
anomaly. Unlike the conventional DTW, $\downarrow$ moves are not available in
our warping paths because each time point should be aligned with only a single
label.

The two key challenges are raised here as follows. 1) We need to carefully
model the cost matrix based on the anomaly score so that it can effectively
capture the normality and abnormality at each point, while considering the
long-term dependency. In addition, 2) we need to obtain sequential labels used
for the DTW alignment by distilling the information from given instance-level
labels.

###  3.3 Anomaly Score Modeling for Cost Matrix

To model the cost function $\delta(z_{l},\mathbf{x}_{t})$ that is required to
define the cost matrix $\Delta(\mathbf{z},\mathbf{X})$, we parameterize the
anomaly score at time $t$ (i.e., the probability that $\mathbf{x}_{t}$ comes
from an anomalous event) by using deep neural networks. Among various types of
autoregressive networks designed for temporal data, we employ the basic
architecture of dilated CNN (DiCNN) introduced by WaveNet [28]. As illustrated
in Figure 2 (Right), DiCNN is basically stacks of dilated causal convolutions,
which is the convolution operation not considering future inputs when
computing the output at each point. Specifically, the $n$-th layer has a
convolution with filter size $k$ and dilation rate $k^{n-1}$, thus the output
vector from layer $n$ at time $t$ (denoted by $\mathbf{h}_{t}$) depends on its
previous $k^{n}$ points in the end; i.e.,
$[\mathbf{x}_{t-k^{n}+1},\ldots,\mathbf{x}_{t}]$ becomes its receptive field.

Using the inner product of the output vector at each point
$\mathbf{h}_{t}\in\mathbb{R}^{d}$ and the anomaly weight vector
$\mathbf{w}\in\mathbb{R}^{d}$, whose parameters are trainable, we compute the
series of local anomaly scores $\mathbf{s}\in\mathbb{R}^{T}$ by

$\small\mathbf{s}=[s_{1},s_{2},\ldots,s_{T}],\text{\ where\ \
}s_{t}=\sigma(\mathbf{w}{}^{\top}\mathbf{h}_{t}).$ (2)

We apply the sigmoid function $\sigma$ to make the score range in $[0,1]$.
Finally, we define the cost $\delta(z_{l},\mathbf{x}_{t})$ by the negative
log-posterior probability using the anomaly score $s_{t}$:

$\small\begin{split}\delta(z_{l},\mathbf{x}_{t})&=-\log
p\left(z_{l}|\mathbf{x}_{t}\right)\\\ &=-\left\\{z_{l}\cdot\log
s_{t}+(1-z_{l})\cdot\log(1-s_{t})\right\\}.\end{split}$ (3)

The cost becomes large when the computed score $s_{t}$ is far from a target
label $z_{l}$. In this sense, DTW can identify the anomalous segments by
finding the optimal point-to-point matching between the series of anomaly
scores and its sequential label, which minimizes the total alignment cost.

###  3.4 Learning with Weak Supervision

Our proposed framework, termed as WETAS, optimizes the DiCNN model by
leveraging only weak supervision for the anomaly segmentation. To fully
utilize the given instance-level labels, two different types of losses are
considered: 1) the classification loss for correctly classifying an input
instance as its instance-level anomaly label, and 2) the alignment loss for
matching the input instance well with the sequential label, which is
synthesized by the model by distilling the instance-level label.

Learning from Instance-level Label.  Similar to the local (or point-level)
anomaly score in Equation (2), we define the global (or instance-level)
anomaly score for its binary classification. The global anomaly score
$s_{*}\in\mathbb{R}$ is computed by using the anomaly weight
$\mathbf{w}\in\mathbb{R}^{d}$ and the global output vector
$\mathbf{h}_{*}\in\mathbb{R}^{d}$, obtained by global max (or average) pooling
on all the output vectors along the time axis,

$\small\begin{split}\mathbf{h}_{*}&=\text{global-pooling}\
(\mathbf{h}_{1},\mathbf{h}_{2},\ldots,\mathbf{h}_{T})\\\
s_{*}&=\sigma(\mathbf{w}{}^{\top}\mathbf{h}_{*}).\end{split}$ (4)

To optimize our model based on the instance-level labels, we define the
classification loss by the binary cross entropy between $y$ and $s_{*}$.
Thereby, we can predict the anomaly label for an input instance based on its
global anomaly score:

$\small\mathcal{L}_{c}=-\sum_{i=1}^{N}\left\\{y^{(i)}\cdot\log
s_{*}^{(i)}+(1-y^{(i)})\cdot\log(1-s_{*}^{(i)})\right\\}.$ (5)

The classification loss optimizes the model to differentiate between normal
patterns and anomalous patterns that are commonly observed in weakly labeled
temporal data.

Learning from Sequential Pseudo-Label.  To begin with, we propose the
technique to infer the sequential anomaly label (i.e., pseudo-label
generation) of an input instance, then utilize the generated sequential
pseudo-label to further improve our DiCNN model so that it can compute more
accurate local anomaly scores. The pseudo-labeling technique is motivated by
temporal class activation map (CAM) [21, 38] which helps to analyze temporal
regions that most influence the instance-level classification. We obtain the
anomaly activation map
$\mathbf{m}=[m_{1},m_{2},\ldots,m_{T}]\in\mathbb{R}^{T}$ by multiplicating
each output vector $\mathbf{h}_{t}$ with the weight vector $\mathbf{w}$ that
is used for the instance-level anomaly classification. That is, $m_{t}$
becomes proportional to how strongly the time point $t$ contributes to
classifying the input instance as the anomaly. For the condition $0\leq
m_{t}\leq 1$, their values are min-max normalized along the time axis within
each instance,

$\small
m_{t}=\frac{\mathbf{w}^{\top}\mathbf{h}_{t}-\min_{t^{\prime}}\mathbf{w}^{\top}\mathbf{h}_{t^{\prime}}}{\max_{t^{\prime}}\mathbf{w}^{\top}\mathbf{h}_{t^{\prime}}-\min_{t^{\prime}}\mathbf{w}^{\top}\mathbf{h}_{t^{\prime}}}.$
(6)

Then, we introduce a pseudo-labeling function
$\bm{\phi}_{L}:\mathbb{R}^{T}\rightarrow\\{0,1\\}^{L}$ which partitions a
whole anomaly activation map into $L$ segments (or disjoint intervals) of the
same length $\lceil T/L\rceil$ and generates a binary label for each segment,
i.e., $\widetilde{\mathbf{z}}=\bm{\phi}_{L}(\mathbf{m})\in\\{0,1\\}^{L}$. The
pseudo-label $\widetilde{z}_{l}$ is determined by whether the maximum
activation value in the $l$-th segment is larger than the anomaly threshold
$\tau$ or not:

$\small\widetilde{z}_{l}=\mathbb{I}\left[\left(\max_{(l-1)\cdot\lceil
T/L\rceil\leq t\leq l\cdot\lceil T/L\rceil}m_{t}\right)\geq\tau\right].$ (7)

This sequential pseudo-label $\widetilde{\mathbf{z}}$ offers information on
how many anomalous segments do exist and also their relative locations. Note
that it is used as the rough segmentation mask for the DTW alignment,
explained in Section 3.2.

The alignment loss is designed to reduce the DTW discrepancy between an input
$\mathbf{X}$ and its sequential pseudo-label $\widetilde{\mathbf{z}}$ for
their better alignment. However, the gradient of DTW is not very well defined
(i.e., not differentiable) with respect to the cost matrix, due to its
discontinuous hard-min operation taking only the minimum value. Thus, we
instead adopt the soft-DTW [8] that provides the continuous relaxation of DTW
by combining it with global alignment kernels [9]. The soft-DTW utilizes the
soft-min operation,

$\small\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})=\text{min}_{\gamma}\left\\{\langle
A,\Delta(\widetilde{\mathbf{z}},\mathbf{X})\rangle,\forall
A\in\mathcal{A}\right\\}$ (8)

where $\text{min}_{\gamma}{}$ with a smoothing parameter $\gamma$ is defined
by

$\small\text{min}_{\gamma}\\{a_{1},a_{2},\ldots,a_{n}\\}=\begin{cases}\;\min_{i\leq
n}a_{i},&\gamma=0\\\
\;-\gamma\log\sum_{i=1}^{n}e^{-a_{i}/\gamma},&\gamma>0.\end{cases}$

The soft-DTW distance in Equation (8) can be obtained by solving a dynamic
program based on Bellman’s recursions. Please refer to [8, 21] for its forward
and backward recursions to compute
$\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$ and
$\nabla_{\Delta(\widetilde{\mathbf{z}},\mathbf{X})}\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$.

We remark that only minimizing
$\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$ causes the problem of
degenerated alignment [7] that assigns each label to a single time point. To
obtain more precise boundaries of segmentation results, we use the
discriminative modeling under the supervision of the obtained pseudo-label. In
detail, we optimize the model so that the alignment with a positive pseudo-
label $\widetilde{\mathbf{z}}_{+}$ costs less than that with a negative
pseudo-label $\widetilde{\mathbf{z}}_{-}$ by a margin $\beta$ as follows.

$\small\mathcal{L}_{a}=\sum_{i=1}^{N}\left[\frac{1}{T}\text{DTW}_{\gamma}(\widetilde{\mathbf{z}}_{+}^{(i)},\mathbf{X}^{(i)})-\frac{1}{T}\text{DTW}_{\gamma}(\widetilde{\mathbf{z}}_{-}^{(i)},\mathbf{X}^{(i)})+\beta\right]_{+},$
(9)

where $[z]_{+}=\max(z,0)$. The positive and negative pseudo-labels are
respectively obtained by
$\widetilde{\mathbf{z}}_{+}=y\cdot\bm{\phi}_{L}(\mathbf{m})$ and
$\widetilde{\mathbf{z}}_{-}=(1-y)\cdot\bm{\phi}_{L}(\mathbf{m})$, thereby the
instance-level label $y$ plays the role of a binary mask. In other words, we
regard the sequential pseudo-label corrupted by the wrong instance-level label
(i.e., $1-y$) as negative. This loss is helpful to produce more accurate local
anomaly scores by making them align better with its sequential pseudo-label.
Consequently, the model is capable of exploiting richer supervision than only
using the instance-level labels.

###  3.5 Optimization and Inference

The final loss is described by the sum of the two losses,
$\mathcal{L}=\mathcal{L}_{c}+\mathcal{L}_{a}$, where we can control the
importance of the alignment loss by adjusting the margin size $\beta$. Note
that the anomaly weight vector $\mathbf{w}$ and the network parameters in the
DiCNN model are effectively optimized by minimizing the final loss, whereas
$L$, $\tau$, and $\beta$ are the hyperparameters of WETAS. Figure 2 shows the
overall framework of WETAS.

The segmentation result for a test instance is obtained by the DTW alignment
with its sequential pseudo-label
$\widetilde{\mathbf{z}}=\hat{y}_{*}\cdot\bm{\phi}_{L}(\mathbf{m})$, where
$\hat{y}_{*}=\mathbb{I}[s_{*}\geq\tau_{*}]$. Since the instance-level label is
not given for the test input, we instead impose the predicted label
$\hat{y}_{*}$ as a binary mask; that is, we filter out normal instances based
on the global anomaly score $s_{*}$. Once the model is trained, $\tau_{*}$ is
automatically determined by its optimal value that achieves the best instance-
level classification performance on the validation set.

##  4 Experiments

###  4.1 Experimental Settings

Dataset.  For our experiments, we use four real-world temporal datasets
collected from a range of tasks for detecting anomalous events, including
multivariate time-series (MTS) and surveillance videos (Table 2). We split the
set of all the instances by 5:2:3 ratio into a training set, a validation set,
and a test set. Note that only the instance-level labels are given for the
training and validation set.

  * •

Electromyography
Dataset555http://archive.ics.uci.edu/ml/datasets/EMG+data+for+gestures (EMG)
[24]: The 8-channel myographic signals recorded by bracelets worn on subjects’
forearms. Among several types of gestures, ulnar deviation is considered as
anomalous events. Each time-series instance contains the signals for 5
seconds, which are downsampled to 500 points.

  * •

Gasoil Plant Heating Loop Dataset666https://kas.pr/ics-research/dataset_ghl_1
(GHL) [10]: The control sequences of a gasoil plant heating loop, which
suffered cyber-attacks. As done in [40], we crop 10 different instances of
length 50,000 from each time-series, then downsample each of them to 1,000
points.

  * •

Server Machine Dataset777https://github.com/smallcowbaby/OmniAnomaly (SMD)
[36]: The 38-metric multivariate time-series from server machines over 5
weeks, collected from an internet company. We split them every 720 points
(i.e., 12 hours) to build the set of time-series instances.

  * •

Subway Exit Dataset (Subway) [2]: The surveillance video for a subway exit
gate, where each anomalous event corresponds to the passenger walking towards
a wrong direction. We extract the visual features of each frame by using the
pre-trained ResNet-34 [13], and make a single video instance include 450
frames.

Baselines.  We compare the performance of WETAS with that of anomaly detection
methods for temporal data, which are based on three different approaches.

  * •

Unsupervised learning: VAE-based methods that compute the point-level anomaly
scores from the reconstruction of temporal inputs — Donut [41], LSTM-VAE [31],
LSTM-NDT [15], and OmniAnomaly [36].

  * •

Semi-supervised learning: The variants of the unsupervised methods, whose
models are trained by using only normal instances in order to make them
utilize given instance-level labels888These methods are categorized as semi-
supervised learning [6] in that they only utilize densely-labeled (normal)
points from normal instances. — Donut++, LSTM-VAE++, LSTM-NDT++, and
OmniAnomaly++.

  * •

Weakly supervised learning: The multiple-instance learning method that can
produce the anomaly prediction for each fixed-length segment — DeepMIL [37].

For fair comparisons, DeepMIL employs the same model architecture with WETAS
(i.e., DiCNN). Moreover, we consider different numbers of the segments in a
single instance, denoted by DeepMIL-4, 8, 16. In case of the video dataset
(i.e., Subway), the additional comparison with other frame-level video anomaly
detection methods [16, 18, 23, 30] is presented in the supplementary material.

Table 2: Statistics of real-world temporal datasets.

Dataset | EMG | GHL | SMD | Subway
---|---|---|---|---
Data Type | MTS | MTS | MTS | Video
#Variables | 8 | 19 | 38 | 1024
#Points (Train) | 211,892 | 240,000 | 354,200 | 32,051
#Points (Valid) | 84,734 | 96,000 | 141,670 | 13,050
#Points (Test) | 127,199 | 144,000 | 212,550 | 19,800
Anomaly Ratio (%) | 5.97 | 0.49 | 4.16 | 2.60
Instance Length | 500 | 1,000 | 720 | 450

Evaluation Metrics.  For quantitative evaluation of the anomaly detection (and
segmentation) results, we measure the F1-score (denoted by F1) and the
intersection over union (denoted by IoU) by using the point-level ground
truth: F1 =
$\frac{2\times\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}$
where Precision = $\frac{TP}{TP+FP}$ and Recall = $\frac{TP}{TP+FN}$, and IoU
= $\frac{TP}{TP+FP+FN}$. Several previous work [31, 37, 41] mainly reported
the area under the ROC curve (AUROC) as their evaluation metric, but it is
known to produce misleading results especially for severely imbalanced
classification with few samples of the minority class [34], such as anomaly
detection tasks.

To compute the precision and recall, the weakly supervised methods (i.e.,
DeepMIL and WETAS) can find the optimal anomaly threshold (applied to
instance-level or segment-level scores) by utilizing the instance-level labels
from the validation set. However, in cases of the unsupervised and semi-
supervised methods, they require point-level anomaly labels to tune the
anomaly threshold (applied to point-level scores), thus we report their F1 and
IoU by using the anomaly threshold that yields the best F1 among all possible
thresholds (denoted by F1-best and IoU-best, respectively). This can be
interpreted as a measure of the discrimination power between normal and
anomalous points. For some of the baselines (i.e., LSTM-NDT and OmniAnomaly)
that have their own techniques to automatically determine the anomaly
threshold, we additionally report their F1 and IoU by using the selected
threshold.

Table 3: Performances of WETAS and other baselines in detecting anomalous
points from real-world temporal data. All the results support statistically
significant improvement of WETAS over the best baseline ($p\leq 0.05$ from the
paired $t$-test).

Methods | EMG | GHL
---|---|---
F1 | IoU | F1-best | IoU-best | F1 | IoU | F1-best | IoU-best
Donut [41] | - | - | 0.1748(0.002) | 0.0958(0.001) | - | - | 0.0363(0.013) | 0.0185(0.007)
LSTM-VAE [31] | - | - | 0.1728(0.000) | 0.0946(0.000) | - | - | 0.0746(0.015) | 0.0388(0.008)
LSTM-NDT [15] | 0.1317(0.016) | 0.0705(0.009) | 0.1880(0.010) | 0.1199(0.001) | 0.0640(0.030) | 0.0332(0.016) | 0.1025(0.023) | 0.0541(0.013)
OmniAnomaly [36] | 0.1574(0.003) | 0.0854(0.002) | 0.1793(0.001) | 0.0985(0.001) | 0.0611(0.033) | 0.0318(0.018) | 0.0743(0.026) | 0.0387(0.014)
Donut++ [41] | - | - | 0.1784(0.001) | 0.0980(0.001) | - | - | 0.0850(0.036) | 0.0447(0.019)
LSTM-VAE++ [31] | - | - | 0.1745(0.000) | 0.0956(0.000) | - | - | 0.0828(0.023) | 0.0433(0.013)
LSTM-NDT++ [15] | 0.1344(0.005) | 0.0720(0.003) | 0.1916(0.011) | 0.1201(0.002) | 0.0984(0.091) | 0.0538(0.053) | 0.1043(0.106) | 0.0578(0.064)
OmniAnomaly++ [36] | 0.1536(0.002) | 0.0832(0.001) | 0.1807(0.002) | 0.0993(0.001) | 0.1209(0.101) | 0.0668(0.058) | 0.2096(0.143) | 0.1231(0.094)
DeepMIL-4 [37] | 0.4699(0.022) | 0.3073(0.019) | - | - | 0.0690(0.035) | 0.0359(0.098) | - | -
DeepMIL-8 [37] | 0.4317(0.029) | 0.2755(0.023) | - | - | 0.0571(0.023) | 0.0323(0.015) | - | -
DeepMIL-16 [37] | 0.3182(0.056) | 0.1902(0.039) | - | - | 0.1497(0.040) | 0.0813(0.023) | - | -
WETAS (ours) | 0.5803(0.068) | 0.4118(0.064) | - | - | 0.2295(0.028) | 0.1298(0.018) | - | -
Methods | SMD | Subway
F1 | IoU | F1-best | IoU-best | F1 | IoU | F1-best | IoU-best
Donut [41] | - | - | 0.3206(0.011) | 0.1909(0.008) | - | - | 0.5080(0.018) | 0.3406(0.016)
LSTM-VAE [31] | - | - | 0.2671(0.018) | 0.1542(0.012) | - | - | 0.5329(0.024) | 0.3635(0.022)
LSTM-NDT [15] | 0.1145(0.018) | 0.0608(0.010) | 0.3588(0.019) | 0.2187(0.014) | 0.0000(0.000) | 0.0000(0.000) | 0.5658(0.005) | 0.3945(0.005)
OmniAnomaly [36] | 0.1176(0.002) | 0.0625(0.001) | 0.1223(0.006) | 0.0651(0.004) | 0.5367(0.043) | 0.3704(0.038) | 0.6065(0.020) | 0.4355(0.020)
Donut++ [41] | - | - | 0.2875(0.067) | 0.1693(0.047) | - | - | 0.5100(0.026) | 0.3449(0.021)
LSTM-VAE++ [31] | - | - | 0.2477(0.015) | 0.1414(0.010) | - | - | 0.5452(0.016) | 0.3749(0.016)
LSTM-NDT++ [15] | 0.1211(0.010) | 0.0645(0.006) | 0.3819(0.020) | 0.2361(0.015) | 0.0000(0.000) | 0.0000(0.000) | 0.5723(0.004) | 0.4009(0.004)
OmniAnomaly++ [36] | 0.1435(0.081) | 0.0790(0.049) | 0.1750(0.077) | 0.0974(0.047) | 0.5479(0.028) | 0.3790(0.026) | 0.6198(0.023) | 0.4494(0.024)
DeepMIL-4 [37] | 0.3561(0.052) | 0.2176(0.038) | - | - | 0.5138(0.081) | 0.3738(0.073) | - | -
DeepMIL-8 [37] | 0.3450(0.032) | 0.2088(0.023) | - | - | 0.6471(0.066) | 0.4885(0.064) | - | -
DeepMIL-16 [37] | 0.3568(0.016) | 0.2173(0.012) | - | - | 0.6102(0.077) | 0.4391(0.072) | - | -
WETAS (ours) | 0.4358(0.046) | 0.2795(0.037) | - | - | 0.7414(0.023) | 0.5907(0.028) | - | -

Table 4: F1-scores of WETAS that ablates each component, Dataset: SMD. C and R
denote the “corruption” and “random sampling” for negative pseudo-labels,
respectively.

Model Arch. | $\mathcal{L}_{c}$ | $\mathcal{L}_{a}$ |  DTW Seg. |  Global Pool. |  Neg. Label |  F1
---|---|---|---|---|---|---
LSTM | ✓ | ✓ |  ✓ |  MAX |  C |  0.3685
DiCNN | ✓ | - |  - |  MAX |  C |  0.1225
| ✓ | ✓ |  - |  MAX |  C |  0.1265
| ✓ | - |  ✓ |  MAX |  C |  0.3384
| ✓ | ✓ |  ✓ |  AVG |  C |  0.3046
| ✓ | ✓ |  ✓ |  MAX |  R |  0.4272
| ✓ | ✓ |  ✓ |  MAX |  C |  0.4358

Implementation Details.  We implement our WETAS and all the baselines using
PyTorch, and train them with the Adam optimizer [20]. For the unsupervised
methods, we tune their hyperparameters in the ranges suggested by the previous
work [36] that considered the same baselines. In case of VAE-based methods, we
set the size of a temporal context for each point (i.e., sliding window) to
128 (for MTS) and 16 (for video). For DiCNN-based methods, we stack 7 (for
MTS) and 4 (for video) layers of dilated convolutions with filter size 2 to
keep the size of its receptive field (=$2^{7}$, $2^{4}$) the same with the
others’. The dimensionality $d$ of hidden (and output) vectors in DiCNN and
the smoothing factor $\gamma$ of soft-DTW are set to 128 and 0.01,
respectively. We provide the in-depth sensitivity analysis on the
hyperparameters (i.e., $L$, $\tau$, and $\beta$) in our supplementary
material.

###  4.2 Experimental Results

Performances on Anomaly Detection.  We first measure the detection performance
of WETAS and the other baselines. In this experiment, we do not consider the
point-adjust approach [36, 41] for our evaluation strategy: if any point in a
ground truth anomalous segment is detected as the anomaly, all points in the
segment are considered to be correctly detected as the anomalies.999This
approach skews the results by excessively increasing the true positives, i.e.,
the F1 and IoU are overestimated. We repeat to train each model five times
using different random seeds, and report the averaged results with their
standard deviations.

Figure 3: Anomaly detection results of WETAS and other baselines on 4
consecutive instances, Dataset: SMD. WETAS correctly detects anomalous points
in terms of the number of ground truth anomalous segments and their
boundaries.  Figure 4: Test instances (upper) and the local anomaly scores
computed by WETAS every 10 epoch (lower), Dataset: SMD. The ground truth and
identified anomalous segments are colored in yellow and red, respectively.

In Table 3, the unsupervised and semi-supervised methods show considerably
worse performance than the weakly supervised methods in terms of both F1 and
IoU. Even considering F1-best and IoU-best, their performance is at most
comparable to that of the weakly supervised methods. In other words, though
they additionally use the point-level labels for finding the best anomaly
threshold that clearly distinguishes anomalous points from normal ones, they
cannot achieve the detection performance as high as the weakly supervised
methods due to the lack of supervision.

Compared to the unsupervised methods that use all training instances
regardless of their instance-level labels, the semi-supervised methods that
selectively use them (i.e., only normal instances) sometimes perform better
and sometimes worse. That is, learning the normality of temporal data from
anomalous instances can degrade the detection performance due to anomalous
patterns that reside in the instances; but in some cases, the increasing
number of training data points is rather helpful to improve the generalization
power despite the existence of anomalous points. This strongly indicates that
how to utilize the instance-level labels does matter and largely affects the
final performance.

Our WETAS achieves the best performance for all the datasets, and notably, it
significantly beats DeepMIL whose performance highly depends on the number of
segments. For the EMG dataset whose instance length is only 500, DeepMIL-16 is
not able to effectively detect anomalies because each input segment is too
short to accurately compute the segment-level anomaly score; for the GHL
dataset, DeepMIL-4 suffers from coarse-grained segmentation, which leads to
the limited F1 and IoU. On the contrary, WETAS successfully finds anomalous
segments by using dynamic alignment with the sequential pseudo-label.

Ablation Study.  To further investigate the effectiveness of WETAS, we measure
the F1-score of several variants that ablate each of the following components:
1) DiCNN (vs. LSTM) for capturing the temporal dependency, 2) DTW-based
alignment loss, 3) DTW-based segmentation, 4) global max (vs. average) pooling
for the classification loss, and 5) corruption by the instance-level label
(vs. random sampling) for the negative pseudo-label
$\widetilde{\mathbf{z}}_{-}$ in Equation (9). In case of WETAS without the
DTW-based segmentation, we measure both the F1101010For this measure, we set
the anomaly threshold (applied to our point-level anomaly scores) to $\tau$
that is used for pseudo-labeling in Equation (7). and F1-best using the series
of local (or point-level) anomaly scores.

Table 4 summarizes the results of our ablation study on the SMD dataset. It is
worth noting that the DTW-based segmentation considerably improves the
performance compared to the case simply using the obtained local anomaly
scores. This is because the alignment with the pseudo-label filters out normal
instances based on the global anomaly score, and also considers the continuity
of anomalous points while limiting the number of anomalous segments.
Nevertheless, their F1-best scores (for Rows 2 and 3) are respectively 0.4472
and 0.4590, which means that WETAS outperforms other point-level detection
methods even without the DTW-based segmentation. In addition, it is obvious
that DiCNN is more effective to capture the long-term dependency of temporal
data than LSTM. The global max pooling and the corrupted pseudo-labels also
turn out to help the DiCNN model to learn useful features from the instance-
level labels, compared to their alternatives.

Qualitative Analyses.  We also qualitatively compare the anomaly detection
results for test instances. Figures 3 and 4 present the results on consecutive
instances for the SMD dataset. We plot only the two series of input variables
that are directly related to observed anomalous events.

Compared to the baselines, WETAS more accurately finds anomalous segments in
terms of their number as well as the boundaries. To be specific, DeepMIL-4
with fewer segments correctly identifies the segments that include anomalous
events, but the boundaries are far from the ground truth due to their fixed
length and location. In contrast, DeepMIL-16 with more segments makes
unreliable predictions, because each input segment is not long enough that the
normality and abnormality of the segment are effectively captured. The point-
level detection method, LSTM-VAE++, finds out non-consecutive anomalous
points, which results in a large number of separate segments. It is obvious
that such discontinuous prediction makes it difficult to investigate the
occurrence of anomalous events.

##  5 Conclusion

This paper proposes a weakly supervised learning framework for temporal
anomaly segmentation, which effectively finds anomalous segments by leveraging
instance-level anomaly labels for model training. For each input instance of
temporal data, the learning objective of WETAS includes its accurate
classification as the instance-level label and also strong alignment with the
sequential pseudo-label. In the end, WETAS utilizes the DTW alignment between
an input instance and its pseudo-label to obtain the temporal segmentation
result. Our empirical evaluation shows that WETAS outperforms all baselines in
detecting temporal anomalies as well as specifying precise segments that
correspond to anomalous events.

Acknowledgement.  This work was supported by the NRF grant (No.
2020R1A2B5B03097210) and the IITP grant (No. 2018-0-00584, 2019-0-01906)
funded by the MSIT.

## References

  * [1] Aisha Abdallah, Mohd Aizaini Maarof, and Anazida Zainal.  Fraud detection system: A survey.  Journal of Network and Computer Applications, 68:90–113, 2016.
  * [2] Amit Adam, Ehud Rivlin, Ilan Shimshoni, and Daviv Reinitz.  Robust real-time unusual event detection using multiple fixed-location monitors.  TPAMI, 30(3):555–560, 2008.
  * [3] Subutai Ahmad, Alexander Lavin, Scott Purdy, and Zuha Agha.  Unsupervised real-time anomaly detection for streaming data.  Neurocomputing, 262:134 – 147, 2017.
  * [4] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger.  Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection.  In CVPR, pages 9592–9600, 2019.
  * [5] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger.  Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings.  In CVPR, pages 4183–4192, 2020.
  * [6] Varun Chandola, Arindam Banerjee, and Vipin Kumar.  Anomaly detection: A survey.  ACM computing surveys, 41(3):1–58, 2009.
  * [7] Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles.  D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation.  In CVPR, pages 3546–3555, 2019.
  * [8] Marco Cuturi and Mathieu Blondel.  Soft-dtw: a differentiable loss function for time-series.  In ICML, pages 894–903, 2017.
  * [9] Marco Cuturi, Jean-Philippe Vert, Oystein Birkenes, and Tomoko Matsui.  A kernel for time series based on global alignments.  In ICASSP, volume 2, pages II–413, 2007.
  * [10] Pavel Filonov, Andrey Lavrentyev, and Artem Vorontsov.  Multivariate industrial time series with cyber-attack simulation: Fault detection using an lstm-based predictive data model.  arXiv preprint arXiv:1612.06676, 2016.
  * [11] Nico Görnitz, Marius Kloft, Konrad Rieck, and Ulf Brefeld.  Toward supervised anomaly detection.  JAIR, 46:235–262, 2013.
  * [12] Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K Roy-Chowdhury, and Larry S Davis.  Learning temporal regularity in video sequences.  In CVPR, pages 733–742, 2016.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [14] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song.  A benchmark for anomaly segmentation.  arXiv preprint arXiv:1911.11132, 2019.
  * [15] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom.  Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding.  In KDD, pages 387–395, 2018.
  * [16] Radu Tudor Ionescu, Sorina Smeureanu, Bogdan Alexe, and Marius Popescu.  Unmasking the abnormal events in video.  In ICCV, pages 2895–2903, 2017.
  * [17] Radu Tudor Ionescu, Sorina Smeureanu, Bogdan Alexe, and Marius Popescu.  Unmasking the abnormal events in video.  In ICCV, pages 2895–2903, 2017.
  * [18] Radu Tudor Ionescu, Sorina Smeureanu, Marius Popescu, and Bogdan Alexe.  Detecting abnormal events in video using narrowed normality clusters.  In WACV, pages 1951–1960, 2019.
  * [19] Hyunjun Ju, Dongha Lee, Junyoung Hwang, Junghyun Namkung, and Hwanjo Yu.  Pumad: Pu metric learning for anomaly detection.  Information Sciences, 523:167–183, 2020.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Dongha Lee, Seonghyeon Lee, and Hwanjo Yu.  Learnable dynamic temporal pooling for time series classification.  In AAAI, volume 35, pages 8288–8296, 2021.
  * [22] Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon.  Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference.  In CVPR, pages 5267–5276, 2019.
  * [23] Yusha Liu, Chun-Liang Li, and Barnabás Póczos.  Classifier two sample test for video anomaly detections.  In BMVC, page 71, 2018.
  * [24] Sergey Lobov, Nadia Krilova, Innokentiy Kastalskiy, Victor Kazantsev, and Valeri A Makarov.  Latent factors limiting the performance of semg-interfaces.  Sensors, 18(4):1122, 2018.
  * [25] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In CVPR, pages 3431–3440, 2015.
  * [26] Weixin Luo, Wen Liu, and Shenghua Gao.  Remembering history with convolutional lstm for anomaly detection.  In ICME, pages 439–444, 2017.
  * [27] Mohsin Munir, Shoaib Ahmed Siddiqui, Andreas Dengel, and Sheraz Ahmed.  Deepant: A deep learning approach for unsupervised anomaly detection in time series.  IEEE Access, 7:1991–2005, 2018.
  * [28] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  arXiv preprint arXiv:1609.03499, 2016.
  * [29] Guansong Pang, Chunhua Shen, and Anton van den Hengel.  Deep anomaly detection with deviation networks.  In KDD, pages 353–362, 2019.
  * [30] Guansong Pang, Cheng Yan, Chunhua Shen, Anton van den Hengel, and Xiao Bai.  Self-trained deep ordinal regression for end-to-end video anomaly detection.  In CVPR, pages 12173–12182, 2020.
  * [31] Daehyung Park, Yuuna Hoshi, and Charles C Kemp.  A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder.  IEEE Robotics and Automation Letters, 3(3):1544–1551, 2018.
  * [32] Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall.  Neuralnetwork-viterbi: A framework for weakly supervised video learning.  In CVPR, pages 7386–7395, 2018.
  * [33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, pages 234–241, 2015.
  * [34] Takaya Saito and Marc Rehmsmeier.  The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets.  PloS one, 10(3):e0118432, 2015.
  * [35] Hiroaki Sakoe and Seibi Chiba.  Dynamic programming algorithm optimization for spoken word recognition.  IEEE transactions on acoustics, speech, and signal processing, 26(1):43–49, 1978.
  * [36] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei.  Robust anomaly detection for multivariate time series through stochastic recurrent neural network.  In KDD, pages 2828–2837, 2019.
  * [37] Waqas Sultani, Chen Chen, and Mubarak Shah.  Real-world anomaly detection in surveillance videos.  In CVPR, pages 6479–6488, 2018.
  * [38] Zhiguang Wang, Weizhong Yan, and Tim Oates.  Time series classification from scratch with deep neural networks: A strong baseline.  In IJCNN, pages 1578–1585, 2017.
  * [39] Qi Wei, Yinhao Ren, Rui Hou, Bibo Shi, Joseph Y Lo, and Lawrence Carin.  Anomaly detection for medical images based on a one-class classification.  In Medical Imaging 2018: Computer-Aided Diagnosis, volume 10575, page 105751M, 2018.
  * [40] Tailai Wen and Roy Keyes.  Time series anomaly detection using convolutional neural networks and transfer learning.  arXiv preprint arXiv:1905.13628, 2019.
  * [41] Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al.  Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications.  In WWW, pages 187–196, 2018.
  * [42] Tan Yu, Zhou Ren, Yuncheng Li, Enxu Yan, Ning Xu, and Junsong Yuan.  Temporal structure mining for weakly supervised action detection.  In ICCV, pages 5522–5531, 2019.
  * [43] Zehuan Yuan, Jonathan C Stroud, Tong Lu, and Jia Deng.  Temporal action localization by structured maximal sums.  In CVPR, pages 3684–3692, 2017.
  * [44] Yong Zhang, Yu Zhang, Zhao Zhang, Jie Bao, and Yunpeng Song.  Human activity recognition based on time series analysis using u-net.  arXiv preprint arXiv:1809.08113, 2018.

##  Appendix A Pseudo-code of WETAS Framework

Algorithm 1 shows the pseudo-code of training the framework. In practice, the
model parameters are updated by using minibatch SGD with the Adam optimizer
[20].

Input: A training set containing $N$ instances of temporal data
$\\{(\mathbf{X}^{(1)},y^{(1)}),\ldots,(\mathbf{X}^{(N)},y^{(N)})\\}$

Output: Updated DiCNN model $f(\cdot;\Theta)$ and the anomaly weight vector
$\mathbf{w}$

1 while  _convergence_ do

2 for  _$i=1,\ldots,N$_ do

$\triangleright$ Compute the local anomaly scores

3
$\mathbf{h}_{1}^{(i)},\ldots,\mathbf{h}_{T}^{(i)}=f(\mathbf{x}_{1}^{(i)},\ldots,\mathbf{x}_{T}^{(i)};\Theta)$

4
$s_{1}^{(i)},\ldots,s_{T}^{(i)}=\sigma(\mathbf{w}^{\top}\mathbf{h}_{1}^{(i)}),\ldots,\sigma(\mathbf{w}^{\top}\mathbf{h}_{T}^{(i)})$

5

$\triangleright$ Compute the global anomaly score

6 $\mathbf{h}_{*}^{(i)}=\text{global-
pooling}(\mathbf{h}_{1}^{(i)},\ldots,\mathbf{h}_{T}^{(i)})$

7 $s_{*}^{(i)}=\sigma(\mathbf{w}^{\top}\mathbf{h}_{*}^{(i)})$

8

$\triangleright$ Obtain the sequential pseudo-label

9 $m_{1}^{(i)},\ldots,m_{T}^{(i)}=$

$\text{min-max-
norm}(\mathbf{w}^{\top}\mathbf{h}_{1}^{(i)},\ldots,\mathbf{w}^{\top}\mathbf{h}_{T}^{(i)})$

10
$\widetilde{\mathbf{z}}_{+}^{(i)}=y^{(i)}\cdot\phi_{L}(m_{1}^{(i)},\ldots,m_{T}^{(i)})$

11
$\widetilde{\mathbf{z}}_{-}^{(i)}=(1-y^{(i)})\cdot\phi_{L}(m_{1}^{(i)},\ldots,m_{T}^{(i)})$

12

$\triangleright$ Calculate the two losses

13 $\mathcal{L}_{c}=-y^{(i)}\log
s_{*}^{(i)}-(1-y^{(i)})\cdot\log(1-s_{*}^{(i)})$

14
$\mathcal{L}_{a}=\max(0,(1/T)\cdot\text{DTW}_{\gamma}(\widetilde{\mathbf{z}}_{+}^{(i)},\mathbf{X}^{(i)})$

$-(1/T)\cdot\text{DTW}_{\gamma}(\widetilde{\mathbf{z}}_{-}^{(i)},\mathbf{X}^{(i)})+\beta)$

15 $\mathcal{L}=\mathcal{L}_{c}+\mathcal{L}_{a}$

16

$\triangleright$ Update all the model parameters

17 $\Theta=\Theta-\eta\cdot{\partial\mathcal{L}}/{\partial\Theta}$

18 $\mathbf{w}=\mathbf{w}-\eta\cdot{\partial\mathcal{L}}/{\partial\mathbf{w}}$

19

20

Algorithm 1 Optimizing the WETAS framework

##  Appendix B Computation of Dynamic Time Warping

The dynamic time warping (DTW) discrepancy (and the optimal alignment matrix)
between two time-series of lengths $M$ and $N$ is usually computed by solving
a dynamic program based on Bellman recursion, which takes a quadratic $O(MN)$
cost. The continuous relaxation of DTW (i.e., soft-DTW), which enables to
calculate the gradient of DTW with respect to its input, also can be computed
in a similar way to the original DTW [8]. Algorithm 2 presents the detailed
algorithms for computing
$\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$ and
$\nabla_{\Delta(\widetilde{\mathbf{z}},\mathbf{X})}\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$
based on Bellman recursion.

In order to obtain the eligible segmentation result, we need to enforce the
constraint that a single time point should not be aligned with multiple
consecutive labels. To this end, our forward recursion which computes
$\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$ does not consider the
$\downarrow$ relation in its recurrence; i.e., $R_{l,t}$ depends on only
$R_{l-1,t-1}$ and $R_{l,t-1}$, excluding $R_{l-1,t}$ (Line 6 in Algorithm 2).
Accordingly, the backward recursion which computes
$\nabla_{\Delta(\widetilde{\mathbf{z}},\mathbf{X})}\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$
also does not allow the $\uparrow$ relation; i.e., $E_{l,t}$ is obtained from
$E_{l,t+1}$ and $E_{l+1,t+1}$, excluding $E_{l+1,t}$ (Line 16 in Algorithm 2).

Note that the gradient of soft-DTW with respect to the cost matrix,
$\nabla_{\Delta(\widetilde{\mathbf{z}},\mathbf{X})}\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$,
can effectively update the anomaly weight vector $\mathbf{w}$ as well as the
model parameters of DiCNN $\Theta$ by the help of the gradient back-
propagation. This is possible because each entry of the cost matrix
$[\Delta(\widetilde{\mathbf{z}},\mathbf{X})]_{lt}=\delta(\widetilde{z}_{l},\mathbf{x}_{t})$
is defined by the binary cross entropy between a pseudo-label
$\widetilde{z}_{l}$ and a local anomaly score
$s_{t}=\sigma(\mathbf{w}^{\top}\mathbf{h}_{t})$,

$\delta(\widetilde{z}_{l},\mathbf{x}_{t})=-\left\\{\widetilde{z}_{l}\cdot\log
s_{t}+(1-\widetilde{z}_{l})\cdot\log(1-s_{t})\right\\}.$

In the end, the gradient optimizes each local anomaly score to be closer to
the pseudo-labels that are softly-aligned with the time point.

1 Function  _$\text{forward}\left(\widetilde{\mathbf{z}},\mathbf{X}\right)$_ :

$\triangleright$ Fill the alignment cost matrix $R\in\mathbb{R}^{L\times T}$

2 $R_{0,0}=0$

3 $R_{:,0}=R_{0,:}=\infty$

4 for  _$l=1,\ldots,L$_ do

5 for  _$t=1,\ldots,T$_ do

6
$R_{l,t}=\delta(\widetilde{z}_{l},\mathbf{x}_{t})+\text{min}_{\gamma}\\{R_{l-1,t-1},R_{l,t-1}\\}$

7

8 return $\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})=R_{L,T}$

9 Function  _$\text{backward}\left(\widetilde{\mathbf{z}},\mathbf{X}\right)$_
:

$\triangleright$ Fill the soft alignment matrix $E\in\mathbb{R}^{L\times T}$

$E_{L+1,T+1}=1$

$\triangleright$ $E_{l,t}:=\partial R_{L,T}/\partial R_{l,t}$

10 $E_{:,T+1}=E_{L+1,:}=0$

11 $R_{:,T+1}=R_{L+1,:}=-\infty$

12 for  _$l=L,\ldots,1$_ do

13 for  _$t=T,\ldots,1$_ do

14
$a=\exp\frac{1}{\gamma}(R_{l,t+1}-R_{l,t}-\delta(\widetilde{z}_{l},\mathbf{x}_{t+1}))$

15
$b=\exp\frac{1}{\gamma}(R_{l+1,t+1}-R_{l,t}-\delta(\widetilde{z}_{l+1},\mathbf{x}_{t+1}))$

16 $E_{l,t}=a\cdot E_{l,t+1}+b\cdot E_{l+1,t+1}$

17

18

19 return
$\nabla_{\Delta(\widetilde{\mathbf{z}},\mathbf{X})}\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})=E$

Algorithm 2 Forward recursion and backward recursion to compute
$\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$ and
$\nabla_{\Delta(\widetilde{\mathbf{z}},\mathbf{X})}\text{DTW}_{\gamma}(\widetilde{\mathbf{z}},\mathbf{X})$

Complexity Analysis.  In terms of efficiency, we analyze that our framework
additionally takes the computational cost of $O(LT)$ for DTW alignment between
$\widetilde{\mathbf{z}}$ and $\mathbf{X}$ (per CNN inference and its gradient
back-propagation), as described in Algorithm 2. Furthermore, because of 1) the
small $L$ value ($L\ll T$), 2) batch-wise computations in the PyTorch
framework, and 3) GPU parallel computations for DTW recursions based on the
numba library, it does not raise any severe efficiency issue.

##  Appendix C Reproducibility

For reproducibility, our implementation is publicly
available111111https://github.com/donalee/wetas, and Table 5 presents the
optimization details of WETAS. We empirically found that the performances are
hardly affected by these hyperparameters for the optimization.

Table 5: Details for the optimization of WETAS.

Batch size | 32 (for EMG, GHL, SMD)
---|---
8 (for Subway)
Optimizer | Adam optimizer
Initial learning rate | 0.0001
Max # epochs | 200
Stopping criterion | Instance-level F1 (on validation)

##  Appendix D Hyperparaemter Search

For our WETAS framework, we search the optimal values of the following
hyperparameters: the length of a sequential pseudo-label $L\in$ {4, 8, 12,
16}, the anomaly threshold for pseudo-labeling $\tau\in$ {0.1, 0.3, 0.5, 0.7},
and the margin size for the alignment loss $\beta\in$ {0.1, 0.5, 1.0, 2.0}.
The optimal values are selected based on instance-level F1-scores on the
validation set. The selected hyperparameter values for reporting the final
performance are listed in Table 6. In case of the smoothing parameter $\gamma$
used for soft-DTW, we fix its value to 0.1 without further tuning.

Table 6: Selected hyperparameter values for WETAS. Datasets |  EMG |  GHL |  SMD |  Subway
---|---|---|---|---
global-pooling |  avg |  max |  max |  avg
$L$ |  4 |  16 |  12 |  12
$\tau$ |  0.3 |  0.1 |  0.5 |  0.5
$\beta$ |  0.1 |  1.0 |  0.5 |  0.5

##  Appendix E Baseline Methods

We describe the details of the anomaly detection methods which are used as the
baseline in our experiments. All of them employ their own deep neural networks
to effectively model the temporal dependency among time points, and compute
the anomaly score for each point or segment.

  * •

Donut [41]: A simple VAE model optimized by the modified evidence lower bound
(M-ELBO). It also uses a sampling-based imputation technique for missing
points, in order to effectively deal with anomalous points during the
detection.

  * •

LSTM-VAE [31]: A VAE model that employs long short term memory (LSTM) for its
encoder and decoder. It is trained to reconstruct the non-anomalous training
data well, and defines the anomaly score by the reconstruction error.

  * •

LSTM-NDT [15]: A LSTM network that is trained to predict the next input (i.e.,
sequence modeling). It additionally adopts the non-parametric dynamic
thresholding (NDT) technique to automatically determine the optimal anomaly
threshold.

  * •

OmniAnomaly [36]: The state-of-the-art point-level anomaly detection model
that uses gated recurrent units (GRU) as the encoder and decoder of VAE. It
incorporates advanced techniques into VAE, including normalizing flows and a
linear Gaussian space model, to consider stochasticity and temporal dependency
among the time points.

  * •

DeepMIL [37]: The multiple-instance learning method that learns from weakly
labeled temporal data. It produces the anomaly prediction for each fixed-
length segment, thus the result can be used for temporal anomaly segmentation.
We consider different numbers of the segments in a single instance, denoted by
DeepMIL-4, 8, 16.

Note that Donut, LSTM-VAE, LSTM-NDT, and OmniAnomaly fall into the category of
unsupervised learning as they do not utilize any anomaly labels for training,
and their variants that use only normal instances are categorized as semi-
supervised learning. DeepMIL is the only existing method that is based on
weakly supervised learning.

##  Appendix F Additional Experiments

Comparison with frame-level video anomaly detectors.  In case of the video
dataset (i.e., Subway), we additionally report the AUROC scores of
unsupervised video anomaly detection methods [17, 18, 23, 30] as a benchmark.
They aim to train the networks that take spatio-temporal (or spatial) inputs
to compute the anomaly score of each video frame. Note that all these methods
produce the frame-level (or point-level) anomaly predictions, without
utilizing the instance-level anomaly labels for training. In Table 7, the
weakly supervised methods121212To compute AUROC for the weakly supervised
methods, we regard the anomaly score of each segment as the score for all
points in the segment (for DeepMIL), and use the local (or point-level)
anomaly scores without the DTW-based segmentation (for WETAS). (i.e., DeepMIL
and WETAS) show better performance than the unsupervised methods. Even though
DeepMIL and WETAS simply use the pre-computed visual features of each frame
(extracted by the pre-trained ResNet), it outperforms the other domain-
specific baseline methods (optimized in an end-to-end manner) by the help of
the instance-level anomaly labels. This indicates that leveraging the weak
supervision can be more effective to discriminate normal and anomalous video
frames compared to fine-tuning the networks for visual feature extraction.

Table 7: Comparison with several recent frame-level video anomaly detection
methods, Dataset: Subway.

Methods |  AUROC | Methods |  AUROC
---|---|---|---
Ionescu et al. [17] |  85.7% |  DeepMIL-4 [37] |  95.7%
Ionescu et al. [18] |  95.1% |  DeepMIL-8 [37] |  96.9%
Liu et al. [23] |  93.1% |  DeepMIL-16 [37] |  96.4%
Pang et al. [30] |  92.7% |  WETAS (ours) |  97.8%

Figure 5: Learning curves of WETAS in terms of training/validation loss and
test/validation F1-score, Datasets: EMG, GHL, SMD, and Subway (from the first
row).

Learning curves.  We plot the learning curves of WETAS by using the EMG, GHL,
SMD, and Subway datasets. In Figure 5, as the number of epochs increases, the
classification loss and alignment loss consistently decrease for both the
training and validation sets. This implies that WETAS can infer more accurate
sequential pseudo-labels as the training progresses, and the alignment loss
better guides its model to output anomaly scores that are well aligned with
the pseudo-label. Similarly, Figure 4 illustrates that the pseudo-labeling and
the DTW-based segmentation collaboratively improve with each other.
Consequently, the instance-level and point-level F1-scores for the test set
increase as well. The results empirically show that the instance-level
F1-score (or the total loss) on the validation set can be a good termination
criterion for the optimization of WETAS, thus we adopt it in our experiments.

Figure 6: F1-scores of WETAS with different hyperparameter values, Dataset:
GHL.

Sensitivity Analyses.  We finally examine the performance changes of WETAS
with respect to the three hyperparameters (i.e., $L$, $\tau$, and $\beta$). In
Figure 6, we observe that the final performance of WETAS is not sensitive to
$\beta$, and it shows higher F1-score when using a smaller $\tau$ and a larger
$L$. Specifically,

  * •

$\beta$ does not much affect the final performance of WETAS, because it simply
controls the margin size in the alignment loss.

  * •

A smaller $\tau$ encourages to find out more anomalous segments, which leads
to a high recall for anomaly detection, by making the sequential pseudo-label
have more number of 1s.

  * •

A larger $L$ allows a finer-grained segmentation by aligning the time points
with more number of 0s and 1s in each sequential anomaly pseudo-label.

Nevertheless, unlike the DeepMIL, the granularity of pseudo-labeling is not a
critical factor for WETAS because the DTW-based segmentation is capable of
dynamically aligning an input instance with its pseudo-label. In conclusion,
the best performing hyperparameter values successfully optimize WETAS under
the weak supervision so that it can identify variable-length anomalous
segments.