# Plotting time: On the usage of CNNs for time series classification

Nuno M. Rodrigues1 João E. Batista1 Leonardo Trujillo2&Bernardo Duarte3,4
&Mario Giacobini5&Leonardo Vanneschi6&Sara Silva1 1LASIGE, Faculdade de
Ciências da Universidade de Lisboa, Campo Grande, 1749-016 Lisboa, Portugal
2Departamento de Ingeniería Eléctrica y Electrónica, Tecnológico Nacional,
México/IT de Tijuana
3MARE - Marine and Environmental Sciences Centre, Faculdade de Ciências da
Universidade de Lisboa, Campo Grande, 1749-016 Lisboa, Portugal
4Departamento de Biologia Vegetal, Faculdade de Ciências da Universidade de
Lisboa, Campo Grande, 1749-016 Lisboa, Portugal
5Data Analysis and Modeling Unit, Department of Veterinary Sciences,
University of Torino, Italy
6NOVA Information Management School (NOVA IMS), Universidade Nova de Lisboa,
Campus de Campolide, 1070-312 Lisboa, Portugal {nmrodrigues, jebatista,
baduarte<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present a novel approach for time series classification where we represent
time series data as plot images and feed them to a simple CNN, outperforming
several state-of-the-art methods.

We propose a simple and highly replicable way of plotting the time series, and
feed these images as input to a non-optimized shallow CNN, without any
normalization or residual connections. These representations are no more than
default line plots using the time series data, where the only pre-processing
applied is to reduce the number of white pixels in the image. We compare our
method with different state-of-the-art methods specialized in time series
classification on two real-world non public datasets, as well as 98 datasets
of the UCR dataset collection. The results show that our approach is very
promising, achieving the best results on both real-world datasets and matching
/ beating the best state-of-the-art methods in six UCR datasets. We argue
that, if a simple naive design like ours can obtain such good results, it is
worth further exploring the capabilities of using image representation of time
series data, along with more powerful CNNs, for classification and other
related tasks.

##  1 Introduction

Time series classification (TSC) problems are highly diverse in nature,
covering a wide range of domains. Recently, this field of study is receiving
more and more attention, arguably due to the creation of large data
repositories such as UCR Dau et al. (2018). Such easy availability of data
motivated the proposal of multiple new algorithms to address TSC.

When looking at the current state-of-the-art (SOTA) of TSC, led by the HIVE-
COTE or TS-CHIEF methods, or even just the more popular algorithms for TSC,
there are two main approaches. Some algorithms like BOSS, BOP or WEASEL use
word and symbol embeddings along with sliding windows to transform the data
Schäfer (2014); Large et al. (2018); Schäfer and Leser (2017). The other
predominant approach is to use different ensemble methods, such as HIVE-COTE
(which includes BOSS), TS-CHIEF or Proximity Forests Lines et al. (2016);
Lucas et al. (2019); Shifaz et al. (2020).

Recently, a new trend is emerging due to the rising popularity of
convolutional neural networks (CNN), which is the use of convolution
operations. We can see this in ROCKET Dempster et al. (2020), which applies
random convolutional kernels to the time series data, or in InceptionTime,
which is an actual computer vision model Karimi-Bidhendi et al. (2018); Fawaz
et al. (2020).

The vast majority of these algorithms share one common trait: they use the
numeric data of the time series to find patterns and induce models. However,
pattern recognition can also be tackled from a computer vision perspective,
using visual representations of the data. As previously mentioned, and further
elaborated, there are already some recent algorithms that use a visual
representation of the data to perform TSC. However, we argue that these works
follow an overly complex path, and we propose a much simpler approach.

Our work demonstrates that, with highly simplistic visual representations of
the time series data, it is possible to achieve results that are equal or even
better than the ones obtained by the best SOTA methods available today.

##  2 Related work

There are two general goals when non-image data is converted to a 2D image
representation. The first one is to understand and interpret the data, since
converting it to a visual representation allows a researcher to analyze the
data in a more human-readable form. These techniques have a long history in
pattern analysis and machine learning Keim (2000), including different types
of geometric representations, pixel-oriented visualizations and dimensionality
reduction techniques. For instance, a popular SOTA method is t-SNE van der
Maaten and Hinton (2008), that allows for 2D and 3D representations of large
datasets in high-dimensional spaces. While t-SNE focuses on visualizing an
entire dataset, General Line Coordinates (GLC), on the other hand, can be used
to represent individual data samples as polylines Kovalerchuk (2018), without
any loss of information.

The second broad reason for transforming non-image data into images is that it
allows researchers to solve the original learning problem using powerful and
widely available computer vision algorithms, such as CNNs. This approach is
much more recent, with a growing but still fairly small literature on the
subject. In Dovhalets et al. (2018) the authors used GLC representations of
data to pose an image recognition problem and solved it with CNNs, achieving
similar performance compared to solving the original problem directly, with
the added benefit that interpreting the results is easier. RNA data is
transformed into 2D images in Lyu and Haque (2018), outperforming previous
works on the same problem, while using a domain specific transformation of the
data. A notable approach is DeepInsight Sharma et al. (2019), that proposes a
general methodology to transform data samples into unique images and
subsequently applies CNN to a variety of problems from different domains.
DeepInsight transforms the data by applying Principal Component Analysis in a
simple but creative manner, and also produces informative visualizations of
the samples in a dataset, intended to provide unique insights on the problem.
Regarding the case of time series classification, Karimi-Bidhendi et al.
(2018) transforms the time series to a Gramian Angular Difference Field
(GADF), a visual representation of the data that is subsequently processed by
Google’s pre-trained Inception CNN, achieving SOTA results.

##  3 Proposed methodology

The methodology we propose is more focused on the transformation of the time
series data into image data, and less on the particulars of the CNN
architecture. Figure 1 illustrates the different steps of the methodology, in
both univariate and multivariate cases. Details of each step are provided
below.

Figure 1: The proposed methodology. For univariate time series, each series is
converted into an image, normalized and given to a CNN. For multivariate time
series, we test two approaches: a) all series are displayed on the same plot
and given to a single CNN; b) different series are displayed in separate plots
and given to siamese CNNs, whose output is concatenated before flattening. We
also split the multivariate cases in multiple univariate problems, for
comparison.

###  3.1 Generating image data from time series

We follow a simple and replicable process of transforming the time series data
into image plots to be used by the CNN. First, the time series data is loaded
without any pre-processing. Then, using matplotlib library Hunter (2007) we
plot each time series adopting the default parameters, obtaining a $432\times
288$ sized image for each series, as seen in Figure 1. The plot axes are
included in the image, along with their problem specific time ($x$-axis) and
value ($y$-axis) numbers, since they can provide important information, as
shown later when we analyse feature maps. Notice that not all the time series
are required to have the same length, since this has no impact when
transforming the data into an image, and does not imply loss of valuable
information (this way avoiding the need to oversample the data).

###  3.2 Normalizing the image data

When loading the plots to use as input to our networks, we perform two kinds
of image pre-processing. We start by the standard rescaling of pixel values to
the $[0,1]$ range, and then we apply a samplewise standard normalization. The
samplewise normalization essentially inverts the colors (produces a negative
of the original image, as shown in Figure 1), which reduces the amount of
empty white space by turning the pixel values to 0 (black). This reduces the
activations of the feature maps in empty areas where there is no information,
while increasing them in the numbers of the axes, which is helpful as they
also contain information. This normalization produces large improvements in
both learning and generalization. Its effect on the feature maps will be
analysed in Section 5.

###  3.3 Feeding the plots to the CNN

For all the problems addressed, we use a simple shallow CNN of five
convolutional and pooling layers, interspersed, and three fully connected
layers with two dropouts in between. As for the hyperparameters, we use
standard values and do not perform any sort of grid-search / bayesian
hyperparameter optimization. Regarding the filters, we start with $16$ and
double the number in each convolutional layer. In the first and second fully
connected layers we use $256$ and $128$ units, respectively. For the optimizer
we use ADAM with a learning rate of $0.001$.

When modeling a problem, if its data is univariate (containing only one time
series) we simply feed the pre-processed images to a CNN like the one
described above (top part of Figure 1). For multivariate problems (data
containing more than one time series) we test different approaches, identified
as a) and b) in the bottom half of Figure 1. In variant a) the different time
series for the same sample are displayed on the same plot and given to a
single CNN as the one used for the univariate approach. In variant b) the
different time series are displayed in separate plots and given to a set of
siamese CNNs. Each input head has the same set of convolutional and pooling
layers described above and, before flattening, the outputs of these networks
are concatenated and fed into the same classifier section, producing a single
output. This method allows for different features to be extracted from the
different time series, however, it is far more computationally expensive.

There is also an additional variant, not shown in the figure, which is to
split the multivariate problems in multiple univariate problems, each one
using only one time series. We have included this last variant because we
wanted to check that drawing more than one time series in a single plot, or
using a set of siamese networks, is not a confounding factor that prevents the
learning of the information contained in each variable. Since our multivariate
datasets have only two time series, it was feasible to do so.

##  4 Experiments

All experiments were performed on a Windows 10 machine with one NVIDIA 2080 TI
GPU with 11GB of RAM. All network based models were implemented in TensorFlow.

###  4.1 Datasets and Problems

Here we describe the datasets used to test our methodology. Since the UCR
dataset is public domain and is already described in its website, we provide
more details on the remaining two datasets and respective prediction tasks.

####  4.1.1 OPTOX

The OPTOX dataset was collected by the Marine and Environmental Sciences
Centre - MARE - Faculty of Sciences, University of Lisbon. Each sample
corresponds to a chlorophyll fluorescence induction curve, and contains the
fluorescence values taken at different time steps from a model diatom used in
ecotoxicology (Phaeodactylum tricornutum) exposed to 13 different emerging
contaminants at different concentrations, following the international
standards for ecotoxicological assays. For each concentration, 30 independent
replicates were obtained. Details of the experiments and samples can be found
in Silva et al. (2020).

With this dataset, we address the challenge of identifying which of the 13
contaminants is detected, using only the fluorescence induction curve. Table 1
contains more detailed information regarding this dataset, including the
identification of the contaminants, number of concentrations considered for
each, and number of samples per class.

Table 1: OPTOX dataset summary, containing the class names (contaminants)
concentrations considered for each contaminant, and distribution of the total
number of samples per class.

Contaminant | #concentrations | #samples
---|---|---
Diclofenac | $6$ | $180$
Fluoxetine | $6$ | $180$
Glyphosate | $6$ | $180$
Ibuprofen | $6$ | $180$
Propranolol | $6$ | $180$
Sodium Dodecyl Sulphate (SDS) | $5$ | $150$
Triclosan | $6$ | $180$
Dissolved Ionic Copper | 4 | 120
Copper Engineered Nanoparticles | 4 | 120
Dissolved Ionic Zinc | 4 | 120
Zinc Engineered Nanoparticles | 4 | 120
Dissolved Ionic Titanium | 4 | 120
Titanium Engineered Nanoparticles | 4 | 120

####  4.1.2 FISIO

The FISIO dataset was collected by the Centre of Preventive Medicine and Sport
- SUISM - University Structure of Hygiene and Sport Sciences, Centre of
excellence of the University of Turin. Data were recorded during an indoor
trial conducted by the centre, where a group of 262 volunteers underwent an
aerobic exercise on a trade mill. The test started with a speed of 5 km/h that
was incremented by 1 km/h every minute. The participant could suspend the test
when he/she felt exhausted. Cardio-respiratory variables were recorded every
10 seconds, namely heart rate (HR) and ventilation (VEN), along with the age,
gender and body mass index (BMI) of each participant. More details on this
dataset and related tests can be found in Azzali et al. (2020).

With this dataset, we addressed two challenges: based on both HR and VEN, we
predict 1) BMI class: two classes were defined, based on the UK NHS BMI charts
for both children and adults (since some of the participants were underage),
where Class 1 contains Underweight and Healthy participants, and Class 2
contains Overweight and Obese participants; 2) Age class: two classes were
defined, with Class 1 for participants younger than 30 years old and Class 2
for all the others. Gender remained unused.

Table 2 contains the information regarding sample distribution per class for
both problems. Since we had both time series (HR and VEN) for each
participant, we decided to use both multivariate and univariate approaches
(Section 3.3).

Table 2: FISIO dataset summary, containing the distributions of samples per
class for the two problems addressed, BMI and Age.

BMI |  | Age
---|---|---
Class 1 | Class 2 |  | Class 1 | Class 2
221 | 41 |  | 128 | 134

####  4.1.3 UCR

The UCR archive contains, as of now, a set of 128 univariate and 30
multivariate datasets Dau et al. (2018). From the 128 univariate problems we
tested our approach on 98 of them. The selection criterion for these 98
problems was two-fold. First, because the results provided by UCR for each
classifier do not include all 128 problems, we decided to use only the
problems that were tested with all the algorithms, ending up with 108
problems. Second, due to time constrains, we decided to not test our approach
on single axis data of multi-axes problems (e.g Cricket X/Y/Z), ending with
the final amount of 98 problems. Regarding the multivariate problems, the
archive does not have results to compare to, so we did not use them.

###  4.2 Methods

In order to have comparison grounds for our methodology, we tested a set of
different classifiers for both the OPTOX and the FISIO datasets: XGBoost Chen
and Guestrin (2016), Random Forests (RF) Breiman (2001), deep learning
classifiers 1D CNN and a deep ANN, and ROCKET.

The FISIO dataset presented an additional problem for these classifiers, as it
consists of a multivariate problem with time series of different lengths. To
address this issue, we truncated each time series to the same length as the
smallest one of that type, therefore assuming the risk of losing information.
This problem serves to highlight the advantage of our methodology in being
able to use the full time series without the need for all the series to have
the same length.

We performed 30 independent runs with all the methods, with an 80/20 split for
training and test, each with a seed equal to the number of the run. For each
run, both XGBoost and RF were optimized by means of grid-search for the
following parameters: max depth; number of estimators; tree method; learning
rate; criterion.

As for UCR, the archive already provides results based on 30 independent runs
from several other algorithms. Due to the limited computational resources and
large amount of problems, we started by performing five independent runs per
problem, and an additional 25 only in those problems where our methodology was
able to outperform or tie as the best result.

##  5 Results and Discussion

Here we compare the results obtained by our approach with the results of
several other methods, including the current SOTA, on the OPTOX, FISIO and UCR
datasets. Statistically significant differences in the results are determined
by the Kruskal-Wallis test at $p<0.01$. We use a non-parametric test because
not all the results follow a normal distribution. When the results presented
in tables identify more than one value as the best, it means the difference
between them is not statistically significant. All the $p$-values are included
in the Appendix, in Table 7 for OPTOX, Tables 8 to 15 for FISIO, and Tables 16
to 21 for the UCR problems where we performed 30 runs (see Section 4.2). This
section also discusses the effect that image data normalization has on the
learned features by examining feature maps extracted from the CNNs.

### OPTOX

For this dataset, we have produced two sets of results from our methodology,
identified as CNN and CNN Log. In CNN, the values of the different time steps
are drawn at equally spaced intervals on the $x$-axis of the plots, even if in
reality the time intervals between every two measurements are not the same for
the entire curve. By convention, when visualizing an induction curve it is
common to use a logarithmic scale on the $x$-axis, which is what CNN Log does.

Table 3 compares all tested classifiers in terms of median accuracy in both
training and test sets for the contaminant prediction task of the OPTOX
dataset. As shown, our algorithm outperformed all the other methods. CNN Log
was the best performing method, significantly better than CNN, which is tied
in second with ROCKET. This difference between CNN and CNN Log highlights the
need to take into consideration domain knowledge when transforming the data
into images. Despite not changing the methodology whatsoever, this change in
scale significantly improved the results.

Table 3: Median overall accuracy of all methods on the OPTOX problem. The best
test result is represented in green.

| RF | XGBoost | ROCKET | ANN | 1D CNN | CNN Log | CNN
---|---|---|---|---|---|---|---
Train | $0.9625$ | $0.9983$ | $0.9991$ | $0.8493$ | $0.9993$ | $0.9807$ | $0.9815$
Test | $0.7461$ | $0.8076$ | $0.9705$ | $0.9089$ | $0.9641$ | $0.9765$ | $0.9732$

### FISIO

Table 4 compares all tested classifiers in terms of median accuracy in both
training and test sets, for the BMI and Age problems of the FISIO dataset. The
results identified as HR and VEN are the ones obtained using only one of the
time series (the additional variant mentioned in Section 3.3). The results
identified as HR+VEN refer to variant a), where both time series are drawn in
the same plot, and the ones identified as Siamese refer to variant b), where
each time series is drawn in a different plot and the two plots are given to
siamese networks.

On the BMI classification problem, the CNNs using only the HR time series
obtained the best median result, but not significantly different from the
results of RF and XGBoost. This provides two interesting findings: for a
multivariate problem, the best results were obtained using only one of the
time series, supporting the hypothesis that using multiple time series may
indeed be a confounding factor (for 1D CNN, HR+VEN certainly is); two standard
classification methods were able to outperform not only all other deep
learning classifiers, but also ROCKET, which is one of the top algorithms for
TSC.

Regarding the Age classification problem, the best median accuracy was
obtained by the CNNs, who achieved the exact same value using only the HR time
series, using both HR+VEN on the same plot, and following the siamese
approach. When using only the HR time series, the CNNs outperformed all other
methods, which did not happen with the other two approaches where, despite
having a higher median accuracy, CNNs were not significantly different from
the other highlighted classifiers.

Table 4: Median overall accuracy from all models in the BMI and Age problems
of the FISIO dataset. The best test result for each problem is represented in
green.

| BMI
---|---
| HR | VEN | HR+VEN | Siamese
| Train | Test | Train | Test | Train | Test | Train | Test
RF | 0.8564 | 0.8301 | 0.9090 | 0.8113 | 0.9138 | 0.8113 | —- | —-
XGBoost | 0.8562 | 0.8301 | 0.9371 | 0.8301 | 0.9251 | 0.8301 | —- | —-
ROCKET | 1 | 0.7264 | 1 | 0.7547 | 1 | 0.7169 | —- | —-
ANN | 0.9330 | 0.7641 | 0.9617 | 0.7924 | 0.9760 | 0.7735 | 0.8492 | 0.8113
1D CNN | 1 | 0.6792 | 1 | 0.6981 | 0.5100 | 0.46 | 1 | 0.7547
CNN | 1 | 0.8461 | 1 | 0.8269 | 1 | 0.8269 | 1 | 0.8365

| Age
---|---
| HR | VEN | HR+VEN | Siamese
| Train | Test | Train | Test | Train | Test | Train | Test
RF | 0.9569 | 0.6037 | 0.9377 | 0.6037 | 0.9808 | 0.6415 | —- | —-
XGBoost | 0.9730 | 0.6037 | 0.9491 | 0.6320 | 1 | 0.6792 | —- | —-
ROCKET | 1 | 0.6132 | 1 | 0.5849 | 1 | 0.6981 | —- | —-
ANN | 0.8947 | 0.6415 | 0.9282 | 0.6037 | 0.9665 | 0.6981 | 0.7775 | 0.6981
1D CNN | 1 | 0.5660 | 1 | 0.5849 | 1 | 0.6698 | 0.9952 | 0.6792
CNN | 1 | 0.7058 | 1 | 0.6568 | 1 | 0.7058 | 1 | 0.7058

### UCR

Table 5 compares the number of times that each method, both the ones provided
by UCR and ours, achieved first, second or third ranking on the set of 98
problems addressed. We can see that, although it is clearly not the best, our
approach is able to challenge the current SOTA methods, beating / matching
them on six problems, and arriving in third on another.

As mentioned in Section 4.2, in order to assess the results on such a large
number of datasets, we performed five independent runs on all addressed
problems, and then compared the best test accuracy achieved on each problem
(between our five runs and UCR’s reported 30 runs per method). The six
problems where CNN matched/beat the other methods on this comparison are BME,
Coffee, Earthquakes, Plane, SmoothSubspace and Trace (see Appendix, Table
LABEL:giga_table). For these six problems, we performed our additional 25
independent runs, and compared the median test accuracy of the 30 runs with
the median results provided by UCR. As shown in Table 6, CNN achieved the best
results in all these problems.

Table 5: Number of times each method achieved first, second or third ranking
spot in the UCR problems.

| CNN | TS-CHIEF | HIVE-COTE | ROCKET | InceptionTime | STC | ResNet | ProximityForest | WEASEL | S-BOSS | cBOSS | BOSS | RISE | TSF | Catch22
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
First | 6 | 35 | 32 | 26 | 31 | 21 | 25 | 23 | 20 | 20 | 20 | 18 | 12 | 14 | 11
Second | 0 | 6 | 9 | 10 | 12 | 2 | 9 | 4 | 7 | 4 | 2 | 4 | 1 | 2 | 1
Third | 1 | 7 | 5 | 12 | 5 | 6 | 11 | 4 | 5 | 4 | 3 | 3 | 0 | 1 | 5

Table 6: Median overall test accuracy in the selected UCR problems. The best
results for each problem are highlighted in green.

| CNN | TS-CHIEF | HIVE-COTE | ROCKET |  Inception Time | STC | ResNet |  Proximity Forest | WEASEL | S-BOSS | cBOSS | BOSS | RISE | TSF | Catch22
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
BME | $1$ | $1$ | $0.9867$ | $1$ | $1$ | $0.9400$ | $1$ | $1$ | $0.9600$ | $0.876$ | $0.7733$ | $0.8667$ | $0.7933$ | $0.9733$ | $0.9133$
Coffee | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$
Earthquakes | $0.7698$ | $0.7482$ | $0.7482$ | $0.7482$ | $0.7410$ | $0.7410$ | $0.7194$ | $0.7518$ | $0.7482$ | $0.7482$ | $0.7482$ | $0.7482$ | $0.7482$ | $0.7482$ | $0.7410$
Plane | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $0.9905$
SmoothSubspace | $1$ | $1$ | $0.9867$ | $0.9767$ | $0.9867$ | $0.9400$ | $0.9933$ |  $1$$0.8600$ | $0.4067$ | $0.4467$ | $0.4033$ | $0.8467$ | $0.9867$ | $0.8533$ |
Trace | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $1$ | $0.9900$ | $1$ | $1$

| (a) | (b) | (c)
---|---|---
|  |
(d) | (e) | (f)
|  |
| (a) | (b) | (c)
---|---|---
|  |
(d) | (e) | (f)
|  |
Figure 2: Feature maps from different convolutional layers taken from two
models trained on the OPTOX dataset. The set of images on the left side was
obtained using input that was normalized, while the right side corresponds to
non normalized images. For both set of plots a) was taken from the first
convolution layer, b) and c) from the second, d) and e) from the third, and f)
from the fourth.

### The effect of normalizing image data

Before deciding to apply the samplewise normalization described in Section
3.2, we performed some trial runs where the only pre-processing of image data
was the rescaling of the pixel values. The main difference we observed was
that, without normalization, there were multiple runs that converged to sub-
optimal solutions during training, particularly on the UCR problems.
Furthermore, even when the models converged to good solutions, they took much
longer to converge than the ones with normalization, as much as $4\times$
longer on the OPTOX dataset.

A possible explanation for this difference is that, without normalization, the
filters have to learn the features from low pixel values (curves and axes)
that are surrounded by high pixel values (white space), a much more difficult
task than the opposite, that is to learn features from the pixels that already
enter the network with the highest values. Indeed, when analysing feature maps
from models trained with non normalized image data, we observed that sometimes
the time series are not even detected in the first convolutional layer. On
later layers there are in fact feature maps showing that the filters
identified key information such as the time series and axes. However, there
are a large number of feature maps where no prominent features are identified,
something that does not happen when the images are normalized.

Figure 2 presents two sets of feature maps. On the left, feature maps obtained
from a model trained with normalized data; on the right, feature maps from a
model trained on the same dataset (and same training/test partition) without
normalization. Comparing both sets, we can see that, with normalization, the
filters begin to detect different important areas of the image, starting on
the first convolutional layer, whereas without normalization there are almost
no activations on the feature maps of the first and second convolutional
layers.

##  6 Conclusions and Future Work

We have proposed a new approach for time series classification where the time
series are represented as plot images and given to a shallow CNN. Our
methodology is very simple and can be applied to a broad variety of problems,
including ones where different time series have different lengths. We tested
our methodology on two real-world non public datasets, where it outperformed
all other methods, and on the UCR archive, where it was able to beat the
current state-of-the-art methods in a small set of problems. We have also
shown that, thanks to the image pre-processing we apply, the models are able
to converge faster and detect important image features earlier during
training. We conclude that, if a simple naive design like ours can obtain such
good results, then there is much to explore when using deep learning methods
that rely on image data.

As future research, we will test our methodology using known network
architectures such as ResNet, VGG or Inception, and alternative image
representations of the data. Finally, we will use the same approach to solve
other classification problems that may not involve time series, but rather
signal data, such as remote sensing applications.

## Acknowledgements

This work was partially supported by FCT through funding of Research Units
LASIGE (UIDB/ 00408/2020 and UIDP/ 00408/2020) and MARE (UIDB/ 04292/2020);
AICE (DSAIPA/DS/ 0113/2019), Projects BINDER (PTDC/CCI-INF/ 29168/2017),
GADgET (DSAIPA/DS/ 0022/2018), INTERPHENO (PTDC/ASP-PLA/ 28726/2017), OPTOX
(PTDC/CTA-AMB/ 30056/2017), PREDICT (PTDC/CCI-CIF/ 29877/2017); PhD
Grant(SFRH/BD/143972/2019); Research Contracts CEECIND/00511/2017 and
CEECIND/02513/2017. The authors also acknowledge Prof. Alberto Rainoldi and
Dr. Marco Ivaldi for the useful discussion on the physiological dataset and
the Centre of Preventive Medicine and Sport - SUISM - University Structure of
Hygiene and Sport Sciences, Centre of Excellence of the University of Torino
for allowing the use of the dataset.

## References

  * Azzali et al. [2020] Irene Azzali, L. Vanneschi, Illya Bakurov, Sara Silva, M. Ivaldi, and M. Giacobini.  Towards the use of vector based gp to predict physiological time series.  Appl. Soft Comput., 89:106097, 2020.
  * Breiman [2001] Leo Breiman.  Random forests.  Machine Learning, 45:5–32, 2001.
  * Chen and Guestrin [2016] T. Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
  * Dau et al. [2018] Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, Gustavo Batista, and Hexagon-ML.  The ucr time series classification archive, October 2018.  https://www.cs.ucr.edu/~eamonn/time_series_data_2018/.
  * Dempster et al. [2020] Angus Dempster, Franccois Petitjean, and Geoffrey I. Webb.  Rocket: exceptionally fast and accurate time series classification using random convolutional kernels.  Data Mining and Knowledge Discovery, 34:1454–1495, 2020.
  * Dovhalets et al. [2018] Dmytro Dovhalets, Boris Kovalerchuk, Szilárd Vajda, and Răzvan Andonie.  Deep learning of 2-d images representing n-d data in general line coordinates.  In International Symposium on Affective Science and Engineering, volume ISASE2018, pages 1–6. Japan Society of Kansei Engineering, 2018.
  * Fawaz et al. [2020] Hassan Ismail Fawaz, B. Lucas, G. Forestier, Charlotte Pelletier, D. Schmidt, Jonathan Weber, Geoffrey I. Webb, L. Idoumghar, Pierre-Alain Muller, and Franccois Petitjean.  Inceptiontime: Finding alexnet for time series classification.  ArXiv, abs/1909.04939, 2020.
  * Hunter [2007] J. D. Hunter.  Matplotlib: A 2d graphics environment.  Computing in Science & Engineering, 9(3):90–95, 2007.
  * Karimi-Bidhendi et al. [2018] S. Karimi-Bidhendi, F. Munshi, and A. Munshi.  Scalable classification of univariate and multivariate time series.  In 2018 IEEE International Conference on Big Data, pages 1598–1605, 2018.
  * Keim [2000] D. A. Keim.  Designing pixel-oriented visualization techniques: theory and applications.  IEEE Transactions on Visualization and Computer Graphics, 6(1):59–78, 2000.
  * Kovalerchuk [2018] Boris Kovalerchuk.  Visual Knowledge Discovery and Machine Learning.  Springer International Publishing, 2018.
  * Large et al. [2018] J. Large, Anthony J. Bagnall, S. Malinowski, and R. Tavenard.  From bop to boss and beyond: Time series classification with dictionary based classifiers.  ArXiv, abs/1809.06751, 2018.
  * Lines et al. [2016] J. Lines, Sarah Taylor, and Anthony J. Bagnall.  Hive-cote: The hierarchical vote collective of transformation-based ensembles for time series classification.  2016 IEEE ICDM, pages 1041–1046, 2016.
  * Lucas et al. [2019] B. Lucas, Ahmed Shifaz, Charlotte Pelletier, Lachlan O’Neill, N. A. Zaidi, B. Goethals, François Petitjean, and Geoffrey I. Webb.  Proximity forest: an effective and scalable distance-based classifier for time series.  Data Mining and Knowledge Discovery, 33:607–635, 2019.
  * Lyu and Haque [2018] Boyu Lyu and Anamul Haque.  Deep learning based tumor type classification using gene expression data.  bioRxiv, 2018.
  * Schäfer and Leser [2017] Patrick Schäfer and U. Leser.  Fast and accurate time series classification with weasel.  Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 2017.
  * Schäfer [2014] Patrick Schäfer.  The boss is concerned with time series classification in the presence of noise.  Data Mining and Knowledge Discovery, 29:1505–1530, 2014.
  * Sharma et al. [2019] Alok Sharma, Edwin Vans, Daichi Shigemizu, Keith A. Boroevich, and Tatsuhiko Tsunoda.  DeepInsight: A methodology to transform a non-image data to an image for convolution neural network architecture.  Scientific Reports, 9(1), August 2019.
  * Shifaz et al. [2020] Ahmed Shifaz, Charlotte Pelletier, F. Petitjean, and Geoffrey I. Webb.  Ts-chief: a scalable and accurate forest algorithm for time series classification.  Data Mining and Knowledge Discovery, 34:742–775, 2020.
  * Silva et al. [2020] M. Silva, Eduardo Feijão, Ricardo da Cruz de Carvalho, Irina A. Duarte, A. Matos, M. T. Cabrita, A. Barreiro, M. Lemos, S. Novais, João H Marques, I. Caçador, P. Reis-Santos, V. Fonseca, and B. Duarte.  Comfortably numb: Ecotoxicity of the non-steroidal anti-inflammatory drug ibuprofen on phaeodactylum tricornutum.  Marine environmental research, 161:105109, 2020.
  * van der Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(86):2579–2605, 2008.

## Appendix

##  Appendix A OPTOX $p$-values

Table 7: Kruskal-Wallis p-values comparing all methods of the contaminant
prediction task of the OPTOX dataset. Above the diagonal, we have the results
for the test set, and below for the training set. Significant differences
($p<0.01$) are represented in green or red when the method on the left is
significantly better or worse, respectively.

| RF | XG | ROCKET | ANN | 1D CNN | CNN Log | CNN |
---|---|---|---|---|---|---|---|---
RF | ——– | $\mathit{1.23e^{-10}}$ | $\mathit{2.58e^{-11}}$ | $\mathit{2.80e^{-11}}$ | $\mathit{1.89e^{-08}}$ | $\mathit{2.68e^{-11}}$ |  $\mathit{2.63e^{-11}}$ Test |
XG | $2.24e^{-11}$ | ——– | $\mathit{2.60e^{-11}}$ | $\mathit{2.82e^{-11}}$ | $\mathit{1.01e^{-07}}$ | $\mathit{2.70e^{-11}}$ | $\mathit{2.65e^{-11}}$ |
ROCKET | $1.88e^{-11}$ | $1.08e^{-05}$ | ——– | $2.59e^{-11}$ | $4.04e^{-03}$ | $\mathit{5.79e^{-04}}$ | $3.64e^{-02}$ |
ANN | $\mathit{2.82e^{-11}}$ | $\mathit{2.26e^{-11}}$ | $\mathit{1.90e^{-11}}$ | ——– | $\mathit{7.04e^{-04}}$ | $\mathit{2.69e^{-11}}$ | $\mathit{2.64e^{-11}}$ |
1D CNN | $2.34e^{-11}$ | $8.89e^{-02}$ | $8.08e^{-01}$ | $2.36e^{-11}$ | ——– | $\mathit{2.68e^{-06}}$ | $\mathit{3.19e^{-04}}$ |
CNN Log | $1.63e^{-09}$ | $\mathit{2.07e^{-11}}$ | $\mathit{1.73e^{-11}}$ | $2.61e^{-11}$ | $\mathit{6.55e^{-09}}$ | ——– | $6.93e^{-03}$ |
CNN | $2.54e^{-11}$ | $\mathit{2.03e^{-11}}$ | $\mathit{1.70e^{-11}}$ | $2.56e^{-11}$ | $\mathit{3.09e^{-08}}$ | $2.55e^{-02}$ | ——– |
| Training |

##  Appendix B FISIO $p$-values

Table 8: Kruskal-Wallis p-values comparing all methods of the BMI_Siamese
prediction task. Above the diagonal, we have the results for the test set, and
below for the training set. Significant results ($p<0.01$) are represented in
green or red when the method on the left is significantly better or worse,
respectively.

BMI_Siamese | ANN | 1D CNN | CNN |
---|---|---|---|---
ANN | ——– | $8.31e^{-10}$ |  $4.76e^{-01}$ Test |
1D CNN | $1.12e^{-11}$ | ——– | $\mathit{1.32e^{-09}}$ |
CNN | $1.04e^{-12}$ | $6.33e^{-04}$ | ——– |
| Training |

Table 9: Kruskal-Wallis p-values comparing all methods of the BMI_HR
prediction task. Above the diagonal, we have the results for the test set, and
below for the training set. Significant results ($p<0.01$) are represented in
green or red when the method on the left is significantly better or worse,
respectively.

BMI_HR | RF | XG | Rocket | ANN | 1D CNN | CNN |
---|---|---|---|---|---|---|---
RF | ——– | $9.94e^{-01}$ | $2.34e^{-08}$ | $7.81e^{-07}$ | $2.25e^{-11}$ |  $6.55e^{-01}$ Test |
XG | $1.00e^{+00}$ | ——– | $3.11e^{-08}$ | $2.96e^{-06}$ | $2.24e^{-11}$ | $7.32e^{-01}$ |
Rocket | $1.10e^{-12}$ | $1.12e^{-12}$ | ——– | $\mathit{3.52e^{-04}}$ | $7.29e^{-05}$ | $\mathit{1.48e^{-08}}$ |
ANN | $5.89e^{-08}$ | $1.30e^{-10}$ | $\mathit{1.06e^{-12}}$ | ——– | $1.22e^{-10}$ | $\mathit{1.86e^{-08}}$ |
1D CNN | $8.88e^{-12}$ | $7.31e^{-12}$ | $\mathit{5.35e^{-03}}$ | $8.18e^{-12}$ | ——– | $\mathit{1.96e^{-11}}$ |
CNN | $1.10e^{-12}$ | $1.12e^{-12}$ | $1.00e^{+00}$ | $1.06e^{-12}$ | $5.35e^{-03}$ | ——– |
| Training |

Table 10: Kruskal-Wallis p-values comparing all methods of the BMI_VEN
prediction task. Above the diagonal, we have the results for the test set, and
below for the training set. Significant results ($p<0.01$) are represented in
green or red when the method on the left is significantly better or worse,
respectively.

BMI_VEN | RF | XG | Rocket | ANN | 1D CNN | CNN |
---|---|---|---|---|---|---|---
RF | ——– | $9.11e^{-01}$ | $2.33e^{-07}$ | $4.21e^{-03}$ | $8.82e^{-11}$ |  $3.97e^{-01}$ Test |
XG | $4.69e^{-01}$ | ——– | $3.99e^{-07}$ | $4.83e^{-03}$ | $2.31e^{-10}$ | $4.06e^{-01}$ |
Rocket | $1.13e^{-12}$ | $1.12e^{-12}$ | ——– | $\mathit{6.07e^{-04}}$ | $3.41e^{-05}$ | $\mathit{8.08e^{-08}}$ |
ANN | $1.34e^{-02}$ | $5.64e^{-05}$ | $\mathit{1.07e^{-12}}$ | ——– | $3.52e^{-09}$ | $\mathit{9.34e^{-03}}$ |
1D CNN | $5.65e^{-11}$ | $1.04e^{-10}$ | $1.54e^{-01}$ | $1.13e^{-09}$ | ——– | $\mathit{2.73e^{-11}}$ |
CNN | $1.13e^{-12}$ | $1.12e^{-12}$ | $1.00e^{+00}$ | $1.07e^{-12}$ | $1.54e^{-01}$ | ——– |
| Training |

Table 11: Kruskal-Wallis p-values comparing all methods of the BMI_HR+VEN
prediction task. Above the diagonal, we have the results for the test set, and
below for the training set. Significant results ($p<0.01$) are represented in
green or red when the method on the left is significantly better or worse,
respectively.

BMI_HR+VEN | RF | XG | Rocket | ANN | 1D CNN | CNN |
---|---|---|---|---|---|---|---
RF | ——– | $8.11e^{-01}$ | $1.77e^{-08}$ | $2.21e^{-05}$ | $2.57e^{-11}$ |  $4.28e^{-01}$ Test |
XG | $8.94e^{-01}$ | ——– | $3.47e^{-09}$ | $2.72e^{-06}$ | $2.49e^{-11}$ | $3.31e^{-01}$ |
Rocket | $1.12e^{-12}$ | $4.28e^{-12}$ | ——– | $\mathit{8.03e^{-04}}$ | $2.60e^{-11}$ | $\mathit{4.55e^{-10}}$ |
ANN | $1.25e^{-03}$ | $7.09e^{-07}$ | $\mathit{1.08e^{-12}}$ | ——– | $2.44e^{-11}$ | $\mathit{7.22e^{-07}}$ |
1D CNN | $\mathit{2.66e^{-11}}$ | $\mathit{2.67e^{-11}}$ | $\mathit{1.06e^{-12}}$ | $\mathit{2.57e^{-11}}$ | ——– | $\mathit{1.69e^{-11}}$ |
CNN | $1.12e^{-12}$ | $4.28e^{-12}$ | $1.00e^{+00}$ | $1.08e^{-12}$ | $1.06e^{-12}$ | ——– |
| Training |

Table 12: Kruskal-Wallis p-values comparing all methods of the Age_Siamese
prediction task. Above the diagonal, we have the results for the test set, and
below for the training set. Significant results ($p<0.01$) are represented in
green or red when the method on the left is significantly better or worse,
respectively.

Age_Siamese | ANN | 1D CNN | CNN |
---|---|---|---|---
ANN | ——– | $5.11e^{-01}$ |  $3.13e^{-01}$ Test |
1D CNN | $2.10e^{-11}$ | ——– | $6.33e^{-02}$ |
CNN | $1.12e^{-12}$ | $5.09e^{-06}$ | ——– |
| Training |

Table 13: Kruskal-Wallis p-values comparing all methods of the Age_HR
prediction task. Above the diagonal, we have the results for the test set, and
below for the training set. Significant results ($p<0.01$) are represented in
green or red when the method on the left is significantly better or worse,
respectively.

Age_HR | RF | XG | Rocket | ANN | 1D CNN | CNN |
---|---|---|---|---|---|---|---
RF | ——– | $6.51e^{-01}$ | $8.99e^{-01}$ | $3.82e^{-01}$ | $1.51e^{-02}$ |  $\mathit{6.23e^{-08}}$ Test |
XG | $1.51e^{-01}$ | ——– | $6.71e^{-01}$ | $1.55e^{-01}$ | $3.41e^{-02}$ | $\mathit{1.76e^{-08}}$ |
Rocket | $1.55e^{-11}$ | $1.53e^{-11}$ | ——– | $2.54e^{-01}$ | $2.75e^{-03}$ | $\mathit{1.55e^{-09}}$ |
ANN | $\mathit{1.18e^{-05}}$ | $\mathit{2.37e^{-07}}$ | $\mathit{1.08e^{-12}}$ | ——– | $1.35e^{-03}$ | $\mathit{4.58e^{-07}}$ |
1D CNN | $4.05e^{-09}$ | $3.04e^{-09}$ | $\mathit{5.32e^{-03}}$ | $8.64e^{-12}$ | ——– | $\mathit{8.02e^{-11}}$ |
CNN | $1.55e^{-11}$ | $1.53e^{-11}$ | $1.00e^{+00}$ | $1.08e^{-12}$ | $5.32e^{-03}$ | ——– |
| Training |

Table 14: Kruskal-Wallis p-values comparing all methods of the Age_VEN
prediction task. Above the diagonal, we have the results for the test set, and
below for the training set. Significant results ($p<0.01$) are represented in
green or red when the method on the left is significantly better or worse,
respectively.

Age_VEN | RF | XG | Rocket | ANN | 1D CNN | CNN |
---|---|---|---|---|---|---|---
RF | ——– | $3.25e^{-01}$ | $6.64e^{-02}$ | $6.57e^{-01}$ | $4.39e^{-02}$ |  $\mathit{7.97e^{-04}}$ Test |
XG | $2.71e^{-01}$ | ——– | $1.19e^{-03}$ | $1.16e^{-01}$ | $2.08e^{-04}$ | $\mathit{2.03e^{-04}}$ |
Rocket | $1.52e^{-11}$ | $3.29e^{-07}$ | ——– | $1.20e^{-01}$ | $7.00e^{-01}$ | $\mathit{5.56e^{-08}}$ |
ANN | $2.13e^{-01}$ | $3.90e^{-02}$ | $\mathit{1.06e^{-12}}$ | ——– | $8.67e^{-02}$ | $\mathit{1.68e^{-05}}$ |
1D CNN | $1.63e^{-07}$ | $4.54e^{-04}$ | $\mathit{5.36e^{-03}}$ | $5.27e^{-10}$ | ——– | $\mathit{6.99e^{-09}}$ |
CNN | $1.52e^{-11}$ | $3.29e^{-07}$ | $1.00e^{+00}$ | $1.06e^{-12}$ | $5.36e^{-03}$ | ——– |
| Training |

Table 15: Kruskal-Wallis p-values comparing all methods of the Age_HR+VEN
prediction task. Above the diagonal, we have the results for the test set, and
below for the training set. Significant results ($p<0.01$) are represented in
green or red when the method on the left is significantly better or worse,
respectively.

Age_HR+VEN | RF | XG | Rocket | ANN | 1D CNN | CNN |
---|---|---|---|---|---|---|---
RF | ——– | $\mathit{7.88e^{-03}}$ | $\mathit{6.68e^{-03}}$ | $\mathit{7.65e^{-04}}$ | $6.61e^{-02}$ |  $\mathit{3.58e^{-04}}$ Test |
XG | $1.99e^{-03}$ | ——– | $5.02e^{-01}$ | $1.68e^{-01}$ | $5.63e^{-01}$ | $1.23e^{-01}$ |
Rocket | $1.79e^{-10}$ | $3.00e^{-04}$ | ——– | $5.73e^{-01}$ | $9.10e^{-02}$ | $3.58e^{-01}$ |
ANN | $8.33e^{-02}$ | $\mathit{2.91e^{-05}}$ | $\mathit{1.10e^{-12}}$ | ——– | $8.70e^{-03}$ | $9.06e^{-01}$ |
1D CNN | $2.25e^{-08}$ | $1.16e^{-02}$ | $7.82e^{-02}$ | $7.44e^{-11}$ | ——– | $\mathit{5.57e^{-03}}$ |
CNN | $1.79e^{-10}$ | $3.00e^{-04}$ | $1.00e^{+00}$ | $1.10e^{-12}$ | $7.82e^{-02}$ | ——– |
| Training |

##  Appendix C UCR $p$-values

Table 16: Kruskal-Wallis p-values comparing the test results of all methods on
the BME prediction task.
Significant results ($p<0.01$) are represented in green or red when the method
on the left is significantly better or worse, respectively.

BME | TS-CHIEF |  HIVE-COTE v1.0 | ROCKET |  Inception Time | STC | ResNet |  Proximity Forest | WEASEL | S-BOSS | cBOSS | BOSS | RISE | TSF | Catch22 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CNN | $1.26e^{-01}$ | $1.35e^{-04}$ | $1.20e^{-01}$ | $3.81e^{-01}$ | $3.02e^{-11}$ | $\mathit{1.02e^{-03}}$ | $\mathit{5.50e^{-03}}$ | $9.16e^{-08}$ | $3.06e^{-11}$ | $3.05e^{-11}$ | $3.08e^{-11}$ | $3.07e^{-11}$ | $4.74e^{-09}$ |  $3.07e^{-11}$ Test |
TS-CHIEF | ——– | $2.75e^{-06}$ | $8.53e^{-01}$ | $3.24e^{-01}$ | $3.18e^{-11}$ | $1.49e^{-02}$ | $1.71e^{-01}$ | $1.10e^{-08}$ | $2.10e^{-11}$ | $2.09e^{-11}$ | $2.11e^{-11}$ | $1.99e^{-11}$ | $4.14e^{-10}$ | $2.35e^{-11}$ |
HIVE-COTE v1.0 | ——– | ——– | $\mathit{1.34e^{-06}}$ | $\mathit{9.38e^{-06}}$ | $8.76e^{-10}$ | $\mathit{5.93e^{-09}}$ | $\mathit{1.28e^{-08}}$ | $6.09e^{-04}$ | $8.83e^{-11}$ | $6.46e^{-11}$ | $7.61e^{-11}$ | $5.55e^{-11}$ | $1.12e^{-03}$ | $1.72e^{-10}$ |
ROCKET | ——– | ——– | ——– | $3.96e^{-01}$ | $2.33e^{-11}$ | $\mathit{7.95e^{-03}}$ | $1.10e^{-01}$ | $6.19e^{-09}$ | $2.24e^{-11}$ | $2.23e^{-11}$ | $2.25e^{-11}$ | $2.24e^{-11}$ | $7.95e^{-11}$ | $2.24e^{-11}$ |
InceptionTime | ——– | ——– | ——– | ——– | $3.19e^{-11}$ | $\mathit{9.81e^{-04}}$ | $1.66e^{-02}$ | $2.06e^{-08}$ | $3.06e^{-11}$ | $3.05e^{-11}$ | $3.08e^{-11}$ | $3.07e^{-11}$ | $2.45e^{-10}$ | $3.07e^{-11}$ |
STC | ——– | ——– | ——– | ——– | ——– | $\mathit{3.95e^{-12}}$ | $\mathit{8.82e^{-12}}$ | $1.03e^{-02}$ | $4.95e^{-06}$ | $5.21e^{-09}$ | $2.19e^{-06}$ | $7.59e^{-11}$ | $\mathit{1.84e^{-04}}$ | $7.67e^{-03}$ |
ResNet | ——– | ——– | ——– | ——– | ——– | ——– | $1.85e^{-01}$ | $1.54e^{-10}$ | $3.79e^{-12}$ | $3.78e^{-12}$ | $3.82e^{-12}$ | $3.80e^{-12}$ | $1.48e^{-11}$ | $3.81e^{-12}$ |
ProximityForest | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $3.76e^{-10}$ | $8.97e^{-12}$ | $8.93e^{-12}$ | $9.03e^{-12}$ | $8.97e^{-12}$ | $8.78e^{-12}$ | $8.99e^{-12}$ |
WEASEL | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $4.93e^{-07}$ | $5.84e^{-09}$ | $2.38e^{-07}$ | $9.13e^{-10}$ | $4.13e^{-01}$ | $4.82e^{-05}$ |
S-BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.58e^{-04}$ | $9.56e^{-01}$ | $3.90e^{-06}$ | $\mathit{1.59e^{-09}}$ | $\mathit{4.65e^{-03}}$ |
cBOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{1.76e^{-05}}$ | $8.39e^{-01}$ | $\mathit{1.99e^{-10}}$ | $\mathit{1.05e^{-07}}$ |
BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.01e^{-07}$ | $\mathit{7.00e^{-10}}$ | $\mathit{2.78e^{-03}}$ |
RISE | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{5.83e^{-11}}$ | $\mathit{8.62e^{-10}}$ |
TSF | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.00e^{-07}$ |

Table 17: Kruskal-Wallis p-values comparing the test results of all methods on
the Coffee prediction task.
Significant results ($p<0.01$) are represented in green or red when the method
on the left is significantly better or worse, respectively.

Coffee | TS-CHIEF |  HIVE-COTE v1.0 | ROCKET |  Inception Time | STC | ResNet |  Proximity Forest | WEASEL | S-BOSS | cBOSS | BOSS | RISE | TSF | Catch22 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CNN | $2.53e^{-03}$ | $2.05e^{-02}$ | $1.00e^{+00}$ | $3.17e^{-01}$ | $5.20e^{-03}$ | $7.79e^{-02}$ | $1.04e^{-02}$ | $2.56e^{-03}$ | $2.25e^{-05}$ | $2.53e^{-03}$ | $5.90e^{-04}$ | $5.90e^{-04}$ | $1.25e^{-03}$ |  $2.32e^{-05}$ Test |
TS-CHIEF | ——– | $3.96e^{-01}$ | $\mathit{2.53e^{-03}}$ | $1.18e^{-02}$ | $8.30e^{-01}$ | $9.69e^{-02}$ | $6.01e^{-01}$ | $9.36e^{-01}$ | $8.17e^{-02}$ | $1.00e^{+00}$ | $4.76e^{-01}$ | $4.76e^{-01}$ | $3.27e^{-01}$ | $7.07e^{-02}$ |
HIVE-COTE v1.0 | ——– | ——– | $2.05e^{-02}$ | $8.49e^{-02}$ | $5.29e^{-01}$ | $4.28e^{-01}$ | $7.49e^{-01}$ | $3.68e^{-01}$ | $1.50e^{-02}$ | $3.96e^{-01}$ | $1.40e^{-01}$ | $1.37e^{-01}$ | $1.38e^{-01}$ | $1.34e^{-02}$ |
ROCKET | ——– | ——– | ——– | $3.17e^{-01}$ | $5.20e^{-03}$ | $7.79e^{-02}$ | $1.04e^{-02}$ | $2.56e^{-03}$ | $2.25e^{-05}$ | $2.53e^{-03}$ | $5.90e^{-04}$ | $5.90e^{-04}$ | $1.25e^{-03}$ | $2.32e^{-05}$ |
InceptionTime | ——– | ——– | ——– | ——– | $2.29e^{-02}$ | $3.04e^{-01}$ | $4.45e^{-02}$ | $1.16e^{-02}$ | $1.11e^{-04}$ | $1.18e^{-02}$ | $2.69e^{-03}$ | $2.69e^{-03}$ | $4.33e^{-03}$ | $1.10e^{-04}$ |
STC | ——– | ——– | ——– | ——– | ——– | $1.58e^{-01}$ | $7.56e^{-01}$ | $7.83e^{-01}$ | $6.37e^{-02}$ | $8.30e^{-01}$ | $3.84e^{-01}$ | $3.79e^{-01}$ | $3.08e^{-01}$ | $5.70e^{-02}$ |
ResNet | ——– | ——– | ——– | ——– | ——– | ——– | $2.65e^{-01}$ | $9.07e^{-02}$ | $1.45e^{-03}$ | $9.69e^{-02}$ | $2.51e^{-02}$ | $2.51e^{-02}$ | $2.70e^{-02}$ | $1.34e^{-03}$ |
ProximityForest | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $5.59e^{-01}$ | $3.18e^{-02}$ | $6.01e^{-01}$ | $2.40e^{-01}$ | $2.36e^{-01}$ | $2.13e^{-01}$ | $2.80e^{-02}$ |
WEASEL | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.08e^{-01}$ | $9.36e^{-01}$ | $5.39e^{-01}$ | $5.32e^{-01}$ | $4.21e^{-01}$ | $9.42e^{-02}$ |
S-BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $8.17e^{-02}$ | $3.35e^{-01}$ | $3.44e^{-01}$ | $6.26e^{-01}$ | $9.03e^{-01}$ |
cBOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $4.76e^{-01}$ | $4.76e^{-01}$ | $3.27e^{-01}$ | $7.07e^{-02}$ |
BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $9.85e^{-01}$ | $8.15e^{-01}$ | $2.94e^{-01}$ |
RISE | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $8.15e^{-01}$ | $3.06e^{-01}$ |
TSF | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $5.18e^{-01}$ |

Table 18: Kruskal-Wallis p-values comparing the test results of all methods on
the Earthquakes prediction task.
Significant results ($p<0.01$) are represented in green or red when the method
on the left is significantly better or worse, respectively.

Earthquakes | TS-CHIEF |  HIVE-COTE v1.0 | ROCKET |  Inception Time | STC | ResNet |  Proximity Forest | WEASEL | S-BOSS | cBOSS | BOSS | RISE | TSF | Catch22 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CNN | $1.47e^{-03}$ | $1.54e^{-03}$ | $1.59e^{-03}$ | $1.38e^{-05}$ | $4.22e^{-04}$ | $2.46e^{-08}$ | $7.31e^{-03}$ | $1.54e^{-03}$ | $1.54e^{-03}$ | $1.54e^{-03}$ | $1.49e^{-03}$ | $1.47e^{-03}$ | $1.72e^{-03}$ |  $3.81e^{-04}$ Test |
TS-CHIEF | ——– | $7.79e^{-02}$ | $3.17e^{-01}$ | $5.40e^{-06}$ | $3.96e^{-06}$ | $2.53e^{-10}$ | $3.27e^{-02}$ | $7.79e^{-02}$ | $7.79e^{-02}$ | $7.79e^{-02}$ | $2.56e^{-03}$ | $1.00e^{+00}$ | $\mathit{1.50e^{-07}}$ | $5.73e^{-07}$ |
HIVE-COTE v1.0 | ——– | ——– | $4.54e^{-02}$ | $3.81e^{-05}$ | $2.42e^{-04}$ | $1.09e^{-09}$ | $1.92e^{-02}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $9.07e^{-02}$ | $7.79e^{-02}$ | $\mathit{2.69e^{-07}}$ | $1.06e^{-05}$ |
ROCKET | ——– | ——– | ——– | $5.54e^{-06}$ | $3.43e^{-06}$ | $3.28e^{-10}$ | $5.32e^{-02}$ | $4.54e^{-02}$ | $4.54e^{-02}$ | $4.54e^{-02}$ | $1.76e^{-03}$ | $3.17e^{-01}$ | $\mathit{1.71e^{-06}}$ | $5.64e^{-07}$ |
InceptionTime | ——– | ——– | ——– | ——– | $3.43e^{-02}$ | $7.25e^{-03}$ | $\mathit{1.84e^{-04}}$ | $\mathit{3.81e^{-05}}$ | $\mathit{3.81e^{-05}}$ | $\mathit{3.81e^{-05}}$ | $\mathit{6.29e^{-04}}$ | $\mathit{5.40e^{-06}}$ | $\mathit{4.38e^{-06}}$ | $2.92e^{-01}$ |
STC | ——– | ——– | ——– | ——– | ——– | $7.84e^{-07}$ | $\mathit{4.54e^{-04}}$ | $\mathit{2.42e^{-04}}$ | $\mathit{2.42e^{-04}}$ | $\mathit{2.42e^{-04}}$ | $2.55e^{-02}$ | $\mathit{3.96e^{-06}}$ | $\mathit{1.96e^{-07}}$ | $1.13e^{-01}$ |
ResNet | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{5.97e^{-08}}$ | $\mathit{1.09e^{-09}}$ | $\mathit{1.09e^{-09}}$ | $\mathit{1.09e^{-09}}$ | $\mathit{8.98e^{-09}}$ | $\mathit{2.53e^{-10}}$ | $\mathit{1.55e^{-09}}$ | $\mathit{1.82e^{-05}}$ |
ProximityForest | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.92e^{-02}$ | $1.92e^{-02}$ | $1.92e^{-02}$ | $6.54e^{-03}$ | $3.27e^{-02}$ | $6.34e^{-01}$ | $1.73e^{-04}$ |
WEASEL | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $9.07e^{-02}$ | $7.79e^{-02}$ | $\mathit{2.69e^{-07}}$ | $1.06e^{-05}$ |
S-BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $9.07e^{-02}$ | $7.79e^{-02}$ | $\mathit{2.69e^{-07}}$ | $1.06e^{-05}$ |
cBOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $9.07e^{-02}$ | $7.79e^{-02}$ | $\mathit{2.69e^{-07}}$ | $1.06e^{-05}$ |
BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{2.56e^{-03}}$ | $\mathit{3.59e^{-07}}$ | $6.30e^{-04}$ |
RISE | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{1.50e^{-07}}$ | $5.73e^{-07}$ |
TSF | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.26e^{-07}$ |

Table 19: Kruskal-Wallis p-values comparing the test results of all methods on
the Plane prediction task.
Significant results ($p<0.01$) are represented in green or red when the method
on the left is significantly better or worse, respectively.

Plane | TS-CHIEF |  HIVE-COTE v1.0 | ROCKET |  Inception Time | STC | ResNet |  Proximity Forest | WEASEL | S-BOSS | cBOSS | BOSS | RISE | TSF | Catch22 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CNN | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.04e^{-02}$ | $7.79e^{-02}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $2.53e^{-03}$ | $5.16e^{-03}$ | $1.00e^{+00}$ | $1.03e^{-02}$ | $2.59e^{-03}$ | $5.97e^{-04}$ |  $3.01e^{-09}$ Test |
TS-CHIEF | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.04e^{-02}$ | $7.79e^{-02}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $2.53e^{-03}$ | $5.16e^{-03}$ | $1.00e^{+00}$ | $1.03e^{-02}$ | $2.59e^{-03}$ | $5.97e^{-04}$ | $3.01e^{-09}$ |
HIVE-COTE v1.0 | ——– | ——– | $1.00e^{+00}$ | $1.04e^{-02}$ | $7.79e^{-02}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $2.53e^{-03}$ | $5.16e^{-03}$ | $1.00e^{+00}$ | $1.03e^{-02}$ | $2.59e^{-03}$ | $5.97e^{-04}$ | $3.01e^{-09}$ |
ROCKET | ——– | ——– | ——– | $1.04e^{-02}$ | $7.79e^{-02}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $2.53e^{-03}$ | $5.16e^{-03}$ | $1.00e^{+00}$ | $1.03e^{-02}$ | $2.59e^{-03}$ | $5.97e^{-04}$ | $3.01e^{-09}$ |
InceptionTime | ——– | ——– | ——– | ——– | $2.21e^{-01}$ | $1.04e^{-02}$ | $1.04e^{-02}$ | $4.40e^{-01}$ | $9.91e^{-01}$ | $1.04e^{-02}$ | $7.92e^{-01}$ | $6.48e^{-01}$ | $3.42e^{-01}$ | $9.20e^{-05}$ |
STC | ——– | ——– | ——– | ——– | ——– | $7.79e^{-02}$ | $7.79e^{-02}$ | $5.40e^{-02}$ | $1.68e^{-01}$ | $7.79e^{-02}$ | $2.81e^{-01}$ | $7.92e^{-02}$ | $1.29e^{-02}$ | $1.44e^{-07}$ |
ResNet | ——– | ——– | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $2.53e^{-03}$ | $5.16e^{-03}$ | $1.00e^{+00}$ | $1.03e^{-02}$ | $2.59e^{-03}$ | $5.97e^{-04}$ | $3.01e^{-09}$ |
ProximityForest | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $2.53e^{-03}$ | $5.16e^{-03}$ | $1.00e^{+00}$ | $1.03e^{-02}$ | $2.59e^{-03}$ | $5.97e^{-04}$ | $3.01e^{-09}$ |
WEASEL | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $3.89e^{-01}$ | $\mathit{2.53e^{-03}}$ | $2.71e^{-01}$ | $6.91e^{-01}$ | $8.34e^{-01}$ | $4.66e^{-03}$ |
S-BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{5.16e^{-03}}$ | $7.55e^{-01}$ | $6.11e^{-01}$ | $1.28e^{-01}$ | $7.64e^{-06}$ |
cBOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.03e^{-02}$ | $2.59e^{-03}$ | $5.97e^{-04}$ | $3.01e^{-09}$ |
BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $4.28e^{-01}$ | $8.16e^{-02}$ | $3.14e^{-06}$ |
RISE | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $4.59e^{-01}$ | $1.95e^{-04}$ |
TSF | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $7.44e^{-03}$ |

Table 20: Kruskal-Wallis p-values comparing the test results of all methods on
the SmoothSubspace prediction task.
Significant results ($p<0.01$) are represented in green or red when the method
on the left is significantly better or worse, respectively.

SmoothSubspace | TS-CHIEF |  HIVE-COTE v1.0 | ROCKET |  Inception Time | STC | ResNet |  Proximity Forest | WEASEL | S-BOSS | cBOSS | BOSS | RISE | TSF | Catch22 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CNN | $9.89e^{-03}$ | $2.53e^{-09}$ | $5.03e^{-12}$ | $6.10e^{-12}$ | $4.99e^{-12}$ | $2.49e^{-05}$ | $3.72e^{-02}$ | $5.03e^{-12}$ | $5.21e^{-12}$ | $5.31e^{-12}$ | $5.13e^{-12}$ | $5.16e^{-12}$ | $8.93e^{-10}$ |  $5.09e^{-12}$ Test |
TS-CHIEF | ——– | $1.89e^{-06}$ | $1.04e^{-10}$ | $1.53e^{-09}$ | $2.54e^{-11}$ | $2.31e^{-02}$ | $5.83e^{-01}$ | $2.43e^{-11}$ | $2.50e^{-11}$ | $2.54e^{-11}$ | $2.47e^{-11}$ | $2.48e^{-11}$ | $3.84e^{-08}$ | $2.45e^{-11}$ |
HIVE-COTE v1.0 | ——– | ——– | $1.49e^{-04}$ | $3.11e^{-01}$ | $1.14e^{-10}$ | $\mathit{7.71e^{-03}}$ | $\mathit{3.46e^{-07}}$ | $5.10e^{-11}$ | $5.25e^{-11}$ | $5.33e^{-11}$ | $5.18e^{-11}$ | $5.20e^{-11}$ | $7.18e^{-01}$ | $5.15e^{-11}$ |
ROCKET | ——– | ——– | ——– | $\mathit{1.79e^{-04}}$ | $5.99e^{-09}$ | $\mathit{9.71e^{-08}}$ | $\mathit{3.73e^{-11}}$ | $5.09e^{-11}$ | $5.24e^{-11}$ | $5.32e^{-11}$ | $5.17e^{-11}$ | $5.19e^{-11}$ | $\mathit{1.24e^{-05}}$ | $5.14e^{-11}$ |
InceptionTime | ——– | ——– | ——– | ——– | $7.86e^{-11}$ | $\mathit{1.02e^{-04}}$ | $\mathit{2.56e^{-10}}$ | $4.07e^{-11}$ | $4.20e^{-11}$ | $4.26e^{-11}$ | $4.14e^{-11}$ | $4.16e^{-11}$ | $3.85e^{-02}$ | $4.11e^{-11}$ |
STC | ——– | ——– | ——– | ——– | ——– | $\mathit{6.29e^{-11}}$ | $\mathit{1.88e^{-11}}$ | $1.48e^{-10}$ | $5.72e^{-11}$ | $5.81e^{-11}$ | $5.65e^{-11}$ | $5.97e^{-11}$ | $\mathit{1.10e^{-10}}$ | $8.95e^{-11}$ |
ResNet | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{6.32e^{-03}}$ | $4.41e^{-11}$ | $4.54e^{-11}$ | $4.61e^{-11}$ | $4.48e^{-11}$ | $4.50e^{-11}$ | $2.60e^{-03}$ | $4.45e^{-11}$ |
ProximityForest | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.89e^{-11}$ | $1.95e^{-11}$ | $1.99e^{-11}$ | $1.93e^{-11}$ | $1.94e^{-11}$ | $1.82e^{-08}$ | $1.91e^{-11}$ |
WEASEL | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $5.76e^{-11}$ | $5.85e^{-11}$ | $5.68e^{-11}$ | $3.24e^{-01}$ | $\mathit{4.88e^{-11}}$ | $7.37e^{-01}$ |
S-BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $6.50e^{-02}$ | $5.22e^{-01}$ | $\mathit{5.88e^{-11}}$ | $\mathit{5.02e^{-11}}$ | $\mathit{5.81e^{-11}}$ |
cBOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $2.87e^{-02}$ | $\mathit{5.96e^{-11}}$ | $\mathit{5.10e^{-11}}$ | $\mathit{5.90e^{-11}}$ |
BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{5.80e^{-11}}$ | $\mathit{4.96e^{-11}}$ | $\mathit{5.73e^{-11}}$ |
RISE | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{4.98e^{-11}}$ | $3.65e^{-01}$ |
TSF | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $4.93e^{-11}$ |

Table 21: Kruskal-Wallis p-values comparing the test results of all methods on
the Trace prediction task.
Significant results ($p<0.01$) are represented in green or red when the method
on the left is significantly better or worse, respectively.

Trace | TS-CHIEF |  HIVE-COTE v1.0 | ROCKET |  Inception Time | STC | ResNet |  Proximity Forest | WEASEL | S-BOSS | cBOSS | BOSS | RISE | TSF | Catch22 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CNN | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ |  $3.17e^{-01}$ Test |
TS-CHIEF | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
HIVE-COTE v1.0 | ——– | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
ROCKET | ——– | ——– | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
InceptionTime | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
STC | ——– | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
ResNet | ——– | ——– | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
ProximityForest | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
WEASEL | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
S-BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
cBOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.00e^{+00}$ | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
BOSS | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $3.63e^{-09}$ | $2.61e^{-03}$ | $3.17e^{-01}$ |
RISE | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $\mathit{1.25e^{-03}}$ | $\mathit{1.27e^{-08}}$ |
TSF | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | ——– | $1.02e^{-02}$ |

##  Appendix D UCR problems

Table 22: Ranking by dataset from the set of algorithms present in UCR and our CNN methodology on the selected 98 UCR problems. | $1^{st}$ | $2^{nd}$
---|---|---
ACSF1 | ResNet (0.9300) | Over 2 methods (0.9200)
Adiac | InceptionTime (0.8517) | ResNet (0.8491)
ArrowHead | TS-CHIEF & ProximityForest (0.9486) |
Beef | STC (1.0000) | WEASEL & RISE (0.9333)
BeetleFly | Over 2 methods (1.0000) |
BirdChicken | Over 2 methods (1.0000) |
BME |  Over 2 methods (1.0000) |
Car | InceptionTime (0.9667) | Over 2 methods (0.9500)
CBF | Over 2 methods (1.0000) |
Chinatown | InceptionTime (0.9883) | ResNet (0.9855)
ChlorineConcentration | InceptionTime (0.8930) | ResNet (0.8703)
CinCECGTorso | HIVE-COTE v1.0 (1.0000) | WEASEL (0.9993)
Coffee |  Over 2 methods (1.0000) |
Computers | InceptionTime (0.9040) | ResNet (0.8920)
Crop | InceptionTime (0.7968) | HIVE-COTE v1.0 (0.7730)
DiatomSizeReduction | InceptionTime (1.0000) | ROCKET & BOSS (0.9935)
DistalPhalanxOutlineAgeGroup | S-BOSS (0.9065) | Over 2 methods (0.8993)
DistalPhalanxOutlineCorrect | ROCKET & ProximityForest (0.8696) |
DistalPhalanxTW | Over 2 methods (0.7410) |
Earthquakes |  CNN (0.7770) | ProximityForest (0.7698)
ECG200 | ResNet (0.9500) | ROCKET & InceptionTime (0.9400)
ECG5000 | TS-CHIEF (0.9520) | WEASEL (0.9513)
ECGFiveDays | Over 2 methods (1.0000) |
ElectricDevices | ROCKET (0.9046) | InceptionTime (0.9042)
EthanolLevel | ResNet (0.9080) | InceptionTime (0.9040)
FaceAll | InceptionTime & ResNet (0.9941) |
FaceFour | Over 2 methods (1.0000) |
FacesUCR | InceptionTime (0.9859) | TS-CHIEF (0.9839)
FiftyWords | TS-CHIEF & InceptionTime (0.8659) |
Fish | Over 2 methods (1.0000) |
FordA | WEASEL (0.9765) | InceptionTime (0.9689)
FreezerRegularTrain | HIVE-COTE v1.0 & STC (1.0000) |
FreezerSmallTrain | HIVE-COTE v1.0 & STC (1.0000) |
GunPoint | Over 2 methods (1.0000) |
GunPointAgeSpan | Over 2 methods (1.0000) |
GunPointMaleVersusFemale | Over 2 methods (1.0000) |
GunPointOldVersusYoung | Over 2 methods (1.0000) |
Ham | ProximityForest (1.0000) | ROCKET (0.9968)
Haptics | HIVE-COTE v1.0 (0.9486) | ResNet (0.9351)
Herring | Catch22 (0.9054) | S-BOSS (0.7344)
HouseTwenty | Over 2 methods (0.9916) |
InlineSkate | HIVE-COTE v1.0 (1.0000) | Over 2 methods (0.9916)
InsectEPGRegularTrain | Over 2 methods (1.0000) |
InsectEPGSmallTrain | Over 2 methods (1.0000) |
ItalyPowerDemand | TS-CHIEF & InceptionTime (0.9728) |
LargeKitchenAppliances | InceptionTime (0.9733) | ResNet (0.9718)
Lightning2 | ResNet (0.9760) | HIVE-COTE v1.0 (0.9520)
Lightning7 | Catch22 (0.9253) | ProximityForest (0.9180)
Mallat | TS-CHIEF (0.9979) | WEASEL (0.9851)
Meat | Over 2 methods (1.0000) |
MedicalImages | Over 2 methods (1.0000) |
MiddlePhalanxOutlineAgeGroup | Catch22 (0.9833) | ROCKET (0.8289)
MiddlePhalanxOutlineCorrect | STC (0.8694) | InceptionTime (0.8660)
MiddlePhalanxTW | ROCKET (0.8660) | ProximityForest (0.8625)
MixedShapesRegularTrain | TS-CHIEF (0.9810) | InceptionTime & WEASEL (0.9753)
MixedShapesSmallTrain | ResNet (0.9790) | HIVE-COTE v1.0 (0.9769)
MoteStrain | TS-CHIEF (0.9688) | HIVE-COTE v1.0 (0.9662)
OliveOil | Over 2 methods (1.0000) |
OSULeaf | ROCKET & ProximityForest (1.0000) |
PhalangesOutlinesCorrect | HIVE-COTE v1.0 (0.9623) | ROCKET (0.9587)
Phoneme | HIVE-COTE v1.0 & ResNet (0.9667) |
PigAirwayPressure | STC (0.9952) | HIVE-COTE v1.0 & ResNet (0.9876)
PigArtPressure | S-BOSS (1.0000) | Over 2 methods (0.9952)
PigCVP | TS-CHIEF (0.9856) | S-BOSS (0.9808)
Plane |  Over 2 methods (1.0000) |
ProximalPhalanxOutlineAgeGroup | ProximityForest (1.0000) | ROCKET (0.9889)
ProximalPhalanxOutlineCorrect | HIVE-COTE v1.0 & ResNet (1.0000) |
ProximalPhalanxTW | HIVE-COTE v1.0 & Catch22 (1.0000) |
RefrigerationDevices | Catch22 (0.9389) | HIVE-COTE v1.0 (0.8878)
Rock | STC (0.9600) | WEASEL (0.9400)
ScreenType | ROCKET (0.9000) | Catch22 (0.8660)
SemgHandGenderCh2 | TS-CHIEF (0.9683) | TSF (0.9650)
SemgHandMovementCh2 | ProximityForest (0.9800) | ROCKET (0.9500)
SemgHandSubjectCh2 | TSF (0.9533) | TS-CHIEF (0.9511)
ShapeletSim | Over 2 methods (1.0000) |
ShapesAll | ROCKET (1.0000) | InceptionTime (0.9600)
SmallKitchenAppliances | HIVE-COTE v1.0 (0.9689) | ROCKET (0.9417)
SmoothSubspace |  Over 2 methods (1.0000) |
SonyAIBORobotSurface1 | ProximityForest & Catch22 (1.0000) |
SonyAIBORobotSurface2 | ROCKET (0.9817) | InceptionTime (0.9748)
StarLightCurves | HIVE-COTE v1.0 & ResNet (1.0000) |
Strawberry | WEASEL (0.9919) | Over 2 methods (0.9892)
SwedishLeaf | ROCKET & ProximityForest (0.9919) |
Symbols | TS-CHIEF (0.9879) | InceptionTime (0.9859)
SyntheticControl | Over 2 methods (1.0000) |
ToeSegmentation1 | ROCKET & ProximityForest (1.0000) |
ToeSegmentation2 | TS-CHIEF & InceptionTime (0.9923) |
Trace |  Over 2 methods (1.0000) |
TwoLeadECG | Over 2 methods (1.0000) |
TwoPatterns | Over 2 methods (1.0000) |
UMD | Over 2 methods (1.0000) |
UWaveGestureLibraryAll | Over 2 methods (1.0000) |
Wafer | Over 2 methods (1.0000) |
Wine | Over 2 methods (1.0000) |
WordSynonyms | ROCKET & ProximityForest (1.0000) |
Worms | HIVE-COTE v1.0 & ResNet (1.0000) |
WormsTwoClass | HIVE-COTE v1.0 & ResNet (1.0000) |
Yoga | cBOSS (0.9373) | InceptionTime (0.9337)


# AdaProj: Adaptively Scaled Angular Margin Subspace Projections for Anomalous
Sound Detection with Auxiliary Classification Tasks

###### Abstract

The state-of-the-art approach for semi-supervised anomalous sound detection is
to first learn an embedding space by using auxiliary classification tasks
based on meta information or self-supervised learning and then estimate the
distribution of normal data. In this work, AdaProj a novel loss function is
presented. In contrast to commonly used angular margin losses, which project
data of each class as close as possible to their corresponding class centers,
AdaProj learns to project data onto class-specific subspaces. By doing so, the
resulting distributions of embeddings belonging to normal data are not
required to be as restrictive as other loss functions allowing a more detailed
view on the data. In experiments conducted on the DCASE2022 and DCASE2023
datasets, it is shown that using AdaProj to learn an embedding space
significantly outperforms other commonly used loss functions and results in a
state-of-the-art performance on the DCASE2023 dataset.

Index Terms—  machine listening, anomaly detection, representation learning,
domain generalization

##  1 Introduction

Semi-supervised anomaly detection is the task of training a system to
differentiate between normal and anomalous data using only normal training
samples [1]. An example application is acoustic machine condition monitoring
for predictive maintenance [2, 3]. Here, normal data corresponds to sounds of
fully functioning machines whereas anomalous sounds indicate mechanical
failure. One of the main difficulties to overcome in acoustic machine
condition monitoring is that it is practically impossible to record isolated
sounds of a target machine. Instead, recordings also contain many other sounds
emitted by non-target machines or humans. Compared to this complex acoustic
scene, anomalous signal components of the target machines are very subtle and
hard to detect without utilizing additional knowledge. Another main difficulty
is that a system should also be able to reliably detect anomalous sounds when
changing the acoustic conditions or machine settings without needing to
collect large amounts of data in these changed conditions or to re-train the
system (domain generalization [4]). One possibility to simultaneously overcome
both difficulties is to learn mapping audio signals into a fixed-dimensional
vector space, in which representations belonging to normal and anomalous data,
called embeddings, can be easily separated. Then, by estimating the
distribution of normal training samples in the embedding space, one can
compute an anomaly score for an unseen test sample to distinguish between
normal and anomalous samples.

To train such an embedding model, the state-of-the-art is to utilize an
auxiliary classification task using provided meta information or self-
supervised learning. This enables the embedding model to closely monitor
target signals and ignore other signals and noise [5]. For machine condition
monitoring, possible auxiliary tasks are classifying between machine types [6,
7, 8] or, additionally, between different machine states and noise settings
[9, 10, 11], recognizing augmented and non-augmented versions of normal data
[6, 12] or predicting the activity of machines [10]. Using an auxiliary task
to learn embeddings is also called outlier exposure (OE) [13] because normal
samples belonging to other classes than a target class can be considered proxy
outliers [14].

The contributions of this work are the following. First and foremost, AdaProj,
a novel angular margin loss function that learns class-specific subspaces for
training an embedding model, is presented. Second, it is proven that AdaProj
has arbitrari large optimal solution spaces allowing to relax the compactness
requirements of the class-specific distributions in the embedding space. Last
but not least, AdaProj is compared to other commonly used loss functions. In
experiments conducted on the DCASE2022 and DCASE2023 anomalous sound detection
(ASD) datasets it is shown that AdaProj outperforms other commonly used loss
functions. As a result, a state-of-the-art performance is achieved on the
DCASE2023 dataset.

###  1.1 Related Work

When training a neural network to solve a classification task, usually the
softmax function in combination with the categorical cross-entropy (CXE) is
used. However, this only reduces inter-class similarity without explicitly
increasing intra-class similarity [15]. When training an embedding model for
anomaly detection, high intra-class similarity is a desired property to
cluster normal data and be able to detect anomalous samples. There are several
loss functions that explicitly increase intra-class similarity: [16] proposed
a compactness loss to project the data into a hypersphere of minimal volume
for one-class classification. However, for machine condition monitoring in
noisy conditions it is known that one-class losses perform worse than losses
that also discriminatively solve an auxiliary classification task [5]. [17]
did this by simultaneously using a so-called descriptiveness loss consisting
of a CXE on another arbitrary dataset than the target dataset, to be able to
learn a better structured embedding space in case no meta information is
available on the target dataset. For machine condition monitoring, often meta
information is available as it can at least be ensured which machine is being
recorded when collecting data. [8] used center loss [18], which minimizes the
distance to learned class centers for each class. Another choice are angular
margin losses that learn an embedding space on the unit sphere while ensuring
a margin between classes, which improves the generalization capabilities.
Specific examples are the additive margin softmax loss [15] as used by [7, 19]
and ArcFace [20] as used by [6, 21, 11]. [22, 23] use the AdaCos loss [24],
which essentially is ArcFace with an adaptive scale parameter, or the sub-
cluster AdaCos loss [25], which utilizes multiple sub-clusters for each class
instead of a single one.

As stated before, the goal of this work is to reduce the restrictions on the
learned distributions in the embedding space by learning class-specific linear
subspaces. There are also other works on losses aiming at learning subspaces
based on orthogonal projections in an embedding space. [26] used orthogonal
projections as a constraint for training an autoencoder based anomaly
detection system. Another example is semi-supervised image classification by
using a combination of class-specific subspace projections with a
reconstructions loss and ensure that they are different by also using a
discriminative loss [27]. Our work focuses on learning an embedding space
through an auxiliary classification task that is well-suited for semi-
supervised anomaly detection.

##  2 Methodology

###  2.1 Notation

Let $\phi:X\rightarrow\mathbb{R}^{D}$ denote a neural network where $X$
denotes some input space, which consists of audio signals in this work, and
$D\in\mathbb{N}$ denotes the dimension of the embedding space. Define the
linear projection of $x\in\mathbb{R}^{D}$ onto the subspace
$\operatorname{span}(\mathcal{C}_{k})\subset\mathbb{R}^{D}$ as
${P_{\operatorname{span}(\mathcal{C}_{k})}(x):=\sum_{c_{k}\in\mathcal{C}_{k}}\langle
x,c_{k}\rangle c_{k}}$. Furthermore, let
${\mathcal{S}^{D-1}=\\{y\in\mathbb{R}^{D}:\lVert
y\rVert_{2}=1\\}\subset\mathbb{R}^{D}}$ denote the $D$-sphere and define
${P_{\mathcal{S}^{D-1}}(x):=\frac{x}{\lVert x\rVert_{2}}\in\mathcal{S}^{D-1}}$
to be the projection onto the $D$-sphere.

###  2.2 AdaProj loss function

Similar to the sub-cluster AdaCos loss [25], the idea of the AdaProj loss is
to enlarge the space of optimal solutions to allow the network to learn less
restrictive distributions of normal data. This relaxation is achieved by
measuring the distance to class-specific subspaces while training the
embedding model instead of measuring the distance to a single or multiple
centers as done for other angular margin losses and may help to differentiate
between normal and anomalous embeddings after training. The reason is that for
some auxiliary classes a strong compactness may be detrimental when aiming to
learn an embedding space that separates normal and anomalous data since both
may be distributed very similarly.

Formally, the definition of the AdaProj loss is as follows.

######  Definition 1 (AdaProj loss).

Let $\mathcal{C}_{k}\subset\mathbb{R}^{D}$ with
$\lvert\mathcal{C}_{k}\rvert=J\in\mathbb{N}$ denote class centers for class
$k\in\\{1,...,N_{\text{classes}}\\}$. Then for the AdaProj loss the logit for
class $k\in\\{1,...,N_{\text{classes}}\\}$ is defined as

$\displaystyle L(x,\mathcal{C}_{k}):=\hat{s}\cdot\lVert
P_{\mathcal{S}^{D-1}}(x)-P_{\mathcal{S}^{D-1}}(P_{{\operatorname{span}(\mathcal{C}_{k})}}(x))\rVert_{2}^{2}$

where $\hat{s}\in\mathbb{R}_{+}$ is the adaptive scale parameter of the AdaCos
loss [24]. Inserting these logits into a softmax function and computing the
CXE yields the AdaProj loss function.

######  Remark.

Note that, by Lemma 5 of [5], it holds that

$\displaystyle\lVert
P_{\mathcal{S}^{D-1}}(x)-P_{\mathcal{S}^{D-1}}(P_{{\operatorname{span}(\mathcal{C}_{k})}}(x))\rVert_{2}^{2}$
$\displaystyle=$ $\displaystyle 2(1-\langle
P_{\mathcal{S}^{D-1}}(x),P_{\mathcal{S}^{D-1}}(P_{{\operatorname{span}(\mathcal{C}_{k})}}(x))\rangle),$

which is equal to the cosine distance in this case and explains why the
AdaProj loss can be called an angular margin loss.

As for other angular margin losses, projecting the embedding space onto the
$D$-sphere has several advantages [5]. Most importantly, if $D$ is
sufficiently large randomly initialized centers are with very high probability
approximately orthonormal to each other [28], i.e. distributed equidistantly,
and sufficiently far away from $\boldsymbol{0}\in\mathbb{R}^{D}$. Therefore,
one does not need to carefully design a method to initialize the centers.
Another advantage is that a normalization may prevent numerical issues,
similar to applying batch normalization [29].

The following Lemma shows that using the AdaProj loss, as defined above,
indeed increases the solution space.

######  Lemma 2.

Let $x\in\mathbb{R}^{D}$ and let $\mathcal{C}\subset\mathbb{R}^{D}$ contain
pairwise orthonormal elements. If
$x\in\operatorname{span}(\mathcal{C})\cap\mathcal{S}^{D-1}$, then

$\displaystyle\lVert
P_{\mathcal{S}^{D-1}}(x)-P_{\mathcal{S}^{D-1}}(P_{\operatorname{span}(\mathcal{C})}(x))\rVert_{2}^{2}=0.$

###### Proof.

Let
$x\in\operatorname{span}(\mathcal{C})\cap\mathcal{S}^{D-1}\subset\mathbb{R}^{D}$
with $\lvert\mathcal{C}\rvert=J$. Therefore, $\lVert x\rVert_{2}=1$ and there
are $\lambda_{j}\in\mathbb{R}$ with $x=\sum_{j=1}^{J}\lambda_{j}c_{j}$. Thus,
it holds that

$\displaystyle x$
$\displaystyle=\sum_{j=1}^{J}\lambda_{j}c_{j}=\sum_{j=1}^{J}\sum_{i=1}^{J}\lambda_{i}\langle
c_{i},c_{j}\rangle
c_{j}=\sum_{j=1}^{J}\langle\sum_{i=1}^{J}\lambda_{i}c_{i},c_{j}\rangle c_{j}$
$\displaystyle=\sum_{j=1}^{J}\langle x,c_{j}\rangle
c_{j}=P_{\operatorname{span}(\mathcal{C})}(x).$

Hence, we obtain

$\displaystyle\lVert
P_{\mathcal{S}^{D-1}}(x)-P_{\mathcal{S}^{D-1}}(P_{\operatorname{span}(\mathcal{C})}(x))\rVert_{2}^{2}=$
$\displaystyle 0.$

∎

######  Remark.

If $\mathcal{C}$ contains randomly initialized elements of the unit sphere and
$D$ is sufficiently large, then the elements of $\mathcal{C}$ are
approximately pairwise orthonormal with very high probability [28].

When inserting the projection onto the $D-1$-sphere as an operation into the
neural network, this Lemma shows that the solution space for the AdaProj loss
function is increased to the whole subspace
$\operatorname{span}(\mathcal{C})$, which has a dimension of
$\lvert\mathcal{C}\rvert$ with very high probability. Because of this, it
should be ensured that $\lvert\mathcal{C}\rvert<D$. Otherwise the whole
embedding space may be an optimal solution and thus the network cannot learn a
meaningful embedding space. In comparison, for the AdaCos loss only the class
centers themselves are optimal solutions and for the sub-cluster AdaCos loss
each sub-cluster is an optimal solution [5].

##  3 Experimental results

###  3.1 Datasets and performance metrics

For the experiments, the DCASE2022 ASD dataset [2] and the DCASE2023 ASD
dataset [3] for semi-supervised machine condition monitoring were used. Both
datasets consist of a development set and an evaluation set that are divided
into a training split containing only normal data and a test split containing
normal as well as anomalous data. Furthermore, both tasks explicitly capture
the problem of domain generalization [4] by defining a source and a target
domain, which differs from the source domain by altering machine parameters or
noise conditions. The task is to detect anomalous samples regardless of the
domain a sample belongs to by training a system with only normal data. As meta
information, the target machine type of each sample is known and for the
training samples, also the domain and additional parameter settings or noise
conditions, called attribute information, are known and thus can be utilized
to train an embedding model.

The DCASE2022 ASD dataset [2] consists of the machine types “ToyCar” and
“ToyTrain” from ToyAdmos2 [30] and “fan”, “gearbox”, “bearing”, “slide rail”
and “valve” from MIMII-DG [31]. For each machine type, there are six different
sections corresponding to different domain shifts and also defining subsets
used for computing the performance. These sections are known for each
recording and can also be utilized as meta information to train the system.
For the source domain of each section, there are $1000$ normal audio
recordings with a duration of $10\text{\,}\mathrm{s}$ and a sampling rate of
$16\text{\,}\mathrm{kHz}$ belonging to the training split. For the target
domain of each section, there are only $10$ normal audio recordings belonging
to the training split. The test splits of each section contain approximately
$100$ normal and $100$ anomalous samples.

The DCASE2023 ASD dataset [3] is similar to the DCASE2022 ASD dataset with the
following modifications. First of all, the development set and the evaluation
set contain mutually exclusive machine types. More concretely, the development
set contains the same machine types as the DCASE2022 dataset and the
evaluation set contains the machine types “ToyTank”, “ToyNscale” and
“ToyDrone” fromToyAdmos2+ [32] and “vacuum”, “bandsaw”, “grinder” and “shaker”
from MIMII-DG [31]. Furthermore, there is only a single section for each
machine type, which makes the auxiliary classification task much easier
resulting in less informative embeddings for the ASD task. Last but not least,
the duration of each recording has a length between $6\text{\,}\mathrm{s}$ and
$18\text{\,}\mathrm{s}$. Overall, all three modifications make this task much
more challenging than the DCASE2022 ASD task.

To measure the performance of the ASD systems the threshold-independent area
under the  curve (AUC) metric is used. In addition, the partial area under the
curve (pAUC) [33], which is the AUC for low false positive rates ranging from
$0$ to $p$, with $p=0.1$, is used. Both performance metrics are computed
domain-independent for every previously defined section of the dataset and the
harmonic mean of all resulting values is the final score used to measure and
compare the performances of different ASD systems.

###  3.2 Anomalous sound detection system

embedding model: jointly trained to discriminatebetween machine types and
attribute informationinput feature representationscomputing anomaly scores
using only the normalsamples of the same machine type as the test samples raw
waveform $(288000)$ compute magnitude spectrogram $(561\times 513)$ apply
temporal mean normalization $(561\times 513)$ compute magnitude spectrum
($144000$) extract embbeding with neural network for spectrograms (256)
extract embbeding with neural network for spectra (256) concatenate and
normalize embeddings (512) evaluate cosine distances with all samples of
target domain (10) evaluate cosine distances with means (k-Means) of source
domain (32) return minimum of distances as anomaly score (1)

Fig. 1: Structure of the ASD system, adapted from Figure 1 in [23].
Representation size in each step is given in brackets.

For all experiments conducted in this work, the state-of-the-art ASD system
presented in [23] is used. An overview of the system can be found in Figure 1.
The system consists of three main components: 1) a feature extractor, 2) an
embedding model and 3) a backend for computing anomaly scores.

In the first processing block, two different feature representations are
extracted from the raw waveforms, namely magnitude spectrograms and the
magnitude spectrum. To make both feature representations a bit more different
the temporal mean is subtracted from the spectrograms, essentially removing
static frequency information that are captured with the highest possible
resolution by the spectra.

For each of the two feature representations, another convolutional sub-network
is trained and the resulting embeddings are concatenated and normalized with
respect to the Euclidean norm to obtain a single embedding. In contrast to the
original architecture, the embedding dimension is doubled from $256$ to $512$.
More details about the subnetwork architectures can be found in [23]. The
network is trained for $10$ epochs using a batch size of $64$ using adam [34]
by utilizing meta information such as machine types and the provided attribute
information as an auxiliary classification task. Different loss functions can
be used for this purpose and will be compared in the next subsection. All loss
functions investigated in this work require class-specific center vectors,
which are initialized randomly using Glorot uniform intialization [35]. To
improve the ASD performance, the randomly initialized class centers are not
adapted during training and no bias terms are used as proposed in [16] for
one-class classification. Furthermore, mixup [36] with a uniformly distributed
mixing coefficient is applied to the waveforms.

As a backend, k-means with $32$ means is applied to the normal training
samples of the source domain. For a given test sample, the smallest cosine
distance to these means and the ten normal training samples of the target
domain is used as an anomaly score. Thus, smaller values indicate normal
samples whereas higher values indicate anomalous samples.

###  3.3 Performance evaluation

Table 1: ASD performance obtained with different loss functions. Harmonic
means of all AUCs and pAUCs over all pre-defined sections of the dataset are
depicted in percent. Arithmetic mean and standard deviation of the results
over ten independent trials are shown. Best results in each column are
highlighted with bold letters.

loss function DCASE2022 dev. set [2] DCASE2022 eval. set [2] DCASE2023 dev.
set [3] DCASE2023 eval. set [3] arithmetic mean AUC pAUC AUC pAUC AUC pAUC
pAUC AUC pAUC pAUC IC compactness loss [16] $79.2\pm 0.9$ $64.7\pm 1.1$
$70.3\pm 0.8$ $58.9\pm 0.8$ $67.7\pm 1.2$ $56.9\pm 0.9$ $64.0\pm 1.5$ $55.8\pm
0.9$ $70.3$ $59.1$ IC compactness loss + CXE [17] $79.0\pm 0.8$ $65.0\pm 0.7$
$72.6\pm 0.4$ $60.3\pm 0.7$ $70.4\pm 1.0$ $\boldsymbol{57.4\pm 1.1}$ $67.5\pm
0.8$ $57.5\pm 1.0$ $72.4$ $60.1$ AdaCos loss [24] $79.8\pm 0.7$
$\boldsymbol{65.5\pm 0.9}$ $73.0\pm 0.4$ $59.7\pm 0.6$ $70.9\pm 0.9$ $56.8\pm
0.9$ $68.0\pm 1.6$ $58.0\pm 1.1$ $72.9$ $60.0$ sub-cluster AdaCos loss [25]
$80.0\pm 1.4$ $65.2\pm 1.1$ $72.9\pm 0.6$ $59.5\pm 0.5$ $70.4\pm 0.9$ $56.3\pm
0.8$ $66.5\pm 1.6$ $56.2\pm 1.0$ $72.5$ $59.3$ proposed AdaProj loss
$\boldsymbol{80.6\pm 0.8}$ $\boldsymbol{65.5\pm 1.3}$ $\boldsymbol{73.6\pm
0.7}$ $\boldsymbol{60.5\pm 0.7}$ $\boldsymbol{71.4\pm 1.0}$ $56.2\pm 0.7$
$\boldsymbol{69.8\pm 1.3}$ $\boldsymbol{60.0\pm 0.5}$ $\boldsymbol{73.9}$
$\boldsymbol{60.6}$

In the first experiment, the ASD performance obtained with the following loss
functions was compared: 1) individual class-specific intra-class (IC)
compactness losses jointly trained on all classes [16] 2) an additional
discriminative CXE loss, similar to the descriptiveness loss used in [17] but
trained on the same dataset, 3) the AdaCos loss [24], 4) the sub-cluster
AdaCos loss [25] with $32$ sub-clusters and 5) the proposed AdaProj loss. Each
experiment was repeated ten times to reduce the variance of the resulting
performances. The results can be found in Table 1.

The main observation to be made is that the proposed AdaProj loss outperforms
all other losses. Especially on the DCASE2023 dataset, there are significant
improvements to be observed. The most likely explanation is that for this
dataset the classification task is less difficult and thus a few classes may
be easily identified leading to embeddings that do not carry enough
information to distinguish between embeddings belonging to normal and
anomalous samples of these classes.

Another interesting observation is that, in contrast to the original results
presented in [25], the sub-cluster AdaCos loss actually performs slightly
worse than the AdaCos loss despite having a higher solution space. A possible
explanation is that in [25], the centers are adapted during training whereas,
in our work, they are not as this has been shown to improve the resulting
performance [23]. Since all centers have approximately the same distance to
each other when being randomly initialized, i.e. the centers belonging to a
target class and the other centers, the network will likely utilize only a
single center for each class that is closest to the initial embeddings of the
corresponding target class. Moreover, a low inter-class similarity is more
difficult to ensure due to the higher total number of sub-clusters belonging
to other classes. This leads to more restrictive requirements when learning
class-specific distributions and thus actually reduces the ability to
differentiate between embeddings belonging to normal and anomalous samples.

###  3.4 Investigating the impact of the subspace dimension on the performance

embedding model: jointly trained to discriminatebetween machine types and
attribute informationinput feature representationscomputing anomaly scores
using only the normalsamples of the same machine type as the test
samples$0$$10$$20$$30$$40$$50$$60$$70$$80$$90$$100$$110$$120$$130$$0.5$$0.55$$0.6$$0.65$$0.7$$0.75$subspace
dimension $\lvert\mathcal{C}_{k}\rvert$harmonic mean of AUCs/pAUCsAUC \-
development setpAUC \- development setAUC \- evaluation setpAUC \- evaluation
set

Fig. 2: Domain-independent performance obtained on the DCASE2023 dataset with
different subspace dimensions. The means over ten independent trials are
shown.

As an ablation study, different choices for the dimension of the subspaces
have been compared experimentally on the DCASE2023 ASD dataset. The results
can be found in Figure 2. It can be seen, that, on the development set, the
results are relatively stable while a larger dimension slightly improves the
performance on the evaluation set without any significant differences. For
subspace dimensions greater than $48$ the performances seem to slightly
degrade again. In conclusion, the subspace dimension should be neither too
high nor too low and a dimension of $32$ as used for the other experiments in
this works appears to be a reasonable choice.

###  3.5 Comparison to other published systems

embedding model: jointly trained to discriminatebetween machine types and
attribute informationinput feature representationscomputing anomaly scores
using only the normalsamples of the same machine type as the test
samples$50$$60$$70$Y6.757e1]pt67.57%Y6.697e1]pt66.97%Y6.639e1]pt66.39%Y6.54031e1]pt65.40%Y6.49115e1]pt64.91%Y6.41043e1]pt64.10%Y6.36449e1]pt63.64%Y6.35032e1]pt63.50%Y6.17655e1]pt61.77%Y6.10508e1]pt61.05%Y5.95395e1]pt59.54%Y5.40172e1]pt54.02%official
score in percentproposed systemrank 1rank 2rank 3rank 4 (own system)rank 5rank
6rank 7rank 8rank 9rank 10challenge baseline

Fig. 3: Comparison of the proposed system to the ten top-performing systems of
the DCASE2023 Challenge. The official evaluation script was used to compute
the score. Rank 4 is the same system as the proposed system but uses a sub-
cluster AdaCos loss instead of the AdaProj loss (cf. [37]).

As a last experiment, the performance of the proposed system using AdaProj is
compared to the ten top-performing systems of the DCASE2023 Challenge. As many
systems utilize ensembles of models, the mean of the anomaly scores belonging
to ten independent trials was used to create an ensemble of ten systems
allowing a fair comparison. The results can be found in Figure 3. It can be
seen that the proposed system outperforms all other published systems and thus
achieves a state-of-the-art performance. This adds confidence to the benefits
of the AdaProj loss function.

##  4 Conclusions

In this work, AdaProj a novel angular margin loss function specifically
designed for semi-supervised anomaly detection with auxiliary classification
tasks was presented. It was proven that this loss function learns an embedding
space with class-specific subspaces of arbitrary dimension. In contrast to
other angular margin losses, which try to project data to individual points in
space, this relaxes the requirements of solving the classification task and
allows for less compact distributions in the embedding space. In experiments
conducted on the DCASE2022 and DCASE2023 ASD datasets, it was shown that using
AdaProj results in better performance than other commonly used loss functions.
In conclusion, the resulting embedding space has a more desirable structure
than the other embedding spaces for differentiating between normal and
anomalous samples. As a result, a state-of-the-art performance on the
DCASE2023 dataset is achieved. For future work, it is planned to evaluate
AdaProj on other datasets and further improve the performance of the ASD
system by utilizing self-supervised learning [38].

## References

  * [1] C. Aggarwal,  Outlier Analysis,  Springer, 2nd edition, 2017.
  * [2] K. Dohi et al.,  “Description and discussion on DCASE 2022 challenge task 2: Unsupervised anomalous sound detection for machine condition monitoring applying domain generalization techniques,”  in DCASE. 2022, pp. 26–30, Tampere University.
  * [3] K. Dohi et al.,  “Description and discussion on DCASE 2023 challenge task 2: First-shot unsupervised anomalous sound detection for machine condition monitoring,”  in DCASE. 2023, pp. 31–35, Tampere University.
  * [4] J. Wang, C. Lan, C. Liu, Y. Ouyang, and T. Qin,  “Generalizing to unseen domains: A survey on domain generalization,”  in IJCAI. 2021, pp. 4627–4635, ijcai.org.
  * [5] K. Wilkinghoff and F. Kurth,  “Why do angular margin losses work well for semi-supervised anomalous sound detection?,”  IEEE/ACM TASLP, vol. 32, pp. 608–622, 2024.
  * [6] R. Giri, S. V. Tenneti, F. Cheng, K. Helwani, U. Isik, and A. Krishnaswamy,  “Self-supervised classification for detecting anomalous sounds,”  in DCASE, 2020, pp. 46–50.
  * [7] J. A. Lopez, H. Lu, P. Lopez-Meyer, L. Nachman, G. Stemmer, and J. Huang,  “A speaker recognition approach to anomaly detection,”  in DCASE, 2020, pp. 96–99.
  * [8] T. Inoue et al.,  “Detection of anomalous sounds for machine condition monitoring using classification confidence,”  in DCASE, 2020, pp. 66–70.
  * [9] S. Venkatesh, G. Wichern, A. Shanmugam Subramanian, and J. Le Roux,  “Improved domain generalization via disentangled multi-task learning in unsupervised anomalous sound detection,”  in DCASE. 2022, pp. 196–200, Tampere University.
  * [10] T. Nishida, K. Dohi, T. Endo, M. Yamamoto, and Y. Kawaguchi,  “Anomalous sound detection based on machine activity detection,”  in EUSIPCO. 2022, pp. 269–273, IEEE.
  * [11] Y. Deng et al.,  “Ensemble of multiple anomalous sound detectors,”  in DCASE. 2022, Tampere University.
  * [12] H. Chen et al.,  “An effective anomalous sound detection method based on representation learning with simulated anomalies,”  in ICASSP. IEEE, 2023.
  * [13] D. Hendrycks, M. Mazeika, and T. G. Dietterich,  “Deep anomaly detection with outlier exposure,”  in ICLR. 2019, OpenReview.net.
  * [14] P. Primus, V. Haunschmid, P. Praher, and G. Widmer,  “Anomalous sound detection as a simple binary classification problem with careful selection of proxy outlier examples,”  in DCASE, 2020, pp. 170–174.
  * [15] F. Wang, J. Cheng, W. Liu, and H. Liu,  “Additive margin softmax for face verification,”  IEEE SPL, vol. 25, no. 7, pp. 926–930, 2018.
  * [16] L. Ruff et al.,  “Deep one-class classification,”  in ICML. 2018, vol. 80 of Proceedings of Machine Learning Research, pp. 4390–4399, PMLR.
  * [17] P. Perera and V. M. Patel,  “Learning deep features for one-class classification,”  IEEE TIP, vol. 28, no. 11, pp. 5450–5463, 2019.
  * [18] Y. Wen, K. Zhang, Z. Li, and Y. Qiao,  “A discriminative feature learning approach for deep face recognition,”  in ECCV. Springer, 2016, pp. 499–515.
  * [19] J. A. Lopez, G. Stemmer, P. Lopez-Meyer, P. Singh, J. A. del Hoyo Ontiveros, and H. A. Cordourier,  “Ensemble of complementary anomaly detectors under domain shifted conditions,”  in DCASE, 2021, pp. 11–15.
  * [20] J. Deng, J. Guo, N. Xue, and S. Zafeiriou,  “ArcFace: Additive angular margin loss for deep face recognition,”  in CVPR. 2019, pp. 4690–4699, IEEE.
  * [21] I. Kuroyanagi, T. Hayashi, Y. Adachi, T. Yoshimura, K. Takeda, and T. Toda,  “An ensemble approach to anomalous sound detection based on conformer-based autoencoder and binary classifier incorporated with metric learning,”  in DCASE, 2021, pp. 110–114.
  * [22] K. Wilkinghoff,  “Combining multiple distributions based on sub-cluster adacos for anomalous sound detection under domain shifted conditions,”  in DCASE, 2021, pp. 55–59.
  * [23] K. Wilkinghoff,  “Design choices for learning embeddings from auxiliary tasks for domain generalization in anomalous sound detection,”  in ICASSP. 2023, IEEE.
  * [24] X. Zhang, R. Zhao, Y. Qiao, X. Wang, and H. Li,  “AdaCos: Adaptively scaling cosine logits for effectively learning deep face representations,”  in CVPR. 2019, pp. 10823–10832, IEEE.
  * [25] K. Wilkinghoff,  “Sub-cluster AdaCos: Learning representations for anomalous sound detection,”  in IJCNN. 2021, IEEE.
  * [26] Q. Yu, M. S. Kavitha, and T. Kurita,  “Autoencoder framework based on orthogonal projection constraints improves anomalies detection,”  Neurocomputing, vol. 450, pp. 372–388, 2021.
  * [27] L. Li, Y. Zhang, and A. Huang,  “Learnable subspace orthogonal projection for semi-supervised image classification,”  in ACCV. 2022, vol. 13843 of Lecture Notes in Computer Science, pp. 477–490, Springer.
  * [28] A. N. Gorban, I. Yu. Tyukin, D. V. Prokhorov, and K. I. Sofeikov,  “Approximation with random bases: Pro et contra,”  Information Sciences, vol. 364-365, pp. 129–145, 2016.
  * [29] S. Ioffe and C. Szegedy,  “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  in ICML, 2015, vol. 37, pp. 448–456.
  * [30] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, M. Yasuda, and S. Saito,  “ToyADMOS2: Another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions,”  in DCASE, 2021, pp. 1–5.
  * [31] K. Dohi et al.,  “MIMII DG: sound dataset for malfunctioning industrial machine investigation and inspection for domain generalization task,”  in DCASE. 2022, pp. 26–30, Tampere University.
  * [32] N. Harada, D. Niizumi, D. Takeuchi, Y. Ohishi, and M. Yasuda,  “ToyADMOS2+: New toyadmos data and benchmark results of the first-shot anomalous sound event detection baseline,”  in DCASE. 2023, pp. 41–45, Tampere University.
  * [33] D. Katzman McClish,  “Analyzing a portion of the ROC curve,”  Medical decision making, vol. 9, no. 3, pp. 190–195, 1989.
  * [34] D. P. Kingma and J. Ba,  “Adam: A method for stochastic optimization,”  in ICLR, Yoshua Bengio and Yann LeCun, Eds., 2015.
  * [35] X. Glorot and Y. Bengio,  “Understanding the difficulty of training deep feedforward neural networks,”  in AISTATS. 2010, vol. 9 of JMLR Proceedings, pp. 249–256, JMLR.org.
  * [36] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz,  “Mixup: Beyond empirical risk minimization,”  in ICLR. 2018, OpenReview.net.
  * [37] K. Wilkinghoff,  “Fraunhofer FKIE submission for task 2: First-shot unsupervised anomalous sound detection for machine condition monitoring,”  Tech. Rep., DCASE2023 Challenge, 2023.
  * [38] Kevin Wilkinghoff,  “Self-supervised learning for anomalous sound detection,”  arXiv:2312.09578, 2024.

  *[OE]: outlier exposure
  *[ASD]: anomalous sound detection
  *[CXE]: categorical cross-entropy
  *[]:
  *[AUC]: area under the   () curve
  *[pAUC]: partial area under the  curve
  *[IC]: intra-class
  *[AUCs]: area under the   () curve
  *[pAUCs]: partial area under the  curve