# Out-of-Distribution Detection in Time-Series Domain: A Novel Seasonal Ratio
Scoring Approach

Taha Belkhouja∗<EMAIL_ADDRESS>
Yan Yan∗ yan.yan1<EMAIL_ADDRESS>
Janardhan Rao Doppa∗<EMAIL_ADDRESS>
∗School of EECS, Washington State University

###### Abstract

Safe deployment of time-series classifiers for real-world applications relies
on the ability to detect the data which is not generated from the same
distribution as training data. This task is referred to as out-of-distribution
(OOD) detection. We consider the novel problem of OOD detection for the time-
series domain. We discuss the unique challenges posed by time-series data and
explain why prior methods from the image domain will perform poorly. Motivated
by these challenges, this paper proposes a novel Seasonal Ratio Scoring (SRS)
approach. SRS consists of three key algorithmic steps. First, each input is
decomposed into class-wise semantic component and remainder. Second, this
decomposition is employed to estimate the class-wise conditional likelihoods
of the input and remainder using deep generative models. The seasonal ratio
score is computed from these estimates. Third, a threshold interval is
identified from the in-distribution data to detect OOD examples. Experiments
on diverse real-world benchmarks demonstrate that the SRS method is well-
suited for time-series OOD detection when compared to baseline methods. Open-
source code for SRS method is provided at
https://github.com/tahabelkhouja/SRS.

##  1 Introduction

Time-series data analytics using deep neural networks Ismail Fawaz et al.,
(2019) enable many real-world applications in health-care (Tripathy and
Acharya,, 2018), finance (Yang,, 2021), smart grids (Zheng et al.,, 2017), and
intrusion detection (Kim et al.,, 2017). However, safe and reliable deployment
of such machine learning (ML) systems require the ability to detect time-
series data which does not follow the distribution of training data, aka in-
distribution (ID). This task is referred to as out-of-distribution (OOD)
detection. If the ML model encounters OOD inputs, it can output wrong
predictions with high confidence. For example, a circumstantial event for an
epilepsy patient or a sudden surge in a smart grid branch result in sensor
readings which deviate from the training distribution. Another important
application of OOD detection for the time-series domain is synthetic data
generation. Many time-series applications suffer from limited or imbalanced
data, which motivates methods to generate synthetic data (Smith and Smith,,
2020). A key challenge is to automatically assess the validity of synthetic
data, which can be alleviated using accurate OOD detectors.

There is a growing body of work on OOD detection for the image domain (Dan
Hendrycks,, 2017; Liu et al.,, 2020; Liang et al.,, 2018; Xiao et al.,, 2020;
Yang* et al.,, 2020; Cao et al.,, 2020) and other types of data such as
genomic sequences (Ren et al.,, 2019). These methods can be categorized into

  * •

Supervised methods which fine-tune the ML system or perform specific training
to distinguish examples from ID and OOD.

  * •

Unsupervised methods which employ Deep Generative Models (DGMs) or unlabeled
data to perform OOD detection.

However, time-series data with its unique characteristics (e.g., sparse peaks,
fast oscillations) pose unique challenges that are not encountered in the
image domain:

  * •

Spatial relations between pixels is not similar to the temporal relations
across different time-steps of time-series signals.

  * •

Pixel variables follow a categorical distribution of values
$\\{0,1,\cdots,255\\}$ where as time-series variables follow a continuous
distribution.

  * •

The semantics of images (e.g., background, edges) do not apply to time-series
data.

  * •

Humans can identify OOD images for fine-tuning purposes, but this task is
challenging for time-series data. Hence, prior OOD methods are not suitable
for the time-series domain.

Figure 1: Overview of the seasonal ratio (SR) scoring algorithm. The semantic
component $S_{\hat{y}}$ for the predicted output $\hat{y}$ is obtained from
the training stage via Seasonal and Trend decomposition using Loess (STL). The
semantic component $S_{\hat{y}}$ is subtracted from the time-series $x$ to
obtain the remainder $r$. The trained CVAE models $\mathcal{M}_{x}$ and
$\mathcal{M}_{r}$ are used to compute the SR score. If the SR score is within
the threshold interval $[\tau_{l},\tau_{u}]$ identified during training, then
$x$ is classified as ID. Otherwise, it is flagged as OOD.

This paper proposes a novel OOD detection algorithm for the time-series domain
referred to as Seasonal Ratio Scoring (SRS). To the best of our knowledge,
this is the first work on OOD detection over time-series data. SRS employs the
Seasonal and Trend decomposition using Loess (STL) (Cleveland et al.,, 1990)
on time-series signals from the ID data to create class-wise semantic pattern
and remainder for each signal. For example, in a human activity recognition
system, SRS would extract a pattern ”running” that describes semantically all
the ”running” windows. If the person trips and falls, SRS would detect that
this event does not belong to the pre-defined classes of activity and flags it
as OOD. For this purpose, we train two separate DGMs over this STL
decomposition to estimate the class-wise conditional likelihoods for time-
series signal and remainder. The seasonal ratio score for each time-series
signal from ID is computed from these two estimates. A threshold interval is
estimated from the statistics of all these scores over ID data. Given a new
time-series input and a classifier at the testing time, the SRS approach
computes the seasonal ratio score for the predicted output and classifies the
time-series signal as OOD example if the score lies outside the threshold
interval. Figure 1 illustrates the SRS algorithm. The effectiveness of SRS
critically depends on the extraction of accurate class-wise semantic
components. Since time-series data is prone to warping and time-shifts, we
also propose a new alignment approach based on dynamic time warping to improve
the output accuracy of STL decomposition. Our experiments on diverse real-
world time-series datasets demonstrate that SRS method is well-suited for
time-series OOD detection when compared to prior methods.

Contributions. The main contribution of this paper is the development and
evaluation of the Seasonal Ratio Scoring (SRS) algorithm for OOD detection in
the time-series domain. Specific contributions include:

  1. 1.

Principled algorithm based on STL decomposition and deep generative models to
compute seasonal ratio score to detect OOD time-series examples.

  2. 2.

Novel time-series alignment algorithm based on dynamic time warping to improve
the effectiveness of seasonal ratio score based OOD detection.

  3. 3.

Formulation of experimental setting for time-series OOD detection.
Experimental evaluation of SRS algorithm on real-world datasets and comparison
with state-of-the-art baselines.

  4. 4.

Open-source code and data for SRS method https://github.com/tahabelkhouja/SRS.

##  2 Problem Setup

Suppose $\mathcal{D}_{in}$ is an in-distribution (ID) time-series dataset with
$d$ examples $\\{(x_{i},y_{i})\\}$ sampled from the distribution $P^{*}$
defined on the joint space of input-output pairs $(\mathcal{X},\mathcal{Y})$.
Each $x_{i}\in\mathbb{R}^{n\times T}$ from $\mathcal{X}$ is a multi-variate
time-series input, where $n$ is the number of channels and $T$ is the window-
size of the signal. $y_{i}\in\mathcal{Y}=\\{1,\cdots,C\\}$ represents the
class label for time-series signal $x_{i}$. We consider a time-series
classifier $F:\mathbb{R}^{n\times T}\rightarrow\\{1,\cdots,C\\}$ learned using
$\mathcal{D}_{in}$. For example, in a health monitoring application using
physiological sensors for patients diagnosed with cardiac arrhythmia, we use
the measurements from wearable devices to predict the likelihood of a cardiac
failure.

OOD samples $(x,y)$ are typically generated from a distribution other than
$P^{*}$. Specifically, we consider a sample $(x,y)$ to be OOD if the class
label $y$ is different from the set of in-distribution class labels, i.e.,
$y\notin\mathcal{Y}$. The classifier $F_{\theta}(x)$ learned using
$\mathcal{D}_{in}$ will assign one of the $C$ class labels from $\mathcal{Y}$
when encountering an OOD sample $(x,y)$. Our goal is to detect such OOD
examples for safe and reliable real-world deployment of time-series
classifiers. We provide a summary of the mathematical notations used in this
paper in the Table 1.

Table 1: Mathematical notations used in this paper.

Variable | Definition
---|---
$\mathcal{D}_{in}$ | dataset of in-distribution time-series signals
$P^{*}$ | True distribution of the time-series dataset
$x_{i}$ | Input time-series signal
$\mathcal{Y}$ | Set of output class labels $y\in\\{1,\cdots,C\\}$
$F_{\theta}$ | Classifier that maps an input$x\in\mathbb{R}^{n\times T}$ to a class label $y\in\mathcal{Y}$
$S_{y}$ | Semantic pattern of a class $y$ according to STL
$SR$ | Seasonal Ratio score

Challenges of time-series data. The unique characteristics of time-series data
(e.g., temporal relation across time-steps, fast oscillations, continuous
distribution of variables) pose unique challenges not seen in the image
domain. Real-world time-series datasets are typically small (relative to image
datasets) and exhibit high class imbalance Dau et al., (2019). Therefore,
estimating a good approximation of in-distribution $P^{*}$ is hard, which
results in failure of prior OOD methods. Indeed, our experiments demonstrate
that prior OOD methods are not suited for the time-series domain. As a
prototypical example, Figure 7 shows the limitation of Likelihood Regret score
(Xiao et al.,, 2020) to identify OOD examples: ID and OOD scores of real-world
time-series examples overlap.

##  3 Background and Preliminaries

In this section, we provide the necessary background on conditional VAE and
STL decomposition, to be able to understand the proposed seasonal ratio score
based OOD detection approach.

Conditional VAE. Variational Auto-Encoders (VAEs) are a class of likelihood-
based generative models with many real-world applications (Doersch,, 2016).
They rely on the encoding of raw input $x$ as a latent Gaussian variable $z$
to estimate the likelihood of $x$. The latent variable $z$ is used to compute
the likelihood of the training data: $p_{\theta}(x)=\int
p_{\theta}(x|z)p(z)dz$. Since the direct computation of this likelihood is
impractical, the principle of evidence lower bound (ELBO) (Xiao et al.,, 2020)
is employed. In this work, we consider the ID data $\mathcal{D}_{in}$ as $d$
input-output samples of the form $(x_{i},y_{i})$. We want to estimate the ID
using both $x_{i}$ and $y_{i}$. Therefore, we propose to use conditional VAE
(CVAE) for this purpose. CVAEs are a class of likelihood-based generative
models (Doersch,, 2016). They rely on the encoding of raw input $(x,y)$ as a
latent Gaussian variable $z$ to estimate the conditional likelihood of $x$
over the class label $y$. CVAE is similar to VAE with the key difference being
the use of conditional probability over both $x_{i}$ and $y_{i}$. The ELBO
objective of CVAE is:

$\mathcal{L}_{\text{ELBO}}\overset{\Delta}{=}\mathbb{E}_{\phi}\big{[}\log
p_{\theta}(x|z,y)\big{]}-D_{\text{KL}}\big{[}q_{\phi}(z|x,y)||p(z|y)\big{]}$

where $q_{\phi}(z|x,y)$ is the variational approximation of the true posterior
distribution $p_{\theta}(x|z,y)$. As CVAE only computes the lower bound of the
log-likelihood of a given input, the exact log-likelihood is estimated using
Monte-Carlo sampling as shown below:

$\mathcal{L}_{M}=\mathbb{E}_{z^{m}\sim
q_{\phi}(z|x,y)}\bigg{[}\log\dfrac{1}{M}\sum_{m=1}^{M}\dfrac{p_{\theta}(x|z^{m},y)p(z^{m})}{q_{\phi}(z^{m}|x,y)}\bigg{]}$
(1)

The intuitive expectation from a DGM learned using training data is to assign
a high likelihood to ID samples and a low likelihood to OOD samples. However,
recent research showed that DGMs tend to assign highly unreliable likelihood
to OOD samples regardless of the different semantics of both ID and OOD data
(Xiao et al.,, 2020). Indeed, our experimental results shown in Table 5
demonstrate that this observation is also true for the time-series domain.

Figure 2: Illustration of STL method for two different classes from the ERing
dataset. Dotted signals are natural time-series signals $x$ and the red signal
is the semantic pattern $S_{y}$.

STL decomposition. STL (Jiang et al.,, 2011) is a statistical method for
decomposing a given time-series signal $x$ into three different components: 1)
The seasonality $x_{s}$ is a fixed regular pattern that recurs in the data; 2)
The trend $x_{t}$ is the increment or the decrement of the seasonality over
time; and 3) The residual $x_{r}$ represents random additive noise. STL
employs Loess (LOcal regrESSion) smoothing in an iterative process to estimate
the seasonality component $x_{s}$ (McKinney et al.,, 2011). The remainder is
the additive residual from the input $x$ after summing both $x_{s}$ and
$x_{t}$. For the proposed SRS algorithm, we assume that there is a fixed
semantic pattern $S_{y}$ for every class label $y\in\mathcal{Y}$, and this
pattern recurs in all examples $(x_{i},y_{i})$ from $\mathcal{D}_{in}$ with
the same class label, i.e., $y_{i}$=$y$. We will elaborate more on this
assumption and a reformulation of the problem that can be used when the
assumption is violated in the next section. Hence, every time-series example
has the following two elements: $x_{i}=S_{y_{i}}+r_{i}$, where $S_{y_{i}}$ is
the pattern for the class label $y$=$y_{i}$ and $r_{i}$ is the remainder noise
w.r.t $S_{y_{i}}$. For time-series classification tasks, ID samples are
assumed to be stationary. Therefore, we propose to average the trend $x_{t}$
component observed during training and include it in the semantic pattern
$S_{y}$. Figure 2 illustrates the above-mentioned decomposition for two
different classes from the ERing dataset.

##  4 Seasonal Ratio Scoring Approach for OOD Detection

Overview of SRS algorithm. The training stage proceeds as follows. We employ
STL decomposition to get the semantic component $S_{y}$ for each class label
$y\in\mathcal{Y}$=$\\{1,\cdots,C\\}$ from the given in-distribution (ID) data
$\mathcal{D}_{in}$. To improve the accuracy of STL decomposition, we apply a
time-series alignment method based on dynamic time warping to address scaling,
warping, and time-shifts. We train two CVAE models $\mathcal{M}_{x}$ and
$\mathcal{M}_{r}$ to estimate the class-wise conditional likelihood of each
time-series signal $x_{i}$ and its remainder component $r_{i}$ w.r.t the
semantic component $S_{y_{i}}$. The seasonal ratio score for each ID example
$(x_{i},y_{i})$ from $\mathcal{D}_{in}$ is computed as the ratio of the class-
wise conditional likelihood estimates for $x_{i}$ and its remainder $r_{i}$:
$SR_{i}(x_{i},y_{i})\overset{\Delta}{=}\frac{p(x_{i}|y=y_{i})}{p(r_{i}|y=y_{i})}$.
We compute the SR scores for all in-distribution examples from
$\mathcal{D}_{in}$ to estimate the threshold interval $[\tau_{l},\tau_{u}]$
for OOD detection. During the inference stage, given a time-series signal $x$
and a trained classifier $F(x)$, we compute the SR score of $x$ with the
predicted output $\hat{y}$=$F(x)\in\mathcal{Y}$ and identify it as an OOD
example if the SR score lies outside the threshold interval
$[\tau_{l},\tau_{u}]$. Figure 1 provides a high-level illustration of the SRS
algorithm.

Below we first provide an intuitive explanation to motivate the seasonal ratio
score. Next, we describe the complete details of the SRS algorithm including
both training and inference stages. Finally, we motivate and describe a time-
series alignment approach based on dynamic time warping to improve the
effectiveness of SRS.

###  4.1 Intuition for Seasonal Ratio Score

We explain the intuition behind the proposed SRS algorithm using STL
decomposition of time-series signals and CVAE models for likelihood
estimation. Current research shows that DGMs alone can fail to identify OOD
samples (Xiao et al.,, 2020). They not only assign high likelihood to OOD
samples, but they also exhibit good reconstruction quality. In fact, we show
in Table 5 that CVAEs trained on a given ID data generally exhibit a low
reconstruction error on most of the OOD samples. Furthermore, we show in Table
8 that using a trained CVAE likelihood output for OOD detection fails to
perform well. These results motivate the new for a new OOD scoring method for
the time-series domain.

Class-wise seasonality via STL decomposition. The proposed SRS algorithm
relies on the following assumption to analyze the time-series space for OOD
detection.

######  Assumption 1.

Each time-series example $(x_{i},y_{i})$ from the in-distribution data
$\mathcal{D}_{in}$ consists of two components. 1) A class-wise semantic
pattern $S_{y}$ for each class label $y\in\mathcal{Y}$ representing the
meaningful semantics of the class label $y$. 2) A remainder noise $r_{i}$
representing an additive perturbation to the semantic portion. Hence,
$\forall(x_{i},y_{i})\in\mathcal{D}_{in}:~{}x_{i}=S_{y_{i}}+r_{i}$

We propose to employ STL decompostion to estimate semantic pattern $S_{y}$ (as
illustrated in Figure 2) and deduce the remainder noise that can be due to
several factors including errors in sensor measurements and noise in
communication channel. These two components are analogous to the foreground
and the background of an image, where the foreground is the interesting
segment of the input that describes it, and the background may not be
necessarily related to the foreground. In spite of this analogy, prior methods
for the image domain are not suitable for time-series as explained in the
related work. In this decomposition, we cannot assume that $S_{y}$ and $r$ are
independent for a given time-series example $(x,y)$, as $S_{y}$ is class-
dependant and $r$ is the remainder of the input $x$ using $S_{y}$. Hence, we
present their conditional likelihoods in the following observation.

######  Observation 1.

Let $x\in\mathbb{R}^{n\times T}$ is a time-series signal and
$y_{i}\in\mathcal{Y}$=$\\{1,\cdots,C\\}$ be the corresponding class label. As
$x$ = $S_{y_{i}}+r$, we have:

$p(x|y_{i})=p(r|y_{i})p(S_{y_{i}}|y_{i})$ (2)

Proof of Observation 1 As $X=S_{y_{i}}+r$, it is intuitive to think that
$p(X|y=y_{i})=p(S_{y_{i}})\times p(r)$. However, we cannot assume that
$S_{y_{i}}$ and $r$ are independent, as $S_{y_{i}}$ is class-dependant and $r$
is the remainder of the input $X$ given $S_{y_{i}}$.

Therefore, we make use of the conditional probabilities of the components. The
likelihood $p(X)$ can be decomposed as follows:

$\begin{split}p(X|y=y_{i})&=p(S_{y_{i}},r|y=y_{i})\\\
&=\frac{p(S_{y_{i}},r,y=y_{i})}{p(y=y_{i})}\\\
&=\frac{p(r,y=y_{i})p(S_{y_{i}}|r,y=y_{i})}{p(y=y_{i})}\end{split}$

For the conditional probability $p(S_{y_{i}}|r,y=y_{i})$, as only the pattern
$S_{y_{i}}$ depends on the class label, and that we have defined $r$ as a non-
meaningful noise to the input, we can assume that $S_{y_{i}}$ and $r$ are
conditionally independent given the class $y_{i}$. Therefore, we have the
following.

$\begin{split}p(X|y=y_{i})&=\frac{p(r,y=y_{i})p(S_{y_{i}}|r,y=y_{i})}{p(y=y_{i})}\\\
&=\frac{p(r,y=y_{i})p(S_{y_{i}}|y=y_{i})}{p(y=y_{i})}\\\
&=\frac{p(r|y=y_{i})p(y=y_{i})p(S_{y_{i}}|y=y_{i})}{p(y=y_{i})}\\\
&=p(r|y=y_{i})p(S_{y_{i}}|y=y_{i})\\\ \end{split}$

OOD detection using CVAEs. Observation 2 shows the relationship between the
conditional likelihood of the input $x$ and its remainder $r$. Since both
likelihoods are conditional, we employ CVAEs. Recall that OOD examples come
from an unknown distribution which is different from the in-distribution
$P^{*}$ and do not belong to any pre-defined class label from $\mathcal{Y}$.
Therefore, we propose to use the following observation for OOD detection in
the time-series domain.

######  Observation 2.

Let $x\in\mathbb{R}^{n\times T}$ is a time-series signal and
$y\in\mathcal{Y}$=$\\{1,\cdots,C\\}$ be the corresponding class label. As $x$
= $S_{y}+r$, $x$ is an OOD example if $p(x|y)\neq p(r|y)$ and an in-
distribution example if $p(x|y)$ = $p(r|y)$.

Observation 2 shows how we can exploit the relationship between the estimated
conditional likelihood of the time-series signal $x$ and its remainder $r$ to
predict whether $x$ is an OOD example or not. This observation relies on the
assumption that $p(S_{y}|y)=1$ for in-distribution data. For ID data, the
semantic pattern $S_{y}$ is a class-dependant signal that defines the class
label $y$. Since the semantic component is guaranteed to be $S_{y}$ for any
time-series example with class label $y$, we have $p(S_{y}|y)$=1. On the other
hand, OOD examples do not belong to any class label from $\mathcal{Y}$, i.e.,
$p(S_{y}|y)\neq 1$ for any $y\in\mathcal{Y}$. To estimate
$p(x|y\in\mathcal{Y})$ and $p(r|y\in{\mathcal{Y}})$ in observation 2, we train
two separate CVAE models using the in-distribution data $\mathcal{D}_{in}$.

Discussion on Assumption 1. The paper acknowledges that this assumption may
fail to hold in some real-world scenarios. However, surprisingly, our
experimental results shown in Table 7 strongly corroborate this key
assumption: the distance between each time-series signal $x_{i}$ and its
semantic pattern $S_{y_{i}}$ is very small. The strong OOD performance of SRS
algorithm in our diverse experiments demonstrates the effectiveness of a
simple approach based on this assumption.

Suppose the assumption does not hold and some class label $y$ can possess
$K>1$ different semantics $\\{S^{k}_{y}\\}_{k\leq K}$. If we take a human
activity recognition example, it is safe to think of a certain activity (e.g.,
running or walking) will have $K>1$ different patterns (e.g., athletic runners
vs. young runners). Therefore, the decomposition in Assumption 1 for a given
time-series example $(x_{i},y_{i})$ will result into a semantic pattern
describing the patterns of the different sub-categories (e.g., a pattern that
describes both athletic runners and young runners). By using Lowess smoothing,
the STL season extracted over a multiple-pattern class is a pattern $S_{y}$
that is a linear combination of $\\{S^{k}_{y}\\}_{k\leq K}$ (for our example,
it describes the combination of both athletic runs and young runs). While for
an in-distribution example, $p(S_{y}|y)$=1 of Observation 2 will not hold,
$p(S_{y}|y)$ is likely to be well-defined from $p(S^{k}_{y}|y)$ as
$\\{S^{k}_{y}\\}_{k\leq K}$ are fixed and natural for the class label $y$.
Hence, we can still rely on the CVAEs to estimate this distribution and to
perform successful OOD detection. Alternatively, we can use a simple
reformulation of the problem by clustering time-series signals of a class
label $y$ (for which the assumption is not satisfied) to identify sub-classes
and apply the SRS algorithm on transformed data. Since we found the assumption
to be true in all our experimental scenarios (see Table 7), we didn’t find the
need to apply this reformulation.

Algorithm 1 Seasonal Ratio Scoring Algorithm for OOD Detection

1:$\mathcal{D}_{in}=\\{(x_{i},y_{i})\\}_{i=1}^{n}$, Input training/validation
time-series samples; $\mathcal{M}_{x}$, CVAE for training data with likelihood
$\mathcal{L}_{\mathcal{M}_{x}}$; $\mathcal{M}_{r}$, CVAE for data remainders
with likelihood $\mathcal{L}_{\mathcal{M}_{r}}$; $F_{\theta}$, Time-series
classifier; $X_{test}$, testing time-series signal;

2:OOD Boolean decision.

3:Pre-process(Offline)

4: Cluster the time-series data in $\mathcal{D}_{in}$, one cluster for each
class label ${y\in\mathcal{Y}=\\{1,\cdots,C\\}}$

5: Perform STL decomposition to compute class-wise semantics $\\{S_{y}\\}$

6: Compute remainder component $r_{i}$ for each time-series example
$(x_{i},y_{i})\in\mathcal{D}_{in}$

7: Train CVAEs $\mathcal{M}_{r}$ and $\mathcal{M}_{x}$ using the decomposed
data

8: Estimate the threshold interval $[\tau_{l},\tau_{u}]$ from sample SR scores
of in-distribution data

9:end Pre-process

10:Compute the predicted class label: $\hat{y}$=$F_{\theta}(X_{test})$

11:Compute the remainder component: $r_{test}$=$X_{test}-S_{y=\hat{y}}$

12:Estimate the input log-likelihood
$l_{x}=\mathcal{L}_{\mathcal{M}_{x}}(X_{test},\hat{y})$

13:Estimate remainder log-likelihood
$l_{r}=\mathcal{L}_{\mathcal{M}_{r}}(r_{test},\hat{y})$

14:Compute the seasonal ratio score: $SR(X_{test})$ = $l_{x}/l_{r}$

15:if $SR(X_{test})\in[\tau_{l},\tau_{u}]$ then

16: return FALSE

17:else

18: return TRUE

19:end if

###  4.2 OOD Detection Approach

One key advantage of SRS method is that it can be directly executed at
inference stage and does not require additional training similar to prior VAE-
based methods such as Likelihood Regret scoring.

Training stage. Our overall training procedure for time-series OOD detection
is as follows:

  1. 1.

Train a CVAE $\mathcal{M}_{x}$ using in-distribution data $\mathcal{D}_{in}$
to estimate the conditional likelihood $p(x|y\in\mathcal{Y})$ of time-series
signal $x$.

  2. 2.

Create one group $\mathcal{D}_{y}$ with all examples from in-distribution data
$\mathcal{D}_{in}$, which have class label $y\in\mathcal{Y}$. Execute STL
decomposition on each group $\mathcal{D}_{y}$ after serializing the data over
the $T$ dimension to extract the corresponding semantic component
$S_{y}\in\mathbb{R}^{n\times T}$.

  3. 3.

Create the remainder for each training example
$(x_{i},y_{i})\in\mathcal{D}_{in}$ using the patterns $S_{y}$ for each class
label $y$ $:~{}r_{i}=x_{i}-S_{y_{i}}$. We train another CVAE $\mathcal{M}_{r}$
using all these remainders to estimate the conditional likelihood
$p(r|y\in{\mathcal{Y}})$.

  4. 4.

Compute seasonal ratio score for each $(x_{i},y_{i})\in D_{in}$ using the
trained CVAEs $\mathcal{M}_{x}$ and $\mathcal{M}_{r}$.

$SR_{i}(x_{i},y_{i})\overset{\Delta}{=}\dfrac{p(x_{i}|y=y_{i})}{p(r_{i}|y=y_{i})}$
(3)

  5. 5.

Compute the mean $\mu_{SR}$ and variance $\sigma_{SR}$ over SR scores of all
in-distribution examples seen during training. Set the OOD detection threshold
interval as $[\tau_{l},\tau_{u}]$ such that $\tau_{l}$ =
$\mu_{SR}-\lambda\times\sigma_{SR}$ and $\tau_{u}$ =
$\mu_{SR}+\lambda\times\sigma_{SR}$, where $\lambda$ is a hyper-parameter.

  6. 6.

Tune the hyper-parameter $\lambda$ on the validation data to maximize OOD
detection accuracy.

The choice of $[\tau_{l},\tau_{u}]$ for OOD detection is motivated by the
fractional nature of the seasonal ratio scores. SRS algorithm assumes that in-
distribution examples satisfy $p(x|y)=p(r|y)$. Hence, we characterize in-
distribution examples with an SR score close to 1, whether from left
($\tau_{l}$) or right ($\tau_{u}$) side. To identify in-distribution examples,
we rely on SR scores that are close to the mean score recorded during
training, whether from left ($\tau_{l}$) or right ($\tau_{u}$) side. This
design choice is based on the fact that SR score is a quotient ideally
centered around the value 1. Indeed, we observe in Figure 3 that the SR score
for OOD examples can go on either the left or the right side of the SR scores
for in-distribution examples. Ideally, the SR score for in-distribution
examples is closest to $\mu_{SR}$ as illustrated in Figure 1. $\lambda$ is
tuned to define the valid range of SR scores for in-distribution examples from
$\mathcal{D}_{in}$. We note that the score can be changed easily to consider
the quantiles of estimated ratios during training stage and use it to separate
the region of OOD and ID score. Therefore, we can redefine
$\tau_{l}=(0.5-\lambda)$ as the $\tau_{l}$-quantile for the lower-limit of ID
score and $\tau_{u}=(0.5+\lambda)$ as the $\tau_{u}$-quantile for the upper-
limit of ID score. Given this definition, we need to tune the hyper-parameter
$0<\lambda\leq 0.5$ on the validation data to maximize OOD detection accuracy.
We have observed in our experiments that both these settings give similar
performance. Therefore, we only consider $\tau_{u,l}$ =
$\mu_{SR}\pm\lambda\times\sigma_{SR}$ for simplicity for our experimental
evaluation.

Figure 3: Histogram showing the ID and OOD scores along the seasonal ratio
score axis. The seasonal ratio scores for OOD examples can be either greater
or less than the seasonal ratio scores for ID examples.

Inference stage. Given a time-series signal $x$, our OOD detection approach
works as follows.

  1. 1.

Compute the predicted class label $\hat{y}$ using the classifier $F(x)$.

  2. 2.

Create the remainder component of $x$ with the predicted label $\hat{y}$: $r$
= $x-S_{\hat{y}}$.

  3. 3.

Compute conditional likelihoods $p(x|\hat{y})$ and $p(r|\hat{y})$ from trained
CVAE models $\mathcal{M}_{x}$ and $\mathcal{M}_{r}$.

  4. 4.

Compute the seasonal ratio score using conditional likelihoods.

$SR(x,\hat{y})=\frac{p(x|y=\hat{y})}{p(r|y=\hat{y})}$

  5. 5.

If the seasonal ratio score $SR(x,\hat{y})$ does not lie within the threshold
interval $[\tau_{l},\tau_{u}]$, then classify $x$ as OOD example. Otherwise,
classify $x$ as in-distribution example.

Algorithm 1 shows the complete pseudo-code including the offline training
stage and online inference stage for new time-series signals. For a given
time-series signal $x$ at the inference stage, we first compute the seasonal
ratio score. If the score is within $[\tau_{l},\tau_{u}]$, then the time-
series signal is classified as in-distribution. Otherwise, we flag it as an
OOD time-series signal.

###  4.3 Alignment method for improving the accuracy of SRS algorithm

In this section, we first motivate the need for pre-processing raw time-series
signals to improve the accuracy of SRS algorithm. Subsequently, we describe a
novel time-series alignment method based on dynamic time warping to achieve
this goal.

Motivation. The effectiveness of the SRS algorithm depends critically on the
accuracy of the STL decomposition. STL method employs fixed-length window over
the serialized data to estimate the recurring pattern. This is a challenge for
real-world time-series signals as they are prone to scaling, warping, and
time-shifts. We illustrate in Figure 4 the challenge of scaling, warping, and
time-shift occurrences in time-series data. The top-left figure depicts a set
of time-series signals with a clear ECG pattern. Due to their misalignment, if
we subtract one fixed ECG pattern from every time-series signal, the remainder
will be inaccurate. The figures in the left and right column show the
difference in the remainder components between the natural data (Left) and the
aligned version of the time-series data (Right). We can clearly observe that
the remainder components from the aligned data are more accurate. If input
time-series data is not aligned, it can significantly affect the estimation of
$p(r_{i}|y=y_{i})$ and the effectiveness of SRS for OOD detection. Hence, we
propose a novel alignment method using the class-wise semantic for the in-
distribution data $\mathcal{D}_{in}$ during both training and inference
stages.

Before alignment

After alignment

Figure 4: Illustration of the challenges in time-series data for STL
decomposition: semantic component and remainder. (Left column) Set of natural
time-series signals with an ECG wave as semantic component $S_{y}$ and the
corresponding remainders w.r.t $S_{y}$. (Right column) Time-series signals and
remainders from STL decomposition after applying the alignment procedure.

Time-series alignment algorithm. The overall goal of our approach is to
produce a class-wise aligned time-series signals using the ID data
$\mathcal{D}_{in}$ so that STL algorithm will produce accurate semantic
components $S_{y}$ for all $y\in\mathcal{Y}$. We propose to employ dynamic
time warping (DTW) (Müller,, 2007) based optimal alignment to achieve this
goal. The optimal DTW alignment describes the warping between two time-series
signals to make them aligned in time. It overcomes warping and time-shifts
issue by developing a one-to-many match over time steps. There are two key
steps in our alignment algorithm. First, we compute the semantic components
$S_{y}$ for all $y\in\mathcal{Y}$ from $\mathcal{D}_{in}$ using STL
decomposition. For each in-distribution example
$(x_{i},y_{i})\in\mathcal{D}_{in}$, we compute the optimal DTW alignment
between $S_{y_{i}}$ and $x_{i}$. Second, we use an appropriate time-series
transformation for each in-distribution example $(x_{i},y_{i})$ to improve the
DTW alignment from the first step. Specifically, we use the time-steps of the
longest one-to-many or many-to-one or sequential one-to-one sequence match to
select the Expand, Reduce, and Translate transformation as illustrated in
Figure 5.

We define these three time-series transformation below. Let
$X^{1}=(t^{1}_{1},t^{1}_{2},\cdots,t^{1}_{T})$ and
$X^{2}=(t^{2}_{1},t^{2}_{2},\cdots,t^{2}_{T})$ be two time-series signals of
length $T$.

  * •

Expand$(X^{1},X^{2})$: We employ this transformation for a one-to-many time-
step matching ($t^{1}_{i}$ is matched with $[t^{2}_{j},\cdots,t^{2}_{j+k}]$).
It duplicates the $t^{1}_{i}$ time-step for $k$ times.

  * •

Reduce$(X^{1},X^{2})$: We employ this transformation in the case of a many-to-
one time-step matching ($[t^{1}_{i},\cdots,t^{1}_{i+k}]$ is matched with
$t^{2}_{j}$). It replaces the time-steps $[t^{1}_{i},\cdots,t^{1}_{i+k}]$ by a
single averaged value.

  * •

Translate$(X^{1},X^{2})$: We employ this transformation in the case of a
sequential one-to-one time-step matching ($[t^{1}_{i},\cdots,t^{1}_{i+k}]$ is
matched one-to-one with $[t^{2}_{j},\cdots,t^{2}_{j+k}]$). It translates
$X_{1}$ to ensure that $t^{1}_{i}=t^{2}_{j}$.

One-to-Many

Many-to-One

One-to-One

Figure 5: Illustration of the use of appropriate transformation to adjust the
alignment between two time-series signals $X^{1}$ (blue signal) and $X^{2}$
(green signal).

We illustrate in Figure 6 two examples of transformation choices for time-
series signal $x$ when aligned with a pattern $S$. The alignment on the left
exhibits that the longest consecutive matching sequence is a one-to-many
($x_{4}$ is matched with $[S_{2},\cdots,S_{7}]$) while the alignment on the
right exhibits that the longest consecutive matching sequence is a sequential
one-to-one ($[x_{4},\cdots,x_{8}]$ is matched with $[S_{3},\cdots,S_{7}]$).

Figure 6: Illustration of two transformation choices for a time-series $x$
aligned with a pattern $S$. (Left) One-to-many as the longest match, calling
to use $Expand$ transformation. (Right) sequential one-to-one as the longest
match, calling to use the $Translate$.

##  5 Related Work

OOD detection via pre-trained models. Employing pre-trained deep neural
networks (DNNs) to detect OOD examples was justified by the observation that
DNNs with ReLU activation can produce arbitrarily high softmax confidence for
OOD examples (Dan Hendrycks,, 2017). Maximum probability over class labels has
been used (Dan Hendrycks,, 2017) to improve the OOD detection accuracy.
Building on the success of this method, temperature scaling and controlled
perturbations were used (Liang et al.,, 2018) to further increase the
performance. The Mahalanobis-based scoring method (Lee et al., 2018b, ) is
used to identify OOD examples with class-conditional Gaussian distributions.
Gram matrices (Sastry and Oore,, 2020) were used to detect OOD examples based
on the features learned from the training data. The effectiveness of these
prior methods depends critically on the availability of a highly-accurate DNN
for the classification task. However, this requirement is challenging for the
time-series domain as real-world datasets are typically small and exhibit high
class imbalance resulting in inaccurate DNNs Wen et al., (2020); Huang et al.,
(2016).

OOD detection via synthetic data. During the training phase, it is impossible
to anticipate the OOD examples that would be encountered during the deployment
of DNNs (Hendrycks et al.,, 2019). Hence, unsupervised methods (Yu and
Aizawa,, 2019) are employed or d synthetic data based on generative models is
created (Lee et al., 2018a, ; Lin et al.,, 2020) to explicitly regularize the
DNN weights over potential OOD examples. It is much more challenging to create
synthetic data for the time-series domain due to the limited data and their
ambiguity to validate by human experts.

OOD detection via deep generative models. The overall idea of using deep
generative models (DGMs) for OOD detection is as follows: 1) DGMs are trained
to directly estimate the in-distribution $P^{*}$; and 2) The learned DGM
identifies OOD samples when they are found lying in a low-density region.
Prior work has used auto-regressive generative models (Ren et al.,, 2019) or
GANs (Wang et al.,, 2020) and proposed scoring metrics such as likelihood
estimates to obtain good OOD detectors. DGMs are shown to be effective in
evaluating the likelihood of input data and to estimate the data distribution,
which makes them a good candidate to identify OOD examples with high accuracy.
However, as shown by (Nalisnick et al.,, 2018), DGMs can assign a high
likelihood to OOD examples. Likelihood ratio (Ren et al.,, 2019) and
likelihood regret (Xiao et al.,, 2020) are proposed to improve OOD detection.
While likelihood regret method can generalize to different types of data,
likelihood ratio is limited to categorical data distributions with the
assumption that the data contains background units (background pixels for
images and background sequences for genomes). Likelihood ratio cannot be
applied to the time-series domain for two reasons: 1) We need to deal with
continuous distributions; and 2) We cannot assume that time-steps (information
unit) can be independently classified as background or semantic content.

OOD detection via time-series anomaly detection. Generic Anomaly Detection
(AD) algorithms (Ruff et al.,, 2021) can be employed to solve OOD problems for
time-series data. We note that there exist some AD methods that can cover the
same setting as the OOD problem for time-series domain. However, both settings
are still considered as two different frameworks with two different goals Yang
et al., (2021). By definition, AD aims to detect and flag anomalous samples
that deviates from a pre-defined normality (Laptev et al.,, 2015; Canizo et
al.,, 2019) estimated during training. Under the AD assumption of normality,
such samples only originate from a covariate shift in the data distribution
(Ruff et al.,, 2021). Semantically, such samples do not classify as OOD
samples (Yang et al.,, 2021). For example, consider an intelligent system
trained to identify the movement of a person (e.g, run, stand, walk, swim),
where stumbling may occur during running. Such event would be classified as an
anomaly as the activity running is still taking place, but in an irregular
manner. However, if the runner slips and falls, such activity should be
flagged as OOD due to the fact that it does not belong to any of the pre-
defined activity classes. In other words, OOD samples must originate from a
different class distribution ($y_{\text{OOD}}\notin\mathcal{Y}$) than in-
distribution examples, while anomalies typically originate from the same
underlying distribution but with anomalous behavior. Additionally, anomalies
can manifest as a single time-step, non-static window-length, but not
generally a complete time-series example in itself. Such differences can be
critical for users and practitioners, which necessitates the study of separate
algorithms for AD and OOD. Unlike anomaly detection, OOD detection focuses on
identifying test samples with non-overlapping labels with in-distribution data
and can generalize to multi-class setting (Yang et al.,, 2021). The main
limitations of time-series AD algorithms Challu et al., (2022) for OOD
detection task are

  * •

OOD samples cannot be used as labeled anomalous examples during training due
to the general definition of the OOD space. For various AD methods such as
nearest neighbors and distance-based, the fine-tuning of the cut-off threshold
between ”normal” and ”anomalous” examples requires anomaly labels during
training. Mainly, window-based techniquesChandola et al., (2010) require both
normal and anomalous sequences during training, and if there are none,
anomalous examples are randomly generated. Such a requirement is not practical
for OOD problem settings as the distribution is ambiguous to define and sample
from.

  * •

AD assumes that normal samples are homogeneous in their observations. This
assumption helps the AD algorithm to detect anomalies. Such assumption cannot
hold for different classes of the in-distribution space for multi-class
settings. Therefore, time-series AD algorithms are prone to fail at detecting
OOD samples. Indeed, our experiments demonstrate the failure of state-of-the-
art time-series AD methods.

##  6 Experiments and Results

In this section, we present experimental results comparing the proposed SRS
algorithm and prior methods on diverse real-world time-series datasets.

###  6.1 Experimental Setup

Datasets. We employ the multivariate benchmarks from the UCR time-series
repository (Dau et al.,, 2019). Due to space constraints, we present the
results on representative datasets from six different pre-defined domains
Motion, ECG, HAR, EEG, Audio and Other. The list of datasets include
Articulary Word Recognition (AWR), Stand Walk Jump (SWJ), Cricket (Ckt), Hand
Movement Direction (HMD), Heartbeat (Hbt), and ERing (ERg). We employ the
standard training/validation/testing splits from these benchmarks.

OOD experimental setting. Prior work formalized the OOD experimental setting
for different domains such as computer vision (Dan Hendrycks,, 2017). However,
there is no OOD setting for the time-series domain. In what follows, we
explain the challenges for the time-series domain and propose a concrete OOD
experimental setting for it.

The first challenge with the time-series domain is the dimensionality of
signals. Let the ID space be $\mathbb{R}^{n_{i}\times T_{i}}$ and the OOD
space be $\mathbb{R}^{n_{o}\times T_{o}}$. Since we train CVAEs on the ID
space, ${n_{o}\times T_{o}}$ needs to match ${n_{i}\times T_{i}}$. Hence, if
$n_{o}>n_{i}$ or $T_{o}>T_{i}$, we window-clip the respective OOD dimension in
order to have $n^{\prime}_{o}=n_{i}$ or $T^{\prime}_{o}=T_{i}$. If
$n_{o}<n_{i}$ or $T_{o}<T_{i}$, we zero-pad the respective OOD dimension in
order to have $n^{\prime}_{o}=n_{i}$ or $T^{\prime}_{o}=T_{i}$. Zero-padding
is based on the assumption that the additional dimension exists but takes null
values. The second challenge is in defining OOD examples. Since the number of
datasets in UCR repository is large, conducting experiments on all
combinations of datasets as ID and OOD is impractical and repetitive (600
distinct configurations for the 25 different datasets considered in this
paper).

Hence, we propose two settings using the notion of domains.

  * •

In-domain OOD: Both ID and OOD datasets belong to the same domain. This
setting helps in understanding the behavior of OOD detectors when real-world
OOD example come from the same application domain. For example, a detector of
Epileptic time-series signals should consider signals resulting from sports
activity (Cricket) as OOD.

  * •

Cross-domain OOD: Both ID and OOD datasets come from two different domains.
This configuration is more intuitive for OOD detectors, where time-series
signals from different applications domains should not confuse the ML model
(e.g., Motion and HAR data).

Our intuition is that in-domain OOD setting is more likely to occur in the
real-world deployment. Hence, we propose to do separate experiments by
treating every dataset from the same domain as OOD. For the cross-domain OOD
setting, we believe that a single representative dataset from the domain can
be used as OOD. In this work, we focus on real-world OOD detection for the
time-series domain. Since random noise does not inherit the characteristics of
time-series data, methods from the computer vision literature have a good
potential in detecting random noise.

For improved readability and ease of understanding, we provide Table 2 and
Table 3 to explain the domain labels and dataset labels used in the
experimental section of our paper along with the corresponding UCR domain name
and dataset name.

  * •

Table 2 shows the label used to represent a given domain for Cross-Domain OOD
setting.

Table 2: List of domain labels used in the experimental section and the corresponding UCR domain name. Domain label | Domain name
---|---
D1 | Motion
D2 | ECG
D3 | HAR
D4 | EEG
D5 | Audio
D6 | Other
  * •

Table 3 shows the label used to represent the dataset used as an OOD source
against a given ID dataset for the In-Domain OOD setting. For example, while
reading Table 8 in the main paper, when AWR is the ID distribution, according
to Table 3, DS1 represents CharacterT. dataset. On the other hand, if HMD is
the ID distribution, DS1 represents FingerM. dataset.

Table 3: Reference table for the In-Domain dataset labels used in the
experimental section and the corresponding UCR dataset name. The second column
shows the average CVAE normalized reconstruction Mean Absolute Error (MAE)
with a negligible variance $\leq$ 0.001 on the in-distribution data.

In-distriution Dataset name | MAE | OOD Dataset label
---|---|---
DS1 | DS2 | DS3 | DS4 | DS5 | DS6 | DS7
ArticW. (Motion) | 0.025 | CharacterT. | EigenW. | PenD. | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$
EigenW. (Motion) | 0.000 | ArticW. | CharacterT. | PenD. | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$
PenD. (Motion) | 0.001 | ArticW. | CharacterT. | EigenW. | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$
AtrialF. (ECG) | 0.005 | StandW. | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$
StandW. (ECG) | 0.012 | AtrialF. | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$
BasicM. (HAR) | 0.024 | Cricket | Epilepsy | Handw. | Libras | NATOPS | RacketS. | UWaveG.
Cricket (HAR) | 0.010 | BasicM. | Epilepsy | Handw. | Libras | NATOPS | RacketS. | UWaveG.
Epilepsy (HAR) | 0.030 | BasicM. | Cricket | Handw. | Libras | NATOPS | RacketS. | UWaveG.
Handw. (HAR) | 0.006 | BasicM. | Cricket | Epilepsy | Libras | NATOPS | RacketS. | UWaveG.
Libras (HAR) | 0.003 | BasicM. | Cricket | Epilepsy | Handw. | NATOPS | RacketS. | UWaveG.
NATOPS (HAR) | 0.046 | BasicM. | Cricket | Epilepsy | Handw. | Libras | RacketS. | UWaveG.
RacketS. (HAR) | 0.026 | BasicM. | Cricket | Epilepsy | Handw. | Libras | NATOPS | UWaveG.
UWaveG. (HAR) | 0.015 | BasicM. | Cricket | Epilepsy | Handw. | Libras | NATOPS | RacketS.
EthanolC. (Other) | 0.001 | ER. | LSST | PEMS-SF | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$
ER. (Other) | 0.044 | EthanolC. | LSST | PEMS-SF | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$
LSST (Other) | 0.008 | EthanolC. | ER. | PEMS-SF | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$
PEMS-SF (Other) | 0.525 | EthanolC. | ER. | LSST | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$
FingerM. (EEG) | 0.048 | HandM. | MotorI. | SelfR1. | SelfR2. | $\emptyset$ | $\emptyset$ | $\emptyset$
HandM. (EEG) | 0.006 | FingerM. | MotorI. | SelfR1. | SelfR2. | $\emptyset$ | $\emptyset$ | $\emptyset$
MotorI. (EEG) | 0.543 | FingerM. | HandM. | SelfR1. | SelfR2. | $\emptyset$ | $\emptyset$ | $\emptyset$
SelfR1. (EEG) | 0.009 | FingerM. | HandM. | MotorI. | SelfR2. | $\emptyset$ | $\emptyset$ | $\emptyset$
SelfR2. (EEG) | 0.012 | FingerM. | HandM. | MotorI. | SelfR1. | $\emptyset$ | $\emptyset$ | $\emptyset$
Heartbeat (Audio) | 0.011 | JapaneseV. | SpokenA. | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$

Evaluation metrics. We employ the following two standard metrics in our
experimental evaluation. 1) AUROC score: The area under the receiver operating
characteristic curve is a threshold-independent metric. This metric (higher is
better) is equal to 1.0 for a perfect detector and 0.5 for a random detector.
2) F1 score: It is the harmonic mean of precision and recall. Due the
threshold dependence of F1 score, we propose to use the highest F1 score
obtained with a variable threshold. This score has a maximum of 1.0 in the
case of a perfect precision and recall.

Configuration of algorithms. We employ a 1D-CNN architecture for the CVAE
models required for seasonal ratio scoring (SR) method. We consider a naive
baseline where the CVAE is trained on the ID data and the likelihood (LL) is
used to detect OOD samples. We consider a variant of SR scoring (SRa) that
works on the aligned time-series data using the method explained in Section
4.3. We evaluate both SR and SRa against state-of-the-art baselines and employ
their publicly available code: Out-of-Distribution Images in Neural networks
(ODIN) (Liang et al.,, 2018) and Gram Matrices (GM) (Sastry and Oore,, 2020)
that have been shown to outperform most of the existing baselines; recently
proposed Likelihood Regret (LR) score (Xiao et al.,, 2020); adaptation of a
very recent time-series AD method referred to as Deep generative model with
hierarchical latent (HL) (Challu et al.,, 2022) that does not require labeled
anomalies for training purposes. We chose HL as the main baseline to represent
time-series AD under OOD setting as it is the state-of-the-art time-series AD
algorithm. HL for time-series was shown (Challu et al.,, 2022) to outperform
nearest-neighbor based methods, LSTM-based methods, and other methods
(Blázquez-García et al.,, 2021; Braei and Wagner,, 2020) in various AD
settings.

  * •

Choice of architecture: We have experimented with 3 different types of CVAE
architecture to decide on the most suitable one for our OOD experiments. We
evaluated 1) fully connected, 2) convolutional, and 3) LSTM based
architectures using the reconstruction error as the performance metric. We
have observed that fully connected networks generally suffer from poor
reconstruction performance especially on high-dimensional data. We have also
observed that LSTM’s runtime during training and inference is relatively
longer than the other architectures. However, CNN-based CVAEs delivered both a
good reconstruction performance and fast runtime.

  * •

1D-CNN CVAE details: To evaluate the the effectiveness of the proposed
seasonal ratio (SR) score, we employed a CVAE that is based on 1D-CNN layers.
The encoder of the CVAE is composed of 1) A minmax normalization layer, 2) A
series of 1D-CNN layers, and 3) A fully-connected layer. At the end of the
encoder, the parameters $\mu_{\text{CVAE}}$ and $\sigma_{\text{CVAE}}$ are
computed to estimate the posterior distribution. A random sample is then
generated from this distribution and passed on to the CVAE decoder along with
the class label. The decoder of the CVAE is composed of 1) A fully-connected
layer, 2) A series of transposed convolutional layer, and 3) A denormalization
layer.

  * •

CVAE Training: We use the standard training, validation, and testing split on
the benchmark datasets to train both CVAEs $\mathcal{M}_{x}$ and
$\mathcal{M}_{r}$. Both CVAEs are trained to maximize the ELBO on the
conditional log-likelihood defined in Section 3 using Adam optimizer with a
learning rate of $10^{-4}$. We employ a maximum number of training iterations
equal to $500$. To ensure the reliability of the performance of the proposed
CVAEs, we report in Table 3 the test reconstruction error of the trained CVAE
on ID data using Mean Absolute Error (MAE). We observe clearly that the
proposed CVAE is able to learn well the ID space as the reconstruction error
is relatively low. To compute the semantic patterns and remainders for in-
distribution examples for training $\mathcal{M}_{x}$ and $\mathcal{M}_{r}$, we
use the STLdecompose111https://github.com/jrmontag/STLDecompose.git python
package.

  * •

Implementation of the baselines: The baseline methods for
ODIN222https://github.com/facebookresearch/odin.git,
GM333https://github.com/VectorInstitute/gram-ood-detection.git,
HL444https://github.com/cchallu/dghl.git and
LR555https://github.com/XavierXiao/Likelihood-Regret.git were implemented
using their respective publicly available code with the recommended settings.
To employ ODIN and GM, we have trained two different DNN models: a 1D-CNN and
a LSTM for classification tasks with different settings. We report the average
performance of the baseline OOD detectors in our experimental setting. To
repurpose HL method from the AD setting to OOD setting, we have serialized the
training data and use it during the training of the generator. For OOD
detection at inference time, we serialize both the test ID data and OOD data
and shuffle them. By providing the window size equal to the time-steps
dimension of the original in-distribution inputs, we execute HL anomaly
detection algorithm and report every anomaly as an OOD sample. We employed the
default parameters of the generator. As recommended by the authors, we use a
hierarchical level equal to 4 and 500 iterations for training and inference.
We lower the learning rate to $10^{-6}$ to prevent exploding gradient occurred
using the default $10^{-3}$ value. For a fair comparison, the VAE for
Likelihood Regret (LR) has the same architecture as the CVAE used to estimate
seasonal ratio (SR) and the naive LL score.

###  6.2 Results and Discussion

Reconstruction error of DGMs. Table 5 shows the test reconstruction error of
the trained CVAE on ID data using Mean Absolute Error (MAE). We clearly
observe that CVAE model is able to learn the ID space as the reconstruction
error is relatively low. Table 5 shows analogous results for the same CVAE on
some OOD data. We observe that DGMs perform well on OOD samples regardless of
the different semantics of both ID and OOD data. The pre-trained CVAEs
performed well on the OOD AWR dataset with a reconstruction error $\leq 0.1$.
For the OOD FingerMovement (Fmv) dataset, only two out of the six CVAEs
exhibited an intuitive high reconstruction error.

OOD detection via pre-trained classifier and DGMs. Our first hypothesis is
that pre-trained DNN classifiers are not well-suited for OOD detection. To
test this hypothesis, we train two DNN models: a 1D-CNN and a RNN classifier.
We use these models for OOD detection using the ODIN and GM baselines. Table 4
shows that AUROC is low on all datasets. For datasets such as HMD and SWJ, the
AUROC score does not exceed 0.6 for any experimental setting. The accuracy of
DNNs for time-series classification is not as high as those for the image
domain for the reasons explained earlier. Hence, we believe that this
uncertainty of DNNs causes the baselines ODIN and GM to fail in OOD detection.

Table 4: Average AUROC results for ODIN and GMM. |  | AWR | SWJ | Ckt | HMD | Hbt | ERg
---|---|---|---|---|---|---|---
In- Domain | ODIN | 0.65$\pm 0.03$ | 0.54$\pm 0.01$ | 0.65$\pm 0.03$ | 0.55$\pm 0.01$ | 0.71$\pm 0.03$ | 0.70$\pm 0.03$
GM | 0.70$\pm 0.03$ | 0.58$\pm 0.01$ | 0.64$\pm 0.03$ | 0.58$\pm 0.02$ | 0.80$\pm 0.01$ | 0.75$\pm 0.03$
Cross- Domain | ODIN | 0.55$\pm 0.01$ | 0.54$\pm 0.01$ | 0.55$\pm 0.01$ | 0.50 | 0.70$\pm 0.01$ | 0.70$\pm 0.01$
GM | 0.56$\pm 0.01$ | 0.54$\pm 0.01$ | 0.65$\pm 0.03$ | 0.55$\pm 0.01$ | 0.75$\pm 0.03$ | 0.78$\pm 0.02$

Our second hypothesis is that DGMs assign high likelihood for OOD samples is
also applicable to time-series. While results in Table 5 corroborated this
hypothesis, we provide the use of a pre-trained CVAE for OOD detection (LL) in
Table 8. We observe that AUROC score of LL does not out-perform any of the
other baselines. Hence, a new scoring method is necessary for CVAE based OOD
detection.

Table 5: Average reconstruction error of CVAE is small on both ID and OOD data. The variance is $\leq 0.001$. ID Train | AWR | SWJ | Ckt | HMD | Hbt | ERg
---|---|---|---|---|---|---
ID Test | 0.025 | 0.012 | 0.010 | 0.118 | 0.011 | 0.045
OOD | AWR | $\emptyset$ | 0.018 | 0.035 | 0.039 | 0.002 | 0.137
Fmv | 1.658 | 0.146 | 0.146 | 0.039 | 0.071 | 6.132

Figure 7: Histogram showing the non-separability of ID and OOD LR scores (Top
row) and the separability using the seasonal ratio method on real-world time-
series data (Bottom row).

Random Noise as OOD. An existing experimental setting for OOD detection tasks
is to detect random noise. For this setting, we generate random noise as an
input sampled from a Gaussian distribution or a Uniform distribution. Table 6
shows the LR baseline performance on detecting the random noise as OOD
examples. We observe in Table 6 that LR has an excellent performance on this
task. This is explainable as time-series noise does not necessarily obey time-
series characteristics. Hence, the existing baselines can perform strongly on
the OOD examples. We motivate our seasonal ratio scoring approach for OOD
detection based on real-world examples. We have shown in the main paper that
existing baselines have poor performance on detecting real-world OOD examples,
where SR has significantly much better performance.

Table 6: Average performance of LR on OOD examples sampled from Gaussian/Uniform distribution. | AWR | SWJ | Ckt | HMD | Hbt | ERg
---|---|---|---|---|---|---
LR | 1.00 | 1.00 | 1.00 | 1.00 | 0.95 | 1.00
Table 7: Results for the validity of Assumption 1. Average distance (MAE and DTW measures) between the semantic pattern from STL $S_{y}$ and time-series example $x$ with label $y$ from the testing data (with a negligible variance$\leq 0.001$). | AWR | SWJ | Ckt | HMD | Hbt | ERg
---|---|---|---|---|---|---
MAE | 0.047 | 0.031 | 0.029 | 0.078 | 0.002 | 0.086
DTW | 0.039 | 0.018 | 0.022 | 0.068 | 0.002 | 0.069

Results for SR score. The effectiveness of SR score depends on the validity of
Assumption 1. Table 7 shows both MAE and DTW measure between semantic pattern
$S_{y}$ from STL and different time-series examples of the same class $y$. We
observe that the average difference measure is low. These results strongly
demonstrate that Assumption 1 holds empirically. For qualitative results, Fig.
7 shows the performance of SR in contrast to the performance of Likelihood
Regret (LR) shown in Fig. 7. This illustration shows that SRS provides
significantly better OOD separability.

Table 8: AUROC results for the baselines, SR, and SR with time-series
alignment (SRa) on different datasets for both in-domain and cross-domain OOD
setting. DSi is a label given to the dataset used as a real-world OOD data
compared to the ID dataset in the first cell of every column. Di is a label
given to the domain used to define the OOD distribution. $\emptyset$ denotes
non-existent setting, e.g., AWR’s corresponding domain has only three
datasets. The last two rows show the percentage of total experiments where SRa
outperforms the baseline methods, and SRa improves over $SR$ performance
respectively.

| In-domain OOD | Cross-domain OOD
---|---|---
| DS1 | DS2 | DS3 | DS4 | DS5 | DS6 | DS7 | D1 | D2 | D3 | D4 | D5 | D6
AWR (Motion) | LL | 0.80 | 0.85 | 0.54 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.81 | 0.80 | 0.57 | 0.59 | 0.81
HL | _0.50_ | 0.96 | 0.94 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | _0.50_ | 0.75 | 0.98 | _0.50_ | 0.56
LR | 0.90 | 0.95 | 0.66 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.61 | 0.84 | 0.56 | 0.72 | 0.61
| SR | 0.90 | 0.97 | 0.95 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 1.00 | 0.97 | 1.00 | 1.00 | 1.00
| SRa | 0.90 | 0.97 | 0.95 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
SWJ (ECG) | LL | 0.55 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.61 | $\emptyset$ | 0.51 | 1.00 | 0.77 | 0.52
HL | _0.50_ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | _0.50_ | $\emptyset$ | _0.50_ | _0.50_ | _0.50_ | _0.50_
LR | 0.97 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.64 | $\emptyset$ | _0.50_ | 1.00 | 0.67 | 0.99
| SR | 0.65 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.70 | $\emptyset$ | 0.61 | 1.00 | 0.96 | 0.61
| SRa | 0.67 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.70 | $\emptyset$ | 0.61 | 1.00 | 0.96 | 0.61
Ckt (HAR | LL | 0.89 | 0.85 | 0.84 | 0.81 | 0.82 | 0.91 | 0.79 | 0.79 | 0.82 | $\emptyset$ | 0.95 | 0.90 | 0.81
HL | 0.97 | _0.50_ | 0.99 | _0.50_ | _0.50_ | 0.51 | _0.50_ | _0.50_ | _0.50_ | $\emptyset$ | 0.52 | 0.94 | _0.50_
LR | 0.81 | 0.75 | 0.74 | 0.71 | 0.74 | 1.00 | 0.78 | 0.77 | 0.72 | $\emptyset$ | 0.95 | 0.88 | 0.71
| SR | 0.99 | 0.98 | 0.99 | 0.99 | 0.99 | 0.98 | 0.98 | 0.98 | 0.99 | $\emptyset$ | 0.99 | 0.98 | 0.98
| SRa | 0.99 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 0.98 | 0.98 | 0.99 | $\emptyset$ | 1.00 | 1.00 | 1.00
HMD (EEG) | LL | 0.88 | 0.88 | 0.89 | 0.89 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.89 | 0.80 | 0.87 | $\emptyset$ | 0.90 | 0.91
HL | 0.93 | 0.57 | 0.78 | 0.87 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.97 | _0.50_ | 0.98 | $\emptyset$ | 0.58 | 0.66
LR | 0.68 | 0.68 | 0.68 | 0.68 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.68 | 0.68 | 0.68 | $\emptyset$ | 0.80 | 0.68
| SR | 0.75 | 0.75 | 0.75 | 0.75 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.75 | 0.75 | 0.75 | $\emptyset$ | 0.83 | 0.75
| SRa | 0.90 | 0.90 | 0.90 | 0.90 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.75 | 0.84 | 0.89 | $\emptyset$ | 0.97 | 0.91
Hbt (Audio) | LL | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.90 | 0.95 | 0.94 | 0.85 | $\emptyset$ | 0.98
HL | _0.50_ | _0.50_ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.96 | _0.50_ | 0.78 | 0.62 | $\emptyset$ | 0.82
LR | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.93 | 0.94 | 0.94 | 0.75 | $\emptyset$ | 0.98
| SR | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.96 | 0.97 | 0.94 | 1.00 | $\emptyset$ | 1.00
| SRa | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.96 | 0.97 | 0.94 | 1.00 | $\emptyset$ | 1.00
ERg (Other) | LL | 0.83 | 0.77 | 0.75 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.88 | 0.86 | 0.82 | 0.77 | 0.76 | $\emptyset$
HL | _0.50_ | _0.50_ | _0.50_ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | _0.50_ | _0.50_ | _0.50_ | _0.50_ | _0.50_ | $\emptyset$
LR | 0.83 | 0.72 | 0.78 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.88 | 0.78 | 0.81 | 0.71 | 0.78 | $\emptyset$
| SR | 1.00 | 0.98 | 0.99 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.89 | 0.99 | 0.94 | 0.99 | 0.99 | $\emptyset$
| SRa | 1.00 | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.95 | 1.00 | 0.95 | 1.00 | 1.00 | $\emptyset$
Absolute | LL 00.0% | HL 05.0% | LR 05.0% | LL 0.00% | HL 06.7% | LR 03.30%
Wins (%) | Ties 20.0% | SRa 70.0% | Ties 16.7% | SRa 73.3%
$SR_{a}$ Improvement (%) | 55.0% | 40.0%

SR score vs. Baselines. Table 8 shows the OOD results for SRS and baseline
methods. For a fair comparison, we use the same architecture for VAEs
computing the LL, LR, and SR scores. We make the following observations. 1)
The naive LL method fails to outperform any other approach, which demonstrates
that DGMs are not reliable on their own as they produce high likelihood for
OOD samples. 2) Time-series anomaly detection method HL fails drastically for
various OOD settings as reflected by the poor AUROC score of 0.5. This
demonstrates that AD methods are not appropriate for OOD detection in the
multi-class setting. 3) SR score outperforms LR score in identifying OOD
examples on 80% of the total experiments. This means improvement is due to
better scoring function. 4) For the in-domain OOD setting, AUROC score of LR
is always lower than SRS. 3) For the cross-domain setting, SRS outperforms LR
in all cases except one experiment on a single dataset SWJ. 4) LR and SRS has
the same performance in 20% of the total experiments. Therefore, we conclude
that SRS is better than LR in terms of OOD performance and execution time (LR
requires new training for every single testing input unlike SR).

Alignment improves the accuracy of SR score. Our hypothesis is that extraction
of accurate semantic component using STL results in improved OOD detection
accuracy. To test this hypothesis, we compare SR and SRa (SR with aligned
time-series data). Table 8 shows the AUROC scores of SR and SRa. SRa improves
the performance of SR for around 50% of the overall experiments. For example,
on HMD dataset, we observe that SRa enhances the performance of SR by an
average of 15% under the in-domain OOD setting. These results strongly
corroborate our hypothesis that alignment improves OOD performance.

SR performance using F1-score. In addition to the AUROC score, we employ F1
score to assess the effectiveness of SR score in detecting OOD. Table 9
provides the results comparing SR score and LR score. Like AUROC score
evaluation, we make similar observation on F1 score. 1) SR score outperforms
LR score in identifying OOD examples on 60% of the total experiments. This
means improvement is due to better scoring function. 2) For the in-domain OOD
setting, F1 score of LR is mostly lower than SR. 3) For the cross-domain
setting, SR outperforms LR in 66% of the cases. Hence, we conclude that SR is
better than LR in terms of OOD performance measured as F1 metric.

Table 9: F1 metric results of LR, $SR_{a}$ on the different datasets for both
In-Domain and Cross-Domain setting. The last two rows show the percentage of
datasets where $SR_{a}$ is out-performing the $LR$ score.

| In-Domain OOD | Cross-Domain OOD
---|---|---
| DS1 | DS2 | DS3 | DS4 | DS5 | DS6 | DS7 | D1 | D2 | D3 | D4 | D5 | D6
AWR (Motion) | LR | 0.58 | 0.99 | 0.80 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.53 | 0.79 | 0.61 | 0.44 | 0.67
SRa | 0.69 | 0.89 | 0.97 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.97 | 0.85 | 1.00 | 0.99 | 1.00
SWJ (ECG) | LR | 0.69 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.98 | $\emptyset$ | 0.82 | 1.00 | 0.97 | 0.97
SRa | 0.69 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.98 | $\emptyset$ | 0.86 | 1.00 | 0.97 | 0.98
Ckt (HAR) | LR | 0.70 | 0.90 | 0.97 | 0.92 | 0.93 | 0.91 | 0.95 | 0.96 | 0.48 | $\emptyset$ | 1.00 | 0.94 | 0.93
SRa | 0.98 | 0.96 | 0.99 | 0.98 | 0.99 | 0.99 | 0.98 | 0.98 | 0.94 | $\emptyset$ | 0.99 | 0.97 | 1.00
HMD (HMD) | LR | 0.82 | 0.81 | 0.84 | 0.81 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.84 | 0.45 | 0.68 | $\emptyset$ | 0.80 | 0.82
SRa | 0.80 | 0.75 | 0.84 | 0.81 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.94 | 0.65 | 0.68 | $\emptyset$ | 0.90 | 0.92
Hbt (Audio) | LR | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.93 | 0.88 | 0.94 | 0.75 | $\emptyset$ | 0.98
SRa | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.93 | 0.88 | 0.94 | 1.00 | $\emptyset$ | 0.98
ERg (Other) | LR | 0.44 | 0.85 | 0.76 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.88 | 0.90 | 0.79 | 0.83 | 0.88 | $\emptyset$
SRa | 0.99 | 0.99 | 0.95 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.87 | 0.94 | 0.88 | 0.98 | 1.00 | $\emptyset$
Wins(%) | LR | 15.0% | 6.7%
Ties | 25.0% | 26.7%
| SRa | 60.0% | 66.6%

AUROC performance of SR scoring on the full multivariate UCR dataset. For the
sake of completeness, Table 10 provides additional results for the performance
of SR on all the UCR multi-variate datasets in terms of the AUROC score. These
results demonstrate that the proposed SR scoring approach is general and
highly-effective for all time-series datasets.

Table 10: AUROC results for LR, and SRa on different datasets for both in-
domain and cross-domain OOD setting. The last two rows shows respectively the
percentage of datasets where $SR_{a}$ is out-performing the $LR$ score.

| In-domain OOD | Cross-domain OOD
---|---|---
| DS1 | DS2 | DS3 | DS4 | DS5 | DS6 | DS7 | D1 | D2 | D3 | D4 | D5 | D6
ArticularyW. | LR | 0.90 | 0.95 | 0.66 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.61 | 0.84 | 0.56 | 0.72 | 0.61
SRa | 0.90 | 0.97 | 0.95 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
EigenW. | LR | 1.00 | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 1.00 | 1.00 | 0.88 | 1.00 | 1.00
SRa | 0.66 | 1.00 | 0.66 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.67 | 0.61 | 1.00 | 0.68 | 0.60
PenD. | LR | 1.00 | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 1.00 | 1.00 | 0.55 | 0.94 | 1.00
SRa | 0.71 | 0.72 | 0.68 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.67 | 0.70 | 0.99 | 0.96 | 0.70
AtrialF. | LR | 0.86 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.99 | $\emptyset$ | 0.92 | 1.00 | 0.97 | 0.58
SRa | 0.67 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.67 | $\emptyset$ | 0.79 | 1.00 | 0.99 | 0.60
StandW. | LR | 0.97 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.64 | $\emptyset$ | 0.50 | 1.00 | 0.67 | 0.99
SRa | 0.67 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.70 | $\emptyset$ | 0.61 | 1.00 | 0.96 | 0.61
BasicM. | LR | 0.54 | 0.54 | 0.54 | 0.55 | 0.54 | 0.54 | 0.54 | 0.54 | 0.57 | $\emptyset$ | 0.51 | 0.74 | 0.56
SRa | 0.55 | 0.55 | 0.55 | 0.57 | 0.53 | 0.53 | 0.52 | 0.55 | 0.62 | $\emptyset$ | 0.99 | 0.97 | 0.74
Cricket | LR | 0.81 | 0.75 | 0.74 | 0.71 | 0.74 | 1.00 | 0.78 | 0.77 | 0.72 | $\emptyset$ | 0.95 | 0.88 | 0.71
SRa | 0.99 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 | 0.98 | 0.98 | 0.99 | $\emptyset$ | 1.00 | 1.00 | 1.00
Epilepsy | LR | 0.71 | 0.84 | 0.91 | 0.62 | 0.76 | 0.79 | 0.89 | 0.65 | 0.53 | $\emptyset$ | 0.90 | 0.60 | 0.72
SRa | 0.75 | 0.59 | 0.69 | 0.82 | 0.80 | 0.70 | 0.63 | 0.71 | 0.74 | $\emptyset$ | 1.00 | 1.00 | 0.82
Handwriting | LR | 0.85 | 0.61 | 0.65 | 0.57 | 0.57 | 0.92 | 0.63 | 0.57 | 0.57 | $\emptyset$ | 0.84 | 0.68 | 0.57
SRa | 0.90 | 0.78 | 0.75 | 0.60 | 0.77 | 0.75 | 0.78 | 0.64 | 0.64 | $\emptyset$ | 1.00 | 1.00 | 0.74
Libras | LR | 0.78 | 0.84 | 0.82 | 0.83 | 0.73 | 0.67 | 0.78 | 0.81 | 0.97 | $\emptyset$ | 0.71 | 0.80 | 0.95
SRa | 0.89 | 0.93 | 0.98 | 0.90 | 1.00 | 0.87 | 0.93 | 0.95 | 0.84 | $\emptyset$ | 0.76 | 0.60 | 0.97
NATOPS | LR | 0.93 | 1.00 | 0.97 | 1.00 | 1.00 | 0.93 | 0.99 | 1.00 | 1.00 | $\emptyset$ | 0.83 | 0.52 | 1.00
SRa | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | $\emptyset$ | 1.00 | 1.00 | 1.00
RacketS. | LR | 0.51 | 0.51 | 0.51 | 0.51 | 0.51 | 0.51 | 0.51 | 0.51 | 0.51 | $\emptyset$ | 0.98 | 0.93 | 0.51
SRa | 0.61 | 0.66 | 0.65 | 0.65 | 0.69 | 0.64 | 0.66 | 0.67 | 0.67 | $\emptyset$ | 0.99 | 0.96 | 0.64
UWaveG. | LR | 0.79 | 0.94 | 0.84 | 0.93 | 0.57 | 0.64 | 0.76 | 0.83 | 0.55 | $\emptyset$ | 0.86 | 0.72 | 0.57
SRa | 0.97 | 0.94 | 0.96 | 0.96 | 0.97 | 0.89 | 0.93 | 0.95 | 0.97 | $\emptyset$ | 1.00 | 1.00 | 0.97
FingerM. | LR | 0.94 | 1.00 | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 1.00 | 1.00 | 1.00 | $\emptyset$ | 0.85 | 1.00
SRa | 0.74 | 0.92 | 0.90 | 0.90 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.90 | 0.90 | 0.90 | $\emptyset$ | 0.96 | 0.90
HandM. | LR | 0.68 | 0.68 | 0.68 | 0.68 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.68 | 0.68 | 0.68 | $\emptyset$ | 0.80 | 0.68
SRa | 0.90 | 0.90 | 0.90 | 0.90 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.75 | 0.84 | 0.75 | $\emptyset$ | 0.97 | 0.91
MotorI. | LR | 0.66 | 0.66 | 0.56 | 0.57 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.86 | 0.89 | 0.82 | $\emptyset$ | 0.98 | 0.83
SRa | 0.70 | 0.71 | 0.60 | 0.60 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.60 | 0.60 | 0.60 | $\emptyset$ | 1.00 | 0.60
SelfR1. | LR | 0.62 | 0.62 | 0.62 | 0.72 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.62 | 0.60 | 0.59 | $\emptyset$ | 0.78 | 0.62
SRa | 0.65 | 0.85 | 0.66 | 0.77 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.62 | 0.60 | 0.61 | $\emptyset$ | 0.99 | 0.77
SelfR2. | LR | 0.98 | 0.85 | 0.61 | 0.93 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.92 | 0.96 | 0.88 | $\emptyset$ | 1.00 | 0.95
SRa | 0.81 | 0.96 | 0.68 | 0.60 | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.83 | 0.83 | 0.91 | $\emptyset$ | 1.00 | 0.83
Heartbeat | LR | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.93 | 0.94 | 0.94 | 0.75 | $\emptyset$ | 0.98
SRa | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.96 | 0.97 | 0.94 | 1.00 | $\emptyset$ | 1.00
ERing | LR | 0.83 | 0.72 | 0.78 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.88 | 0.78 | 0.81 | 0.71 | 0.78 | $\emptyset$
SRa | 1.00 | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.95 | 1.00 | 0.95 | 1.00 | 1.00 | $\emptyset$
EthanolC. | LR | 0.99 | 0.99 | 0.99 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | $\emptyset$
SRa | 1.00 | 1.00 | 1.00 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | $\emptyset$
LSST | LR | 0.79 | 0.58 | 0.58 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.58 | 0.57 | 0.57 | 0.51 | 0.58 | $\emptyset$
SRa | 0.60 | 0.58 | 0.61 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.60 | 0.60 | 0.60 | 0.55 | 0.60 | $\emptyset$
PEMS-SF | LR | 0.77 | 0.94 | 0.84 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 0.92 | 1.00 | 0.95 | 0.97 | 0.99 | $\emptyset$
SRa | 1.00 | 1.00 | 0.97 | $\emptyset$ | $\emptyset$ | $\emptyset$ | $\emptyset$ | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | $\emptyset$
Wins(%) | LR | 22.3% | 17.7%
Ties | 15.7% | 20.0%
SRa | 62.0% | 62.3%

##  7 Summary and Future Work

We introduced a novel seasonal ratio (SR) score to detect out-of-distribution
(OOD) examples in the time-series domain. SR scoring relies on Seasonal and
Trend decomposition using Loess (STL) to extract class-wise semantic patterns
and remainders from time-series signals; and estimating class-wise conditional
likelihoods for both input time-series and remainders using deep generative
models. The SR score of a given time-series signal and the estimated threshold
interval from the in-distribution data enables OOD detection. Our strong
experimental results demonstrate the effectiveness of SR scoring and alignment
method to detect time-series OOD examples over prior methods. Immediate future
work includes applying seasonal ratio score based OOD detection to generating
synthetic time-series data for small-data settings.

##  8 Acknowledgments

The authors would like to thank Alan Fern for useful discussions regarding the
key assumption behind the seasonal ratio scoring approach.

## References

  * Blázquez-García et al., (2021) Blázquez-García, A., Conde, A., Mori, U., and Lozano, J. A. (2021).  A review on outlier/anomaly detection in time series data.  ACM Computing Surveys (CSUR).
  * Braei and Wagner, (2020) Braei, M. and Wagner, S. (2020).  Anomaly detection in univariate time-series: A survey on the state-of-the-art.  arXiv preprint arXiv:2004.00433.
  * Canizo et al., (2019) Canizo, M., Triguero, I., Conde, A., and Onieva, E. (2019).  Multi-head cnn–rnn for multi-time series anomaly detection: An industrial case study.  Neurocomputing.
  * Cao et al., (2020) Cao, T., Huang, C.-W., Hui, D. Y.-T., and Cohen, J. P. (2020).  A benchmark of medical out of distribution detection.  arXiv preprint arXiv:2007.04250.
  * Challu et al., (2022) Challu, C., Jiang, P., Wu, Y. N., and Callot, L. (2022).  Deep generative model with hierarchical latent factors for time series anomaly detection.  International Conference on Artificial Intelligence and Statistics (AISTATS).
  * Chandola et al., (2010) Chandola, V., Banerjee, A., and Kumar, V. (2010).  Anomaly detection for discrete sequences: A survey.  IEEE transactions on knowledge and data engineering, 24(5):823–839.
  * Cleveland et al., (1990) Cleveland, R. B., Cleveland, W. S., McRae, J. E., and Terpenning, I. (1990).  Stl: A seasonal-trend decomposition.  J. Off. Stat, 6(1):3–73.
  * Dan Hendrycks, (2017) Dan Hendrycks, K. G. (2017).  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In 5th International Conference on Learning Representations, ICLR.
  * Dau et al., (2019) Dau, H. A., Bagnall, A., Kamgar, K., Yeh, C.-C. M., Zhu, Y., Gharghabi, S., Ratanamahatana, C. A., and Keogh, E. (2019).  The ucr time series archive.  IEEE/CAA Journal of Automatica Sinica.
  * Doersch, (2016) Doersch, C. (2016).  Tutorial on variational autoencoders.  arXiv preprint arXiv:1606.05908.
  * Hendrycks et al., (2019) Hendrycks, D., Basart, S., Mazeika, M., Mostajabi, M., Steinhardt, J., and Song, D. (2019).  Scaling out-of-distribution detection for real-world settings.  arXiv preprint arXiv:1911.11132.
  * Huang et al., (2016) Huang, C., Li, Y., Loy, C. C., and Tang, X. (2016).  Learning deep representation for imbalanced classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Ismail Fawaz et al., (2019) Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., and Muller, P.-A. (2019).  Deep learning for time series classification: a review.  Data mining and knowledge discovery.
  * Jiang et al., (2011) Jiang, K., Huang, C., and Liu, B. (2011).  Part decomposing algorithm based on stl solid model used in shape deposition manufacturing process.  The International Journal of Advanced Manufacturing Technology.
  * Kim et al., (2017) Kim, J., Shin, N., Jo, S. Y., and Kim, S. H. (2017).  Method of intrusion detection using deep neural network.  In 2017 IEEE International Conference on Big Data and Smart Computing (BigComp). IEEE.
  * Laptev et al., (2015) Laptev, N., Amizadeh, S., and Flint, I. (2015).  Generic and scalable framework for automated time-series anomaly detection.  KDD ’15. Association for Computing Machinery.
  * (17) Lee, K., Lee, H., Lee, K., and Shin, J. (2018a).  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  In International Conference on Learning Representations (ICLR).
  * (18) Lee, K., Lee, K., Lee, H., and Shin, J. (2018b).  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In Advances in Neural Information Processing Systems (NeurIPS).
  * Liang et al., (2018) Liang, S., Li, Y., and Srikant, R. (2018).  Enhancing the reliability of out-of-distribution image detection in neural networks.  International Conference on Learning Representations (ICLR).
  * Lin et al., (2020) Lin, Z., Jain, A., Wang, C., Fanti, G., and Sekar, V. (2020).  Using gans for sharing networked time series data: Challenges, initial promise, and open questions.  In Proceedings of the ACM Internet Measurement Conference (IMC).
  * Liu et al., (2020) Liu, W., Wang, X., Owens, J., and Li, Y. (2020).  Energy-based out-of-distribution detection.  In Advances in Neural Information Processing Systems (NeurIPS).
  * McKinney et al., (2011) McKinney, W., Perktold, J., and Seabold, S. (2011).  Time series analysis in python with statsmodels.  Jarrodmillman Com, pages 96–102.
  * Müller, (2007) Müller, M. (2007).  Dynamic time warping.  Information retrieval for music and motion, pages 69–84.
  * Nalisnick et al., (2018) Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., and Lakshminarayanan, B. (2018).  Do deep generative models know what they don’t know?  arXiv preprint arXiv:1810.09136.
  * Ren et al., (2019) Ren, J., Liu, P. J., Fertig, E., Snoek, J., Poplin, R., Depristo, M., Dillon, J., and Lakshminarayanan, B. (2019).  Likelihood ratios for out-of-distribution detection.  In Advances in Neural Information Processing Systems (NeurIPS).
  * Ruff et al., (2021) Ruff, L., Kauffmann, J. R., Vandermeulen, R. A., Montavon, G., Samek, W., Kloft, M., Dietterich, T. G., and Müller, K.-R. (2021).  A unifying review of deep and shallow anomaly detection.  Proceedings of the IEEE.
  * Sastry and Oore, (2020) Sastry, C. S. and Oore, S. (2020).  Detecting out-of-distribution examples with Gram matrices.  In Proceedings of the 37th International Conference on Machine Learning (ICML).
  * Smith and Smith, (2020) Smith, K. E. and Smith, A. O. (2020).  Conditional gan for timeseries generation.  arXiv preprint arXiv:2006.16477.
  * Tripathy and Acharya, (2018) Tripathy, R. and Acharya, U. R. (2018).  Use of features from rr-time series and eeg signals for automated classification of sleep stages in deep neural network framework.  Biocybernetics and Biomedical Engineering.
  * Wang et al., (2020) Wang, Z., Dai, B., Wipf, D., and Zhu, J. (2020).  Further analysis of outlier detection with deep generative models.  In Advances in Neural Information Processing Systems (NeurIPS).
  * Wen et al., (2020) Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., and Xu, H. (2020).  Time series data augmentation for deep learning: A survey.  arXiv preprint arXiv:2002.12478.
  * Xiao et al., (2020) Xiao, Z., Yan, Q., and Amit, Y. (2020).  Likelihood regret: An out-of-distribution detection score for variational auto-encoder.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems (NeurIPS).
  * Yang et al., (2021) Yang, J., Zhou, K., Li, Y., and Liu, Z. (2021).  Generalized out-of-distribution detection: A survey.  arXiv preprint arXiv:2110.11334.
  * Yang, (2021) Yang, S. (2021).  A novel study on deep learning framework to predict and analyze the financial time series information.  Future Generation Computer Systems.
  * Yang* et al., (2020) Yang*, Y.-Y., Rashtchian*, C., Zhang, H., Salakhutdinov, R., and Chaudhuri, K. (2020).  A closer look at accuracy vs. robustness.  In Proceedings of Neural Information Processing Systems (NeurIPS).
  * Yu and Aizawa, (2019) Yu, Q. and Aizawa, K. (2019).  Unsupervised out-of-distribution detection by maximum classifier discrepancy.  In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  * Zheng et al., (2017) Zheng, Z., Yang, Y., Niu, X., Dai, H.-N., and Zhou, Y. (2017).  Wide and deep convolutional neural networks for electricity-theft detection to secure smart grids.  IEEE Transactions on Industrial Informatics.