#  Diversify: A General Framework for Time Series Out-of-distribution
Detection and Generalization

Wang Lu, Jindong Wang, Xinwei Sun, Yiqiang Chen, , Xiangyang Ji, , Qiang Yang,
, and Xing Xie Wang Lu and Xiangyang Ji are with Tsinghua University. E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Jindong Wang and Xing Xie are
with Microsoft Research Asia. E-mail: {jindong.wang<EMAIL_ADDRESS>Xinwei Sun is with Fudan University, Shanghai 200437, China. E-mail:
<EMAIL_ADDRESS>Yiqiang Chen is with Beijing Key Laboratory of Mobile
Computing and Pervasive Devices, CAS. E-mail<EMAIL_ADDRESS>Qiang Yang is
with Hong Kong University of Science and Technology, Hong Kong. E-mail:
<EMAIL_ADDRESS>Correspondence to: Jindong Wang
(jindong.wang@microsoft.com). Manuscript received April 19, 2021; revised
August 16, 2021.

###### Abstract

Time series remains one of the most challenging modalities in machine learning
research. The out-of-distribution (OOD) detection and generalization on time
series tend to suffer due to its non-stationary property, i.e., the
distribution changes over time. The _dynamic_ distributions inside time series
pose great challenges to existing algorithms to identify invariant
distributions since they mainly focus on the scenario where the domain
information is given as prior knowledge. In this paper, we attempt to exploit
subdomains within a whole dataset to counteract issues induced by non-
stationary for generalized representation learning. We propose Diversify, a
general framework, for OOD detection and generalization on dynamic
distributions of time series. Diversify takes an iterative process: it first
obtains the _‘worst-case’_ latent distribution scenario via adversarial
training, then reduces the gap between these latent distributions. We
implement Diversify via combining existing OOD detection methods according to
either extracted features or outputs of models for detection while we also
directly utilize outputs for classification. In addition, theoretical insights
illustrate that Diversify is theoretically supported. Extensive experiments
are conducted on seven datasets with different OOD settings across gesture
recognition, speech commands recognition, wearable stress and affect
detection, and sensor-based human activity recognition. Qualitative and
quantitative results demonstrate that Diversify learns more generalized
features and significantly outperforms other baselines.

###### Index Terms:

Domain Generalization, Out-of-distribution, OOD Detection, Time Series,
Representation Learning

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE

##  Sec. 1 Introduction

Time series analysis is one of the most challenging problems in machine
learning [1, 2]. For years, there have been tremendous efforts for time series
classification, such as hidden Markov models [3], RNN-based methods [4], and
Transformer-based approaches [5, 6]. Time series has wide applications in a
wide spectrum of applications, e.g., industrial process [7], stock predicting
[8], and clinical and remote health [9]. There are several active research
areas related to time series, including classification, forecasting,
clustering, multivariate analysis, and high-frequency time series analysis.

The primary focus of this paper is to learn general representations for time
series for better out-of-distribution (OOD) detection [10] and generalization
[11]. The main difference between OOD detection and generalization is the
different types of distribution shift that cause the problem: label shift and
feature shift, according to variables that change in distributions. On the one
hand, label shift normally occurs in classes, which means unseen targets can
contain classes not present in the training data, which is studied extensively
under the name of anomaly detection and OOD detection111We noticed that there
are some methods performing OOD detection under feature shift [12, 13], but
this paper mainly deals with OOD detection caused by label shift, following
existing work [10, 14].; On the other hand, feature shift typically happens in
inputs and the corresponding research field is OOD generalization that has
been extensively studied [11].

OOD detection attempts to solve label shift and it has also attracted much
attention recently [15, 16, 17, 18]. OOD detection can be viewed as a special
classification task that distinguishes between in-distribution (ID) classes
and OOD classes [10, 14]. For example, Hendrycks et al. [19] trained OOD
detectors against an auxiliary dataset of outliers to improve deep OOD
detection while BATS [20] rectified the feature into its typical set and
calculated the OOD score with the typical features to achieve reliable
uncertainty estimation. As for OOD generalization, existing approaches assume
the existence of several predefined domains and then endeavor to bridge gaps
among domains to learn domain-invariant representations that can be seamlessly
transferred to the unseen target distribution. The key in existing algorithms
is to exploit the given domain information (i.e., domain index) to guide the
domain-invariant representation learning. For instance, GILE [21] learned to
automatically disentangle domain-agnostic and domain-specific features for
generalizable sensor-based cross-person activity recognition while SDMix [22]
provided a semantic data augmentation method to solve a similar problem. These
methods still heavily rely on domain information.

Can we directly adopt existing OOD detection and generalization algorithms for
time series? Unfortunately, the answer is no. Non-stationary property [23],
i.e. statistical features changing over time, bring new challenges to time
series OOD detection and generalization. Besides common spatial shifts in
feature space, non-stationary property leads to another features shifts named
temporal shifts that can occur in one class for the same subject at different
times. Temporal shifts are often _latent_ , _dynamic_ , and _variable_ , which
makes it difficult to pre-split data and leads manual splits according to
priors inaccurate. Moreover, different from computer vision, few time-series
datasets are well pre-partitioned. To the best of our knowledge, no work
studies OOD representation by considering temporal shifts for time series OOD
detection and generalization simultaneously.

Figure 1: Illustration of Diversify: (a) Domain generalization for image data
requires known domain labels. (b) Domain labels are unknown for time series.
(c) If we treat the time series data as one single domain, the sub-domains are
misclassified. Different colors and shapes correspond to different classes and
domains. Axes represent data values. (d) Finally, our Diversify can
effectively learn the latent distributions. X-axis represents data numbers
while Y-axis represents values.

Figure 1 shows an illustrative example. OOD generalization in image
classification often involves several domains whose domain labels are static
and known (subfigure (a)), which can be employed to build OOD models. However,
Figure 1(b) shows that in EMG time series data [24], the distribution is
changing dynamically over time and its domain information is _unavailable_. If
no attention is paid to exploring its _latent_ distributions (i.e., sub-
domains), predictions may fail in face of diverse sub-domain distributions
(subfigure (c)). This will dramatically impede existing OOD algorithms due to
their reliance on domain information.

In this work, we propose Diversify, a general representation learning
framework for time series OOD detection and generalization by characterizing
the latent distributions inside the data. The ultimate idea of Diversify is to
characterize the latent distributions inside time series without domain
labels, which can then be leveraged to perform OOD detection and
generalization. Concretely speaking, Diversify consists of a min-max
adversarial game: on one hand, it learns to segment the time series data into
several latent sub-domains by maximizing the segment-wise distribution gap to
preserve diversities, i.e., the ‘worst-case’ distribution scenario; on the
other hand, it learns domain-invariant representations by reducing the
distribution divergence between the obtained latent domains. Such latent
distributions naturally exist in time series, e.g., the activity data from
multiple people follow different distributions. Additionally, our experiments
show that even the data of one person still has such diversity: it can also be
split into several latent distributions. After obtaining the characterized
latent distributions, we can establish different implementations for
downstream purposes such as OOD detection and generalization. Specifically for
OOD detection, we provide two implementations, where Diversify-MAH utilizes
the Mahalanobis distance with learned generalized representations while
Diversify-MCP makes use of logit outputs of models. With a simple softmax
activation, Diversify can be easily used for generalization. Since Diversify
can provide better representations and better predictions, all proposed
implementations can significantly outperform other methods.

This paper extends our previous paper published at ICLR 2023 [15], which
focuses only on OOD generalization. Compared to the previous version, this
version makes substantial extensions by formulating Diversify as a general
framework for both OOD detection and generalization, and then develops novel
algorithms for OOD detection with more experiments and analysis.

To summarize, our contributions are four-fold:

  * •

General framework: We propose a general framework, Diversify, to solve OOD
detection and generalization simultaneously. Diversify can identify the latent
distributions and learn generalized representations. We provide the
theoretical insights behind Diversify to analyze its design philosophy.

  * •

Specific implementations: For detection, we provide two implementations,
Diversify-MAH and Diversify-MCP. For classification, we directly utilize
outputs of Diversify with softmax activation.

  * •

Superior performance and insightful results: Qualitative and quantitative
results demonstrate the superiority of Diversify in several challenging
scenarios: difficult tasks, significantly diverse datasets, and limited data.
More importantly, Diversify can successfully characterize the latent
distributions within a time series dataset.

  * •

Extensibility: Besides implementations proposed in the paper, Diversify is an
extensible framework, which means it can implement with more methods, e.g.
ODIN [25]. Thereby, Diversify can be applied to more applications and be
further improved with more latest methods.

The remainder of this paper is organized as follows. We will introduce related
work in section 2 and elaborate on the proposed method and offer a clear
summary in section 3. Then, the experimental implementations and results are
presented to demonstrate the superiority of Diversify for detection and
generalization in section 4 and section 5 respectively while experimental
analyses are provided in section 6. section 7 provides some limitations and
discussions. Finally, conclusions and some possible future directions can be
found in section 8.

##  Sec. 2 Related Work

###  2.1 Time series analysis

Time series classification is a challenging problem. Existing researches
mainly focus on the modeling of temporal relations using either RNN-based
methods [26] or the recently-proposed Transformer architecture [6]. MiniRocket
[27] transformed input time series using convolutional kernels and used the
transformed features to train a linear classifier. MI-ShaRNN [26] was proposed
to induce long-term dependencies, and yet admit parallelization. In this
architecture, the first layer split inputs and ran several independent RNNs
while the second layer consumed the outputs using a second RNN. PatchTST [28]
segmented time series into subseries-level patches which served as input
tokens to Transformer and utilized channel-independence where each channel
contained a single univariate time series that shared the same embedding and
Transformer weights across all the series. More related work and details can
be found in the following surveys [29, 30, 31]. However, few studies pay
attention to latent subdomains in time series to learn generalized features
for OOD detection and generalization.

###  2.2 Domain/OOD generalization

Transfer learning [32, 33] trains a model on a source task and aims to enhance
the performance of the model on a different but related target task. Domain
generalization (DG)/out-of-distribution generalization can be viewed as a
branch of transfer learning but aims at learning models that can be
generalized to unseen targets whose distribution can be different from
training [11, 34]. Existing methods can be categorized into three groups,
namely: data manipulation [35, 36], representation learning [37, 38], and
learning strategy [39, 40]. Data manipulation mainly focuses on manipulating
the inputs and generating more diversified data or representations to enhance
models’ generalizability. CROSSGRAD[35] utilized a Bayesian network to model
dependence between label, domain, and input instance, and it parallelly a
label and a domain classifier on examples perturbed by loss gradients of each
other’s objectives. FACT [36] assumed that the Fourier phase information
contained high-level semantics and was not easily by domain shifts. And
thereby, it utilized an amplitude mix that linearly interpolated between the
amplitude spectrum of two images to force the model to capture phase
information. Representation learning is the most popular category in domain
generalization and it can be further split into two sub-groups, domain-
invariant representation learning and feature disentanglement. CIAN [37], a
conditional invariant adversarial network, learned class-wise adversarial
network for DG. StableNet [38] utilized a novel nonlinear feature
decorrelation approach based on Random Fourier features with linear
computational complexity and it could effectively partial out the irrelevant
features and leverage truly relevant features for classification. Learning
strategy focuses on exploiting special learning strategy to promote
generalization capability. Fishr [39] introduced a new regularization and
enforced domain invariance in the space of the gradients via the gradient
covariance similar to CORAL [41]. SelfReg [40] proposed a new regularization
method for domain generalization based on contrastive learning and it only
utilized positive data pairs. More details can be found in the survey [11]. We
will introduce more related work and illustrate their difference from ours.

Most domain/OOD generalization methods typically assume the availability of
domain labels for training [42, 43]. Specifically, [44] also studied DG
without domain labels by clustering with the style features for images, which
is not applied to time series and is not end-to-end trainable. Single domain
generalization is similar to our problem setting which also involves one
training domain [45, 46, 47, 48]. However, they treated the single domain as
one distribution and did not explore latent distributions. Multi-domain
learning is similar to DG which also trains on multiple domains but also tests
on training distributions. [49] proposed sparse latent adapters to learn from
unknown domain labels, but their work does not consider the min-max worst-case
distribution scenario and optimization. In domain adaptation, [50] proposed
the notion of domain index and further used variational models to learn them
[51], but took a different modeling methodology since they did not consider
min-max optimization. Mixture models [52] are models representing the presence
of subpopulations within an overall population, e.g., Gaussian mixture models.
Our approach has a similar formulation but does not use generative models.
Subpopulation shift is a new setting [53] that refers to the case where the
training and test domains overlap, but their relative proportions differ. Our
problem does not belong to this setting since we assume that these
distributions do not overlap. Distributionally robust optimization [54] shares
a similar paradigm with our work, whose paradigm is also to seek a
distribution that has the worst performance within a range of the raw
distribution. GroupDRO [55] studied DRO at a group level. However, we study
the internal distribution shift instead of seeking a global distribution close
to the original one. To our best knowledge, there is only one recent work [8]
that studied time series from the distribution level. However, AdaRNN is a
two-stage non-differential method that is tailored for RNN and it is mainly
designed for prediction.

###  2.3 OOD detection

OOD detection has been extensively studied with a plethora of methods
developed in the past few years. In simple terms, OOD detection aims to find
OOD samples that belong to the classes not present in training data. Existing
methods can be categorized into several groups, post-hoc methods, training-
time regularization, training with Outlier Exposure, and some other methods.
Comprehensive surveys can be found in [10, 14].

Post-hoc [25, 56, 57] is one of the most popular directions for its simplicity
and extensibility. Odin [25] utilized temperature scaling and added small
perturbations to the input to separate the softmax score distributions between
ID and OOD images for more effective detection while RankFeat [56] removed the
rank-1 matrix composed of the largest singular value and the associated
singular vectors from the high-level feature. Another post-hoc method, ASH
[57] removed a large portion of a sample’s activation at a late layer and
simply adjusted the rest at inference time. Compared to post-hoc methods,
training-time regularization methods require training with custom-designed
goals [58, 59]. CIDER [58] jointly optimized a dispersion loss and a
compactness loss to promote strong ID-OOD separability for exploiting better
hyperspherical embeddings while NPOS [59] generated artificial OOD training
data and facilitated learning a reliable decision boundary between ID and OOD
data. Training with Outlier Exposure [60, 61] makes use of a set of collected
OOD samples during training to learn the discrepancy between ID and OOD. [60]
proposed a two-head deep convolutional neural network (CNN) and maximized the
discrepancy between the two classifiers to detect OOD inputs while [61]
proposed unsupervised dual grouping (UDG) to leverage an external unlabeled
set for the joint modeling of ID and OOD data. Few studies consider feature
shifts in OOD detection. [62] introduced full-spectrum OOD detection and took
into account both covariate shift and semantic shift but it mainly focused on
computer vision. For time series OOD detection, there still lacks effective
techniques to address both two types of distribution shifts simultaneously.

##  Sec. 3 Methodology

###  3.1 Problem Formulation

A time-series training dataset $\mathcal{D}^{tr}$ can be often pre-processed
using sliding window222Sliding window is a common technique to segment one
time series data into fixed-size windows. Each window is a minimum instance.
We focus on fixed-size inputs for its popularity in time series [63]. to $N$
inputs: $\mathcal{D}^{tr}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{N}$, where
$\mathbf{x}_{i}\in\mathcal{X}\subset\mathbb{R}^{p}$ is the $p$-dimensional
instance and $y_{i}\in\mathcal{Y}=\\{1,\ldots,C\\}$ is its label. We use
$\mathbb{P}^{tr}(\mathbf{x},y)$ on $\mathcal{X}\times\mathcal{Y}$ to denote
the joint distribution of the training dataset. Our goal is to learn a
generalized model from $\mathcal{D}^{tr}$ to predict well on an _unseen_
target dataset, $\mathcal{D}^{te}$, which is inaccessible in training. In our
problem, the training and test datasets have the same input but different
distributions, i.e., $\mathcal{X}^{tr}=\mathcal{X}^{te}$, but
$\mathbb{P}^{tr}(\mathbf{x},y)\neq\mathbb{P}^{te}(\mathbf{x},y)$.

OOD detection: The testing datasets contain more classes than training
datasets, i.e. $\mathcal{Y}^{tr}\subset\mathcal{Y}^{te}$. We denote the
classes present in the training datasets as ID classes,
$C_{ID}=\\{1,2,\cdots,C_{n}\\}$, while the rest classes that only exist in the
testing datasets are the OOD class, $C_{OOD}=\\{C_{n}+1\\}$. We aim to train a
model $h$ from $\mathcal{D}^{tr}$ to detect OOD classes and achieve minimum
error on $\mathcal{D}^{te}$for ID classes.

OOD generalization: the training and test datasets share the same output
space, i.e. $\mathcal{Y}^{tr}=\mathcal{Y}^{te}$. We aim to train a model $h$
from $\mathcal{D}^{tr}$ to achieve minimum error on $\mathcal{D}^{te}$.

###  3.2 Motivation

What are domain and feature distribution shifts in time series? Time series
may consist of several unknown latent distributions (domains). For instance,
data collected by sensors of three persons may belong to two different
distributions due to their dissimilarities. This can be termed a spatial
distribution shift. Surprisingly, we even find temporal distribution shifts
that distributions of one person can also change at different times. Those
shifts widely exist in time series, as suggested by [64, 65]. Figure 2 gives
an example where Figure 2(a) indicates that the distribution in EMG time
series data [24] is changing dynamically over time and its domain information
is _unavailable_ while Figure 2(b) shows that the sensor data collected during
walking follow different distributions across different persons. In addition,
Figure 2(c) provides an example of label shifts, where standing, running, and
cycling can be in-distribution (ID) classes present in the training data while
falling down is an OOD class only present in the targets.

Latent domain characterization is indispensable for OOD detection and
generalization. Due to the non-stationary property, naive approaches that
treat time series as one distribution fail to capture domain-invariant (OOD)
features since they ignore the diversities inside the dataset. In Figure 1(d),
we assume the training domain contains two sub-domains (circle and plus
points). Directly treating it as one distribution via existing OOD approaches
may generate the black margin. Red star points are misclassified to the green
class when predicting on the OOD domain (star points) with the learned model.
Thus, multiple diverse latent distributions in time series should be
characterized to learn better OOD features which are essential factors
affecting performance of both OOD detection and generalization when
encountering non-stationary. We name distribution shifts in Figure 1(b)
spatial distribution shifts, for which we can group data into different
domains according to some specific characteristics, e.g., persons, positions,
and some other factors. However, in real scenarios, the information can be
missing or not suitable for grouping and we only have access to a whole
dataset without splits.

Figure 2: Categories of distribution shifts.

A brief formulation of latent domain characterization. Following the above
discussions, according to feature shifts, a time series may consist of $K$
unknown latent domains333We assume $K\in[N]$ in this work, as a smaller $K$ is
too coarse to show the diversities in distributions while a larger $K$ brings
difficulties to optimization. $K$ is tuned here but we hope to learn it in the
future.444A domain is a set of data samples following a certain distribution
and we use them interchangeably hereafter. rather than a fixed one, i.e.,
$\mathbb{P}^{tr}(\mathbf{x},y)=\sum_{i=1}^{K}\pi_{i}\mathbb{P}^{i}(\mathbf{x},y)$,
where $\mathbb{P}^{i}(\mathbf{x},y)$ is the distribution of the $i$-th latent
one with weight $\pi_{i}$, $\sum_{i=1}^{K}\pi_{i}=1$.555We use the notations
$\pi_{i}$ and $\mathbb{P}^{i}$ to only describe the problem, but do not
formalize it. There could be infinite ways to obtain $\mathbb{P}^{i}$s and our
goal is to learn the ‘worst-case’ distribution scenario where the distribution
divergence between each $\mathbb{P}^{i}$ and $\mathbb{P}^{j}$ is maximized.
Why the ‘worst-case’ scenario? It will maximally preserve the diverse
information of each latent distribution, thus benefiting generalization.

###  3.3 Diversify

Figure 3: The framework of Diversify.

In this paper, we propose Diversify to learn OOD representations for time
series OOD detection and generalization. The core of Diversify is to
characterize the latent distributions and then minimize the distribution
divergence between each two. Concretely speaking, an iterative process is
utilized: it first obtains the ’worst-case’ distribution scenario from a given
dataset, then bridges the distribution gaps between each pair of latent
distributions. It mainly contains four steps, where step $2\sim 4$ are
iterative:

  1. 1.

Pre-processing: this step adopts the sliding window to split the entire
training dataset into fixed-size windows. We argue that the data from one
window is the smallest domain unit.

  2. 2.

Fine-grained feature update: this step updates the feature extractor using the
proposed pseudo domain-class labels as the supervision.

  3. 3.

Latent distribution characterization: it aims to identify the domain label for
each instance to obtain the latent distribution information. It maximizes the
different distribution gaps to enlarge diversity.

  4. 4.

Domain-invariant representation learning: this step utilizes pseudo domain
labels from the last step to learn domain-invariant representations and train
a generalizable model.

Fine-grained Feature Update. Before characterizing latent distributions, we
perform fine-grained feature updates to obtain fine-grained representation. As
shown in Figure 3 (blue), we propose a new concept: pseudo domain-class label
to fully utilize the knowledge contained in domains and classes, which serves
as the supervision for the feature extractor. Features are more fine-grained
w.r.t. domains and labels, instead of only attached to domains or labels.

At the first iteration, there is no domain label $d^{\prime}$ and we simply
initialize $d^{\prime}=0$ for all samples. We treat per category per domain as
a new class with label $s\in\\{1,2,\cdots,S\\}$. We have $S=K\times C$666For
detection, replace $C$ with $C_{n}$. where $K$ is the pre-defined number of
latent distributions that can be tuned in experiments. We perform pseudo
domain-class label assignment to get discrete values for supervision:
$s=d^{\prime}\times C+y.$

Let $h_{f}^{(2)},h_{b}^{(2)},h_{c}^{(2)}$ be feature extractor, bottleneck,
and classifier, respectively (we use superscripts to denote step number).
Then, the supervised loss is computed using the cross-entropy loss $\ell$:

$\mathcal{L}_{super}=\mathbb{E}_{(\mathbf{x},y)\sim\mathbb{P}^{tr}}\ell\left(h_{c}^{(2)}(h_{b}^{(2)}(h_{f}^{(2)}(\mathbf{x}))),s\right).$
(1)

Latent Distribution Characterization. This step characterizes the latent
distributions contained in one dataset. As shown in Figure 3 (green), we
propose an adapted version of adversarial training to disentangle the domain
labels from the class labels. However, there are no actual domain labels
provided, which hinders such disentanglement. Inspired by [66], we employ a
self-supervised pseudo-labeling strategy to obtain domain labels.

First, we attain the centroid for each domain with class-invariant features:

$\tilde{\mu}_{k}=\frac{\sum_{\mathbf{x}_{i}\in\mathcal{X}^{tr}}\delta_{k}(h_{c}^{(3)}(h_{b}^{(3)}(h_{f}^{(3)}(\mathbf{x}_{i}))))h_{b}^{(3)}(h_{f}^{(3)}(\mathbf{x}_{i}))}{\sum_{\mathbf{x}_{i}\in\mathcal{X}^{tr}}\delta_{k}(h_{c}^{(3)}(h_{b}^{(3)}(h_{f}^{(3)}(\mathbf{x}_{i}))))},$
(2)

where $h_{f}^{(3)},h_{b}^{(3)},h_{c}^{(3)}$ are feature extractor, bottleneck,
and classifier, respectively. $\tilde{\mu}_{k}$ is the initial centroid of the
$k^{th}$ latent domain while $\delta_{k}$ is the $k^{th}$ element of the logit
soft-max output. Then, we obtain the pseudo domain labels via the nearest
centroid classifier using a distance function $D$:

$\tilde{d}^{\prime}_{i}=\arg\min_{k}D(h^{(3)}_{b}(h^{(3)}_{f}(\mathbf{x}_{i})),\tilde{\mu}_{k}).$
(3)

Then, we compute the centroids and obtain the updated pseudo domain labels:

$\displaystyle\mu_{k}=\frac{\sum_{\mathbf{x}_{i}\in\mathcal{X}^{tr}}\mathbb{I}(\tilde{d}^{\prime}_{i}=k)h^{(3)}_{b}(h^{(3)}_{f}(\mathbf{x}))}{\sum_{\mathbf{x}_{i}\in\mathcal{X}^{tr}}\mathbb{I}(\tilde{d}^{\prime}_{i}=k)},$
(4) $\displaystyle
d^{\prime}_{i}=\arg\min_{k}D(h^{(3)}_{b}(h^{(3)}_{f}(\mathbf{x}_{i})),\mu_{k}),$

where $\mathbb{I}(a)=1$ when $a$ is true, otherwise $0$. After obtaining
$d^{\prime}$, we can compute the loss of step 2:

$\displaystyle\mathcal{L}_{self}+\mathcal{L}_{cls}=$
$\displaystyle\mathbb{E}_{(\mathbf{x},y)\sim\mathbb{P}^{tr}}\ell(h^{(3)}_{c}(h^{(3)}_{b}(h^{(3)}_{f}(\mathbf{x}))),d^{\prime})$
(5)
$\displaystyle+\ell(h^{(3)}_{adv}(R_{\lambda_{1}}(h^{(3)}_{b}(h^{(3)}_{f}(\mathbf{x})))),y),$

where $h^{(3)}_{adv}$ is the discriminator for step 3 that contains several
linear layers and one classification layer. $R_{\lambda_{1}}$ is the gradient
reverse layer with hyperparameter $\lambda_{1}$ [67]. After this step, we can
obtain the pseudo domain label $d^{\prime}$ for $\mathbf{x}$.

Domain-invariant Representation Learning. After obtaining the latent
distributions, we learn domain-invariant representations for generalization.
In fact, this step (purple in Figure 3) is simple: we borrow the idea from
DANN [67] and directly use adversarial training to update the classification
loss $\mathcal{L}_{cls}$ and domain classifier loss $\mathcal{L}_{dom}$ using
gradient reversal layer (GRL) (a common technique that facilitates adversarial
training via reversing gradients) [67]:

$\displaystyle\mathcal{L}_{cls}+\mathcal{L}_{dom}=$
$\displaystyle\mathbb{E}_{(\mathbf{x},y)\sim\mathbb{P}^{tr}}\ell(h_{c}^{(4)}(h_{b}^{(4)}(h_{f}^{(4)}(\mathbf{x}))),y)$
(6)
$\displaystyle+\ell(h_{adv}^{(4)}(R_{\lambda_{2}}(h_{b}^{(4)}(h_{f}^{(4)}(\mathbf{x})))),d^{\prime}),$

where $\ell$ is the cross-entropy loss and $R_{\lambda_{2}}$ is the gradient
reverse layer with hyperparameter $\lambda_{2}$ [67]. We will omit the details
of GRL and adversarial training here since they are common techniques in deep
learning.

Training and Complexity. We repeat these steps until convergence or max
epochs. Different from existing methods, the last two steps only optimize the
last few independent layers. Most of the trainable parameters are shared
between modules, indicating that Diversify has the same model size as existing
methods. The modules from the last step are utilized for inference.

###  3.4 Diversify for OOD detection

Diversify attempts to exploit subdomains and learn generalized
representations. In this section, we provide two implementations of Diversify,
Diversify-MAH and Diversify-MCP, for time series OOD detection. In these two
implementations, we first train the model with steps in section 3.3 and then
combine post-hoc methods using representations and logits respectively when
testing.

Diversify-MAH. Mahalanobis distance-based confidence score is a popular metric
to detect OOD samples [68] and it is mainly influenced by the
features/representations given by the model. We can obtain the class
conditional Gaussian distributions with respect to features of the deep models
under Gaussian discriminant analysis (GDA) which result in a confidence score
based on the Mahalanobis distance. Diversify-MAH obtains the features of the
corresponding samples using the outputs of the bottleneck,

$\mathbf{z}=h_{b}^{(4)}(h_{f}^{(4)}(\mathbf{x})).$ (7)

Diversify-MAH assumes that the class-conditional distributions of $\mathbf{z}$
follow multivariate Gaussian distributions,

$\mathbb{P}(\mathbf{z}|y=c)\sim\mathcal{N}(\mathbf{\mu}_{c},\Sigma),$ (8)

where $\mu_{c}$ is the mean of multivariate Gaussian distribution of ID class
$c$ while $\Sigma$ is a tied covariance matrix,

$\displaystyle\mathbf{\mu}_{c}$
$\displaystyle=\frac{1}{N_{c}}\sum_{i:y_{i}=c}\mathbf{z}_{i},$ (9)
$\displaystyle\Sigma$
$\displaystyle=\frac{1}{N}\sum_{c}\sum_{i:y_{i}=c}(\mathbf{z}_{i}-\mathbf{\mu}_{c})(\mathbf{z}_{i}-\mathbf{\mu}_{c})^{T},$

where $N_{c}$ denotes the number of samples in class $c$. According to a
simple theoretical connection between GDA and the softmax classifier [69, 70,
68], the posterior distribution defined by the generative classifier under GDA
with tied covariance assumption is equivalent to the softmax classifier. Now,
we can utilize the Mahalanobis distance between a test sample $\mathbf{x}$ and
the closest class-conditional distribution as a confidence score,

$M(\mathbf{x})=\max_{c}-(\mathbf{z}-\mathbf{\mu}_{c})^{T}\Sigma^{-1}(\mathbf{z}-\mathbf{\mu}_{c}).$
(10)

The larger $M(\mathbf{x})$ is, the more likely $\mathbf{x}$ belongs to class
$c$, and thereby $\mathbf{x}$ is more likely to be an ID sample. Conversely, a
small $M(\mathbf{x})$ illustrates that the sample can be an OOD sample. Since
Diversify-MAH does not rely on the predictions of the model, it can avoid the
impact of high-confidence outputs from the deep learning models.

Diversify-MCP. Maximum class probability (MCP) [19] is a popular baseline for
OOD detection. It is extremely influenced by the predictions of the model
since it utilizes the logit outputs of the model directly, which means that
better predictions can bring better performance with MCP. Combining with MCP,
we implement Diversify-MCP.

Diversify-MCP obtains an estimation vector of data as:

$\mathbf{y}^{\prime}=h_{c}^{(4)}(h_{b}^{(4)}(h_{f}^{(4)}(\mathbf{x}))).$ (11)

Via a softmax activation, $\mathbf{y}^{\prime}$ can be converted into a vector
ranging from $0$ to $1$,
$\tilde{\mathbf{y}}^{\prime}=\mathrm{softmax}(\mathbf{y}^{\prime})$, which can
reflect the confidence of the model’s confidence to some extent. Commonly, we
can view $\tilde{\mathbf{y}}^{\prime}$ as a probability estimation of
classes.The probability $v=\max_{y}\mathbb{P}(y|\mathbf{x})$ of the most
likely class is the final prediction and it can also serve as an ID score. The
larger $v$ is, the more confident the model is, and thereby the sample is more
likely to be an ID sample. Conversely, a small $v$ indicates possible OOD
inputs. Since modern NNs have been shown to often make over-confident softmax
outputs, we can also utilize a temperature hyperparameter in the softmax
activation to generate smoother predictions [71],777In this paper, we simply
set $T=1$.

$\tilde{\mathbf{y}^{\prime}}=\mathrm{softmax}(\mathbf{y}^{\prime},T),\text{
and
}\mathbf{\tilde{y}}^{\prime}_{i}=\frac{\exp(\mathbf{y}^{\prime}_{i}/T)}{\sum_{j}\exp(\mathbf{y}^{\prime}_{j}/T)}.$
(12)

###  3.5 Diversify for OOD generalization

Diversify can also be utilized for classification. We can directly utilize the
outputs of step 4 with a softmax activation to obtain predictions,

$\mathbf{y}^{\prime}=\mathrm{softmax}(h_{c}^{(4)}(h_{b}^{(4)}(h_{f}^{(4)}(\mathbf{x})))).$
(13)

###  3.6 Theoretical Insights

Preliminary For a distribution $\mathbb{P}$ with an ideal binary labeling
function $h^{*}$ and a hypothesis $h$, we define the error
$\varepsilon_{\mathbb{P}}(h)$ in accordance with [72] as:

$\varepsilon_{\mathbb{P}}(h)=\mathbb{E}_{\mathbf{x}\sim\mathbb{P}}|h(\mathbf{x})-h^{*}(\mathbf{x})|.$
(14)

We also give the definition of $\mathcal{H}$-divergence according with [72].
Given two distributions $\mathbb{P},\mathbb{Q}$ over a space $\mathcal{X}$ and
a hypothesis class $\mathcal{H}$,

$d_{\mathcal{H}}(\mathbb{P},\mathbb{Q})=2\sup_{h\in\mathcal{H}}|Pr_{\mathbb{P}}(I_{h})-Pr_{\mathbb{Q}}(I_{h})|,$
(15)

where $I_{h}=\\{\mathbf{x}\in\mathcal{X}|h(\mathbf{x})=1\\}$. We often
consider the $\mathcal{H}\Delta\mathcal{H}$-divergence in [72] where the
symmetric difference hypothesis class $\mathcal{H}\Delta\mathcal{H}$ is the
set of functions characteristic to disagreements between hypotheses.

######  Theorem 3.1.

(Theorem 2.1 in [73], modified from Theorem 2 in [72]). Let $\mathcal{X}$ be a
space and $\mathcal{H}$ be a class of hypotheses corresponding to this space.
Suppose $\mathbb{P}$ and $\mathbb{Q}$ are distributions over $\mathcal{X}$.
Then for any $h\in\mathcal{H}$, the following holds

$\varepsilon_{\mathbb{Q}}(h)\leq\lambda^{\prime\prime}+\varepsilon_{\mathbb{P}}(h)+\frac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{Q},\mathbb{P})$
(16)

with $\lambda^{\prime\prime}$ the error of an ideal joint hypothesis for
$\mathbb{Q},\mathbb{P}$.

Theorem 3.1 provides an upper bound on the target-error.
$\lambda^{\prime\prime}$ is a property of the dataset and hypothesis class and
is often ignored. Theorem 3.1 demonstrates the necessity to learn domain
invariant features.

Why Diversify can learn better representations and achieve better predictions?
We present some theoretical insights to show that our approach is well
motivated.

######  Proposition 3.2.

Let $\mathcal{X}$ be a space and $\mathcal{H}$ be a class of hypotheses
corresponding to this space. Let $\mathbb{Q}$ and the collection
$\\{\mathbb{P}_{i}\\}_{i=1}^{K}$ be distributions over $\mathcal{X}$ and let
$\\{\varphi_{i}\\}_{i=1}^{K}$ be a collection of non-negative coefficients
with $\sum_{i}\varphi_{i}=1$. Let $\mathcal{O}$ be a set of distributions s.t.
$\forall\mathbb{S}\in\mathcal{O}$, the following holds

$d_{\mathcal{H}\Delta\mathcal{H}}(\sum_{i}\varphi_{i}\mathbb{P}_{i},\mathbb{S})\leq\max_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j}).$
(17)

Then, for any $h\in\mathcal{H}$,

$\displaystyle\varepsilon_{\mathbb{Q}}(h)\leq$
$\displaystyle\lambda^{\prime}+\sum_{i}\varphi_{i}\varepsilon_{\mathbb{P}_{i}}(h)+\frac{1}{2}\min_{\mathbb{S}\in\mathcal{O}}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q})$
(18)
$\displaystyle+\frac{1}{2}\max_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j}),$

where $\lambda^{\prime}$ is the error of an ideal joint hypothesis.
$\varepsilon_{\mathbb{P}}(h)$ is the error for a hypothesis $h$ on a
distribution $\mathbb{P}$.
$d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P},\mathbb{Q})$ is
$\mathcal{H}$-divergence which measures differences in distribution [72].

###### Proof.

On one hand, with Theorem 3.1, we have

$\displaystyle\varepsilon_{\mathbb{Q}}(h)\leq\lambda^{\prime}_{1}+\varepsilon_{\mathbb{S}}(h)+\frac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q}),\forall
h\in\mathcal{H},\forall\mathbb{S}\in\mathcal{O}.$ (19)

On the other hand, with Theorem 3.1, we have

$\displaystyle\varepsilon_{\mathbb{S}}(h)\leq$
$\displaystyle\lambda^{\prime}_{2}+\varepsilon_{\sum_{i}\varphi_{i}\mathbb{P}_{i}}(h)+$
(20)
$\displaystyle\frac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\sum_{i}\varphi_{i}\mathbb{P}_{i},\mathbb{S}),\forall
h\in\mathcal{H}.$

Since
$d_{\mathcal{H}\Delta\mathcal{H}}(\sum_{i}\varphi_{i}\mathbb{P}_{i},\mathbb{S})\leq\max_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j})$,
and
$\varepsilon_{\sum_{i}\varphi_{i}\mathbb{P}_{i}}(h)=\sum_{i}\varphi_{i}\varepsilon_{\mathbb{P}_{i}}(h)$,
we have

$\displaystyle\varepsilon_{\mathbb{Q}}(h)\leq$
$\displaystyle\lambda^{\prime}+\sum_{i}\varphi_{i}\varepsilon_{\mathbb{P}_{i}}(h)+\frac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q})$
(21)
$\displaystyle+\frac{1}{2}\max_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\sum_{i}\varphi_{i}\mathbb{P}_{i},\mathbb{S}),\forall
h\in\mathcal{H},\forall\mathbb{S}\in\mathcal{O},$

where $\lambda^{\prime}=\lambda^{\prime}_{1}+\lambda^{\prime}_{2}$. Equation
21 for all $\mathbb{S}\in\mathcal{O}$ holds. Therefore, we complete the proof.
∎

The first item in (18), $\lambda^{\prime}$, is often neglected since it is
small in reality. The second item,
$\sum_{i}\varphi_{i}\varepsilon_{\mathbb{P}_{i}}(h)$, exists in almost all
methods and can be minimized via supervision from class labels with cross-
entropy loss in (6). Our main purpose is to minimize the last two items in
(18). Here $\mathbb{Q}$ corresponds to the unseen out-of-distribution target
domain.

The last term
$\frac{1}{2}\max_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j})$
is common in OOD theory which measures the maximum differences among source
domains. This corresponds to step 4 in our approach.

Finally, the third item,
$\frac{1}{2}\min_{\mathbb{S}\in\mathcal{O}}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q})$,
explains why we exploit sub-domains. Since our goal is to learn a model which
can perform well on an unseen target domain, we cannot obtain $\mathbb{Q}$. To
minimize
$\frac{1}{2}\min_{\mathbb{S}\in\mathcal{O}}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q})$,
we can only enlarge the range of $\mathcal{O}$. We have to
$\max_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j})$
according to (17), corresponding to step 3 in our method which tries to
segment the time series data into several latent sub-domains by maximizing the
segment-wise distribution gap to preserve diversities, i.e. the ‘worst-case’
distribution scenario. Better representations and predictions bring
improvements on OOD detection and generalization.

##  Sec. 4 Experiments on OOD detection

We perform evaluations on three diverse time series detection tasks: gesture
recognition, wearable stress&affect detection, and sensor-based activity
recognition. TABLE I shows the statistical information on datasets that we
use.

TABLE I: Information on datasets.

Dataset | Subjects | Sensors | Classes | Samples
---|---|---|---|---
EMG | 36 | 1 | 7 | 33,903,472
WESAD | 15 | 8 | 4 | 63,000,000
DSADS | 8 | 3 | 19 | 1,140,000
USC-HAD | 14 | 2 | 12 | 5,441,000
UCI-HAR | 30 | 2 | 6 | 1,310,000
PAMAP2 | 9 | 3 | 18 | 3,850,505

###  4.1 Setup

We utilize the sliding window technique to split data. As its name suggests,
this technique involves taking a subset of data from a given array or
sequence. Two main parameters of the sliding window technique are the window
size, describing a subset length, and the step size, describing moving forward
distance each time.

Time series OOD detection algorithms with feature shifts are currently less
studied and we combine existing DG methods with MCP and Mahalanobis distance.
We compare with five state-of-the-art methods. DANN [67] is a method that
utilizes adversarial training to force the discriminator unable to classify
domains for better domain-invariant features. It requires domain labels and
splits data in advance while ours is a universal method. CORAL [41] utilizes
the covariance alignment in feature layers for better domain-invariant
features. It also requires domain labels and splits data in advance. GroupDRO
[55] is a method that seeks a global distribution with the worst performance
within a range of the raw distribution for better generalization. Ours study
the internal distribution shift instead of seeking a global distribution close
to the original one. ANDMask [74] is a gradient-based optimization method that
belongs to special learning strategies.

For fairness, all methods use a feature net with two blocks and each block has
one convolution layer, one pooling layer, and one batch normalization layer,
following [75]. All methods are implemented with PyTorch [76]. The maximum
training epoch is set to 150. The Adam optimizer with weight decay $5\times
10^{-4}$ is used. The learning rate for the rest methods is $10^{-2}$ or
$10^{-3}$. For the pooling layer, we utilize MaxPool2d.The kernel size is
$(1,2)$ ad the stride is 2.

Some OOD methods require the domain labels known in training while ours does
not, which is more challenging and practical. For these methods that require
domain labels, we randomly assign domain labels to them in batches. We conduct
the training-domain-validation strategy and the training data are split by
$8:2$ for training and validation. We tune all methods to report the average
best performance of three trials. $K$ in Diversify is treated as a
hyperparameter and we tune it to record the best OOD performance.888There
might be no optimal $K$ for a dataset. We perform a grid search in $[2,10]$ to
get the best performance. We utilize three metrics on the testing datasets for
evaluation, including ID accuracy which evaluates generalized classification
capability and the Area Under the Receiver Operating Characteristic curve
(AUROC) and the Area Under the Precision-Recall curve (AUPR) for evaluating
OOD detection capability following [77].

###  4.2 Gesture Recognition

First, we evaluate Diversify on EMG for gestures Data Set [24].
Electromyography (EMG) is a typical time-series data that is based on
bioelectric signals. EMG for gestures Data Set [24] contains raw EMG data
recorded by MYO Thalmic bracelet. The bracelet is equipped with eight sensors
equally spaced around the forearm that simultaneously acquire myographic
signals. EMG data are scene and device-dependent, which means the same person
may generate different data when performing the same activity with the same
device at a different time (i.e., distribution shift across time [78, 79]) or
with different devices at the same time. Data from 36 subjects are collected
while they performed a series of static hand gestures and the number of
instances is $~{}40,000-50,000$ recordings in each column. It contains 7
classes and we select 6 common classes performed by all subjects for our
experiments. Ulnar deviations is selected as OOD class while the rests serve
as the ID classes.

For EMG, we set the window size to 200 and the step size to 100, which means
there exist $50\%$ overlaps between two adjacent samples. We normalize each
sample with
$\tilde{\mathbf{x}}=\frac{\mathbf{x}-\min\mathbf{X}}{\max\mathbf{X}-\min\mathbf{X}}$.
$\mathbf{X}$ contains all $\mathbf{x}$. The final dimension is $8\times
1\times 200$. We randomly divide 36 subjects into four domains (i.e.,
$0,1,2,3$) without overlapping and each domain contains data of 9 persons.

TABLE II: ID accuracy, AUROC, and AUPR on EMG. Bold means the best while
underline means the second-best.

Targets | 0 | 1 | 2 | 3 | AVG | AVG (+MCP) | AVG (+MAH)
---|---|---|---|---|---|---|---
Metrics | ID ACC | AUROC | AUPR | AUROC | AUPR
ERM | 62.59 | 70.56 | 77.45 | 69.96 | 70.14 | 54.85 | 85.98 | 49.28 | 85.17
CORAL | 66.88 | 82.71 | 82.53 | 73.88 | 76.50 | 61.42 | 89.02 | 64.32 | 90.71
DANN | 69.69 | 76.17 | 80.74 | 75.80 | 75.60 | 61.54 | 88.31 | 58.79 | 89.08
GroupDRO | 74.19 | 77.90 | 81.53 | 69.18 | 75.70 | 66.09 | 90.33 | 62.33 | 90.59
ANDMask | 65.61 | 78.30 | 75.73 | 55.02 | 68.67 | 51.67 | 84.71 | 59.75 | 88.37
Diversify | 81.43 | 90.39 | 87.26 | 86.19 | 86.32 | 70.76 | 92.66 | 78.38 | 94.65

The results are shown in Table II and we gain the following observation. 1)
Our method achieves the best performance on each task and has an improvement
of about $10\%$ on average compared to the second-best method, which
demonstrates that our method has a good capability of ID classification. 2)
For methods with MCP, Diversify-MCP achieves the best AUROC and AUPR on
average. Compared to the second-best method, Diversify-MCP has improvements of
$4.67\%$ and $2.33\%$ respectively. For methods with Mahalanobis distance
Diversify-MAH achieves the best AUROC and AUPR on average. Compared to the
second-best method, Diversify-MAH has improvements of $14.06\%$ and $3.94\%$
respectively. These results demonstrate that Diversify has a good ability to
detect anomalies. 3) In most situations, ID accuracy has a positive relation
to AUROC and AUPR, but there also exist some counterexamples. CORAL achieves
better ID accuracy but worse AUROC(MCP) on average than GroupDRO. Therefore,
we might need to select different hyperparameters and methods for different
purposes. 4) Compared to Diversify-MCP, Diversify-MAH has another remarkable
improvement but some methods, e.g. ERM-MAH and ERM-MCP, perform more terribly.
These results demonstrate that our method can learn better generalized
representations. And for time series OOD detection task, Mahalanobis distance
is not influenced by the over-confidence of deep models while MCP might
require further tuning.

###  4.3 Wearable Stress and Affect Detection

We further evaluate Diversify on a larger dataset, Wearable Stress and Affect
Detection (WESAD) [80]. WESAD is a public dataset that contains physiological
and motion data of 15 subjects with $63,000,000$ instances. We utilize sensor
modalities of chest-worn devices including electrocardiogram, electrodermal
activity, electromyogram, respiration, body temperature, and three-axis
acceleration. We split 15 subjects into four domains. We utilize the same
preprocessing as EMG and select class stress as the OOD class.

TABLE III: ID accuracy, AUROC, and AUPR on WESAD. Bold means the best while
underline means the second-best.

Targets | 0 | 1 | 2 | 3 | AVG | AVG (+MCP) | AVG (+MAH)
---|---|---|---|---|---|---|---
Metrics | ID ACC | AUROC | AUPR | AUROC | AUPR
ERM | 52.83 | 60.77 | 59.01 | 55.53 | 57.04 | 49.66 | 80.54 | 59.68 | 83.56
CORAL | 51.66 | 55.22 | 50.19 | 52.49 | 52.39 | 69.98 | 87.97 | 72.67 | 90.94
DANN | 49.22 | 57.12 | 56.47 | 59.96 | 55.69 | 60.34 | 85.97 | 65.97 | 86.86
GroupDRO | 51.63 | 67.12 | 63.96 | 54.01 | 59.18 | 66.20 | 88.32 | 77.92 | 92.12
ANDMask | 50.20 | 54.80 | 57.30 | 56.62 | 54.73 | 53.30 | 81.30 | 66.12 | 84.17
Diversify | 48.79 | 73.18 | 60.49 | 75.01 | 64.37 | 77.24 | 92.57 | 88.68 | 96.47

The results are shown in Table III and we have the following observation. 1)
Similar to EMG, our method has the best ID accuracy, AUROC, and AUPR on
average, which demonstrates that our method has excellent abilities of
classification and OOD detection. 2) For some tasks, our method obtains worse
ID accuracy than other methods, which can be caused by two reasons. On the one
hand, our method relies on adapted DANN to exploit subdomains and learn
representations. When DANN performs terribly, our method can be influenced. On
the other hand, $K$ is a hyperparameter on the current method, and we can miss
the best results due to limited searches. 3) No matter how the method performs
in accuracy, Diversify-MAH performs the best, which indicates that the
features Diversify extracted are outstanding. Moreover, compare to Diversify-
MCP, Diversify-MAH still has another remarkable improvement.

###  4.4 Sensor-based Human Activity Recognition

Finally, we construct three diverse OOD settings by leveraging four sensor-
based human activity recognition datasets: DSADS [81], USC-HAD [82], UCI-HAR
[83], and PAMAP2 [84]. UCI daily and sports dataset (DSADS) [81] consists of
19 activities collected from 8 subjects wearing body-worn sensors on 5 body
parts. USC-SIPI human activity dataset (USC-HAD) [82] is composed of 14
subjects (7 male, 7 female, aged 21 to 49) executing 12 activities with a
sensor tied on the front right hip. UCI-HAR [83] is collected by 30 subjects
performing 6 daily living activities with a waist-mounted smartphone. PAMAP2
[84] contains data from 18 activities, performed by 9 subjects wearing 3
sensors. These datasets are collected from different people and positions
using accelerometer and gyroscope, with $11,741,000$ instances in total. For
DSADS, we directly utilize data split by the providers. The final dimension
shape is $45\times 1\times 125$. $45=5\times 3\times 3$ where $5$ means five
positions, the first $3$ means three sensors, and the second $3$ means each
sensor has three axes. For USC-HAD, the window size is 200 and the step size
is 100. The final dimension shape is $6\times 1\times 200$. For PAMAP2, the
window size is 200 and the step size is 100. The final dimension shape is
$27\times 1\times 200$. For UCI-HAR, we directly utilize data split by the
providers. The final dimension shape is $6\times 1\times 128$. (1) X-person
generalization (Cross-person) aims to learn generalized models for different
persons. This setting utilizes DSADS, USC-HAD, and PAMAP2. Within each
dataset, we randomly split the data into four groups. For DSADS, we choose
running, ascending stairs, descending stairs, rope jumping, and playing
basketball as OOD classes. For USC-HAD, we choose Running Forward and Jumping
Up as OOD classes. For PAMAP2, we choose running, Nordic walking, and rope
jumping as OOD classes. (2) X-position generalization (Cross-position) aims to
learn generalized models for different sensor positions. This setting uses
DSADS and data from each position denotes a different domain. Thereby, a
sample is split into five samples in the first dimension and the final
dimension shape is $9\times 1\times 125$. We choose running, ascending stairs,
descending stairs, rope jumping, and playing basketball as OOD classes. (3)
X-dataset generalization (Cross-dataset) aims to learn generalized models for
different datasets. This setting uses all four datasets, and each dataset
corresponds to a different domain. Six common classes are selected. Two
sensors from each dataset that belong to the same position are selected and
data is down-sampled to have the same dimension. The final dimension shape is
$6\times 1\times 50$. We choose ascending and descending as OOD classes.

TABLE IV: ID accuracy under X-Person, X-Position, and X-Dataset settings. Bold
means the best while underline means the second-best.

Task | Datasets | DSADS | USC-HAD | PAMAP2
---|---|---|---|---
X-Person | Targets | 0 | 1 | 2 | 3 | AVG | 0 | 1 | 2 | 3 | AVG | 0 | 1 | 2 | 3 | AVG
ERM | 86.37 | 72.32 | 80.77 | 76.85 | 79.08 | 81.85 | 58.99 | 71.61 | 58.25 | 67.68 | 86.84 | 80.62 | 49.65 | 85.64 | 75.69
CORAL | 85.06 | 81.25 | 89.82 | 72.98 | 82.28 | 78.44 | 57.71 | 74.26 | 60.35 | 67.69 | 89.82 | 79.72 | 59.58 | 84.56 | 78.42
DANN | 87.14 | 77.44 | 81.49 | 76.13 | 80.55 | 80.69 | 57.50 | 72.35 | 59.77 | 67.58 | 89.92 | 77.50 | 51.83 | 87.85 | 76.78
GroupDRO | 85.18 | 76.25 | 83.75 | 75.71 | 80.22 | 79.71 | 62.50 | 74.30 | 58.43 | 68.73 | 91.32 | 78.23 | 60.68 | 88.73 | 79.74
ANDMask | 85.00 | 75.89 | 84.40 | 71.13 | 79.11 | 76.77 | 59.98 | 72.13 | 56.24 | 66.28 | 89.82 | 77.82 | 50.80 | 90.16 | 77.15
Diversify | 87.68 | 82.26 | 90.30 | 86.85 | 86.77 | 81.61 | 66.08 | 75.82 | 71.15 | 73.66 | 91.25 | 82.70 | 62.62 | 90.93 | 81.88
X-Position | Targets | 0 | 1 | 2 | 3 | 4 | AVG | X-Dataset | Targets | DSADS | USC-HAD | HAR | PAMAP2 | AVG |  |
ERM | 33.38 | 21.62 | 35.43 | 22.50 | 19.88 | 26.56 | ERM | 37.11 | 53.05 | 58.84 | 34.34 | 45.83 |  |
CORAL | 41.79 | 27.77 | 32.16 | 28.14 | 26.07 | 31.18 | CORAL | 40.00 | 51.47 | 59.57 | 62.18 | 53.31 |  |
DANN | 40.04 | 29.64 | 30.74 | 22.59 | 23.93 | 29.39 | DANN | 40.96 | 54.49 | 61.44 | 66.24 | 55.78 |  |
GroupDRO | 31.64 | 24.02 | 32.74 | 29.17 | 20.94 | 27.70 | GroupDRO | 40.00 | 55.77 | 60.99 | 72.10 | 57.22 |  |
ANDMask | 31.93 | 24.03 | 33.41 | 23.93 | 23.45 | 27.35 | ANDMask | 39.57 | 48.08 | 58.39 | 61.68 | 51.93 |  |
Diversify | 43.10 | 30.57 | 37.57 | 31.76 | 26.89 | 33.98 | Diversify | 70.70 | 60.48 | 64.00 | 74.28 | 67.36 |  |

TABLE V: Average AUROC and AUPR under X-Person, X-Position, and X-Dataset
settings. Bold means the best.

| Datasets | DSADS | USC-HAD | PAMAP2
---|---|---|---|---
X-Person | Metrics | AUROC | AUPR | AUROC | AUPR | AUROC | AUPR
ERM-MCP | 66.82 | 80.52 | 15.50 | 78.71 | 45.95 | 91.85
CORAL-MCP | 75.54 | 89.33 | 23.00 | 80.59 | 60.79 | 94.84
DANN-MCP | 70.11 | 85.56 | 28.82 | 82.45 | 65.59 | 96.01
GroupDRO-MCP | 78.02 | 90.42 | 27.98 | 82.47 | 65.50 | 95.59
ANDMask-MCP | 69.60 | 85.41 | 17.71 | 79.35 | 51.15 | 93.51
Diversify-MCP | 82.38 | 92.62 | 37.50 | 85.55 | 75.02 | 97.16
ERM-MAH | 65.51 | 84.28 | 98.67 | 99.80 | 74.59 | 96.52
CORAL-MAH | 67.96 | 85.02 | 99.22 | 99.89 | 88.31 | 98.89
DANN-MAH | 63.33 | 78.88 | 99.36 | 99.91 | 86.15 | 98.61
GroupDRO-MAH | 73.60 | 87.88 | 99.25 | 99.89 | 87.37 | 98.77
ANDMask-MAH | 51.51 | 75.11 | 98.74 | 99.82 | 85.71 | 98.57
Diversify-MAH | 93.90 | 97.36 | 99.56 | 99.94 | 90.37 | 99.14
X-Position | Metrics | AUROC | AUPR | X-Dataset | AUROC | AUPR |
ERM-MCP | 42.23 | 67.65 | 40.03 | 75.91 |
CORAL-MCP | 43.43 | 68.75 | 44.60 | 77.80 |
DANN-MCP | 43.20 | 68.33 | 41.21 | 77.97 |
GroupDRO-MCP | 42.95 | 69.11 | 47.91 | 79.24 |
ANDMask-MCP | 39.14 | 66.57 | 38.97 | 74.38 |
Diversify-MCP | 54.58 | 76.78 | 60.23 | 85.67 |
ERM-MAH | 87.92 | 95.40 | 55.66 | 80.49 |
CORAL-MAH | 87.35 | 95.26 | 75.17 | 90.77 |
DANN-MAH | 87.04 | 95.01 | 65.07 | 85.46 |
GroupDRO-MAH | 87.51 | 95.24 | 73.66 | 89.70 |
ANDMask-MAH | 86.33 | 94.37 | 53.83 | 77.03 |
Diversify-MAH | 91.85 | 97.04 | 85.50 | 94.86 |

The ID classification results are shown in Table IV while OOD detection
results are shown in Table V. We have the following observation from these
results. 1) Similar to EMG, our method has the best ID accuracy, AUROC, and
AUPR on average, which demonstrates that our method has excellent abilities of
classification and OOD detection. 2) When the task is difficult: in the
X-Person setting, USC-HAD may be the most difficult task. Although it has more
samples, it contains 14 subjects with only two sensors in one position, which
may bring more difficulty in learning. The results prove the above argument
that all methods perform terribly on this benchmark while ours has the largest
improvement. 3) When datasets are significantly more diverse: compared to
X-Person and X-Position settings, X-Dataset may be more difficult since all
datasets are totally different and samples are influenced by subjects,
devices, sensor positions, and some other factors. In this setting, our method
is substantially better than others. 4) Limited data: for tasks on DSADS, the
number of training data is limited, In this case, enhancing diversity can
still bring a remarkable improvement and our method can boost the performance.
5) For time series OOD detection, it is better to utilize Mahalanobis distance
since it is less influenced by over-confidence in deep learning models. For
USC-HAD, Diversify-MAH even achieves an improvement of over $62\%$ on average
AUROC compared to Diversify-MCP.

##  Sec. 5 Experiments on OOD generalization

We perform evaluations on four diverse time series classification tasks:
gesture recognition, speech commands recognition, wearable stress&affect
detection, and sensor-based activity recognition. Settings and implementations
are similar to OOD detection. Besides the comparison methods mentioned above,
we add more latest methods designed for classification. Mixup [85] is a method
that utilizes interpolation to generate more data for better generalization.
Ours mainly focuses on generalized representation learning. RSC [86] is a
self-challenging training algorithm that forces the network to activate
features as much as possible by manipulating gradients. It belongs to gradient
operation-based DG while ours is to learn generalized features. GILE [21] is a
disentanglement method designed for cross-person human activity recognition.
It is based on VAEs and requires domain labels. AdaRNN [8] is a method with a
two-stage that is non-differential and is tailored for RNN. A specific
algorithm is designed for splitting. Ours is universal and is differential
with better performance. Per-segment accuracy is the evaluation metric.

###  5.1 Gesture Recognition

First, we evaluate Diversify on EMG for gestures Data Set [24]. We randomly
divide 36 subjects into four domains (i.e., $0,1,2,3$). Figure 4(a) shows that
with the same backbone, our method achieves the best average performance and
is $\mathbf{4.3}\%$ better than the second-best method. Diversify even
outperforms AdaRNN which has a stronger backbone.

TABLE VI: Accuracy on cross-person generalization. “Target” $0\sim 4$ denotes
the unseen test set.

| DSADS | USC-HAD | PAMAP2 | ALL
---|---|---|---|---
Target | 0 | 1 | 2 | 3 | AVG | 0 | 1 | 2 | 3 | AVG | 0 | 1 | 2 | 3 | AVG | AVG
ERM | 83.1 | 79.3 | 87.8 | 71.0 | 80.3 | 81.0 | 57.7 | 74.0 | 65.9 | 69.7 | 90.0 | 78.1 | 55.8 | 84.4 | 77.1 | 75.7
DANN | 89.1 | 84.2 | 85.9 | 83.4 | 85.6 | 81.2 | 57.9 | 76.7 | 70.7 | 71.6 | 82.2 | 78.1 | 55.4 | 87.3 | 75.7 | 77.7
CORAL | 91.0 | 85.8 | 86.6 | 78.2 | 85.4 | 78.8 | 58.9 | 75.0 | 53.7 | 66.6 | 86.2 | 77.8 | 49.0 | 87.8 | 75.2 | 75.7
Mixup | 89.6 | 82.2 | 89.2 | 86.9 | 87.0 | 80.0 | 64.1 | 74.3 | 61.3 | 69.9 | 89.4 | 80.3 | 58.4 | 87.7 | 79.0 | 78.6
GroupDRO | 91.7 | 85.9 | 87.6 | 78.3 | 85.9 | 80.1 | 55.5 | 74.7 | 60.0 | 67.6 | 85.2 | 77.7 | 56.2 | 85.0 | 76.0 | 76.5
RSC | 84.9 | 82.3 | 86.7 | 77.7 | 82.9 | 81.9 | 57.9 | 73.4 | 65.1 | 69.6 | 87.1 | 76.9 | 60.3 | 87.8 | 78.0 | 76.9
ANDMask | 85.0 | 75.8 | 87.0 | 77.6 | 81.4 | 79.9 | 55.3 | 74.5 | 65.0 | 68.7 | 86.7 | 76.4 | 43.6 | 85.6 | 73.1 | 74.4
GILE | 81.0 | 75.0 | 77.0 | 66.0 | 74.7 | 78,0 | 62.0 | 77.0 | 63.0 | 70.0 | 83.0 | 68.0 | 42.0 | 76.0 | 67.5 | 70.7
AdaRNN | 80.9 | 75.5 | 90.2 | 75.5 | 80.5 | 78.6 | 55.3 | 66.9 | 73.7 | 68.6 | 81.6 | 71.8 | 45.4 | 82.7 | 70.4 | 73.2
Diversify | 90.4 | 86.5 | 90.0 | 86.1 | 88.2 | 82.6 | 63.5 | 78.7 | 71.3 | 74.0 | 91.0 | 84.3 | 60.5 | 87.7 | 80.8 | 81.0

TABLE VII: Classification accuracy on cross-position, cross-dataset, and one-
to-another generalization.

| Cross-position generalization | Cross-dataset generalization | One-Person-To-Another
---|---|---|---
Target | 0 | 1 | 2 | 3 | 4 | AVG | 0 | 1 | 2 | 3 | AVG | DSADS | USC-HAD | PAMAP2 | AVG
ERM | 41.5 | 26.7 | 35.8 | 21.4 | 27.3 | 30.6 | 26.4 | 29.6 | 44.4 | 32.9 | 33.3 | 51.3 | 46.2 | 53.1 | 50.2
DANN | 45.4 | 25.3 | 38.1 | 28.9 | 25.1 | 32.6 | 29.7 | 45.3 | 46.1 | 43.8 | 41.2 | - | - | - | -
CORAL | 33.2 | 25.2 | 25.8 | 22.3 | 20.6 | 25.4 | 39.5 | 41.8 | 39.1 | 36.6 | 39.2 | - | - | - | -
Mixup | 48.8 | 34.2 | 37.5 | 29.5 | 29.9 | 36.0 | 37.3 | 47.4 | 40.2 | 23.1 | 37.0 | 62.7 | 46.3 | 58.6 | 55.8
GroupDRO | 27.1 | 26.7 | 24.3 | 18.4 | 24.8 | 24.3 | 51.4 | 36.7 | 33.2 | 33.8 | 38.8 | 51.3 | 48.0 | 53.1 | 50.8
RSC | 46.6 | 27.4 | 35.9 | 27.0 | 29.8 | 33.3 | 33.1 | 39.7 | 45.3 | 45.9 | 41.0 | 59.1 | 49.0 | 59.7 | 55.9
ANDMask | 47.5 | 31.1 | 39.2 | 30.2 | 29.9 | 35.6 | 41.7 | 33.8 | 43.2 | 40.2 | 39.7 | 57.2 | 45.9 | 54.3 | 52.5
Diversify | 47.7 | 32.9 | 44.5 | 31.6 | 30.4 | 37.4 | 48.7 | 46.9 | 49.0 | 59.9 | 51.1 | 67.6 | 55.0 | 62.5 | 61.7

###  5.2 Speech Commands

(a) EMG

(b) Speech commands

(c) WESAD

Figure 4: Results on EMG, Speech commands and WESAD.

Then, we adopt a regular speech recognition task, the Speech Commands dataset
[87]. It consists of one-second audio recordings of both background noise and
spoken words such as ‘left’ and ‘right’. It is collected from more than 2,000
persons, thus is more complicated. Following [88], we use 34,975 time series
corresponding to ten spoken words to produce a balanced classification
problem. Since this dataset is collected from multiple persons, the training
and test distributions are different, which is also an OOD problem with one
training domain. We do not split samples due to too many subjects and few
audios per subject. Figure 4(b) shows the results on two different backbones.
Compared with GroupDRO, Diversify has over $\mathbf{1}\%$ improvement with a
basic CNN backbone and over $\mathbf{0.6}\%$ improvement with a strong
backbone MatchBoxNet3-1-64 [89]. It demonstrates the superiority of our method
on a regular time-series benchmark containing massive distributions.

###  5.3 Wearable Stress and Affect Detection

We further evaluate Diversify on WESAD. We split 15 subjects into four
domains. Results Figure 4(c) showed that our method achieves the best
performance compared to other state-of-the-art methods with an improvement of
over $\textbf{8}\%$ on this larger dataset.

###  5.4 Sensor-based Human Activity Recognition

Finally, we construct _four_ diverse OOD settings on Sensor-based Human
Activity Recognition. Besides the settings mentioned above, we add another
difficult setting, One-Person-To-Another. One-Person-To-Another aims to learn
generalized models for different persons from data of a single person. This
setting adopts DSADS, USC-HAD, and PAMAP2. In each dataset, we randomly select
four pairs of persons where one is the training and the other is the test.
Four tasks are $1\rightarrow 0,3\rightarrow 2,5\rightarrow 4,$ and
$7\rightarrow 6$. Each number corresponds to one subject. And the final
dimension shape is $45\times 1\times 125,6\times 1\times 200,$ and $27\times
1\times 200$ for DSADS, USC-HAD, and PAMAP2 respectively. 999In the One-
Person-To-Another setting, we only report the average accuracy of four tasks
on each dataset.

TABLE VI and VII show the results on four settings for HAR, where our method
significantly outperforms the second-best baseline by $\mathbf{2.4}\%$,
$\mathbf{1.4}\%$, $\mathbf{9.9}\%$, and $\mathbf{5.8}\%$ respectively. All
results show the superiority of Diversify.

We observe more insightful conclusions similar to OOD detection. (1) _When the
task is difficult:_ In the Cross-Person setting, USC-HAD may be the most
difficult task. The results prove the above argument that all methods perform
terribly on this benchmark while ours has the largest improvement. (2) _When
datasets are significantly more diverse:_ Compared to Cross-Person and Cross-
Position settings, Cross-Dataset may be more difficult. In this setting, our
method is substantially better than others. (3) _Limited data:_ Compared with
the Cross-Person setting, One-Person-To-Another is more difficult since it has
fewer data samples. In this case, enhancing diversity can bring a remarkable
improvement and our method can boost the performance.

(a) Class-invariant feat.

(b) Update feat. with $d^{\prime}$

(c) Update feature with $y$

(d) #Domain $K$ (EMG)

Figure 5: Ablation study of Diversify for detection.

(a) Class-invariant feat.

(b) Update feat. with $d^{\prime}$

(c) Update feature with $y$

(d) #Domain $K$ (EMG)

Figure 6: Ablation study of Diversify for generalization.

##  Sec. 6 Analysis

###  6.1 Ablation study

We present ablation study to answer the following three questions. (1) _Why
obtaining pseudo domain labels with class-invariant features in step 3?_ If we
obtain pseudo domain labels with common features, domain labels may have
correlations with class labels, which may introduce contradictions when
learning domain-invariant representations and lead to common performance. This
is certified by the results in Figure 5(a). (2) _Why using fine-grained
domain-class labels in step 2?_ If we utilize pseudo domain labels to update
the feature net, it may make the representations seriously biased towards
domain-related features and thereby leads to terrible performance on
classification, which is proved in Figure 5(b). If we only utilize class
labels to update the feature net, it may make representations biased to class-
related features, thus Diversify is unable to obtain true latent sub-domains,
as shown in Figure 5(c). Hence, we should employ fine-grained domain-class
labels to obtain representations with both domain and class information. (3)
_The more latent domains, the better?_ More latent domains may not bring
better results (Figure 5(d)) since a dataset may only have a few latent
domains and introducing more may contradict its intrinsic data property. Plus,
more latent domains also make it harder to obtain pseudo domain labels and
learn domain-invariant features. For generalization, we can obtain similar
observations to OOD detection from Figure 6.

(a) #Latent domains $K$

(b) $\lambda_{1}$

(c) $\lambda_{2}$

(d) Local epoch and Round

Figure 7: Parameter sensitivity analysis (EMG) for detection.

(a) #Latent domains $K$

(b) $\lambda_{1}$

(c) $\lambda_{2}$

(d) Local epoch and Round

Figure 8: Parameter sensitivity analysis (EMG) for generalization.

(a) ERM

(b) ANDMask

(c) Ours (d) ERM (e) ANDMask (f) Ours Figure 9: t-SNE visualization (EMG) for
detection. For (a)-(c) black points are misclassified samples. For (d)-(e),
black points mean OOD samples.

(a) Initial domain split

(b) Our domain split (c) ANDMask (d) Our classification Figure 10: t-SNE
visualizations for domain splits ((a) (b)) and classification ((c) (d)) on EMG
data.

###  6.2 Parameter sensitivity

There are mainly four hyperparameters in our method: $K$ which is the number
of latent sub-domains, $\lambda_{1}$ for the adversarial part in step 3,
$\lambda_{2}$ for the adversarial part in step 4, and local epochs and total
rounds. For fairness, the product of local epochs and total rounds is the same
value. We evaluate the parameter sensitivity of our method for detection in
Figure 7 where we change one parameter and fix the other to record the
results. From these results, we can obtain the following observations. 1) For
ID accuracy, our method achieves better performance in a wide range,
demonstrating that our method is robust. 2) For AUROC, our method achieves
better performance for a wide range in most situations, demonstrating that our
method is robust. Sometimes, we need to tune hyperparameters for better AUROC
carefully. We also evaluate the parameter sensitivity of Diversify on OOD
generalization and the results are shown in Figure 8. From these results, we
can see that our method achieves better performance in a wide range,
demonstrating that our method is robust.

###  6.3 Visualization study

We present some visualizations to show the rationales of Diversify. For
detection, Figure 9(a) and 9(c) show that Diversify can learn better margins
and generate few misclassifications while Figure 9(d) and 9(f) show that
Diversify can compact and discriminate OOD samples. For generalization, data
points with different initial domain labels are mixed together in Figure 10(a)
while Diversify can characterize different latent distributions and separate
them well in Figure 10(b). Figure 10(d) and 10(c) show that Diversify can
learn better domain-invariant representations compared to the latest method
ANDMask. To sum up, Diversify can find better representations to enhance
generalization.

(a) Temporal shift

(b) Spatial shift (c) Temporal shift (d) Spatial shift Figure 11: Latent
distributions obtained by our method on two datasets. X-axis is data numbers
while Y-axis is its values. (a)(b) are for detection while (c)(d) are for
generalization.

###  6.4 Existence of latent subdomains

What exactly can our Diversify learn? In Figure 11(a), for the first subject
in EMG, there is more than one latent distribution for class wrist extension,
showing the existence of temporal distribution shifts: the distribution of the
same activity could change. For spatial distribution shift, Figure 11(b) on
EMG dataset shows that our algorithm found two latent distributions from the
EMG data of the first person and the third person. These results indicate the
existence of latent distributions with both temporal and spatial distribution
shifts. For generalization, there exist similar phenomena.

(a) Initial splits

(b) Our splits

Figure 12: $\mathcal{H}$-divergence among domains with initial splits and our
splits on PAMAP2. Axes are domain numbers.

###  6.5 Quantitative analysis for ‘worst-case’ distributions

We present quantitative analysis by computing the $\mathcal{H}$-divergence
[72] to show the effectiveness of our ‘worst-case distribution’. In Figure
12(a) and 12(b), compared to initial domain splits, latent sub-domains
generated by our method have larger $\mathcal{H}$-divergence among each other.
According to Prop. 3.2, larger $\mathcal{H}$-divergence among domains brings
better generalization. This again shows the efficacy of Diversify in computing
the ’worst-case’ distribution scenario.

(a) Extensibility (b) Different CNNs (c) Transformers (d) Time complexity (e)
Convergence Figure 13: Experimental analysis. (a) Extensibility on EMG and
WESAD. (b)-(c) Results with different backbones on EMG and X-dataset. (d)-(e)
Time complexity and convergence.

###  6.6 Extensibility

To demonstrate that our method is extensible, we also provide implementations
with ODIN [25]. The OOD detection results are shown in Figure 13(a). We have
the following observations. 1) Our method still achieves the best average
AUROC and AUPR on both EMG and WESAD, which demonstrates the superiority of
Diversify. 2) Implementations with ODIN perform similarly to implementations
with MCP, since both ODIN and MCP are dependent on models’ predictions. For
some datasets, e.g. EMG, implementations with ODIN bring improvements compared
to MCP but for some datasets, e.g. WESAD, implementations with ODIN perform
worse. Therefore, for specific applications, we need to select the best
detection techniques.

###  6.7 Varying backbones

For generalization, we attempt to demonstrate that Diversify is robust to
varying backbones. Figure 13(b) shows the results using small, medium, and
large backbones, respectively (we implement them with different numbers of
layers.). Results indicate that larger models tend to achieve better OOD
generalization performance. Our method outperforms others in all backbones,
showing that Diversify presents consistently strong OOD performance in
different architectures. We also try Transformer [90] as the backbone for
comparisons. As shown in [91], Transformer often has a better generalization
ability compared to CNN, which implies improving with Transformer is more
difficult. From Figure 13(c), we can see that each method with Transformer has
a remarkable improvement on the first task of cross-dataset. Compared to ERM,
DANN and RSC have no improvements but ours still has further improvements and
achieves the best performance. DANN even performs worse than ERM, which
demonstrates the importance of more accurate sub-domain labels. Overall, for
all architectures, our method achieves the best performance.

###  6.8 Time Complexity and Convergence Analysis

We also provide some analysis on time complexity and convergence. Since we
only optimize the feature extractor in Step 2, our method does not cost too
much time. And the results in Figure 13(d) prove this argument empirically.
The convergence results are shown in Figure 13(e). Our method is convergent.
Although there are some little fluctuations, these fluctuations exist widely
in all domain generalization methods due to different distributions of
different samples.

##  Sec. 7 Limitation and Discussion

Diversify could be more perfect by pursuing the following avenues. 1) Estimate
the number of latent distributions $K$ automatically: we currently treat it as
a hyperparameter. 2) Seek the semantics behind latent distributions: can
adding more human knowledge obtain better latent distributions? 3) Extend
Diversify beyond detection and generalization but for forecasting problems.

Moreover, we argue that dynamic distributions not only exist in time series
but also in general machine learning data such as images and text [49, 51].
Thus, it is of great interest to apply our approach to these domains to
further improve their performance.

##  Sec. 8 Conclusion

We proposed Diversify, a universe framework, to learn generalized
representation for time series detection and generalization. Diversify employs
an adversarial game that maximizes the ‘worst-case’ distribution scenario
while minimizing their distribution divergence. We provide Diversify-MAH and
Diversify-MCP via representations and predictions respectively for detection
while we directly utilize Diversify for generalization. We demonstrated its
effectiveness in different applications. We are surprised that one dataset can
contain several latent distributions. Characterizing such latent distributions
will greatly improve the generalization performance on unseen datasets.

## References

  * [1] Q. Xiao, B. Wu, Y. Zhang, S. Liu, M. Pechenizkiy, E. Mocanu, and D. C. Mocanu, “Dynamic sparse network for time series classification: Learning what to “see”,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 16 849–16 862, 2022.
  * [2] W. Tang, G. Long, L. Liu, T. Zhou, M. Blumenstein, and J. Jiang, “Omni-scale cnns: a simple and effective kernel size configuration for time series classification,” in _International Conference on Learning Representations_ , 2022.
  * [3] B. D. Fulcher and N. S. Jones, “Highly comparative feature-based time-series classification,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 26, no. 12, pp. 3026–3037, 2014.
  * [4] M. Hüsken and P. Stagge, “Recurrent neural networks for time series classification,” _Neurocomputing_ , vol. 50, pp. 223–235, 2003.
  * [5] S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y.-X. Wang, and X. Yan, “Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting,” _Advances in Neural Information Processing Systems_ , vol. 32, pp. 5243–5253, 2019.
  * [6] A. Drouin, É. Marcotte, and N. Chapados, “Tactis: Transformer-attentional copulas for time series,” in _International Conference on Machine Learning, ICML_ , vol. 162, 2022, pp. 5447–5493.
  * [7] Y. Wang, C. Qian, and S. J. Qin, “Attention-mechanism based dipls-lstm and its application in industrial process time series big data prediction,” _Computers & Chemical Engineering_, p. 108296, 2023.
  * [8] Y. Du, J. Wang, W. Feng, S. Pan, T. Qin, R. Xu, and C. Wang, “Adarnn: Adaptive learning and forecasting of time series,” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 402–411.
  * [9] F. Di Martino and F. Delmastro, “Explainable ai for clinical and remote health applications: a survey on tabular and time series data,” _Artificial Intelligence Review_ , vol. 56, no. 6, pp. 5261–5315, 2023.
  * [10] J. Yang, K. Zhou, Y. Li, and Z. Liu, “Generalized out-of-distribution detection: A survey,” _arXiv preprint arXiv:2110.11334_ , 2021.
  * [11] J. Wang, C. Lan, C. Liu, Y. Ouyang, W. Zeng, and T. Qin, “Generalizing to unseen domains: A survey on domain generalization,” _IEEE Transactions on Knowledge and Data Engineering (TKDE)_ , 2022.
  * [12] A. Lehner, S. Gasperini, A. Marcos-Ramiro, M. Schmidt, M.-A. N. Mahani, N. Navab, B. Busam, and F. Tombari, “3d-vfield: Adversarial augmentation of point clouds for domain generalization in 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 17 295–17 304.
  * [13] V. Vidit, M. Engilberge, and M. Salzmann, “Clip the gap: A single domain generalization approach for object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 3219–3229.
  * [14] J. Yang, P. Wang, D. Zou, Z. Zhou, K. Ding, W. Peng, H. Wang, G. Chen, B. Li, Y. Sun _et al._ , “Openood: Benchmarking generalized out-of-distribution detection,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 32 598–32 611, 2022.
  * [15] Q. Wang, J. Ye, F. Liu, Q. Dai, M. Kalander, T. Liu, H. Jianye, and B. Han, “Out-of-distribution detection with implicit outlier transformation,” in _The Eleventh International Conference on Learning Representations_ , 2023\.
  * [16] Y. Wang, J. Zou, J. Lin, Q. Ling, Y. Pan, T. Yao, and T. Mei, “Out-of-distribution detection via conditional kernel independence model,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 36 411–36 425, 2022.
  * [17] J. Ren, J. Luo, Y. Zhao, K. Krishna, M. Saleh, B. Lakshminarayanan, and P. J. Liu, “Out-of-distribution detection and selective generation for conditional language models,” in _The Eleventh International Conference on Learning Representations_ , 2023.
  * [18] Z. Fang, Y. Li, J. Lu, J. Dong, B. Han, and F. Liu, “Is out-of-distribution detection learnable?” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 37 199–37 213, 2022.
  * [19] D. Hendrycks, M. Mazeika, and T. Dietterich, “Deep anomaly detection with outlier exposure,” in _International Conference on Learning Representations_ , 2019.
  * [20] Y. Zhu, Y. Chen, C. Xie, X. Li, R. Zhang, H. Xue, X. Tian, Y. Chen _et al._ , “Boosting out-of-distribution detection with typical features,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 20 758–20 769, 2022.
  * [21] H. Qian, S. J. Pan, C. Miao, H. Qian, S. Pan, and C. Miao, “Latent independent excitation for generalizable sensor-based cross-person activity recognition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 13, 2021, pp. 11 921–11 929.
  * [22] W. Lu, J. Wang, Y. Chen, S. J. Pan, C. Hu, and X. Qin, “Semantic-discriminative mixup for generalizable sensor-based cross-domain activity recognition,” _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 6, no. 2, pp. 1–19, 2022.
  * [23] V. Kuznetsov and M. Mohri, “Learning theory and algorithms for forecasting non-stationary time series,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [24] S. Lobov, N. Krilova, I. Kastalskiy, V. Kazantsev, and V. A. Makarov, “Latent factors limiting the performance of semg-interfaces,” _Sensors_ , vol. 18, no. 4, p. 1122, 2018.
  * [25] S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” in _International Conference on Learning Representations_ , 2018.
  * [26] D. Dennis, D. A. E. Acar, V. Mandikal, V. S. Sadasivan, V. Saligrama, H. V. Simhadri, and P. Jain, “Shallow rnn: accurate time-series classification on resource constrained devices,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [27] A. Dempster, D. F. Schmidt, and G. I. Webb, “Minirocket: A very fast (almost) deterministic transform for time series classification,” in _Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining_, 2021, pp. 248–257.
  * [28] Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long-term forecasting with transformers,” in _The Eleventh International Conference on Learning Representations_ , 2023.
  * [29] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller, “Deep learning for time series classification: a review,” _Data mining and knowledge discovery_ , vol. 33, no. 4, pp. 917–963, 2019.
  * [30] K. Benidis, S. S. Rangapuram, V. Flunkert, Y. Wang, D. Maddix, C. Turkmen, J. Gasthaus, M. Bohlke-Schneider, D. Salinas, L. Stella _et al._ , “Deep learning for time series forecasting: Tutorial and literature survey,” _ACM Computing Surveys_ , vol. 55, no. 6, pp. 1–36, 2022.
  * [31] Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun, “Transformers in time series: A survey,” _arXiv preprint arXiv:2202.07125_ , 2022.
  * [32] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Transactions on knowledge and data engineering_ , vol. 22, no. 10, pp. 1345–1359, 2009.
  * [33] D. L. Silver, Q. Yang, and L. Li, “Lifelong machine learning systems: Beyond learning algorithms,” in _2013 AAAI spring symposium series_ , 2013.
  * [34] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [35] S. Shankar, V. Piratla, S. Chakrabarti, S. Chaudhuri, P. Jyothi, and S. Sarawagi, “Generalizing across domains via cross-gradient training,” in _International Conference on Learning Representations_ , 2018.
  * [36] Q. Xu, R. Zhang, Y. Zhang, Y. Wang, and Q. Tian, “A fourier-based framework for domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 383–14 392.
  * [37] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao, “Deep domain generalization via conditional invariant adversarial networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 624–639.
  * [38] X. Zhang, P. Cui, R. Xu, L. Zhou, Y. He, and Z. Shen, “Deep stable learning for out-of-distribution generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5372–5382.
  * [39] A. Rame, C. Dancette, and M. Cord, “Fishr: Invariant gradient variances for out-of-distribution generalization,” in _International Conference on Machine Learning_ , 2022, pp. 18 347–18 377.
  * [40] D. Kim, Y. Yoo, S. Park, J. Kim, and J. Lee, “Selfreg: Self-supervised contrastive regularization for domain generalization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9619–9628.
  * [41] B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep domain adaptation,” in _European conference on computer vision_. Springer, 2016, pp. 443–450.
  * [42] X. Peng, Z. Huang, X. Sun, and K. Saenko, “Domain agnostic learning with disentangled representations,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5102–5112.
  * [43] H. Zhang, Y.-F. Zhang, W. Liu, A. Weller, B. Schölkopf, and E. P. Xing, “Towards principled disentanglement for domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8024–8034.
  * [44] T. Matsuura and T. Harada, “Domain generalization using a mixture of multiple latent domains,” in _AAAI_ , 2020.
  * [45] X. Fan, Q. Wang, J. Ke, F. Yang, B. Gong, and M. Zhou, “Adversarially adaptive normalization for single domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8208–8217.
  * [46] L. Li, K. Gao, J. Cao, Z. Huang, Y. Weng, X. Mi, Z. Yu, X. Li, and B. Xia, “Progressive domain expansion network for single domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 224–233.
  * [47] Z. Wang, Y. Luo, R. Qiu, Z. Huang, and M. Baktashmotlagh, “Learning to diversify for single domain generalization,” in _ICCV_ , 2021.
  * [48] R. Zhu and S. Li, “Crossmatch: Cross-classifier consistency regularization for open-set single domain generalization,” in _International Conference on Learning Representations_ , 2022.
  * [49] L. Deecke, T. Hospedales, and H. Bilen, “Visual representation learning over latent domains,” in _International Conference on Learning Representations_ , 2022.
  * [50] H. Wang, H. He, and D. Katabi, “Continuously indexed domain adaptation,” in _Proceedings of the 37th International Conference on Machine Learning_ , 2020, pp. 9898–9907.
  * [51] Z. Xu, G.-Y. Hao, H. He, and H. Wang, “Domain-indexing variational bayes: Interpretable domain index for domain adaptation,” in _The Eleventh International Conference on Learning Representations_ , 2023.
  * [52] C. E. Rasmussen _et al._ , “The infinite gaussian mixture model.” in _NIPS_ , vol. 12. Citeseer, 1999, pp. 554–560.
  * [53] P. W. Koh, S. Sagawa, S. M. Xie, M. Zhang, A. Balsubramani _et al._ , “Wilds: A benchmark of in-the-wild distribution shifts,” in _ICML_ , 2021, pp. 5637–5664.
  * [54] E. Delage and Y. Ye, “Distributionally robust optimization under moment uncertainty with application to data-driven problems,” _Operations research_ , vol. 58, no. 3, pp. 595–612, 2010.
  * [55] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, “Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * [56] Y. Song, N. Sebe, and W. Wang, “Rankfeat: Rank-1 feature removal for out-of-distribution detection,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 17 885–17 898, 2022.
  * [57] A. Djurisic, N. Bozanic, A. Ashok, and R. Liu, “Extremely simple activation shaping for out-of-distribution detection,” in _The Eleventh International Conference on Learning Representations_ , 2023.
  * [58] Y. Ming, Y. Sun, O. Dia, and Y. Li, “How to exploit hyperspherical embeddings for out-of-distribution detection?” in _The Eleventh International Conference on Learning Representations_ , 2023.
  * [59] L. Tao, X. Du, J. Zhu, and Y. Li, “Non-parametric outlier synthesis,” in _The Eleventh International Conference on Learning Representations_ , 2023\.
  * [60] Q. Yu and K. Aizawa, “Unsupervised out-of-distribution detection by maximum classifier discrepancy,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 9518–9526.
  * [61] J. Yang, H. Wang, L. Feng, X. Yan, H. Zheng, W. Zhang, and Z. Liu, “Semantically coherent out-of-distribution detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8301–8309.
  * [62] J. Yang, K. Zhou, and Z. Liu, “Full-spectrum out-of-distribution detection,” _International Journal of Computer Vision_ , pp. 1–16, 2023.
  * [63] G. Das, K.-I. Lin, H. Mannila, G. Renganathan, and P. Smyth, “Rule discovery from time series.” in _KDD_ , vol. 98, no. 1, 1998, pp. 16–22.
  * [64] W. Zhang, M. Ragab, and R. Sagarna, “Robust domain-free domain generalization with class-aware alignment,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 2870–2874.
  * [65] M. Ragab, Z. Chen, W. Zhang, E. Eldele, M. Wu, C.-K. Kwoh, and X. Li, “Conditional contrastive domain generalization for fault diagnosis,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 71, pp. 1–12, 2022.
  * [66] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 132–149.
  * [67] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _The journal of machine learning research_ , vol. 17, no. 1, pp. 2096–2030, 2016.
  * [68] K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [69] J. A. Lasserre, C. M. Bishop, and T. P. Minka, “Principled hybrids of generative and discriminative models,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 1. IEEE, 2006, pp. 87–94.
  * [70] K. P. Murphy, _Machine learning: a probabilistic perspective_. MIT press, 2012.
  * [71] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [72] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, “A theory of learning from different domains,” _Machine learning_ , vol. 79, no. 1, pp. 151–175, 2010.
  * [73] A. Sicilia, X. Zhao, and S. J. Hwang, “Domain adversarial neural networks for domain generalization: When it works and how to improve,” _arXiv preprint arXiv:2102.03924_ , 2021.
  * [74] G. Parascandolo, A. Neitz, A. Orvieto, L. Gresele, and B. Schölkopf, “Learning explanations that are hard to vary,” in _ICLR_ , 2021.
  * [75] J. Wang, Y. Chen, S. Hao, X. Peng, and L. Hu, “Deep learning for sensor-based activity recognition: A survey,” _Pattern recognition letters_ , vol. 119, pp. 3–11, 2019.
  * [76] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” vol. 32, 2019, pp. 8026–8037.
  * [77] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in _International Conference on Learning Representations_ , 2017.
  * [78] G. Wilson, J. R. Doppa, and D. J. Cook, “Multi-source deep domain adaptation with weak supervision for time-series sensor data,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 1768–1778.
  * [79] S. Purushotham, W. Carvalho, T. Nilanon, and Y. Liu, “Variational recurrent adversarial deep domain adaptation,” 2016.
  * [80] P. Schmidt, A. Reiss, R. Duerichen, C. Marberger, and K. Van Laerhoven, “Introducing wesad, a multimodal dataset for wearable stress and affect detection,” in _Proceedings of the 20th ACM international conference on multimodal interaction_ , 2018, pp. 400–408.
  * [81] B. Barshan and M. C. Yüksek, “Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units,” _The Computer Journal_ , vol. 57, no. 11, pp. 1649–1667, 2014.
  * [82] M. Zhang and A. A. Sawchuk, “Usc-had: a daily activity dataset for ubiquitous activity recognition using wearable sensors,” in _Proceedings of the 2012 ACM conference on ubiquitous computing_ , 2012, pp. 1036–1043.
  * [83] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz, “Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine,” in _International workshop on ambient assisted living_. Springer, 2012, pp. 216–223.
  * [84] A. Reiss and D. Stricker, “Introducing a new benchmarked dataset for activity monitoring,” in _2012 16th international symposium on wearable computers_. IEEE, 2012, pp. 108–109.
  * [85] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _International Conference on Learning Representations_ , 2018.
  * [86] Z. Huang, H. Wang, E. P. Xing, and D. Huang, “Self-challenging improves cross-domain generalization,” in _ECCV_ , 2020, pp. 124–140.
  * [87] P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” _arXiv preprint arXiv:1804.03209_ , 2018.
  * [88] P. Kidger, J. Morrill, J. Foster, and T. Lyons, “Neural Controlled Differential Equations for Irregular Time Series,” _Advances in Neural Information Processing Systems_ , 2020.
  * [89] S. Majumdar and B. Ginsburg, “Matchboxnet: 1d time-channel separable convolutional neural network architecture for speech commands recognition,” in _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_. ISCA, 2020, pp. 3356–3360.
  * [90] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [91] C. Zhang, M. Zhang, S. Zhang, D. Jin, Q. Zhou, Z. Cai, H. Zhao, X. Liu, and Z. Liu, “Delving deep into the generalization of vision transformers under distribution shifts,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 7277–7286.