# Examining the Effect of Pre-training Followed by Fine-tuning on Time Series
Classification

Jiashu Pu<EMAIL_ADDRESS>
Shiwei Zhao<EMAIL_ADDRESS>
Fuxi AI Lab  Netease Inc  Ling Cheng<EMAIL_ADDRESS>
Singapore Management University  Yongzhu Chang<EMAIL_ADDRESS>
Runze Wu<EMAIL_ADDRESS>
Tangjie Lv<EMAIL_ADDRESS>
Rongsheng Zhang<EMAIL_ADDRESS>
Fuxi AI Lab Corresponding Author Netease Inc

###### Abstract

Although the pre-training followed by fine-tuning paradigm is used extensively
in many fields, there is still some controversy surrounding the impact of pre-
training on the fine-tuning process. Currently, experimental findings based on
text and image data lack consensus. To delve deeper into the unsupervised pre-
training followed by fine-tuning paradigm, we have extended previous research
to a new modality: time series. In this study, we conducted a thorough
examination of 150 classification datasets derived from the Univariate Time
Series (UTS) and Multivariate Time Series (MTS) benchmarks. Our analysis
reveals several key conclusions. (i) Pre-training can only help improve the
optimization process for models that fit the data poorly, rather than those
that fit the data well. (ii) Pre-training does not exhibit the effect of
regularization when given sufficient training time. (iii) Pre-training can
only speed up convergence if the model has sufficient ability to fit the data.
(iv) Adding more pre-training data does not improve generalization, but it can
strengthen the advantage of pre-training on the original data volume, such as
faster convergence. (v) While both the pre-training task and the model
structure determine the effectiveness of the paradigm on a given dataset, the
model structure plays a more significant role.

###### keywords:

Time-Series Classification, Unsupervised Pre-training, Optimization

##  1 Introduction

_The pre-training then fine-tuning paradigm_ continues to shine in the Natural
Language Processing field, owing to the immense data and extra-large model
sizes, with ultra-large models dominating the SuperGlue Benchmark Wang et al.
(2019). Recently, the masked autoencoding pre-training scheme has also
demonstrated its viability in Computer Vision He et al. (2022). Deep Learning
seems to be moving towards a grand unification of pre-training. Nonetheless,
there is still a lack of consensus on how exactly this paradigm manifests
itself. While earlier work Bengio et al. (2006) has shown that unsupervised
pre-training helps optimization, the later milestone research Erhan et al.
(2010) argues that unsupervised pre-training improves generalization by acting
as a form of regularization. In contrast, a recent study He et al. (2019)
claims that supervised pre-training only improves convergence speed without
benefiting generalization. Another study Abnar et al. (2021) showcases a
scenario where the downstream performance is at odds with the supervised pre-
training accuracy. Other works suggest pre-training offers more obvious
advantages of combating label noise Hendrycks et al. (2019), alleviating
catastrophic forgetting Mehta et al. (2021), and dealing with imbalanced
datasets Liu et al. (2021).

The divergent conclusions drawn from prior research have prompted us to
investigate this issue using a different approach. Our curiosity lies in
determining which findings demonstrate cross-modal consistency. Time series
possess two distinct properties that complicate their analysis. Firstly, the
features of time series data vary significantly across different domains,
posing a challenge to domain transfer Eldele et al. (2021). Secondly, within
the same domain, the distribution of time series data can shift over time
Tonekaboni et al. (2020), rendering the use of long-standing data less
effective. It is likely due to these factors that _the pre-training and then
fine-tuning paradigm_ has yet to prosper in the field of time series.
Nonetheless, we believe that further investigation into this direction is
worthwhile. Specifically, we focus on the widely concerned problem of Time
Series Classification (TSC) to understand how this paradigm operates.
Notwithstanding its high accuracy on the UCR benchmark Dau et al. (2019), the
current best ensemble model of Time Series Classification, HIVE-COTE 2.0
Middlehurst et al. (2021), suffers from slow training speed and challenging
deployment. Moreover, given the complexity of time series data Tonekaboni et
al. (2020), it is difficult for non-specialists to directly annotate the raw
time series data Eldele et al. (2021); Zerveas et al. (2021), even as time
series data relevant to our daily lives continues to accumulate at an
unprecedented rate
111https://www.forbes.com/sites/forbescommunicationscouncil/2022/06/16/the-
ubiquity-of-time-series-data-isnt-coming-its-already-here. We posit that _the
unsupervised pre-training and fine-tuning paradigm_ holds promise for the
future of Time Series Classification. Considering the characteristics of time
series data, we design a pre-training setup with in-domain datasets,
conducting experiments on 150 datasets with three model structures and five
pre-training tasks. Our study makes a threefold contribution. Firstly, we
verify the feasibility of the paradigm of unsupervised pre-training followed
by fine-tuning on TSC. Secondly, we re-validate some existing conclusions
concerning the impact of unsupervised pre-training on fine-tuning for time
series data. Additionally, we provide novel insights into which factors - the
pre-training task or model structure - are more critical in enhancing the
efficacy of pre-training on fine-tuning. Lastly, we attempt to find correlates
of successful pre-training. Our key findings are as follows:
\- Pre-training can enhance optimization for under-fitted models of few
parameters (Consistent with Bengio et al. (2006) under constraints), but it
does not improve optimization for for models that already have adequate
ability to fit the data.
\- Given sufficient training time, pre-training does not significantly improve
a model’s generalization ability (Consistent with He et al. (2019)), i.e., it
has no regularization effect (Contradict with Erhan et al. (2010)).
\- Pre-training can accelerate the convergence speed (Consistent with He et
al. (2019)), but only if the model has the capability to fit the data well.
\- Increasing the amount of pre-training data does not aid generalization
(Contradict with Paine et al. (2014)), but it can strengthen the existing
advantages of pre-training for the original data sizes.
\- The effectiveness of _the pre-training followed by fine-tuning paradigm_ on
a given dataset is determined by both the model structure and the pre-training
task, with the model structure being more crucial.

##  2 Related Work

Unsupervised pre-training is gaining more and more attention in the field of
time series. Feature-wise, most of the work is based on temporal features,
considering both inter-sample dissimilarity and intra-temporal resemblance to
design pre-training tasks around contrastive learning Yue et al. (2022);
Eldele et al. (2021); Tonekaboni et al. (2020). The time-frequency
interchangeability motivates other work to exploit the frequency domain to
obtain proper representations of the seasonal trend Woo et al. (2021); Zhang
et al. (2022). However, most of the work above uses linear probing to verify
the effectiveness of representations on tasks such as classification Eldele et
al. (2021); Franceschi et al. (2019), forecasting Yue et al. (2022),
regression Zerveas et al. (2021), and anomaly detection Yue et al. (2022).In
the field of time series, no work has investigated how unsupervised pre-
training affects fine-tuning.

In other fields such as Natural Language Processing and Computer Vision, it is
still controversial how unsupervised pre-training works on fine-tuning. Some
work suggests that unsupervised pre-training is beneficial for optimization
Hao et al. (2019); Neyshabur et al. (2020), while other work argued that
unsupervised pre-training only serves the function of regularization Erhan et
al. (2010). There is also evidence suggesting that unsupervised pre-training
can only speed up convergence He et al. (2019). In the face of adversarial
samples, imbalanced datasets Liu et al. (2021), and label corruption, some
work also argues that pre-training has a great advantage and can improve the
model’s uncertainty estimates as well Hendrycks et al. (2019). In addition, in
the life-long learning scenario, one work suggests that pre-training can
alleviate the effects of catastrophic forgetting Mehta et al. (2021). Our work
extends the above efforts by validating some existing ideas in the field of
time series while doing new research on finding potential correlates of
effective pre-training, the inductive bias of pre-training tasks, etc.

##  3 Problem Formulation

Our paper focuses on the validity of the _paradigm of unsupervised pre-
training followed by fine-tuning_ for the Time Series Classification task. To
this end, we verify its effectiveness on 150 datasets. In the following
paragraphs, we first introduce the notations of the time series data and the
encoder, then we present how experiments are designed and what issues we
specifically analyze.

Time Series Data We define a group of continuous variables changing over time
as $s$, where $s\in\mathbb{R}^{d}$, with $d$ corresponding to the number of
variables or, alternatively, the dimension of the features. Based on this, we
define a time series of length $\ell$ as $S=(s_{1},\ldots,s_{\ell})$. When $d$
is $1$, the time series $S$ belongs to the UTS, while when $d$ is greater than
$1$, the time series $S$ belongs to the MTS.

Time Series Encoder and Classification Head We define the time series encoder
as $f_{\theta_{1}}:S\rightarrow E$, where $E$ is the output of the encoder’s
last layer. The output $E=(e_{1},\ldots,e_{\ell})$, where
$e_{i}\in\mathbb{R}^{h}$ and $h$ is the hidden dimension of the output layer.
To adapt to different classification datasets, we introduce the classification
head $g_{\theta_{2}}:E\rightarrow\\{1,\ldots,C\\}$ such that
$(g_{\theta_{2}}\circ f_{\theta_{1}}):S\rightarrow y\in\\{1,\ldots,C\\}$,
where $C$ is the total number of categories and $y$ denotes the categorical
output. More specifically, the classification head $g_{\theta_{2}}$ consists
of a one-dimensional convolutional layer followed by a Multiple Layer
Perceptron (MLP). By considering the length $\ell$ of the sequence as the
number of channels, we apply the one-dimensional convolutional layer to
transform the encoder’s output tensor $\mathrm{T_{E}}\in\mathbb{R}^{\ell\times
h}$ into a length uncorrelated vector $\bm{v}_{E}\in\mathbb{R}^{h}$. Using the
vector $\bm{v}_{E}\in\mathbb{R}^{h}$ as the input, the MLP layer produces the
output as the predicted class label $y$.

Research Problems and Experimental Procedure We chose three sequence encoder
structures and five pre-training tasks for our study, with a pre-training task
expressed as $\mathcal{PT}$. We denote an unlabelled pre-training dataset as
$\mathcal{D}_{pre}$, where $\mathcal{D}_{pre}=(S_{1},\ldots,S_{|D_{pre}|})$.
Similarly, we define the training set and test set of a classification dataset
as $\mathcal{D}_{train}$ and $\mathcal{D}_{test}$. There are $150$
experimented datasets in total, for each of them, we iterate through all
combinations of different model structures and pre-training tasks. For each
combination, we first pre-train $f_{\theta_{1}}$ on $\mathcal{D}_{pre}$ with
$\mathcal{PT}$ and obtain a specific set of pre-trained parameters
$\theta_{pre}$, after which the encoder $f_{\theta_{1}}$ initialized with
$\theta_{pre}$ and $g_{\theta_{2}}$ are simultaneously fine-tuned on
$\mathcal{D}_{train}$ (The domain of $\mathcal{D}_{pre}$ is the same as
$\mathcal{D}_{train}$, and we set
$\mathcal{D}_{train}\subseteq\mathcal{D}_{pre}$ for every dataset, the reason
for which is described in Section 1). Based on the results of all test sets
and the recorded training processes, we perform a significance analysis on
whether pre-training is advantageous over random initialization in various
aspects—including optimization, generalization, convergence speed, etc. We
verify whether the role of pre-training changes when the model is
significantly under-fitted, and whether adding additional pre-training data is
beneficial. In addition, we also analyze the results to answer a question —
which aspect has more influence on the fine-tuning results given an arbitrary
dataset, the model structure, or the pre-training task?

##  4 Research Subjects

###  4.1 Time Series Encoder

While ensemble models continue to set new records on the UCR benchmark
Middlehurst et al. (2021), we select three classical model structures of
moderate complexity as the time series encoder (the number of parameters is
around 500,000). There are two reasons. First, models designed for time series
need to be scalable and efficient because signals are often long and have a
high dimension Tonekaboni et al. (2020); Franceschi et al. (2019). Second,
under limited computational resources, we chose to do experiments on as many
datasets as possible to make conclusions informative and credible. It is worth
noting that the model structures presented below all model the time series in
both directions.

LSTM: Previous work Sagheer and Kotb (2019) shows the usefulness of
unsupervised pre-training of LSTM-based autoencoder for MTS prediction tasks.
Here we have simplified the model by constructing a vanilla bidirectional LSTM
of two layers.

Dilated Convolutional Neural Network (D.Conv): The Convolutional Neural
Network performs well on time series forecasting Yue et al. (2022) and
demonstrates its strengths of representing learning on the UTS and the MTS
datasets Franceschi et al. (2019). _D.Conv_ consists of layers of dilated
convolutions. Compared to the previous work Franceschi et al. (2019), we do
not use causal convolutions, so as to incorporate information from both before
and after time step $i$ when conducting convolution operations.

Time-Series Transformer (TsTransformer): The Time-Series Transformer has
proved a success in representing the MTS data type Zerveas et al. (2021). It
has the same structure as the original transformer encoder Vaswani et al.
(2017), except that it replaces the Layer Normalization layer with the Batch
Normalization layer and the embedding layer with the linear projection layer.

###  4.2 Pre-training Task

Representation learning of time series is becoming more and more sought-after
in recent years. We select some recently proposed unsupervised pre-training
tasks of relative efficacy, while also taking into account diversity.

  * (Baseline) Random-Cls Maennel et al. (2020): _Radom-Cls_ is the abbreviation of training with Random Class Labels. Because one paper claims that pre-training the model with random labels can effectively improve the convergence speed when fine-tuning Maennel et al. (2020), we choose it as the baseline of the pre-training task.

  * Ts2Vec Yue et al. (2022): Ts2vec is a contrastive learning-based approach. It highlights the construction of positive and negative samples by taking into account both sample and temporal differences. The other strength of Ts2Vec lies in modeling time series at hierarchical levels of granularity. This method has been evaluated in great detail on the UTS and MTS datasets, but it only validates its effectiveness in the linear-probing setting.

  * Ts-Tcc Eldele et al. (2021): Ts-Tcc is another contrastive learning framework, integrating two contrasting modules — temporal and contextual. It starts with generating a strong and a weak view from the time series $S$ via augmentation. During temporal contrasting, the model predicts future series ($S_{>t}$) of one view from another ($S_{\leq t}^{\prime}$), where $S^{\prime}$ denotes another view of $S$ and $t$ is the time step. While the contextual contrasting is to minimize the distance between $f_{C}(S_{\leq t})$ and $f_{C}(S_{\leq t}^{\prime})$, where $f_{C}$ summarize the context till $t$, at the same time, maximize the distance between $f_{C}(S_{\leq t})$ and the context of other augmented views of different instances.

  * Mvts Zerveas et al. (2021): The abbreviation Mvts comes from the tile ‘A Transformer-based Framework for Multivariate Time Series Representation Learning’. Given a sample of time series $S$, Mvts obtains corrupted inputs by zeroing. For instance, we can mask (zero) every variable independently (for the MTS data type) or mask a subseries of $S$. The pre-training task is to impute these zeroed inputs, which is analogous to the Masked Language Modeling Devlin et al. (2019) in NLP. The loss function of Mvts is the Mean Squared Error.

  * Srlt Franceschi et al. (2019): The abbreviation Srlt comes from the paper title — Unsupervised Scalable Representation Learning for Multivariate Time Series. Inspired by how word2vec Mikolov et al. (2013) is trained, this method proposes a novel triplet loss for modeling the time series data. Roughly speaking, given a time series $S$, its subseries of varying lengths are selected as positive samples, while other subseries of some arbitrary time series are selected as negative samples.

###  4.3 Given a dataset, how to weigh the picking of the model structure and
the design of the pre-training task?

The pre-training tasks for time series are built on some implicit or explicit
assumptions. Ts-Tcc, Ts2Vec, and Mvts all assume that time series are
contextually related and predictable, and Ts-Tcc assumes that time series have
stretch and shuffle invariance properties. Therefore, we believe that
different pre-training tasks have their specific applicability scenarios and
limitations. Likewise, different model structures also have different
inductive biases Tuli et al. (2021). The model structure determines the
parameter space, while the pre-training process determines the starting point
of fine-tuning. Given an arbitrary dataset, which one is more critical for
successful fine-tuning? The pre-training task or the model structure?

To reveal potential correlates, we calculate the Spearman’s rank correlation
coefficient Schober et al. (2018) between different variables and the relative
test set accuracy gaps, which equals the test-set accuracy of the pre-trained
model $f_{\theta_{pre}}$ minus the accuracy of randomly initialized model
$f_{\theta_{1}}$. We take sequence length as an example. We denote $M$ as the
total number of datasets and $b_{i}$ as the average sequence length of $i$-th
data set. We have the variable of sequence length $B$, where
$B=(b_{1},\ldots,b_{M})$. The other variable of the accuracy gap $A$ can be
defined similarly. If we calculate the correlation coefficient between $A$ and
$B$ as $0.55$, with the $p$-value being $0.03$ ($<0.05$). We can conclude that
$A$ is positively correlated with $B$ and the result is significant enough.

###  4.4 Can we determine in advance whether we need to pre-train and when can
we trust the pre-trained parameters?

In the practical application of time series data, we may have abundant
unlabeled data, while often not having sufficient annotated data Eldele et al.
(2021). When the real demand is for fast iteration or instant migration to new
scenarios, it becomes critical to understand in advance whether $\theta_{pre}$
has the potential to fit the data of the downstream task and generalize
beyond. To this end, we collect potential correlates from three
perspectives—the data, the pre-training process, and the model parameters,
hoping to uncover some indications for effective pre-training. The correlation
factors include: the length of time series, the size of the pre-training data,
$\ell_{2}$ norms Neyshabur et al. (2017), $\ell_{2}$-path norm Jiang et al.
(2019), the sharpness value Mehta et al. (2021); Hao et al. (2019), the
convergence state of pre-training and the $\ell_{2}$ distance $\theta_{pre}$
traveled from the initial point. The $\ell_{2}$ norms, the $\ell_{2}$-path
norm, and the sharpness value are three generalization measures validated by
previous work. Recent work Gouk et al. (2020); Mao (2020) suggest that
traveled distance is associated with generalization performance.

To reveal potential correlates, we calculate the Spearman’s rank correlation
coefficient Schober et al. (2018) between different variables and the relative
test set accuracy gaps, which equals the test-set accuracy of the pre-trained
model $f_{\theta_{pre}}$ minus the accuracy of randomly initialized model
$f_{\theta}$. We take sequence length as an example. We denote $M$ as the
total number of datasets and $b_{i}$ as the average sequence length of $i$-th
data set. We have the variable of sequence length $B$, where
$B=(b_{1},\ldots,b_{M})$. The other variable of the accuracy gap $A$ can be
defined similarly. If we calculate the correlation coefficient between $A$ and
$B$ as $0.55$, with the $p$-value being $0.03$ ($<0.05$). We can conclude that
$A$ is positively correlated with $B$ and the result is significant enough.

##  5 Datasets and Experimental settings

We use a total of 150 datasets, 125 of which are from the UTS data type and 25
from the MTS data type. Due to hardware constraints, we discard six datasets
that contain excessively long and highly dimensional sequences. In most of the
datasets, the sequence lengths are equal, while in other few datasets where
the sequences are not equal, we pad the sequences to the maximum length with
zeros. We present basic statistics of the UTS and MTS datasets in Table 1.

| Num.Class | Num.Sample | Series.Length | Feature.Dim
---|---|---|---|---
UTS | 9/2/60 | 473/16/8926 | 794/15/13167 | 1/1/1
MTS | 9/2/39 | 1866/12/25000 | 1159/8/17984 | 99/2/1345
Table 1: Basic statistics of the UTS and the MTS datasets. We present the
average/min/max value for each aspect. Feature.Dim means feature dimension.

Due to the complexity, diversity, and timeliness of time series data Zhang et
al. (2022), collecting in-domain data depends upon strong expert knowledge
Eldele et al. (2021). Thus it is impractical to collect a large amount of
unlabeled data for each dataset, to simplify the experimental setup, we set
$\mathcal{D}_{train}$ as $\mathcal{D}_{pre}$ in our main experiments, which is
a setup focusing on the low-resource scenario — an important topic in today’s
machine learning community Rotman and Reichart (2019). We divide each
$\mathcal{D}_{pre}$ into a training set $\mathcal{D}_{pre}^{train}$ and a
validation set $\mathcal{D}_{pre}^{val}$ ($10\%$ of $\mathcal{D}_{pre}$). To
avoid over-fitting, we retain the parameter with the lowest validation loss as
$\theta_{pre}$. We set the learning rate, batch size, and epoch of pre-
training to $1e^{-4}$, $32$, and $100$ respectively. Some pre-training works
mention the sensitivity of their methods to hyperparameters Yue et al. (2022);
Zerveas et al. (2021), so we randomly select $30$ UTS datasets, perform a gird
search of hyperparameters for each pre-training task and pick the best ones.

For fine-tuning, we split $10\%$ of $\mathcal{D}_{train}$ as the validation
set, on which we trace the model with the highest accuracy. Learning rates of
$f_{\theta_{1}}$ and $g_{\theta_{2}}$ are both set to $1e^{-3}$, and batch
size is set to $32$. We apply gradient clipping and set the clip value to
$4.0$. We train encoders and classifiers for $200$ epochs to ensure
convergence, with the cross-entropy loss. To evaluate our model, we use the
official split Dau et al. (2019) of training and test data for each dataset.
We train the model five times with different random seeds (control weight
initialization) for a given set of model structures and pre-training tasks.
The final test result for each dataset is the average of the five runs. The
pre-training and supervised training process share the same optimizer of Adam
Kingma and Ba (2014). For other detailed configuration of the model structure
and hyperparameters please refer to code repository.

##  6 Experimental Results and Analysis

The Wilcoxon signed rank test Rey and Neuhäuser (2011) is performed between
the results of each pre-trained model and the results of the train-from-
scratch counterpart. In Table 2 and Table 3, the symbol ${\dagger}$ indicate
$p$-value $<0.05$, and the down arrow $\downarrow$ indicates that the pre-
trained result is significantly worse ($p<0.05$) than the non-pre-trained one.

###  6.1 Effect of pre-training

|  | Not.P | R.Cls | Ts2Vec | Ts-Tcc | Mvts | Srlt
---|---|---|---|---|---|---|---
| Training Loss (Min)
| LSTM | 0.045 | 0.043 |  0.044 | 0.046 |  0.044 |  0.044
| TsTransformer | 0.389 | 0.391 | 0.384 | 0.401 | 0.429 |  0.388
MTS | D.Conv | 0.012 |  0.010 |  0.010 | 0.008 | 0.012 | 0.008
| LSTM |  0.038 | 0.039 | 0.037 | 0.037 | 0.041 | 0.037
| TsTransformer | 0.487 | 0.501 | 0.476 | 0.490 |  0.483 | 0.487
UTS | D.Conv |  0.012 | 0.011 | 0.013 | 0.015 | 0.013 | 0.020
| Accuracy (Early Stopping)
| LSTM | 0.679 | 0.727 | 0.683 |  0.724 | 0.716 | 0.708
| TsTransformer | 0.685 | 0.700 | 0.707† | 0.689 |  0.695 | 0.693
MTS | D.Conv |  0.728 | 0.716 | 0.723 | 0.711 | 0.751† |  0.728
| LSTM | 0.704 | 0.696 | 0.692 |  0.700 | 0.683$\downarrow$ | 0.692
| TsTransformer |  0.665 | 0.660 | 0.662 | 0.661 | 0.653 | 0.668
UTS | D.Conv | 0.724 | 0.721 | 0.743† | 0.732 |  0.741 | 0.726
| Accuracy (Last Epoch)
| LSTM | 0.701 | 0.711 | 0.714 | 0.716 | 0.722 |  0.719
| TsTransformer | 0.680 | 0.672 | 0.696† |  0.685 | 0.680 | 0.681
MTS | D.Conv | 0.730 | 0.728 |  0.732 | 0.724 | 0.742 | 0.723
| LSTM | 0.698 | 0.688 | 0.683 | 0.683 | 0.677 |  0.689
| TsTransformer | 0.660 | 0.650 | 0.660 |  0.656 | 0.648$\downarrow$ | 0.648
UTS | D.Conv | 0.739 | 0.740 |  0.746 | 0.745 | 0.754 | 0.738
| Accuracy (Max)
| LSTM | 0.754 | 0.764 | 0.769 |  0.772 | 0.770 | 0.775†
| TsTransformer | 0.744 | 0.739 |  0.750 | 0.746 | 0.748 | 0.753
MTS | D.Conv | 0.779 | 0.772 |  0.783 | 0.779 | 0.790 | 0.775
| LSTM |  0.770 | 0.765 | 0.763 | 0.764 | 0.761$\downarrow$ | 0.772
| TsTransformer |  0.727 | 0.722 | 0.730 | 0.725 | 0.721$\downarrow$ | 0.726
UTS | D.Conv | 0.794 | 0.796 | 0.804† | 0.799† |  0.800 | 0.790
| Accuracy (Epoch 1)
| LSTM | 0.355 | 0.376 | 0.456† | 0.395 |  0.444† | 0.435†
| TsTransformer |  0.411 | 0.386 | 0.414 | 0.361$\downarrow$ | 0.389 | 0.397
MTS | D.Conv | 0.441 | 0.436 |  0.470 | 0.468† | 0.461 | 0.476
| LSTM | 0.338 | 0.340 | 0.416† | 0.388† | 0.396† | 0.385†
| TsTransformer | 0.399 | 0.361$\downarrow$ |  0.392 | 0.374 | 0.377 | 0.374
UTS | D.Conv | 0.421 | 0.409 | 0.474† |  0.460† | 0.458† |  0.460†
Table 2: Main results of using different pre-training tasks on the UTS (125) and MTS (25) datasets. ‘R.Cls’ refers to the pre-training task of Random-Cls. ‘Not.P’ refers to the Pytorch (https://pytorch.org) default weight initialization scheme. The Best results are in bold and the second best is underlined. |  | Not.P | R.Cls | Ts2Vec | Ts-Tcc | Mvts | Srlt
---|---|---|---|---|---|---|---
Training Loss (Min)
| LSTM-underfit | 0.391 | 0.372† | 0.378 | 0.391 | 0.411$\uparrow$ | 0.387
MTS | D.Conv-underfit | 0.306 | 0.311 | 0.319 | 0.317 | 0.319 | 0.317
| LSTM-underfit | 0.490 | 0.485 | 0.468 | 0.476 | 0.462† | 0.464†
UTS | D.Conv-underfit | 0.414 | 0.396 | 0.401† | 0.410 | 0.404 | 0.404
Accuracy (Epoch 1)
| LSTM-underfit | 0.264 | 0.260 | 0.265 | 0.265 | 0.239 | 0.283
MTS | D.Conv-underfit | 0.258 | 0.276 | 0.271 | 0.297† | 0.308† | 0.303†
| LSTM-underfit | 0.312 | 0.303 | 0.320 | 0.313 | 0.312 | 0.311
UTS | D.Conv-underfit | 0.312 | 0.301 | 0.323 | 0.334 | 0.321 | 0.334
Table 3: Training loss and epoch-$1$ accuracy of under-fitted LSTM and D.Conv.

#### Effects on optimization

Non-convex optimization in high-dimensional space has been a major challenge
in deep learning since the overall process is affected by many factors
Goodfellow et al. (2016), such as the initial parameters, the choice of the
optimizer, the model structure, etc. Here we study whether the pre-trained
parameters $\theta_{pre}$ lead to a lower training loss at convergence. Some
studies suggest pre-training simplifies optimization Hao et al. (2019);
Neyshabur et al. (2020), while other work Erhan et al. (2010) claim pre-
training does not benefit the optimization procedure.

Analyzing Table 2, we can find that all pre-training tasks do not
significantly obtain lower training loss compared to random initialization,
that is, the pre-trained parameters do not enable the model to fit the data
better.

We also analyse whether pre-training helps in optimization by considering two
types of under-fitted models. Both types are poorly fitted (high training
loss) but vary in structural complexity. The first type is the model with a
complex structure, i.e., the TsTransformer (with a parameter count of around
400,000). The other type is the model with a simpler structure, including
LSTM-underfit and D.Conv-underfit (number of parameters around 3,000). LSTM-
underfit and D.Conv-underfit share the same structure with LSTM and D.Conv but
differs in hidden dimensions and number of layers. Interestingly, observing
Table 3 and Table 2, we find that when pre-training the models with simple
structures, pre-training can, in some cases, improve the model’s fitting
ability of the data. In contrast, provided with a poorly fit model with a
large number of parameters, pre-training does not bring any benefit to the
final optimization result.

#### Availability of regularization

A more general definition of regularization is a technology aimed at improving
the generalization ability of a model Tian and Zhang (2022); Goodfellow et al.
(2016). One previous work states that the main role of pre-training is to
regularize Erhan et al. (2010). To verify this belief, we chose three metrics
to reflect the generalization ability of the model - including the accuracy in
the early stopping condition, the accuracy of the last epoch, and the highest
accuracy among all epochs.

We can see in Table 2 that no pre-training task significantly improves the
generalization ability of the model on both the UTS and the MTS datasets.
Relatively speaking, Ts2Vec is the most effective pre-training scheme for
improving generalization, but its effectiveness is only limited to some
datasets and model structures, while the Mvts pre-training task even
significantly degrades the generalization performance in some settings — e.g.,
it degrades the performance of LSTM on the UTS dataset in the early stopping
condition. Although Ts2Vec is effective in some cases, we conclude that most
of the pre-training tasks for time series fail to improve the generalization
ability of the model, i.e., the regularization effect of pre-training is not
significant.

Figure 1: Gaps between each unsupervised pre-training task and random initialization in terms of convergence speed and generalization. Only the results on the UTS datasets are shown (the trend is similar on the MTS datasets). Each point is averaged across all datasets, with the length of the error bars being a $95\%$ confidence interval for the mean. The _Top row_ presents the decrease in training loss of the pre-trained model compared to the randomly initialized model. The _Bottom row_ depicts the increase in test-set accuracy. | Ts2vec | Ts-Tcc | Mvts | Srlt
---|---|---|---|---
Training Loss (Min)
LSTM | 0.010/0.006 | 0.012/0.007 | 0.012/0.008† | 0.013/0.009
LSTM-underfit | 0.184/0.176 | 0.193/0.181 | 0.194/0.180† | 0.193/0.179†
TsTransformer | 0.302/0.282 | 0.301/0.287 | 0.302/0.288 | 0.303/0.296
D.Conv | 0.038/0.024 | 0.046/0.025 | 0.045/0.026 | 0.056/0.021†
D.Conv-underfit | 0.239/0.179† | 0.250/0.201† | 0.247/0.197† | 0.246/0.202†
Accuracy (Early Stopping)
LSTM | 0.726/0.769 | 0.737/0.787 | 0.741/0.778 | 0.736/0.780
LSTM-underfit | 0.712/0.724 | 0.721/0.737 | 0.732/0.741 | 0.736/0.739
TsTransformer | 0.756/0.756 | 0.745/0.745 | 0.743/0.748 | 0.746/0.742
D.Conv | 0.793/0.800 | 0.786/0.807 | 0.790/0.815 | 0.783/0.819†
D.Conv-underfit | 0.736/0.755 | 0.725/0.762 | 0.722/0.761 | 0.728/0.759
Accuracy (Epoch1)
LSTM | 0.513/0.557 | 0.487/0.512 | 0.473/0.508† | 0.459/0.500†
LSTM-underfit | 0.391/0.394 | 0.378/0.376 | 0.381/0.385 | 0.377/0.383
TsTransformer | 0.433/0.453 | 0.414/0.437 | 0.421/0.436 | 0.421/0.427
D.Conv | 0.525/0.567 | 0.506/0.569† | 0.505/0.565† | 0.502/0.569†
D.Conv-underfit | 0.397/0.422 | 0.408/0.413 | 0.406/0.417 | 0.405/0.414
Table 4: The values before and after the slash(/) are the dataset-averaged (21
in total) accuracy of using original pre-training datasets and of using the
expanded ones. The symbol ${\dagger}$ indicates a significant advantage of
using extra data.

#### Effects on convergence speed

As we can see in Figure 1, except when applied to TsTransformer, all pre-
training schemes improve accuracy in the first few epochs and also reduce the
training loss. In addition, observing the accuracy of the first epoch in Table
2, most of the pre-training tasks significantly boost LSTM and D.Conv at the
beginning of fine-tuning. In contrast, the pre-training for TsTransformer does
not show similar advantages. Since TsTransformer is under-fitted (high
training loss) on most datasets, we conjecture that pre-training cannot speed
up the convergence of the model when it is under-fitted. The results in Table
3 largely validate our conjecture. Only a small number of cases of under-
fitted D.Conv are accelerated by pre-training (using Ts-Tcc, Mvts, and srlt
pre-training schemes). We summarize as follows: when the model fitting ability
is sufficient, pre-training mostly improves the convergence speed and entails
fast generalization, a conclusion similar to previous observations in the CV
domain He et al. (2019), while when the model is under-fitted, pre-training
usually fails to improve the convergence rate.

#### Effects of extra pre-training data

We experiment with adding more pre-training data on three data types— _Food_ ,
_ECG_ and _Image_ (all UTS). We specifically select these three types because
they contain datasets that are relatively similar to each other, which does
not violate the experimental setup of in-domain pre-training. _Food_ , _ECG_
and _Image_ data type contains $6$, $7$ and $32$ datasets respectively. On
average, the amount of pre-training data is increased by $31.41$, $27.5$, and
$37.8$ folds. We illustrate the expansion process with an example. Suppose the
target dataset, Beef, falls under the _Food_ type, we add samples of other
datasets (containing both training and test sets) of the _Food_ type to
$\mathcal{D}_{pre}$.

In Table 4 we show the impact of adding extra pre-training data. Two key
points can be summarized. First, in the majority of cases, increasing the pre-
training data by several times does not markedly improve the generalization
ability of the model, which contradicts the previous work Paine et al. (2014)
that claims unsupervised pre-training helps when the ratio of unsupervised to
supervised samples is high. Second, except for Ts2vec, adding more pre-
training data reinforces the advantages stemming from pre-training. For
instance, pre-training on the original dataset speed up the convergence of
D.Conv in the early stage (Table 2). With additional pre-training data, the
convergence speed is further accelerated.

###  6.2 Which factor, the model structure or the pre-training task, is more
critical for pre-training to work on specific dataset?

Figure 2:  We present the intersection ratio $\varphi_{ij}$ of advantageous
dataset between different pre-training tasks with the fixed model structure.
The shades of color present the values of $\varphi_{ij}$. The value in
parentheses represents the size of the advantageous dataset
$\mathcal{A}_{pt}$. _Rand-init_ in this figure and _R.Init_ in Table 5
correspond to sets of randomly initialized parameters that do not contribute
to $acc_{max}$.

Since the training of neural networks is strongly influenced by the
initialized parameters Summers and Dinneen (2021), sometimes the pre-trained
model outperforms the randomly initialized one by a large margin most likely
because the initialized position leads to bad local minima. Thus we need a
stronger baseline for locating true advantageous datasets of the pre-training
task. To this end, for each dataset, we additionally train with random
initialization four more times for each model structure and select the highest
accuracy among the five sets of test results as the new baseline, noted as
$acc_{max}$. Given a pre-training task $\mathcal{PT}$, we define its
advantageous dataset set $\mathcal{A}_{pt}$ of a determined model structure as
follows: for any dataset $\mathcal{D}$, if the pre-trained accuracy $acc_{pt}$
achieves a relative improvement of more than 15% compared to $acc_{max}$, then
$\mathcal{D}\in\mathcal{A}_{pt}$. We define
$\varphi_{ij}=|\mathcal{A}_{pt_{i}}\cap\mathcal{A}_{pt_{j}}|/max(|\mathcal{A}_{pt_{i}}|,|\mathcal{A}_{pt_{j}}|)$
as the intersection ratio between two sets of advantageous dataset
$\mathcal{A}_{pt_{i}}$ and $\mathcal{A}_{pt_{j}}$, derived from two different
pre-training tasks—$\mathcal{PT}_{i}$ and $\mathcal{PT}_{j}$. We also define
their intersection set as
$\Omega_{ij}=\mathcal{A}_{pt_{i}}\cap\mathcal{A}_{pt_{j}}$. We analogously
define the model-wise intersection ratio $\omega_{ij}$ by substituting the
pre-training task for the model structure, where $i$ and $j$ indicate
different model structures.

In Figure 2, given the same model structure, except for TsTransformer, there
is a high intersection ratio $\varphi_{ij}$ between each pair of the pre-
training tasks (random-cls and rand-init excluded), mostly above $0.4$ and in
some cases up to $0.8$ or more. More interestingly, according to statistics,
such overlap is not due to the close vicinity between the pre-trained
parameters. In fact, for some datasets within $\Omega_{ij}$, the $\ell_{2}$
distances between parameter $\theta_{i}$ and parameter $\theta_{j}$ are even
further apart than their average distance across all datasets. We also
calculate the common subset ratio $\phi$ among all pre-training tasks (random-
cls and rand-init excluded), which is formally defined as

$\phi=|\bigcap^{n}_{i=1}{A}_{pt_{i}}|/|\bigcup^{n}_{i=1}{A}_{pt_{i}}|,$ (1)

where $n$ is the number of pre-training tasks. On the UTS dataset, the common
subset ratio $\phi$ of LSTM, D.Conv, and TsTransformers are $0.1186$, $0.06$,
and $0.1568$ respectively, which is relatively low compared with pair-wise
intersection ratio $\varphi$. This result shows that the model is not the only
determinant and that different pre-training tasks can bring out their
advantages on different datasets.

| Ts2Vec | Ts-Tcc | Mvts | Srlt | R.Cls | R.Init
---|---|---|---|---|---|---
MTS | .233$\pm$.047 | .125$\pm$.177 |  .151$\pm$.117 | .125$\pm$.102 | .139$\pm$.104 | .042$\pm$.059
UTS | .252$\pm$.086 | .193$\pm$.099 | .301$\pm$.085 |  .272$\pm$.046 | .198$\pm$.070 | .136$\pm$.037
Table 5: We present overlap ratio $\omega_{ij}$ of advantageous datasets
between different model structures when the pre-training task is fixed. Here
we interpret $i$ and $j$ as two different model structures.

Analyzed from another perspective, Table 5 shows that the model-wise overlap
$\omega_{ij}$ is relatively small when we fix the pre-training task.
Nevertheless, we still find that Mvts and Ts2Vec methods significantly
outperform two baselines — R.Cls and R.Init, demonstrating a potential of
luring the model to perform better on certain datasets regardless of its
structure.

Observing Table 5 and Figure 2, if the question is how to make pre-training
benefit fine-tuning on a certain dataset, it is more important to consider the
fit of the model structure rather than designing a specific pre-training task,
because the model structure has a much greater impact than the pre-training
task for squeezing the potential of unsupervised pre-training on a given
dataset.

###  6.3 Correlation Factors

|  | Seq.L | Pre.S |  $\ell_{2}$-norm |  $\ell_{2}$-path | Sharp. | P.T | P.V | Dis.
---|---|---|---|---|---|---|---|---|---
Mvts
UTS | Lstm | 0.045 | -0.020 | 0.017 | 0.017 | -0.095 | 0.162 | 0.198† | 0.027
Transf. | -0.036 | 0.137 | -0.078 | -0.086 | 0.172 | -0.069 | -0.080 | -0.027
D.CNN | 0.0270 | -0.147 | 0.093 | 0.094 | 0.229† | 0.109 | 0.115 | -0.130
Srlt
UTS | Lstm | -0.037 | 0.058 | 0.040 | 0.037 | 0.174 | -0.060 | -0.043 | 0.045
Transf. | -0.002 | -0.035 | 0.049 | 0.047 | -0.133 | 0.001 | -0.106 | -0.102
D.CNN | -0.127 | -0.004 | -0.040 | -0.042 | 0.112 | -0.009 | -0.071 | -0.050
Ts-Tcc
UTS | Lstm | -0.070 | 0.080 | 0.182 | 0.218† | 0.085 | 0.002 | -0.052 | 0.009
Transf. | 0.0381 | -0.061 | -0.022 | -0.021 | -0.094 | 0.176 | -0.047 | 0.016
D.CNN | -0.037 | -0.067 | 0.049 | 0.051 | 0.131 | -0.109 | -0.066 | -0.046
Ts2Vec
UTS | Lstm | -0.211 | 0.053 | 0.039 | 0.041 | 0.169 | -0.024 | -0.024 | 0.040
Transf. | -0.010 | 0.078 | -0.002 | 0.001 | 0.039 | -0.005 | 0.033 | -0.066
D.CNN | 0.057 | -0.225 | -0.104 | -0.109 | 0.046 | 0.087 | 0.102 | -0.195†
Table 6: Spearman correlation coefficient between different factors and
accuracies (early-stopping) of the test set. The results in this table are
derived from the UTS datasets only, But the situation is similar on the MTS
datasets. Seq.L is sequence length. Pre.S is the size of the pre-training set.
Sharp. means sharpness. P.V represents the convergence state of the model on
the pre-training set and is equal to the lowest training loss divided by the
training loss of the first epoch. P.V is defined similarly on the validation
set. Dis. is the $\ell_{2}$ distance between the pre-trained parameters and
its initial point.

From Table 6, we can see that there is not a single factor that is
significantly associated with the effectiveness of pre-training. There are
some sporadic significant associations, but they are only observed on a small
number of model structures, which do not provide much of a reference. This
result is rather disappointing, and we have yet to find an indicator to guide
us on whether to trust the pre-trained parameters or to decide whether to pre-
train on certain datasets.

##  7 Conclusion and Future Work

This paper focuses on the study of whether unsupervised pre-training is
beneficial for fine-tuning on the Time Series Classification task, making an
empirical contribution to the study of when and how unsupervised pre-training
helps fine-tuning. We conclude that pre-training does not significantly
enhance the generalization performance but can improve the convergence speed
when the model fits the data well. Also, it can improve the optimization
process for simple models with a small number of parameters. Increasing the
amount of pre-training data does not benefit generalization, but amplifies the
existing advantages of pre-training, such as fast convergence. When dealing
with a new time series dataset and aiming to enhance the model using pre-
training, it’s more important to focus on creating an appropriate model
architecture than on creating the pre-training task itself.

The experimental procedures and conclusions presented in this paper are
primarily grounded in low-resource settings, utilizing medium-sized models and
limited pre-training data. This approach is particularly relevant to the
current discourse surrounding artificial intelligence, as the development of
AI with lower resources has become a pressing concern for various fields,
including natural language processing Rotman and Reichart (2019), healthcare
Wahl et al. (2018); Pokaprakarn et al. (2022), edge computing Merenda et al.
(2020), and green AI Schwartz et al. (2020). To extend the scope of our work,
we encourage future studies to explore larger models and more data in this
domain, as well as to do new research on catastrophic forgetting Mehta et al.
(2021), intrinsic dimension Aghajanyan et al. (2021), and other aspects not
covered in this paper.

## References

  * Abnar et al. (2021) Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi.  Exploring the limits of large scale pre-training.  In _International Conference on Learning Representations_ , 2021.
  * Aghajanyan et al. (2021) Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7319–7328, 2021.
  * Bengio et al. (2006) Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle.  Greedy layer-wise training of deep networks.  _Advances in neural information processing systems_ , 19, 2006.
  * Dau et al. (2019) Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh.  The ucr time series archive.  _IEEE/CAA Journal of Automatica Sinica_ , 6(6):1293–1305, 2019.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, 2019.
  * Eldele et al. (2021) Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong, Xiaoli Li Kwoh, and Cuntai Guan.  Time-series representation learning via temporal and contextual contrasting.  2021\.
  * Erhan et al. (2010) Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent.  Why does unsupervised pre-training help deep learning?  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 201–208. JMLR Workshop and Conference Proceedings, 2010.
  * Franceschi et al. (2019) Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi.  Unsupervised scalable representation learning for multivariate time series.  _Advances in neural information processing systems_ , 32, 2019.
  * Goodfellow et al. (2016) Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep Learning_.  MIT Press, Cambridge, MA, USA, 2016.  http://www.deeplearningbook.org.
  * Gouk et al. (2020) Henry Gouk, Timothy Hospedales, et al.  Distance-based regularisation of deep networks for fine-tuning.  In _International Conference on Learning Representations_ , 2020.
  * Hao et al. (2019) Yaru Hao, Li Dong, Furu Wei, and Ke Xu.  Visualizing and understanding the effectiveness of bert.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4143–4152, 2019.
  * He et al. (2019) Kaiming He, Ross Girshick, and Piotr Dollár.  Rethinking imagenet pre-training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4918–4927, 2019.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Hendrycks et al. (2019) Dan Hendrycks, Kimin Lee, and Mantas Mazeika.  Using pre-training can improve model robustness and uncertainty.  In _International Conference on Machine Learning_ , pages 2712–2721. PMLR, 2019.
  * Jiang et al. (2019) Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio.  Fantastic generalization measures and where to find them.  In _International Conference on Learning Representations_ , 2019.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Liu et al. (2021) Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma.  Self-supervised learning is more robust to dataset imbalance.  In _NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications_ , 2021.
  * Maennel et al. (2020) Hartmut Maennel, Ibrahim M Alabdulmohsin, Ilya O Tolstikhin, Robert Baldock, Olivier Bousquet, Sylvain Gelly, and Daniel Keysers.  What do neural networks learn when trained with random labels?  _Advances in Neural Information Processing Systems_ , 33:19693–19704, 2020.
  * Mao (2020) Huanru Henry Mao.  A survey on self-supervised pre-training for sequential transfer learning in neural networks.  _arXiv preprint arXiv:2007.00800_ , 2020.
  * Mehta et al. (2021) Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell.  An empirical investigation of the role of pre-training in lifelong learning.  _arXiv preprint arXiv:2112.09153_ , 2021.
  * Merenda et al. (2020) Massimo Merenda, Carlo Porcaro, and Demetrio Iero.  Edge machine learning for ai-enabled iot devices: A review.  _Sensors_ , 20(9):2533, 2020.
  * Middlehurst et al. (2021) Matthew Middlehurst, James Large, Michael Flynn, Jason Lines, Aaron Bostrom, and Anthony Bagnall.  Hive-cote 2.0: a new meta ensemble for time series classification.  _Machine Learning_ , 110(11):3211–3243, 2021.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ , 2013.
  * Neyshabur et al. (2017) Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro.  Exploring generalization in deep learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Neyshabur et al. (2020) Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang.  What is being transferred in transfer learning?  _Advances in neural information processing systems_ , 33:512–523, 2020.
  * Paine et al. (2014) Tom Le Paine, Pooya Khorrami, Wei Han, and Thomas S Huang.  An analysis of unsupervised pre-training in light of recent advances.  _arXiv preprint arXiv:1412.6597_ , 2014.
  * Pokaprakarn et al. (2022) Teeranan Pokaprakarn, Juan C Prieto, Joan T Price, Margaret P Kasaro, Ntazana Sindano, Hina R Shah, Marc Peterson, Mutinta M Akapelwa, Filson M Kapilya, Yuri V Sebastião, et al.  Ai estimation of gestational age from blind ultrasound sweeps in low-resource settings.  _NEJM Evidence_ , 1(5):EVIDoa2100058, 2022.
  * Rey and Neuhäuser (2011) Denise Rey and Markus Neuhäuser.  Wilcoxon-signed-rank test.  In _International encyclopedia of statistical science_ , pages 1658–1659. Springer, 2011.
  * Rotman and Reichart (2019) Guy Rotman and Roi Reichart.  Deep contextualized self-training for low resource dependency parsing.  _Transactions of the Association for Computational Linguistics_ , 7:695–713, 2019.
  * Sagheer and Kotb (2019) Alaa Sagheer and Mostafa Kotb.  Unsupervised pre-training of a deep lstm-based stacked autoencoder for multivariate time series forecasting problems.  _Scientific reports_ , 9(1):1–16, 2019.
  * Schober et al. (2018) Patrick Schober, Christa Boer, and Lothar A Schwarte.  Correlation coefficients: appropriate use and interpretation.  _Anesthesia & Analgesia_, 126(5):1763–1768, 2018.
  * Schwartz et al. (2020) Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni.  Green ai.  _Communications of the ACM_ , 63(12):54–63, 2020.
  * Summers and Dinneen (2021) Cecilia Summers and Michael J Dinneen.  Nondeterminism and instability in neural network optimization.  In _International Conference on Machine Learning_ , pages 9913–9922. PMLR, 2021.
  * Tian and Zhang (2022) Yingjie Tian and Yuqi Zhang.  A comprehensive survey on regularization strategies in machine learning.  _Information Fusion_ , 80:146–166, 2022.
  * Tonekaboni et al. (2020) Sana Tonekaboni, Danny Eytan, and Anna Goldenberg.  Unsupervised representation learning for time series with temporal neighborhood coding.  In _International Conference on Learning Representations_ , 2020.
  * Tuli et al. (2021) Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L Griffiths.  Are convolutional neural networks or transformers more like human vision?  _arXiv preprint arXiv:2105.07197_ , 2021.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wahl et al. (2018) Brian Wahl, Aline Cossy-Gantner, Stefan Germann, and Nina R Schwalbe.  Artificial intelligence (ai) and global health: how can ai contribute to health in resource-poor settings?  _BMJ global health_ , 3(4):e000798, 2018.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32, 2019.
  * Woo et al. (2021) Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi.  Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting.  In _International Conference on Learning Representations_ , 2021.
  * Yue et al. (2022) Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu.  Ts2vec: Towards universal representation of time series.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 8980–8987, 2022.
  * Zerveas et al. (2021) George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff.  A transformer-based framework for multivariate time series representation learning.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 2114–2124, 2021.
  * Zhang et al. (2022) Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik.  Self-supervised contrastive pre-training for time series via time-frequency consistency.  _arXiv preprint arXiv:2206.08496_ , 2022.