# Label-efficient Time Series Representation Learning: A Review

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh and
Xiaoli Li This work was supported by the Agency of Science Technology and
Research under its AME Programmatic (Grant No. A20H6b0151) and its Career
Development Award (Grant No. C210112046).Emadeldeen Eldele is with the School
of Computer Science and Engineering, Nanyang Technological University,
Singapore and the Centre for Frontier AI Research, Agency for Science,
Technology and Research, Singapore (E-mail: emad0002@ntu.edu.sg).Mohamed Ragab
is with the Institute for Infocomm Research, Agency for Science, Technology
and Research, Singapore, Centre for Frontier AI Research, Agency for Science,
Technology and Research, Singapore (E-mail: mohamedr002@e.ntu.edu.sg).Zhenghua
Chen is with the Institute for Infocomm Research, Agency for Science,
Technology and Research, Singapore and the Centre for Frontier AI Research,
Agency for Science, Technology and Research, Singapore (E-mail:
chen0832@e.ntu.edu.sg).Min Wu is with the Institute for Infocomm Research,
Agency for Science, Technology and Research, Singapore (E-mail:
wumin@i2r.a-star.edu.sg).Chee-Keong Kwoh is with the School of Computer
Science and Engineering, Nanyang Technological University, Singapore (E-mail:
asckkwoh@ntu.edu.sg).Xiaoli Li is with the Institute for Infocomm Research,
Agency for Science, Technology and Research, Singapore, Centre for Frontier AI
Research, Agency for Science, Technology and Research, Singapore, and also
with the School of Computer Science and Engineering at Nanyang Technological
University, Singapore (E-mail: xlli@i2r.a-star.edu.sg).Min Wu is the
corresponding author.

###### Abstract

The scarcity of labeled data is one of the main challenges of applying deep
learning models on time series data in the real world. Therefore, several
approaches, e.g., transfer learning, self-supervised learning, and semi-
supervised learning, have been recently developed to promote the learning
capability of deep learning models from the limited time series labels. In
this survey, for the first time, we provide a novel taxonomy to categorize
existing approaches that address the scarcity of labeled data problem in time
series data based on their dependency on external data sources. Moreover, we
present a review of the recent advances in each approach and conclude the
limitations of the current works and provide future directions that could
yield better progress in the field.

###### Index Terms:

Label-efficient learning, Self-supervised learning, Semi-supervised learning,
domain adaptation.

Deep learning models often require large amount of labeled data for training,
which is usually impractical in real-world scenarios. Therefore, in this work,
we provide a high-level survey to the available solutions to the label
scarcity problem in different scenarios. By identifying three real-world
scenarios, i.e., having few labeled samples, or only unlabeled time series
data, or a combination of both, the paper offers a roadmap that links each
scenario to appropriate solutions within the in-domain and cross-domain
categories. This survey not only provides a valuable resource for researchers
and practitioners working in the field but also offers a rough guideline for
selecting the most suitable solution for specific labeling problems. We shed
light on the challenges, trends, and future prospects, which contributes to
the advancement and applicability of deep learning models in real-world
settings.

##  I Introduction

Deep learning has shown significant success in diverse time series
applications. Unfortunately, the applicability of deep learning models in the
real world is tied to the availability of large and labeled datasets, which
are usually lacking for time series applications. For example, we can collect
a massive amount of sensory data from manufacturing sensors, while only few of
them are being annotated by domain experts. Also, in some healthcare
applications, it can be difficult to collect or access enough data and we end
up with only few samples. In either situation, training efficient deep
learning models with these data and annotation limitations can be very
challenging.

Over recent years, a significant body of research has emerged to address the
label scarcity challenges, with methodologies spanning various conceptual
paradigms. These methodologies can be categorized into: in-domain methods and
cross-domain methods. In-domain methods focus on deriving representations from
the available time series data, without the reliance on external data sources.
Strategies within this category include data augmentation, which increases the
number of labeled samples. Additionally, self-supervised learning allows
learning from fully unlabeled data, while semi-supervised learning provides a
harmonized approach to learning from both labeled and unlabeled instances
within the same domain. Conversely, cross-domain methods learn representations
from an external, labeled source domain, subsequently transferring this
acquired knowledge to the target domain of interest. The common strategy is to
use transfer learning approaches, either by direct fine-tuning the pretrained
model on the target domain or by adapting the source and target domains to
have similar distributions and class boundaries.

In this paper, we explore three specific real-world scenarios: having only a
few labeled samples, having solely unlabeled time series data, and a
combination of both labeled and unlabeled data. We systematically associate
each of these scenarios with its corresponding solution, classified under
either in-domain or cross-domain categories, as illustrated in Fig. 1.
Throughout surveying recent advances within these categories, we aim to
provide a coherent roadmap that guides the effective application of deep
learning models, even in the face of these data limitations. It’s important to
note that our intention is not to provide an exhaustive discussion of each
individual method within every category. Instead, we strive to offer a
panoramic view, presenting a high-level understanding of the challenges and
available solutions. By doing so, we enable readers to grasp the overarching
concepts and approaches, facilitating further exploration and in-depth
investigations according to their specific research interests.

Figure 1: Graphical overview of the different scenarios of data availability
and the suitable solution. Note that the directions given by arrows are not
the only solutions for each respective scenario, but can be the best way to
achieve the best performance.

In overall, the contributions in this paper can be summarized as follows:

  * •

We provide a comprehensive survey of label-efficient techniques for time
series representation learning across a spectrum of scenarios, including few
labeled samples, unlabeled data, and combinations of both, offering readers a
holistic understanding of the challenges posed by limited labeled data.

  * •

We introduce a systematic categorization of solutions into in-domain and
cross-domain strategies, elucidating the diverse array of approaches available
to address the label scarcity issue in time series analysis.

  * •

We highlight the challenges, the new trends, and the future prospects in this
area of research, with the goal of fostering new solutions.

##  II Related Surveys

In the realm of label-efficient time series representation learning, several
surveys have delved into specific solutions tailored to address the challenges
of limited labeled data. While these surveys have significantly contributed to
the field, they tend to focus on specific techniques or solutions, limiting
their scope to a singular aspect of label-efficient time series representation
learning. Our survey, on the other hand, synthesizes these individual
perspectives into a cohesive narrative that encompasses a broader range of
strategies.

We first discuss surveys related to in-domain solutions. Wen et al. [1]
comprehensively summarize time series data augmentation techniques across key
tasks such as forecasting, anomaly detection, and classification. The authors
systematically review different augmentations from simple time domain
transformations to more intricate methods in transformed frequency and time-
frequency domains. Also, Deldari et al. [2] explore the self-supervised
representation learning to learn from unlabeled time series data across
various modalities, with an emphasis on temporal data and cross-modal learning
models. The authors present a classification scheme to benchmark, and
categorize various studies, as a guide for model selection.

Furthermore, Zhang et al. [3] present an exhaustive review of state-of-the-art
self-supervised learning techniques specifically tailored for time series
data. They categorize these methods into three distinct paradigms: generative-
based, contrastive-based, and adversarial-based., and they delve into various
frameworks and techniques within each category. Similarly, Ma et al. [4] offer
a thorough examination of time series pre-trained models, where they discuss
time series modeling tasks and the associated deep learning models. They also
discuss supervised, unsupervised, and self-supervised pre-training techniques.

Regarding the cross-domain related surveys, inspired by the success of
transfer learning in computer vision, Ismail Fawaz et al. [5] explored the
potential of transfer learning for time series classification, and conducted
exhaustive experiments to draw conclusions. Their findings underscore the
presence of transferable low-level features in time series, an their
transferability can be measured by the Dynamic Time Warping technique. Last,
Shi et al. [6] offer a thorough discussion about unsupervised domain
adaptation in the context of time series sensor data. They categorize these
studies by industry, detailing application backgrounds, frequently used
sensors, data discrepancy sources, and prominent research datasets.

Concluding our discussion on the related surveys, it’s imperative to highlight
that, to the best of our knowledge, our work stands as the pioneering effort
that tackles the challenge of label-efficient representation learning for time
series data from a multi-faceted viewpoint. We pride ourselves in offering a
panoramic understanding, shedding light on diverse scenarios and articulating
their respective solutions in a comprehensive manner.

##  III Deep Learning Architectures

Selecting an appropriate architecture is foundational to the construction of a
label-efficient deep learning model. In this section, we discuss various deep
learning architectures tailored for time series data, concisely outlining
their merits and limitations.

###  III-A Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are one of the common choices in the time
series classification tasks. The widely used CNN architectures are the 1D-CNN
[7], the 1D-ResNet [8], and the temporal convolutional network (TCN) [9]. In
addition, other CNN designs have been proposed and then adopted in different
works. For example, Ismail Fawaz et al. [10] propose InceptionTime, which
ensembles five randomly initialized Inception modules. Also, Tang et al. [11]
design Omni-Scale CNN blocks that automatically decide the best size and
number to the kernel size used by CNNs.

CNNs show superiority in time series classification tasks, due to their high
ability to detect patterns and trends in data. In addition, they are less
prone to overfitting as they use a reduced number of parameters. However, they
are less efficient in detecting the temporal relations in data, which makes
them less favorable in time series forecasting tasks.

###  III-B Recurrent Neural Networks

Recurrent Neural Networks (RNN) are specially designed to learn the temporal
features in time series data. They include a memory component that allows them
to take into account the past timesteps when processing the current one. Long
Short-Term Memory (LSTM), one type of RNNs, has been widely adopted in time
series forecasting tasks [12]. Nevertheless, they can also be involved in
classification, either side by side with a CNN model [13], or following CNN
layers [14].

RNNs improve the learning capability of the model on time series data, as it
learns from historical data. However, they are ineffectual with relatively
long sequences, besides being computationally expensive due to sequential
processing.

###  III-C Attention Mechanism

The attention mechanism was proposed to allow the model to focus on specific
parts of the input. It was used to enhance LSTM performance in many
applications by assigning attention scores for LSTM hidden states to determine
the importance of each state in the final prediction [15]. For example,
TCACNet proposed a dual attention mechanism to lean temporal relations and
information distribution over channels from electroencephalogram (EEG) data
[16]. Self-attention was also leveraged in many works to replace the LSTM
network due to its fast parallel processing [17, 18]. Recently, the
Transformer model, which is based on the self-attention mechanism, is being
increasingly used as a backbone architecture for time series [19].

Attention mechanisms can be more effective with long sequences, and they can
improve the interpretability of the model. However, they can be more
computationally expensive due to their larger number of parameters, which also
makes them prone to overfitting when it comes to data scarcity problem.

##  IV In-domain Representation Learning

In this section, we discuss three approaches to learn representations from the
data in hand. First, we assume possessing only few labeled samples which can
be addressed with data augmentation. Second, we assume the existence of a
large cohort of unlabeled data and discuss how self-supervised learning can be
utilized. Third, we may have some labeled samples besides the massive
unlabeled data, where we can apply the semi-supervised learning techniques.

###  IV-A Data Augmentation

Data augmentation is usually adopted as a way to artificially scale up the
training dataset, by applying several transformations to the input data. This
can be particularly useful when it is difficult or expensive to collect more
real-world data. By adding more samples, data augmentation aims to provide the
model with diverse instances that are representative of the underlying data
distribution. These transformed samples not only contribute to better
generalization, but also alleviate the need for an excessive volume of labeled
data. Aside from increasing the sample size, the variety of augmentations
applied to the time series data can generate new sequences that are realistic
variations of the original data, mimicking natural fluctuations and nuances.
Thus, the model can become robust against different orientations, scales,
noise, and other variations that might be present in real-world data. In
addition, some works utilize data augmentation to upsample the minority
classes to overcome the class imbalance issue in some time series applications
[20]. Fig. 2 illustrates the principle of data augmentation applied to a
single time series example. The different augmentations shown in the figure
elucidate how one original sequence can be transformed into multiple variants,
each serving as a distinct training example.

Figure 2: Increasing the number of samples through augmentations. For each
input sample, we apply three (or more) augmentations to generate new samples

There are several data augmentation techniques proposed for time series data.
[1] provide a comprehensive set of time series data augmentations that can be
applied in the time domain and the frequency domain. Similarly, Iwana and
Uchida [21] conduct a survey of data augmentation for empirically evaluating
the classification tasks, where they explore data transformations and the
different ways to mix time series patterns. In general, some augmentations are
more widely employed, e.g., scaling, shifting, rotation, masking, and noise
injection.

Besides applying transformations, synthetic data generation is yet another
technique to artificially generate data using simulation. This technique can
be particularly useful in preserving data privacy, as it might not be viable
to access real-world data in some cases. Therefore, generating synthetic data
provides a way to use the data without compromising privacy. There are several
different methods to generate synthetic data, but we mainly explore the
Generative Adversarial Networks (GANs), being the most commonly used approach.
For example, ActivityGAN generates time series sensory data for human activity
recognition with 1D-CNN networks by training a GAN [22]. They find that
performance increases by including more synthetic data to real data.
Similarly, DCGAN generates artificial EEG signals to improve the performance
of the Brain-Computer Interface (BCI) tasks [23]. Their model witnessed a
significant improvement by including the generated data.

###  IV-B Self-supervised Learning

Self-supervised learning involves training a model to learn representations
from data using only the unlabeled samples. Instead of relying on human-
provided labels, we define a new task to be solved by the model or generate
pseudo labels based on the input data. The general process of self-supervised
learning is shown in Fig. 3. The training process is usually performed on two
phases. In the first phase, i.e., the pre-training phase, we feed the encoder
model with the unlabeled data and train it with the self-supervised task.
Next, we use the few labeled labeled samples in the second phase, i.e., the
fine-tuning, to tailor the model to the downstream task, and improve its
performance on these few labels, compared to fully supervised training with
the few labels.

Self-supervised learning methods can be categorized into auxiliary- or
contrastive-based methods. In both categories, the unlabeled data is used to
learn an additional task that is not the main focus of the model but can
provide additional supervision for learning useful representations of the
data.

Figure 3: Pipeline of self-supervised learning process. The pretraining phase
receives the input time series, pass it to the encoder to extract the
features, and perform the self-supervised learning task. Next, the pretrained
encoder is fine-tuned with the available few labeled data.

####  IV-B1 Auxiliary-based Methods

These methods propose an auxiliary task that can be related to the downstream
task or the characteristics of the data, which helps the pretrained model to
perform well on the downstream task. For example, Banville et al. [24]
introduce two auxiliary tasks, i.e., relative positioning and temporal
shuffling, to learn the underlying temporal characteristics in the EEG
signals. In the first task, a window of the signal is used as an anchor to
measure the relative position of other windows, while in the second task, a
window of the signal is shuffled to teach the model to fix the temporal order
of the signal. Similarly, Saeed et al. [25] propose some auxiliary tasks,
e.g., feature prediction from a masked window, temporal shift prediction,
modality denoising, and blend detection, to learn representations from human
activity recognition data. In addition, TST [19] and STraTS [26] propose to
predict the masked part in the signal as an auxiliary task to pre-train a
Transformer model.

Another example of auxiliary tasks is to utilize data augmentations to provide
several transformations to the time series data. Each transformation is
assigned a pseudo label, and the model is trained to classify them with a
standard cross-entropy loss. Several works followed this approach in different
applications, e.g., electrocardiogram (ECG) emotion recognition [27] and human
activity recognition [28].

Even though auxiliary tasks have been shown to be effective, they restrict the
generality of the learned representations [29]. Therefore, most time series
self-supervised learning works have recently adopted contrastive learning
instead.

####  IV-B2 Contrastive-based Methods

Contrastive-based self-supervised learning methods aim to form positive and
negative pairs, where the contrastive loss maximizes the similarity among
positive pairs while minimizing the similarity among negative pairs. Previous
works propose different strategies to choose the positive and negative pairs
to improve the quality of the learned representations. Additionally, most
works attempt to learn the temporal relations within time series in the
pretraining phase.

In this section, we provide a new time series-specific categorization for
contrastive-based self-supervised learning methods. Particularly, we
categorize them into intra-sample contrastive methods, inter-sample
contrastive methods, and frequency domain contrastive methods.

##### Inter-sample Contrasting (Contextual Contrasting)

In inter-sample contrastive learning, the model uses the input signal as a
whole without splitting or windowing. One of the well-performing visual
contrastive methods is SimCLR [30], which relies on data augmentations to
define positive and negative pairs. In specific, the positive pairs are the
augmented views of an anchor sample, while the negative pair are all the
augmented views of other samples. Many works followed SimCLR for time series
data, such as [31, 32] that apply SimCLR-like methods for sleep stage
classification and fault diagnosis tasks respectively.

Mixup Contrastive Learning (MCL) proposes to replace the traditional
contrastive loss with a mixup-based contrastive loss [33]. In specific, MCL
generates new mixed samples in each mini-batch using the mixup operation, and
the proposed loss predicts the mixup ratio among each pair of samples. Also,
Chang et al. [34] propose another loss that searches for positive and negative
pairs for each anchor sample in both the input and embedding spaces. The
contrastive loss is then applied to the selected triplet to learn a causal
temporal CNN.

##### Intra-sample Contrasting (Temporal Contrasting)

Intra-sample contrastive methods aim to learn the temporal relations in time
series data in the pretraining phase. To do so, they process the signal at the
timestep level, either in the embedding space or in the input space. One of
the earliest works is Contrastive Predictive Coding (CPC), which learns the
temporal relations by splitting the embedding space of the signal into a past
part and a future part [29]. Then, an autoregressive model is trained to
produce a context vector from the past part and use it to predict the future
part using contrastive learning. Similarly, SleepDPC generates multiple
context vectors and uses each one to predict the next timestep in the future
part for the sleep stage classification task [35]. Also, TS-TCC applies strong
and weak augmentations for each input signal, generates the embeddings for
each, and uses the past part of each view to predict the future part of the
other view in a tough cross-view prediction task [7]. Last, ContrNP replaces
the autoregressive model used by CPC with a neural process [36].

Other methods learn the temporal relations among timesteps in the input space.
For example, Temporal Neighborhood Coding (TNC) assumes that the distribution
of one window of timesteps is similar to its neighboring windows and can be
more distinguishable than the non-neighboring windows [37]. Therefore, TNC
forms positive pairs among neighboring windows and negative pairs among non-
neighboring windows. Another example is TS2Vec, which applies data
augmentation on the input signal and considers the current timestep from the
augmented views as positives, while the different timesteps from the same time
series are considered as negatives [38].

##### Frequency Domain Contrastive Methods

A recent trend in time series contrastive learning is to utilize the frequency
domain characteristics in representation learning, being one main
distinguishing properties of time series data. For instance, Bilinear
Temporal-Spectral Fusion (BTSF) applies an iterative bilinear fusion between
feature embeddings of both time and frequency representations of time series
[39]. Specifically, BTSF generates different views of time series and applies
iterative aggregation modules to perform cyclic refinement to the temporal and
spectral features. Similarly, both STFNets [40] and TF-C [41] apply time
domain and frequency domain augmentations, and push the time domain and
frequency domain representations of the same sample closer to each other while
pushing them apart from representations of other signals.

###  IV-C Semi-supervised Learning

Semi-supervised learning is a type of learning that involves training a model
using both labeled and unlabeled data. It is used in situations where it is
difficult or expensive to obtain a large amount of labeled data, and only a
few labeled samples are available with a large amount of unlabeled data. One
of the challenges in semi-supervised learning is how to achieve the best
balance of learning from both labeled and unlabeled samples. Fig. 4 provides a
visual illustration to the semi-supervised training process. In specific, we
feed the model with both the labeled and unlabeled samples, apply the
traditional cross-entropy classification loss to the labeled data, and train
the model on the unlabeled data with an unsupervised learning technique. In
this survey, we categorize semi-supervised learning approaches into self-
training-based methods and generic regularization-based methods.

####  IV-C1 Self-training Methods

Self-training involves assigning labels to the unlabeled data to train the
model. It can be implemented either by pseudo labeling or self-supervised
pretraining.

##### Pseudo Labeling

Pseudo labeling utilizes the few labeled data to train the model and create
pseudo labels for the unlabeled data that are likely to be correct. The common
way to create pseudo labels is to apply the softmax function on the model
predictions. For example, Wang et al. [42] generate pseudo labels from the
model predictions to the unlabeled data and use least-square regression to
learn both shapelets and classification boundaries. Another way is to use
clustering techniques as in SUCCESS, which proposes a hierarchical clustering
scheme to assign pseudo labels to samples according to their respective
clusters [43]. It calculates the distance to centroids on the basis of the
dynamic time warping (DTW) distance metric.

A different way of utilizing the pseudo labels is proposed in FixMatch, which
applies two augmentations to the image data, where one augmentation is weak
and the other is strong [44]. Then, if the prediction of the weak-augmented
view is beyond a threshold value, it uses its pseudo label for the strong
augmented view. By showing success with image data, several methods applied
FixMatch-like approaches on time series data, e.g., PARSE [45] for EEG-based
emotion recognition. Also, SelfMatch uses Gaussian noised signals as a weak
augmentation and timecut as a strong augmentation [46]. It also applies self-
distillation to guide the lower-level blocks of the feature extractor with the
knowledge obtained in the output layer.

Figure 4: Illustration of semi-supervised learning process.Usually, the
classification cross-entropy loss is applied to the labeled portion of data,
while an unsupervised learning technique is applied to learn useful and
supportive representations from the unlabeled data.

##### Self-supervised Learning

Instead of pseudo labeling the unlabeled portion of the data, we can leverage
self-supervised learning to learn their representations, and then fine-tune
the pretrained model with the available labeled portion. For example, Jawed et
al. [47] propose a forecasting task on the unlabeled data to provide a
substitute supervision signal. This task has been jointly optimized along with
the supervised task on the labeled part of the data. Also, CA-TCC extends TS-
TCC in semi-supervised settings by fine-tuning the pretrained TS-TCC model
with the labeled samples [48]. This fine-tuned model is then used to generate
pseudo labels of the unlabeled sample set, which are then deployed to train
the model with a supervised contrastive loss. In addition, iTimes learns from
unlabeled samples by applying several transformations to the time series via
irregular time sampling techniques and learns the model to predict the
transformation type to learn the temporal structure [49].

Contrastive learning has also been widely used for self-supervised pretraining
in semi-supervised learning. For example, SemiTime uses contrastive learning
by splitting the signal into two parts, i.e., past and future parts [50]. It
forms positive pairs among the past and the future parts of the same signal,
and negative pairs between the past of the signal and the future of other
signals. Moreover, MtCLSS applies contrastive learning between the feature
representations of the labeled and unlabeled EEG samples for the sleep stage
classification problem [51]. Last, Semi-DeepConvNet uses a SimCLR-like
contrastive learning technique to learn from EEG-based motor imagery data
[52]. It also eliminates the bias to the data from the different subjects by
using an adversarial training scheme.

####  IV-C2 Generic regularization

Apart from self-training, another stream of methods adopt different techniques
to learn from data, e.g., ensembling and generative models.

##### Ensemble-based Methods

These methods train two or more models separately on the same or on different
views of the data. For example, TEBLSTM proposes temporal ensembling for human
activity recognition data based on an LSTM model [53]. In addition to the
supervised cross-entropy loss on the labeled samples, they train the unlabeled
data by comparing their predictions with the ensembled predictions of the past
epochs. Moreover, [54] cluster labeled samples using k-means to train multi-
modal classifiers. If the ensemble of these classifiers reaches an agreement
on predicting an unlabeled sample, it is assigned a label and added to the
labeled set. The newly labeled samples are then added to the training set and
the models are retrained on the expanded dataset.

##### Reconstruction-based Methods

These methods train the model to reconstruct input data from a latent
representation assuming that the model will learn useful representations from
this process. For example, HCAE trains an autoencoder on industrial motor
bearing datasets to reconstruct the spectrogram of the signals [55]. They
train a supervised loss on the labeled samples besides the reconstruction
loss. Similarly, AnoVAE trains a variational autoencoder on normal EEG signals
and marks the anomalous signals by the trained model as seizures [56]. Also,
SMATE employs a reconstruction loss to learn from data, uses the labeled
samples to create centroids, then classifies the unlabeled data according to
their distance to the centroids [57].

Moreover, other methods use GANs to reconstruct and learn from data. For
example, REG-GAN is a semi-supervised regression method that uses GAN to learn
representations by regenerating the signals of the whole data and applying
supervised regression loss on the labeled part [58]. SRGAN proposes a GAN
model to learn useful information from suspension histories to improve the
remaining useful life prediction performance of industrial machines [59].

##  V Cross-domain Learning

Cross-domain learning refers to train a model on a labeled domain (i.e., the
source domain), and transfer the knowledge to a related domain (i.e., the
target domain), which is the domain of interest. This can be a challenging
problem, as we may need to train the model to adapt the different
distributions across the two domains.

###  V-A Transfer Learning

Transfer learning is used when only few labeled data are available in the
target domain, while more label information is provided in the source domain.
Thus, the model can be pretrained on the source domain data, and then directly
fine-tuned with the few labeled samples in the target domain, as illustrated
in Fig. 5. Yosinski et al. [60] perform one of the first works to study the
impact of transfer learning, where they found that the features produced by
the first layer in deep neural networks tend to be general and can be applied
to many datasets and tasks. The idea behind transfer learning is
straightforward, yet found to be effective, and has been directly applied in
different time series applications. For example, Xu et al. [61] demonstrate
the positive impact of transfer learning to improve the performance on EEG-
based stroke rehabilitation for an effective motor imagery system. Also, Zhong
et al. [62] conclude that transfer learning helps reducing the model
parameters while increasing its accuracy in the fault diagnosis problem.

Figure 5: The phases of transfer learning process. In the first phase, we
train the model on the labeled source domain data. In the second, we fine-tune
the pretrained model on the available few target domain sample. Last, we test
the model on the unlabeled tareget domain data.

Transfer learning can be particularly impactful in time series applications
that include multiple users/subjects, such as healthcare or human activity
recognition. In such cases, we can improve the model performance on new unseen
users with transfer learning instead of training the model from scratch [63].
For example, Li et al. [64] propose to train a single-channel neural network
using single-channel data from the source domain on human activity recognition
datasets. Then, they replicate these single-channel nets for each target
domain channel, showing improved performance.

It is worth noting that the positive impact of transfer learning is not
guaranteed in all cases. It has been found that the effect of transfer
learning depends on the relationship between the source and target
distributions. To demonstrate this phenomenon, Ismail Fawaz et al. [5]
evaluate the impact of transfer learning on 85 UCR datasets by training on one
dataset and testing on all the others. They find that transfer learning is not
effective in all the cross-dataset scenarios. Therefore, they propose DTW as a
measure to predict the effectiveness of transfer learning given a source and
target pair. Following up, Ye and Dai [65] also use DTW and Jensen-Shannon
divergence to select the suitable source domain depending on the inner
distribution dissimilarity to the target domain. Similarly, Meiseles and
Rokach [66] rank the multiple source domains according to the Mean Silhouette
Coefficient, which measures how well each source domain can cluster the target
domain in the embedding space.

###  V-B Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) is a subset of transfer learning, that is
required when no target domain labels are available and when a distribution
shift exists between source and target domains. UDA adapts a model trained on
a labeled source domain to a shifted unlabeled, but related, target domain, to
minimize the domain shift. In general, methods proposed for time series UDA
can be categorized into distance-based and adversarial-based methods. The
training process of UDA is illustrated in Fig. 6.

Figure 6: The unsupervised domain adaptation technique. We feed the model with
both the labeled source domain data, and the unlabeled target domain data. The
model is trained to reduce the domain shift between the two domain by either
reducing a distance measure, or applying adversarial training strategy.

####  V-B1 Distance-based Methods

The main idea behind distance-based UDA is to minimize a distance measure
between the source and target distributions so that the model can learn to
generalize to the target domain. There are several distance metrics, e.g.,
Maximum Mean Discrepancy (MMD) [67] and Correlation Alignment (CORAL) [68],
that have been used in the visual UDA works. However, since they are found to
be less effective in mitigating the large distribution shift, some works
modified these measures to achieve better performance. For example, AdvSKM
refines the MMD measure to be suitable for time series, by updating the inner
kernel of MMD to become a hybrid spectral kernel network [69]. In addition,
Ott et al. [70] pretrain the model on the unlabeled target domain data, then
train an optimal transport jointly with CORAL to transform the source to the
target domain. They also measure the class-wise similarity between the source
and target embeddings to select the best transformation and use it to classify
transformed embedding with the target domain model. Also, SASA considers the
variant information in time series and learns the sparse associative structure
by studying the causal structure of time series variables, with building on
top of an LSTM architecture [71].

The distance-based UDA methods are usually simpler in implementation and
faster in training. However, they may not be as effective as adversarial-based
methods in reducing the large domain shift between source and target domains.
In addition, they may not efficiently capture the underlying structure of the
data as well as adversarial methods.

####  V-B2 Adversarial-based Methods

Adversarial-based UDA bridges the gap between source and target domains with
generative adversarial training. In specific, the source and target data are
passed through a discriminator network, which is trained to make the two
domains indistinguishable. The common architecture for this discriminator is
either a fully connected or a CNN-based network. However, SLARDA proposes an
autoregressive domain discriminator that considers the temporal features when
distinguishing source and target domains [72]. It also pretrains the encoder
with a forecasting task to boost the performance. Finally, a mean-teacher
model is used to learn the fine-grained class distribution of the target
domain.

In most works, the feature extractor architecture is usually shared between
source and target domains. However, Domain Adaptation Forecaster (DAF) uses an
unshared feature extractor and a shared attention-based architecture for time
series UDA forecasting [73]. The shared attention module keeps the domain-
specific features, and the domain discriminator leverages the attention output
to perform the adversarial training. Similarly, ADAST proposes a shared
feature extractor and an unshared domain-specific attention model [74]. Also,
ContrasGAN leverages a bidirectional GAN to adapt the source and target
domains with two unshared generators and discriminators to regenerate samples
between domains [75]. They also discriminate classes by leveraging contrastive
learning with adaptation to minimize the intra-class discrepancy.

To benefit from the availability of multiple source domains in human activity
recognition, CoDATS proposes a weakly supervised multi-source UDA [76]. It
learns domain invariant features with adversarial training, based on a
gradient reversal layer (GRL) to flip the gradients when being back-
propagated.

Adversarial-based UDA methods are computationally expensive and can be more
difficult to implement than distance-based UDA methods. However, they are more
effective at reducing the large domain shift between source and target
domains.

Another line of works attempt to address the UDA problem from a different
perspective. For example, Eldele et al. [77] propose a contrastive-based
approach to mitigate the domain shift. The authors attempt to move the source
and target domains to an intermediate domain with a constrastive loss.
Furthermore, Ragab et al. [78] present a pioneering approach to adapt the
unlabeled target domain to a labeled source domain in a source-free manner,
i.e., the adaptation occurs without accessing the source domain data. Such a
method promotes the data privacy.

###  V-C Semi-supervised Domain Adaptation

Semi-supervised domain adaptation is used to adapt a model trained on a source
domain to a related out-of-distribution target domain, in which only a small
amount of labeled data is available for the target domain. The labeled samples
in the target domain can be helpful in two aspects. First, it can be used to
select the best target model hyperparameters. Second, these samples can be
used to boost the adaptation performance of the model. The training process of
the semi-supervised domain adaptation is shown in Fig. 7, where we apply the
classification loss to the labeled samples from both the source and target
domains, and use a domain alignment strategy to align the distribution of both
domains.

Figure 7: The semi-supervised domain adaptation process. The process is
similar to the unsupervised domain adaptation, but with including a
classification loss the labeled samples from both the source and target
domains.

Most previous semi-supervised UDA works have adopted the adversarial-based
approach. For example, Heremans et al. [79] generate pseudo labels to boost
the performance along with the target domain labeled data while performing
adversarial domain adaptation for EEG signals. Also, Wang et al. [80] propose
nuclear norm maximization and consistency loss with virtual adversarial
training to measure the discrepancy between the predictions of the unlabeled
samples and their perturbations, which helps to maintain diversity and
discriminability. Those components are added along with the adversarial
training on fault diagnosis data.

Semi-supervised UDA is less explored despite being more practical in real-
world scenarios. This can be regarded to the increased challenge in labeling
the target domain samples.

##  VI Discussion and Future Directions

In this section, we outline potential future research directions that stem
from the insights garnered through our review of label-efficient techniques
for time series representation learning. As the field continues to evolve,
several key avenues emerge that hold promise for advancing the state-of-the-
art in efficient and effective time series analysis.

###  VI-A Incorporating Hybrid Approaches

One intriguing direction lies in the fusion of various label-efficient
techniques, which has the potential to harness the complementary strengths of
each approach. This concept stems from the recognition that no single
technique can address all the challenges associated with time series analysis,
especially when dealing its diverse and complex structure. For example,
combining data augmentation, self-supervised learning, and semi-supervised
learning within a unified framework has the potential to harness the
complementary strengths of each approach. Such hybrid strategies could enable
models to leverage both the rich information encoded in the raw data and the
intrinsic relationships within the labeled samples. Exploring the synergies
between these techniques and assessing their combined impact on performance
and efficiency presents an exciting area of research.

This has been studied in [81], who generate synthetic time series data with
diversified characteristics from the original dataset and use that as a source
dataset for transfer learning tasks. Also, Hu et al. [82] generate augmented
data to increase the patterns variability in the dataset, and then deploy
domain adversarial transfer learning for time series forecasting. Besides
augmentations, self-supervised learning has also shown a high learning
capability from unlabeled data, which promotes it for learning from unlabeled
data in any scenario. Hence, numerous semi-supervised learning and transfer
learning-based methods integrate self-supervised learning in their
methodologies, as in [48, 75].

Notably, this fusion approach can raise some challenges, e.g., the increased
model complexity, which require careful design, tuning, and validation to
avoid overfitting or other unintended consequences. In addition, the
combination of different techniques might impose higher computational
requirements, necessitating optimization both in algorithmic design and
computational resources. Last, it is important to investigate how can we
tailor the fused models to specific domains, such as finance or healthcare, to
increase its effectiveness in addressing domain-specific challenges.

###  VI-B In-Domain vs. Cross-Domain Solutions: A Strategic Decision

The landscape of label-efficient time series representation learning offers
both in-domain and cross-domain solutions, each with unique characteristics
and applications. In deciding whether to rely on in-domain or cross-domain
solutions, it is important to consider the context, goals, and challenges
specific to the data. This section explores the factors that guide the
selection between these two paradigms, the potential synergies, and the areas
where future research is needed.

On one hand, the in-domain solutions are usually tailored to the
characteristics of the specific time series domain, allowing them to capture
nuances, patterns, and context unique to that domain. This can lead to more
accurate and relevant representations. In addition, it helps the preservation
of the domain structure. Techniques like data augmentation and self-supervised
learning can preserve the inherent structure of the data, enabling models to
learn from perturbed versions of real-world instances. However, in-domain
techniques may struggle to generalize to unseen domains or scenarios,
especially when the labeled data is scarce or specific to a narrow context. On
the other hand, cross-domain solutions capitalize on knowledge transfer from
related domains, which make models robust across different domains, enabling
them to perform well on new and unseen data distributions. However, the
success of cross-domain solutions heavily depends on the degree of similarity
between the source and target domains. Specifically, we may suffer from the
risk of negative transfer if domains are not sufficiently aligned, besides the
increased computational complexity in matching and adapting across domains.

Therefore, the decision between in-domain and cross-domain solutions is not a
binary one, but rather a strategic choice shaped by the nature of the task,
the availability of data, and the need for generalizability. Future research
that explores the synergies between these paradigms, the development of
decision-making frameworks, and the ethical and practical implications will
contribute to a more nuanced understanding of label-efficient time series
representation learning.

###  VI-C Choosing the Best Method within Each Category

Selecting the most suitable label-efficient method within each category, e.g.,
self-supervised learning, presents a challenge due to the diversity of
techniques available. The decision should be guided by the specific context of
the problem, the available resources, and the desired outcome. For instance,
assume a scenario where we have unlabeled data and we decided to use
contrastive self-supervised learning. It might be confusing to chose between
inter-sample or intra-sample contrasting methods. This could be regarded to
the lack of consistency in the backbone networks, the evaluation schemes, and
the evaluating datasets among existing methods. Therefore, it can be
challenging to identify the best-performing methods under each category [83].

The creation of benchmarks is a forward step into addressing this challenge.
Benchmarks provide standardized evaluation platforms that enable fair
comparisons between different methods. In addition, they facilitate insightful
comparisons of strengths, weaknesses, and limitations across different
techniques. Moreover, they help researchers identify key challenges and gaps,
driving the development of new methods that address specific issues
highlighted by benchmark performance. One example of these benchmarks is
AdaTime [84], which provides a unified benchmarking suite to the UDA
algorithms on time series data. Therefore, the development of dedicated
benchmarking suites is a promising future direction.

###  VI-D Active Learning

Active learning allows the model to interact with its environment and actively
seek labels on data that it is uncertain about, instead of relying solely on
passively provided labeled data. In this way, we allow the model to actively
select the most informative data points during training, which can improve its
performance and reduce the amount of labeled data needed. Despite its
potential, few works have been proposed for active learning on time series
data. For example, Bota et al. [85] propose a semi-supervised active learning
approach based on human activity recognition. They rely on self-training to
select the most relevant samples for annotation and training the model. Also,
the Transfer Active Learning (TAL) method selects the candidate samples by
evaluating their informativeness and representativeness [86]. These criteria
are calculated by mapping the input sample into the embedding space from both
sample and sample-label views.

Designing task-specific active learning techniques for time series data,
exploring synergies with self-supervised learning, and establishing evaluation
protocols that reflect real-world scenarios can be promising future directions
in this field.

###  VI-E Prototypical networks

Prototypical Networks is another approach for addressing the label scarcity
issue in various machine learning applications, including time series
representation learning. These networks build on the idea of learning a metric
space where representations of examples are computed to reflect the underlying
class structure [87]. Prototypical Networks rely on the concept of learning a
prototypical representation for each class, derived from the embeddings of
examples within that class. Prototypes are then computed as the mean of the
embeddings of the examples within a class. After that, unlabeled instances are
compared to these prototypes using a distance metric, e.g., Euclidean
distance, and classification is performed based on proximity to the
prototypes.

With this design, Prototypical Networks are inherently can operate effectively
with few labeled examples, making them an appealing solution for labels
scarcity. Despite that, few works has adopted this approach. For example,
Zhang et al. [88] propose TapNet; and attentional prototype network for
multivariate time series data. While the potential of prototypical networks in
time series representation learning is substantial, challenges such as
handling long sequences, addressing high-dimensional data, and mitigating the
impact of noise must be addressed.

##  VII Conclusion

The scarcity of annotated data presents a significant barrier to the effective
deployment of deep learning models in real-world scenarios. This paper aimed
to offer a holistic perspective, seamlessly connecting disparate research
works and carving out a clear roadmap for tackling various label-deficient
situations and their potential remedies. Our discourse underscores the
importance of selecting the appropriate deep learning architecture, and the
imperative to understand the nature of available time series data when
determining the suitability of in-domain versus cross-domain approaches. Our
investigation further highlights the potential of fusing multiple techniques
to yield a better performance. Moreover, this survey underscores the necessity
for standardized benchmarks, which would streamline the decision-making
process when navigating the plethora of methodologies within each category.

## References

  * Wen et al. [2021] Q. Wen, L. Sun, F. Yang, X. Song, J. Gao, X. Wang, and H. Xu, “Time series data augmentation for deep learning: A survey,” in _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , 8 2021, pp. 4653–4660, survey Track. [Online]. Available: https://doi.org/10.24963/ijcai.2021/631
  * Deldari et al. [2022] S. Deldari, H. Xue, A. Saeed, J. He, D. V. Smith, and F. D. Salim, “Beyond just vision: A review on self-supervised representation learning on multimodal and temporal data,” _arXiv preprint arXiv:2206.02353_ , 2022.
  * Zhang et al. [2023] K. Zhang, Q. Wen, C. Zhang, R. Cai, M. Jin, Y. Liu, J. Zhang, Y. Liang, G. Pang, D. Song, and S. Pan, “Self-supervised learning for time series analysis: Taxonomy, progress, and prospects,” 2023.
  * Ma et al. [2023] Q. Ma, Z. Liu, Z. Zheng, Z. Huang, S. Zhu, Z. Yu, and J. T. Kwok, “A survey on time-series pre-trained models,” 2023.
  * Ismail Fawaz et al. [2018] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller, “Transfer learning for time series classification,” in _IEEE International Conference on Big Data_ , 2018, pp. 1367–1376.
  * Shi et al. [2022] Y. Shi, X. Ying, and J. Yang, “Deep unsupervised domain adaptation with time series sensor data: A survey,” _Sensors_ , vol. 22, no. 15, 2022. [Online]. Available: https://www.mdpi.com/1424-8220/22/15/5507
  * Eldele et al. [2021a] E. Eldele, M. Ragab, Z. Chen, M. Wu, C. K. Kwoh, X. Li, and C. Guan, “Time-series representation learning via temporal and contextual contrasting,” in _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI_ , 2021, pp. 2352–2359.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 770–778.
  * Bai et al. [2018] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” _arXiv preprint arXiv:1803.01271_ , 2018.
  * Ismail Fawaz et al. [2020] H. Ismail Fawaz, B. Lucas, G. Forestier, C. Pelletier, D. F. Schmidt, J. Weber, G. I. Webb, L. Idoumghar, P.-A. Muller, and F. Petitjean, “Inceptiontime: Finding alexnet for time series classification,” _Data Mining and Knowledge Discovery_ , vol. 34, no. 6, pp. 1936–1962, 2020.
  * Tang et al. [2021] W. Tang, G. Long, L. Liu, T. Zhou, M. Blumenstein, and J. Jiang, “Omni-scale cnns: a simple and effective kernel size configuration for time series classification,” in _International Conference on Learning Representations, ICLR_ , 2021.
  * Rangapuram et al. [2018] S. S. Rangapuram, M. W. Seeger, J. Gasthaus, L. Stella, Y. Wang, and T. Januschowski, “Deep state space models for time series forecasting,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018\.
  * Khan et al. [2021] M. Khan, H. Wang, A. Riaz, A. Elfatyany, and S. Karim, “Bidirectional lstm-rnn-based hybrid deep learning frameworks for univariate time series classification,” _The Journal of Supercomputing_ , vol. 77, no. 7, pp. 7021–7045, 2021.
  * Supratak et al. [2017] A. Supratak, H. Dong, C. Wu, and Y. Guo, “Deepsleepnet: a model for automatic sleep stage scoring based on raw single-channel eeg,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 25, no. 11, pp. 1998–2008, 2017.
  * Li et al. [2019] Y. Li, Z. Zhu, D. Kong, H. Han, and Y. Zhao, “Ea-lstm: Evolutionary attention-based lstm for time series prediction,” _Knowledge-Based Systems_ , vol. 181, p. 104785, 2019.
  * Liu et al. [2022a] X. Liu, R. Shi, Q. Hui, S. Xu, S. Wang, R. Na, Y. Sun, W. Ding, D. Zheng, and X. Chen, “Tcacnet: Temporal and channel attention convolutional network for motor imagery classification of eeg-based bci,” _Information Processing & Management_, vol. 59, no. 5, p. 103001, 2022.
  * Huang et al. [2019] S. Huang, D. Wang, X. Wu, and A. Tang, “Dsanet: Dual self-attention network for multivariate time series forecasting,” in _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , ser. CIKM ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 2129–2132. [Online]. Available: https://doi.org/10.1145/3357384.3358132
  * Eldele et al. [2021b] E. Eldele, Z. Chen, C. Liu, M. Wu, C.-K. Kwoh, X. Li, and C. Guan, “An attention-based deep learning approach for sleep stage classification with single-channel eeg,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 29, pp. 809–818, 2021.
  * Zerveas et al. [2021] G. Zerveas, S. Jayaraman, D. Patel, A. Bhamidipaty, and C. Eickhoff, “A transformer-based framework for multivariate time series representation learning,” in _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 2021, pp. 2114–2124.
  * Fan et al. [2020] J. Fan, C. Sun, C. Chen, X. Jiang, X. Liu, X. Zhao, L. Meng, C. Dai, and W. Chen, “Eeg data augmentation: towards class imbalance problem in sleep staging tasks,” _Journal of Neural Engineering_ , vol. 17, no. 5, p. 056017, oct 2020.
  * Iwana and Uchida [2021] B. K. Iwana and S. Uchida, “An empirical survey of data augmentation for time series classification with neural networks,” _Plos one_ , vol. 16, no. 7, p. e0254841, 2021.
  * Li et al. [2020a] X. Li, J. Luo, and R. Younes, “Activitygan: Generative adversarial networks for data augmentation in sensor-based human activity recognition,” in _Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers_ , ser. UbiComp-ISWC ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 249–254. [Online]. Available: https://doi.org/10.1145/3410530.3414367
  * Fahimi et al. [2021] F. Fahimi, S. Dosen, K. K. Ang, N. Mrachacz-Kersting, and C. Guan, “Generative adversarial networks-based data augmentation for brain–computer interface,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 9, pp. 4039–4051, 2021.
  * Banville et al. [2021] H. J. Banville, O. Chehab, A. Hyvärinen, D.-A. Engemann, and A. Gramfort, “Uncovering the structure of clinical eeg signals with self-supervised learning.” _Journal of Neural Engineering_ , vol. 18, no. 4, p. 46020, 2021.
  * Saeed et al. [2021] A. Saeed, V. Ungureanu, and B. Gfeller, “Sense and learn: Self-supervision for omnipresent sensors,” _Machine Learning with Applications_ , vol. 6, p. 100152, 2021.
  * Tipirneni and Reddy [2022] S. Tipirneni and C. K. Reddy, “Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series,” _ACM Trans. Knowl. Discov. Data_ , vol. 16, no. 6, jul 2022. [Online]. Available: https://doi.org/10.1145/3516367
  * Sarkar and Etemad [2022] P. Sarkar and A. Etemad, “Self-supervised ecg representation learning for emotion recognition,” _IEEE Transactions on Affective Computing_ , vol. 13, no. 3, pp. 1541–1554, 2022.
  * Saeed et al. [2019] A. Saeed, T. Ozcelebi, and J. Lukkien, “Multi-task self-supervised learning for human activity detection,” _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , vol. 3, no. 2, 2019.
  * van den Oord et al. [2018] A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv: Learning_ , 2018.
  * Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International Conference on Machine Learning, ICML_ , vol. 1, 2020, pp. 1597–1607.
  * Jiang et al. [2021] X. Jiang, J. Zhao, B. Du, and Z. Yuan, “Self-supervised contrastive learning for eeg-based sleep staging,” in _International Joint Conference on Neural Networks (IJCNN)_ , 2021, pp. 1–8.
  * Pöppelbaum et al. [2022] J. Pöppelbaum, G. S. Chadha, and A. Schwung, “Contrastive learning based self-supervised time-series analysis,” _Applied Soft Computing_ , vol. 117, p. 108397, 2022.
  * Wickstrøm et al. [2022] K. Wickstrøm, M. Kampffmeyer, K. Øyvind Mikalsen, and R. Jenssen, “Mixing up contrastive learning: Self-supervised representation learning for time series,” _Pattern Recognition Letters_ , vol. 155, pp. 54–61, 2022.
  * Chang et al. [2022] Y.-C. Chang, D. Subramanian, R. Pavuluri, and T. Dinger, “Time series representation learning with contrastive triplet selection,” in _5th Joint International Conference on Data Science & Management of Data_, 2022, p. 46–53. [Online]. Available: https://doi.org/10.1145/3493700.3493711
  * Xiao et al. [2021] Q. Xiao, J. Wang, J. Ye, H. Zhang, Y. Bu, Y. Zhang, and H. Wu, “Self-supervised learning for sleep stage classification with predictive and discriminative contrastive coding,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 1290–1294.
  * Kallidromitis et al. [2021] K. Kallidromitis, D. Gudovskiy, K. Kazuki, O. Iku, and L. Rigazio, “Contrastive neural processes for self-supervised learning,” in _Asian Conference on Machine Learning_. PMLR, 2021, pp. 594–609.
  * Tonekaboni et al. [2021] S. Tonekaboni, D. Eytan, and A. Goldenberg, “Unsupervised representation learning for time series with temporal neighborhood coding,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=8qDwejCuCN
  * Yue et al. [2022] Z. Yue, Y. Wang, J. Duan, T. Yang, C. Huang, Y. Tong, and B. Xu, “Ts2vec: Towards universal representation of time series,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, 2022, pp. 8980–8987.
  * Yang and Hong [2022] L. Yang and S. Hong, “Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion,” in _Proceedings of the 39th International Conference on Machine Learning, ICML_ , ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 17–23 Jul 2022, pp. 25 038–25 054.
  * Liu et al. [2021] D. Liu, T. Wang, S. Liu, R. Wang, S. Yao, and T. Abdelzaher, “Contrastive self-supervised representation learning for sensing signals from the time-frequency perspective,” in _International Conference on Computer Communications and Networks (ICCCN)_ , 2021, pp. 1–10.
  * Zhang et al. [2022a] X. Zhang, Z. Zhao, T. Tsiligkaridis, and M. Zitnik, “Self-supervised contrastive pre-training for time series via time-frequency consistency,” in _NeurIPS_ , 2022.
  * Wang et al. [2019] H. Wang, Q. Zhang, J. Wu, S. Pan, and Y. Chen, “Time series feature learning with labeled and unlabeled data,” _Pattern Recognition_ , vol. 89, pp. 55–66, 2019.
  * Marussy and Buza [2013] K. Marussy and K. Buza, “Success: A new approach for semi-supervised classification of time-series,” in _Artificial Intelligence and Soft Computing_ , 2013, pp. 437–447.
  * Sohn et al. [2020] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 596–608.
  * Zhang et al. [2022b] G. Zhang, V. Davoodnia, and A. Etemad, “Parse: Pairwise alignment of representations in semi-supervised eeg learning for emotion recognition,” _IEEE Transactions on Affective Computing_ , pp. 1–16, 2022.
  * Xing et al. [2022] H. Xing, Z. Xiao, D. Zhan, S. Luo, P. Dai, and K. Li, “Selfmatch: Robust semisupervised time-series classification with self-distillation,” _International Journal of Intelligent Systems_ , vol. 37, no. 11, pp. 8583–8610, 2022. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/int.22957
  * Jawed et al. [2020] S. Jawed, J. Grabocka, and L. Schmidt-Thieme, “Self-supervised learning for semi-supervised time series classification,” in _Advances in Knowledge Discovery and Data Mining_ , 2020, pp. 499–511.
  * Eldele et al. [2022] E. Eldele, M. Ragab, Z. Chen, M. Wu, C. K. Kwoh, X. Li, and C. Guan, “Self-supervised contrastive representation learning for semi-supervised time-series classification,” _arXiv preprint arXiv:2208.06616_ , 2022.
  * Liu et al. [2022b] X. Liu, F. Zhang, H. Liu, and H. Fan, “itimes: Investigating semi-supervised time series classification via irregular time sampling,” _IEEE Transactions on Industrial Informatics_ , pp. 1–9, 2022.
  * Fan et al. [2021] H. Fan, F. Zhang, R. Wang, X. Huang, and Z. Li, “Semi-supervised time series classification by temporal relation prediction,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 3545–3549.
  * Li et al. [2022] Y. Li, S. Luo, H. Zhang, Y. Zhang, Y. Zhang, and B. Lo, “Mtclss: Multi-task contrastive learning for semi-supervised pediatric sleep staging,” _IEEE Journal of Biomedical and Health Informatics_ , pp. 1–9, 2022.
  * Han et al. [2021] J. Han, X. Gu, and B. Lo, “Semi-supervised contrastive learning for generalizable motor imagery eeg classification,” in _IEEE 17th International Conference on Wearable and Implantable Body Sensor Networks (BSN)_ , 2021, pp. 1–4.
  * Zhu et al. [2019] Q. Zhu, Z. Chen, and Y. C. Soh, “A novel semisupervised deep learning method for human activity recognition,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 7, pp. 3821–3830, 2019.
  * Chen et al. [2020] K. Chen, L. Yao, D. Zhang, X. Wang, X. Chang, and F. Nie, “A semisupervised recurrent convolutional attention model for human activity recognition,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 31, no. 5, pp. 1747–1756, 2020.
  * Wu et al. [2021] X. Wu, Y. Zhang, C. Cheng, and Z. Peng, “A hybrid classification autoencoder for semi-supervised fault diagnosis in rotating machinery,” _Mechanical Systems and Signal Processing_ , vol. 149, p. 107327, 2021.
  * You et al. [2022] S. You, B. Hwan Cho, Y.-M. Shon, D.-W. Seo, and I. Y. Kim, “Semi-supervised automatic seizure detection using personalized anomaly detecting variational autoencoder with behind-the-ear eeg,” _Computer Methods and Programs in Biomedicine_ , vol. 213, p. 106542, 2022.
  * Zuo et al. [2021] J. Zuo, K. Zeitouni, and Y. Taher, “Smate: Semi-supervised spatio-temporal representation learning on multivariate time series,” in _IEEE International Conference on Data Mining (ICDM)_ , 2021, pp. 1565–1570.
  * Rezagholiradeh and Haidar [2018] M. Rezagholiradeh and M. A. Haidar, “Reg-gan: Semi-supervised learning based on generative adversarial networks for regression,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 2806–2810.
  * He et al. [2022] R. He, Z. Tian, and M. J. Zuo, “A semi-supervised gan method for rul prediction using failure and suspension histories,” _Mechanical Systems and Signal Processing_ , vol. 168, p. 108657, 2022.
  * Yosinski et al. [2014] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” in _Advances in Neural Information Processing Systems_ , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014.
  * Xu et al. [2021] F. Xu, Y. Miao _et al._ , “A transfer learning framework based on motor imagery rehabilitation for stroke,” _Scientific Reports_ , 2021.
  * Zhong et al. [2022] H. Zhong, Y. Lv, R. Yuan, and D. Yang, “Bearing fault diagnosis using transfer learning and self-attention ensemble lightweight convolutional neural network,” _Neurocomputing_ , vol. 501, pp. 765–777, 2022.
  * An et al. [2021] S. An, G. Bhat, S. Gumussoy, and U. Ogras, “Transfer learning for human activity recognition using representational analysis of neural networks,” _arXiv: Learning_ , 2021.
  * Li et al. [2020b] F. Li, K. Shirahama, M. A. Nisar, X. Huang, and M. Grzegorzek, “Deep transfer learning for time series data based on sensor modality classification,” _Sensors_ , vol. 20, no. 15, 2020. [Online]. Available: https://www.mdpi.com/1424-8220/20/15/4271
  * Ye and Dai [2021] R. Ye and Q. Dai, “Implementing transfer learning across different datasets for time series forecasting,” _Pattern Recognition_ , vol. 109, p. 107617, 2021.
  * Meiseles and Rokach [2020] A. Meiseles and L. Rokach, “Source model selection for deep learning in the time series domain,” _IEEE Access_ , vol. 8, pp. 6190–6200, 2020.
  * Gretton et al. [2012] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A kernel two-sample test,” _Journal of Machine Learning Research_ , vol. 13, no. 1, pp. 723–773, 2012.
  * Sun et al. [2016] B. Sun, J. Feng, and K. Saenko, “Return of frustratingly easy domain adaptation,” in _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ , 2016, pp. 2058–2065.
  * Liu and Xue [2021] Q. Liu and H. Xue, “Adversarial spectral kernel matching for unsupervised time series domain adaptation,” in _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , 8 2021, pp. 2744–2750. [Online]. Available: https://doi.org/10.24963/ijcai.2021/378
  * Ott et al. [2022] F. Ott, D. Rügamer, L. Heublein, B. Bischl, and C. Mutschler, “Domain adaptation for time-series classification to mitigate covariate shift,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, p. 5934–5943.
  * Cai et al. [2021] R. Cai, J. Chen, Z. Li, W. Chen, K. Zhang, J. Ye, Z. Li, X. Yang, and Z. Zhang, “Time series domain adaptation via sparse associative structure alignment,” in _AAAI_ , 2021.
  * Ragab et al. [2022] M. Ragab, E. Eldele, Z. Chen, M. Wu, C.-K. Kwoh, and X. Li, “Self-supervised autoregressive domain adaptation for time series data,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * Jin et al. [2022] X. Jin, Y. Park, D. Maddix, H. Wang, and Y. Wang, “Domain adaptation for time series forecasting via attention sharing,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 10 280–10 297.
  * Eldele et al. [2023a] E. Eldele, M. Ragab, Z. Chen, M. Wu, C.-K. Kwoh, X. Li, and C. Guan, “Adast: Attentive cross-domain eeg-based sleep staging framework with iterative self-training,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , vol. 7, no. 1, pp. 210–221, 2023.
  * Sanabria et al. [2021] A. R. Sanabria, F. Zambonelli, S. Dobson, and J. Ye, “Contrasgan: Unsupervised domain adaptation in human activity recognition via adversarial and contrastive learning,” _Pervasive and Mobile Computing_ , vol. 78, p. 101477, 2021.
  * Wilson et al. [2020] G. Wilson, J. R. Doppa, and D. J. Cook, “Multi-source deep domain adaptation with weak supervision for time-series sensor data,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, p. 1768–1778. [Online]. Available: https://doi.org/10.1145/3394486.3403228
  * Eldele et al. [2023b] E. Eldele, M. Ragab, Z. Chen, M. Wu, C.-K. Kwoh, and X. Li, “Contrastive domain adaptation for time-series via temporal mixup,” _IEEE Transactions on Artificial Intelligence_ , pp. 1–10, 2023.
  * Ragab et al. [2023a] M. Ragab, E. Eldele, M. Wu, C.-S. Foo, X. Li, and Z. Chen, “Source-free domain adaptation with temporal imputation for time series data,” in _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , ser. KDD ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 1989–1998. [Online]. Available: https://doi.org/10.1145/3580305.3599507
  * Heremans et al. [2022] E. R. Heremans, H. Phan, P. Borzée, B. Buyse, D. Testelmans, and M. De Vos, “From unsupervised to semi-supervised adversarial domain adaptation in electroencephalography-based sleep staging,” _Journal of Neural Engineering_ , vol. 19, no. 3, p. 036044, 2022.
  * Wang et al. [2020] X. Wang, F. Liu, and D. Zhao, “Cross-machine fault diagnosis with semi-supervised discriminative adversarial domain adaptation,” _Sensors_ , vol. 20, no. 13, 2020. [Online]. Available: https://www.mdpi.com/1424-8220/20/13/3753
  * Rotem et al. [2022] Y. Rotem, N. Shimoni, L. Rokach, and B. Shapira, “Transfer learning for time series classification using synthetic data generation,” in _Cyber Security, Cryptology, and Machine Learning_. Cham: Springer International Publishing, 2022, pp. 232–246.
  * Hu et al. [2020] H. Hu, M. Tang, and C. Bai, “Datsing: Data augmented time series forecasting with adversarial domain adaptation,” in _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. New York, NY, USA: Association for Computing Machinery, 2020, p. 2061–2064. [Online]. Available: https://doi.org/10.1145/3340531.3412155
  * Eldele et al. [2023c] E. Eldele, M. Ragab, Z. Chen, M. Wu, C. K. Kwoh, and X. Li, “Self-supervised learning for label-efficient sleep stage classification: A comprehensive evaluation,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 31, pp. 1333–1342, 2023.
  * Ragab et al. [2023b] M. Ragab, E. Eldele, W. L. Tan, C.-S. Foo, Z. Chen, M. Wu, C.-K. Kwoh, and X. Li, “Adatime: A benchmarking suite for domain adaptation on time series data,” _ACM Trans. Knowl. Discov. Data_ , vol. 17, no. 8, may 2023. [Online]. Available: https://doi.org/10.1145/3587937
  * Bota et al. [2019] P. Bota, J. Silva, D. Folgado, and H. Gamboa, “A semi-automatic annotation approach for human activity recognition,” _Sensors_ , vol. 19, no. 3, 2019\. [Online]. Available: https://www.mdpi.com/1424-8220/19/3/501
  * Kinyua and Jouandeau [2021] P. Kinyua and N. Jouandeau, “Sample-label view transfer active learning for time series classification,” in _Artificial Neural Networks and Machine Learning_. Cham: Springer International Publishing, 2021, pp. 600–611.
  * Snell et al. [2017] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017.
  * Zhang et al. [2020] X. Zhang, Y. Gao, J. Lin, and C.-T. Lu, “Tapnet: Multivariate time series classification with attentional prototypical network,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, 2020, pp. 6845–6852.