# Generative adversarial networks in time series: A survey and taxonomy

Eoin Brophy
Infant Research Centre & School of Computing
Dublin City University
Ireland
<EMAIL_ADDRESS>
&Zhengwei Wang
ByteDance AI Lab
China
&Qi She
ByteDance AI Lab
China

&Tomás Ward
Insight SFI Research Centre for Data Analytics
Dublin City University
Ireland

###### Abstract

Generative adversarial networks (GANs) studies have grown exponentially in the
past few years. Their impact has been seen mainly in the computer vision field
with realistic image and video manipulation, especially generation, making
significant advancements. While these computer vision advances have garnered
much attention, GAN applications have diversified across disciplines such as
time series and sequence generation. As a relatively new niche for GANs,
fieldwork is ongoing to develop high quality, diverse and private time series
data. In this paper, we review GAN variants designed for time series related
applications. We propose a taxonomy of discrete-variant GANs and continuous-
variant GANs, in which GANs deal with discrete time series and continuous time
series data. Here we showcase the latest and most popular literature in this
field; their architectures, results, and applications. We also provide a list
of the most popular evaluation metrics and their suitability across
applications. Also presented is a discussion of privacy measures for these
GANs and further protections and directions for dealing with sensitive data.
We aim to frame clearly and concisely the latest and state-of-the-art research
in this area and their applications to real-world technologies.

_K_ eywords Generative Adversarial Networks $\cdot$ Time Series $\cdot$
Discrete-variant GANs $\cdot$ Continuous-variant GANs

##  1 Introduction

This review paper is designed for those interested in GANs applied to time
series data generation. We provide a review of current state-of-the-art and
novel time series GANs and their solutions to real-world problems. The
applicability of GANs to this type of data can solve many issues that current
dataset holders face. Data shortage is often an issue, and GANs can augment
smaller datasets by generating new, previously unseen data. Data can be
missing or corrupted in cases; GANs can impute data, i.e. replace the
artefacts with information representative of clean data. GANs are also capable
of denoising signals in the case of corrupted data. Data protection, privacy,
and sharing have become heavily regulated; GANs can ensure an extra layer of
data protection by generating differentially private datasets containing no
risk of linkage from source to generated datasets.

Several methods have been used in the past to generate synthetic data. One
such method is the autoencoder (AE) which is designed to efficiently learn an
informative representation of an input in a small dimensional space and
reconstruct the encoded data back such that the reconstructed input is similar
as possible to the original one. The AE model is made of an encoder and
decoder neural network, as shown in Figure 1. However, other generative models
have emerged as front-runners due to the quality of the generated data and
inherent privacy protection measures.

Figure 1: Autoencoder Model

Generative Adversarial Networks have been gaining a lot of traction amongst
the deep learning research community since their inception in 2014 [1]. Their
ability to generate and manipulate data across multiple domains has
contributed to their success. While the main focus of GANs to date has been in
the computer vision (CV) domain, they have also been successfully applied to
others, such as natural language processing (NLP). There has also been a
movement towards the use of GANs for time series and sequential data
generation, and forecasting.

A GAN is a generative model consisting of a generator and discriminator,
typically two neural networks (NN) models. In recent years GANs have
demonstrated their ability to produce high-quality image and video generation,
style-transfer, and image completion. They have also been successfully used
for audio generation, sequence forecasting, and imputation.

However, one of the significant challenges of GANs lies in their inherent
instability, which makes it difficult to train. GAN models suffer from issues
such as non-convergence, diminishing/vanishing gradients, and mode collapse. A
non-converging model does not stabilise and continuously oscillates, causing
it to diverge. Diminishing gradients prevents the generator from learning
anything as the discriminator becomes too successful. Mode collapse is when
the generator collapses, producing only uniform samples with little to no
variety.

The second challenge of GANs lies in its evaluation process. With image-based
GANs, researchers have reached a loose consensus [2] surrounding the
evaluation of the generated distribution estimated from the training data
distribution. Unfortunately for time series GANs, due to the comparatively low
numbers of papers published, there has not been an agreement reached on the
generated data’s evaluation metrics. There have been different approaches put
forward, but none established as a front runner in the metrics space as of
yet.

We define a time series as a sequence of vectors dependant on time $(t)$ and
can be represented as $xt={x1,...,xn}$ for continuous/real-time and discrete-
time. The time series’ values can either be defined as continuous or discrete
and, depending on the number of values recorded, are univariate or
multivariate. In most cases, the time series will take either an integer value
or a real value.

As Dorffner states, a time series can be viewed, from a practical perspective,
as a value sampled at discrete steps in time [3]. This time-step can be as
long as years to as short as milliseconds, for example. We define a continuous
time series as a signal sampled from a continuous process, i.e. the function’s
domain is from an uncountable set. In contrast, a discrete time series has a
countable domain.

In this review, we present the first complete review and taxonomy of time
series GANs, namely discrete and continuous variants, their applications,
architecture, loss functions and how they have improved on their predecessors
in terms of variety and quality of their generated data. We also contribute by
including experiments for the majority of time series GAN architectures
applied to time series synthesis.

##  2 Related Work

There has been a handful of high-quality GAN review papers published in the
past few years. For example, Wang et al. takes a taxonomic approach to GANs in
Computer Vision [4]. The authors split GANs into architecture variants and
loss variants. While they include applications of GANs and mention their
applicability to sequential data generation, the work is heavily focused on
media manipulation and generation. The authors in [5] breakdown GANs into
their constituent parts. They begin by discussing the algorithms and
architecture of various GANs and their evaluation metrics, then list their
surrounding theory and problems such as mode collapse, amongst others.
Finally, they discuss the applications of GANs and provide a very brief
account of GANs used for sequential data. Gonog and Zhou [6] provide a short
introduction to GANs, their theory and explores the variety of plausible
models, again listing their applications in image and video manipulation with
a mention of sequential data (NLP). In another review, paper [7] the authors
give an overview of GAN fundamentals, variants, and applications. Sequential
data applications are mentioned in the form of music and speech synthesis.

As with most review papers, Yinka-Banjo and Ugot give an introduction and
overview of Generative Adversarial Networks [8]. However, they also review
GANs as adversarial detectors and discuss their limitations applied to
cybersecurity. Yi, Walia, and Babyn [9] give a review of GANs and their
applications in medical imaging, how they can be used in clinical research and
potentially deployed to help practising clinicians. There is no mention of
time series data use cases.

A recurring theme in these papers focuses on GAN variants which have mostly
been applied to the computer vision domain. To the best of our knowledge, no
review paper has been conducted with the main focus on time series GANs. While
these reviews have mentioned the application of these GANs in generating
sequential data, they have scratched the surface of what is becoming a growing
body of research.

We contribute to lessening this gap by presenting our work which is concerned
with presenting the latest up-to-date research around time series GANs, their
architecture, loss functions, evaluation metrics, trade-offs and approaches to
privacy preservation of their datasets.

##  3 Generative Adversarial Networks

###  3.1 Background

The introduction of GANs facilitated a significant breakthrough in the
generation of synthetic data. These deep learning models typically consist of
two neural networks, a generator and a discriminator. The generator G takes in
random noise $\textbf{{z}}\in\mathbb{R}^{r}$ and attempts generates synthetic
data that is similar to the training data distribution. The discriminator D
attempts to determine if the generated data is real or fake. The generator
aims to maximise the failure rate of the discriminator, while the
discriminator aims to minimise it. Figure 2 shows a simple example of the GAN
architecture and the game that the neural network models play. The two
networks are locked in a two-player minimax game defined by the value function
V(G,D) (1), where D(x) is the probability that x comes from the real data
rather than the generated data [1].

Figure 2: Generative Adversarial Network

$\mathop{min}_{G}\mathop{max}_{D}V(G,D)=\mathbb{E}_{x\sim
p_{data}(x)}[logD(\textbf{x})]+\mathbb{E}_{z\sim
p_{\textbf{z}}(z)}[log(1-D(G(\textbf{z})))]$ (1)

GANs belong to the family of generative models and are an alternative method
of generating synthetic data that do not require domain expertise. They were
conceived in the paper by Goodfellow in 2014, where a multi-layer perceptron
was used for both the discriminator and the generator [1]. Radford et al.
(2015) subsequently developed the deep convolutional generative adversarial
network (DCGAN) to generate synthetic images [10]. Since then, researchers
have continuously improved on the early GAN architectures, loss functions, and
evaluation metrics while innovating on their potential contributions to real-
world applications. To appreciate why there has been such concerted activity
in the further development of GAN technologies it is important to understand
the limitations of early architectures and the challenges these presented. We
describe these next, and in so doing, prepare the reader for the particular
manifestation of these challenges in the more specific context of time series.

###  3.2 Challenges

There are three main challenges in the area of time series GANs, i.e.,
training stability, evaluation and privacy risk associated with synthetic data
created by GANs. We are going to explain these three challenges as follows.

Training stability. The original work [1] has already proved the global
optimality and the convergence of GANs during training. It still highlights
the instability problem that can arise when training a GAN. Two problems are
well-studied in the literature 1. vanishing gradients and 2. mode collapse.
The vanishing gradient is caused by directly optimizing loss presented in
equation (1). When $D$ reaches the optimality, optimizing the equation (1) for
$G$ can be converted to minimising the Jensen-Shannon (JS) divergence (details
of derivation can refer to section 5 in [4]) between $p_{r}$ and $p_{g}$:

$\mathcal{L}_{G}=2\cdot JS(p_{r}\|p_{g})-2\cdot\mathrm{log}2$ (2)

JS divergence stays constant ($log2=0.693$) when there is no overlap between
$p_{r}$ and $p_{g}$, which indicates the gradient for $G$ using this loss is 0
in this situation. Non-zero gradient for $G$ only exists when $p_{r}$ and
$p_{g}$ have substantial overlap. In practice, the possibility that $p_{r}$
and $p_{g}$ are not intersected or have negligible overlap is very high [11].
In order to get rid of the vanishing gradient problem for $G$, the original
GAN work [1] highlights that the minimization of

$\mathcal{L}_{G}=-\mathbb{E}_{\mathbf{x}\sim
p_{g}}\mathrm{log}[D(\mathbf{x})]$ (3)

for updating $G$. This strategy is able to avoid the vanishing gradient
problem but lead to the mode collapse issue. Optimizing equation (3) can be
converted to optimizing the reverse Kullback–Leibler (KL) divergence i.e.,
$KL(p_{g}||p_{r})$ (details can also refer to [4]). When $p_{r}$ contains
multiple modes, $p_{g}$ chooses to recover a single mode and ignores other
modes when optimizing the reverse KL divergence. Considering this case, $G$
trained using equation (3) might be only able to generate few modes from real
data. These problems can be amended by changing the architecture or the loss
function, which are reviewed by Wang et al. [4] in detail.

Evaluation. A wide range of evaluation metrics has been proposed to evaluate
the performance of GANs [12, 13, 14, 15]. Current evaluations of GANs in
computer vision are normally designed to consider two perspectives i.e.,
quality and quantity of generated data. The most representative qualitative
metric is to use human annotation to determine the visual quality of the
generated images. Quantitative metrics compare statistical properties between
generated and real images i.e., two-sample tests such as maximum mean
discrepancy (MMD) [16], Inception Score [17] and Fréchet Inception Distance
(FID) [18]. Contrary to evaluating image-based GANs, it is difficult to
evaluate time series data from human psycho-perceptual sense qualitatively. In
terms of qualitatively evaluating time series based GANs, it normally conducts
t-SNE [19] and PCA [20] analyses to visualize how well the generated
distributions resemble the original distributions [21]. Quantitative
evaluation for time series based GANs can be done by deploying two-sample
tests similar to image-based GANs.

Privacy risk. Apart from evaluating the performance of GANs, a wide range of
methods have been used to asses the privacy risk associated with synthetic
data created by GANs. Choi et al. performed tests for presence disclosure and
attribute disclosure. In contrast, others utilised a three-sample test on the
training, test, and synthetic data to identify if the synthetic data has
overfitted to the training data [22, 23]. It has been shown that common
methods of de-identifying data do not prevent attackers from re-identifying
individuals using additional data [24, 25]. Sensitive data is usually de-
identified by removing personally identifiable information (PII). However,
work is ongoing to create frameworks to link different sources of publicly
available information together using alternative information to PII. Malin et
al. developed a software program, REID, to connect individuals contained in
publicly available hospital discharge data with their unique DNA records [25].
Culnane et al. re-identified individuals in a de-identified open dataset of
Australian medical billing records using unencrypted parts of the records and
known information about individuals from other sources [26]. Hejblum et al.
developed a probabilistic method to link de-identified EHR data of patients
with rheumatoid arthritis [27]. The re-identification of individuals in
publicly available datasets can lead to the exposure of their sensitive health
information. Health data has been categorised as special personal data by
General Data Protection Regulation (GDPR) and is subject to a higher level of
protection under the Data Protection Act 2018 (Section36(2)) [28].
Consequently, concerned researchers must find alternative methods of
protecting sensitive health data to minimise the risk of re-identification.
This will be addressed in Section 7.

###  3.3 Popular Datasets

Unlike image-based datasets (CIFAR, MNIST, ImageNet [29, 30, 31]) there are no
standardised or commonly used benchmarking datasets for time series
generation. However, we have compiled a list of some of the more popular
datasets implemented in the reviewed works, and they are listed in Table 1.
There exist two repositories; the UCR Time Series Classification/Clustering
database [32], and the UCI Machine Learning repository [33] that make
available several time series datasets. Despite this, there is still no
consensus on a standardised dataset used for benchmarking time series GANs,
which may be due to the ‘continuous’ nature of the architecture dimensions.
GANs designed for continuous time series generation often differ in the length
of their input sequence due to either author preference or the constraints
placed on their architecture for the generated data’s downstream tasks.

Table 1: Popular Datasets used in the reviewed works. Name (Year) |  Data Type |  Instances |  Attributes
---|---|---|---
Oxford-Man Institute "realised library" (Updated Daily) |  Real Multivariate Time Series |  >2,689,487 |  5
EEG Motor Movement/Imagery Dataset (2004) |  Real Multivariate Time Series |  1,500 |  64
ECG 200 (2001) |  Real Univariate Time Series |  200 |  1
Epileptic Seizure Recognition Dataset (2001) |  Real Multivariate Time Series |  11,500 |  179
TwoLeadECG (2015) |  Real Multivariate Time Series |  1,162 |  2
MIMIC-III (-) |  Real, Integer & Categorical Multivariate Time Series |  - |  -
EPILEPSIAE project database (-) |  Real Multivariate Time Series |  30 |  -
PhysioNet/CinC (2015) |  Real Multivariate Time Series |  750 |  4
Wrist PPG During Exercise (2017) |  Real Multivariate Time Series |  19 |  14
MIT-BIH Arrhythmia Database (2001) |  Real Multivariate Time Series |  201 |  2
PhysioNet/CinC (2012) |  Real, Integer & Categorical Multivariate Time Series |  12000 |  43
KDD Cup Dataset (2018) |  Real, Integer & Categorical Multivariate Time Series |  282 |  3
PeMS Database (Updated Daily) |  Integer & Categorical Multivariate Time Series |  - |  8
Nottingham Music Database (2003) |  Special Text Format Time Series |  1000 |  -

##  4 Taxonomy of Time Series based GANs

We propose a taxonomy of the following time series based GANs based on two
distinct variant types: discrete variants (discrete time series) and
continuous variants (continuous time series). A discrete time series consists
of data points separated by time intervals. This type of data might have 1. a
data-reporting interval that is infrequent (e.g., 1 point per minute) or
irregular (e.g., whenever a user logs in) and 2. gaps where values are missing
due to reporting interruptions (e.g., intermittent server or network downtime
in a network traffic application). Discrete time series generation involves
generating sequences that may have a temporal dependency but contain discrete
tokens; these can be commonly found in electronic health records
(International Classification of Diseases 9 codes) and text generation. A
continuous time series has a data value corresponding to every moment in time.
Continuous data generation is concerned with generating a real-valued signal x
with temporal dependencies where x $\in\mathbb{R}$. See Figure 3 for examples
of discrete and continuous time series signals.

Figure 3: Example plots of discrete (left) and continuous time series (right).

Challenges with discrete time series generation. GANs struggle with discrete
data generation due to the zero gradient nearly everywhere, i.e., the
distribution on discrete objects are not differentiable with respect to their
parameters [34, 35]. This limitation makes the generator untrainable using
backpropagation alone. The generator starts with a random sampling and a
deterministic transform guided via the gradient of the loss from the
discriminator with respect to the output produced by $G$ and the training
dataset. This loss leads to a slight change in $G$’s output, pushing it closer
to the desired output. Making slight changes to continuous numbers make sense;
adding 0.001 to a value of 10 in financial time series data will bring it to
10.001. However, a discrete token such as the word ‘penguin’ cannot simply
undergo the addition of 0.001 as the sum ‘penguin+0.001’ makes no sense.
What’s important here is the impossibility for the generator to jump from one
discrete token to the next because the small change gives the token a new
value that does not correspond to any other token over that limited discrete
space [36]. This is because there exists 0 probability in the space between
these tokens, unlike with continuous data.

Challenges with continuous time series generation. Modelling continuous time
series data presents a different problem for GANs, which are inherently
designed to model continuous data, albeit most commonly in the form of images.
The temporal nature of continuous data in time series presents an extra layer
of difficulty. Complex correlations exist between the temporal features and
their attributes, e.g., if using multichannel biometric/physiological data,
the ECG characteristics will depend on the individual’s age and/or health.
Also, long-term correlations exist in the data, which are not necessarily
fixed in dimension compared to image-based data under a fixed dimension.
Transforming image dimensions may lead to a degradation in image quality, but
it is a recognised practice. This operation becomes more difficult with
continuous time series data as there is no standardised dimension used across
time series GANs architectures, which means that benchmarking their
performances becomes difficult.

Since their inception in 2014, GANs have shown great success in generating
high-quality synthetic images indistinguishable from real images [37, 38, 39].
While the focus to date has been on developing GANs for improved media
generation, there is a growing consensus that GANs can be used for more than
image generation and manipulation, which has led to a movement towards
generating time series data with GANs.

Recurrent neural networks (RNNs) (Figure 4), due to their loop-like structure,
are perfect for sequential data applications but by themselves lack the
ability to learn long-term dependencies that might be crucial in forecasting
future values based on past. Long-short Term Memory networks (LSTM) (Figure 4
are a specific kind of RNN that have the ability to remember information for
long periods of time and, in turn, learn these long-term dependencies that the
standard RNN is not capable of doing. In most work reviewed in this paper, the
majority of the RNN based architectures are utilising the LSTM cell.

Figure 4: Block Diagram of (left) a standard RNN and (right) LSTM cell

RNNs can model sequential data such as financial data, medical data, text, and
speech, and they have been the foundational architecture for time series GANs.
A recurrent GAN (RGAN) was first proposed in 2016. The generator contained a
recurrent feedback loop that used both the input and hidden states at each
time step to generate the final output [40]. Recurrent GANs often utilise Long
Short-Term Memory neural networks in their generative models to avoid the
vanishing gradient problem associated with more traditional recurrent networks
[41]. In the section that follows, we chronologically present time series GANs
that have either contributed significantly to this space or have made some of
the most recent novel advancements.

###  4.1 Discrete-variant GANs

####  4.1.1 Sequence GAN (SeqGAN) (Sept. 2016)

Yu et al. proposed a sequential data generation framework [34] that could
address the issues with generating discrete data as previously mentioned in 4.
This approach outperformed previous methods for generative modelling on real-
world tasks, including; a maximum likelihood estimation (MLE) trained LSTM,
scheduled sampling [42], and Policy Gradient with bilingual evaluation
understudy (PG-BLEU) [43]. SeqGAN’s generative model comprises RNNs with LSTM
cells, and its discriminative model is a convolutional neural network (CNN).
Given a dataset of structured sequences the authors train $G$ to produce a
synthetic sequence $Y_{1:T}=(y_{1}...,y_{t}...,y_{T}),y_{t}\in\mathcal{Y}$
where $\mathcal{Y}$ is defined as the vocabulary of candidate tokens. $G$ is
updated by a policy gradient and Monte Carlo (MC) search on the expected
reward from $D$, see Figure 5. The authors used two datasets for their
experiments. A Chinese poem dataset [44] and a Barack Obama Speech dataset
[45] with Adam optimisers and a batch size of 64. Their experiments are
available online111SeqGAN GitHub: https://github.com/LantaoYu/SeqGAN/.

Although the purpose of SeqGAN is to generate discrete sequential data, it
opened the door to other GANs in generating continuous sequential and time
series data. The authors use a synthetic dataset whose distribution is
generated from a randomly initialised LSTM following a normal distribution.
They also compare the generated data to real-world examples of poems, speech-
language and music. SeqGAN showed competitive performance in generating the
sequences and contributed heavily towards the further development of the
continuous sequential GANs.

Figure 5: SeqGAN: D (left) is trained over real and generated data, whereas G
(right) is trained by policy gradient where the final reward signal is
provided by D and is passed back to the intermediate action value via Monte
Carlo search [34].

####  4.1.2 Quant GAN (Jul. 2019)

Quant GAN is a data-driven model that aims to capture long-range dependencies
in financial time series data such as volatility clusters. Both the generator
and discriminator use Temporal Convolutional Networks (TCN) with skip
connections [46] which are essentially dilated causal convolutional networks.
They have the advantage of being suited to model long-range dependencies in
continuous sequential data. The generator function is a novel stochastic
volatility NN (SVNN) that consists of a volatility and drift TCN. Temporal
blocks are the modules used in the TCN that consist of two dilated causal
convolutions layers (Figure 6) and two Parametric Rectified Linear Units
(PReLU) as activation functions. Data generated by $G$ is passed to $D$ to
produce outputs, which can then be averaged to give an MC estimate of $D$’s
loss function. The authors used a dataset of daily spot-prices of the S&P 500
from May 2009 until December 2018.

Figure 6: Dilated causal convolutional layer.

The authors aim to capture long-range dependencies in financial time series;
however, modelling the series in continuous time over these long time frames
would blow up the models’ computational complexity. Therefore, this method
models the time series in discrete time. The authors report that this approach
is capable of outperforming more conventional models from mathematical finance
(Constrained SVNN and generalised autoregressive conditional
heteroskedasticity (GARCH) [47]) but state that there remain issues that need
to be resolved for this approach to become widely adopted. One such issue
concerns the need for a unified metric for quantifying the performance of
these GANs, a point we discuss further in Section 6.

###  4.2 Continuous-variant GANs

####  4.2.1 Continuous RNN-GAN (C-RNN-GAN) (Nov. 2016)

In previous works, RNNs have been applied to modelling music but have
generally used a symbolic representation to model this type of sequential
data. Mogren proposed the C-RNN-GAN (Figure 7), one of the first examples of
using GANs to generate continuous sequential data[48]. The generator is an
RNN, and the discriminator a bidirectional RNN, which allows the discriminator
to take the sequence context in both directions. The RNNs used in this work
were two stacked LSTMs layers, with each cell containing 350 hidden units. The
loss functions can be seen in (4,5), where $z^{(i)}$ is a sequence of uniform
random vectors in [0,1]k, and $x^{(i)}$ is a sequence from the training data.
$k$ is the dimensionality of the data in the random sequence.

Figure 7: Structure of C-RNN-GAN’s generator and discriminator.

${L}_{G}=\frac{1}{m}\sum_{i=1}^{m}log(1-D(G(z^{(i)})))$ (4)

${L}_{D}=\frac{1}{m}\sum_{i=1}^{m}[-logD(x^{(i)})-log(1-D(G(z^{(i)})))]$ (5)

The C-RNN-GAN is trained with backpropagation through time (BPTT) and mini-
batch stochastic gradient descent with L2 regularisation on the weights of
both $G$ and $D$. Freezing was applied to both $G$ and $D$ when one network
becomes too strong relative to the other. The dataset used was 3697 midi files
from 160 different composers of classical music with a batch size of 20. Adam
and Gradient Descent Optimisers were used during training; full implementation
details are available online222C-RNN-GAN GitHub:
https://github.com/olofmogren/c-rnn-gan/. Overall the C-RNN-GAN was capable of
learning the characteristics of continuous sequential data and, in turn,
generate music. However, the author stated that their approach still needs
work, particularly in rigorous evaluation of the generated data quality.

####  4.2.2 Recurrent Conditional GAN (RCGAN) (2017)

RCGAN for continuous data generation [23] differs architecturally from the
C-RNN-GAN. Although the RNN LSTM is used, the discriminator is unidirectional,
and the outputs of $G$ are not fed back as inputs at the next time step. There
is also additional information that the model is conditioned on, which makes
for a conditional RGAN; see the layout of the model in Figure 8. The purpose
of the RCGAN and RGAN in this work is to generate continuous time series with
a focus on medical data intended for use in downstream tasks, and this was one
of the first works in this area. The loss functions can be seen in Equations
(6, 7) where CE is the average cross-entropy between two sequences. $X_{n}$
are samples drawn from the training dataset. $y_{n}$ is the adversarial ground
truth; for real sequences, it is a vector of 1s, and conversely, for generated
or synthetic sequences, it is a vector of 0s. $Z_{n}$ is a sequence of points
sampled from the latent space, and the valid adversarial ground truth is
written here as 1.

Figure 8: RCGAN architecture with conditional input c, input data x and latent
variable z.

${D}_{loss}(X_{n},y_{n})=-CE(D(X_{n}),y_{n})$ (6)

${G}_{loss}(Z_{n})=D_{loss}(G(Z_{n}),\textbf{1})=-CE(D(G(Z_{n})),\textbf{1})$
(7)

In the conditional case, the inputs to $D$ and $G$ are concatenated with some
conditional information $c_{n}$. This variant of an RNN-GAN facilitates the
generation of a synthetic continuous time series dataset with associated
labels. Experiments were carried out on generated sine waves, smooth functions
sampled from a Gaussian process with a zero-valued mean function, MNIST
dataset as a sequence, and the Philips eICU database [49]. A batch size of 28
with Adam and Gradient Descent Optimisers were used for training. The authors
propose a novel method for evaluating their model, which is discussed further
in Section 6. Full experimental details can be found online333RCGAN GitHub:
https://github.com/ratschlab/RGAN/.

####  4.2.3 Sequentially Coupled GAN (SC-GAN) (Apr. 2019)

SC-GAN aims to generate patient-centric medical data to inform of a patient’s
current state and generate a recommended medication dosage based on the state
[50]. It consists of two coupled generators tasked with producing two
outcomes, one for the current state of an individual and the other for a
recommended medication dosage based on the individual’s state. The
discriminator is a two-layer bidirectional LSTM, and the coupled generators
are both two-layer unidirectional LSTMs. See Figure 9 for further details of
the architecture.

Figure 9: SC-GAN architecture.

$G_{1}$ generates the recommended medication dosage data
($\textbf{a}_{1},\textbf{a}_{2},...,\textbf{a}_{T}$) with a combined input of
the sequential continuous patient state data
($\textbf{s}_{0},\textbf{s}_{1},...,\textbf{s}_{T-1}$) and a random noise
sequence
($\hat{\textbf{z}}_{0}^{a},\hat{\textbf{z}}_{1}^{a},...,\hat{\textbf{z}}_{T-1}^{a}$)
sampled from a uniform distribution. At each time step $t$ the input
$\textbf{z}_{t}^{a}$ of $G_{1}$ is the concatenation of $\textbf{s}_{t}$ and
$\hat{\textbf{z}}_{t}^{a}$.

Conversely, $G_{2}$ is tasked with generating the patient state data s_t and
at each time step the input $\textbf{z}_{t}^{s}$ is the concatenation of
$\textbf{s}_{t-1}$, $\textbf{a}_{t-1}$ and $\hat{\textbf{z}}_{t}^{s}$. This
means that the outputs from $G_{1}$ and $G_{2}$ are the inputs to one another.
Combining the generators together leads to the following loss function:

$L_{G}=\frac{1}{N}\frac{1}{T}\sum_{i=1}^{N}\sum_{t=1}^{T}log(1-D(G(\textbf{z}_{i,t})))$
(8)

$G(\textbf{z}_{i,t})=[G_{1}(\textbf{z}_{i,t}^{a});G_{2}(\textbf{z}_{i,t}^{s})]$
(9)

Where N is the number of patients and T is the time length of the patient
record. The SC-GAN has a supervised pretraining step for the generators to
avoid an excessively strong $D$ that uses the least-squares loss.

The discriminator is tasked with classifying the sequential patient-centric
records as real or synthetic, and the loss function is defined as:

$L_{D}=-\frac{1}{N}\frac{1}{T}\sum_{i=1}^{N}\sum_{t=1}^{T}(logD(\textbf{x}_{i,t})+log(1-D(G(\textbf{z}_{i,t}))))$
(10)

where $\textbf{x}_{i,t}=[\textbf{s}_{t};\textbf{a}_{t}]$. This model contains
novel coupled generators that coordinate to generate patient state and
medication dosage data. It has performance close to real data for the
treatment recommendation task. The dataset used in this experiment is MIMIC-
III [51]. The authors benchmark their SC-GAN against variants of SeqGAN,
C-RNN-GAN, and RCGAN and observe their model to be the best performing for
this specific use case.

###  4.3 Noise Reduction GAN (NR-GAN) (Oct. 2019)

NR-GAN is designed for noise reduction in continuous time series signals but
more specifically has been implemented for noise reduction in mice
electroencephalogram (EEG) signals [52]. This dataset was provided by the
International Institute for Integrative Sleep Medicine (IIIS). EEG is the
measure of the brain’s electrical activity, and it commonly contains
significant noise artefact. NR-GAN’s core idea is to reduce or remove the
noise present in the frequency domain representation of an EEG signal. The
architecture of $G$ is a two-layer 1-D CNN with a fully connected layer at the
output. $D$ contains almost the same two-layer 1-D CNN structure with the
fully-connected layer replaced by a softmax layer to calculate the probability
the input belongs to the training set. The loss functions are defined in
(11,12) as:

${G}_{loss}=\sum_{x\in S_{ns}}[log(1-D(G(x)))+\alpha\|x-G(x)\|^{2}]$ (11)

${D}_{loss}=\sum_{x\in S_{ns}}[log(D(G(x)))]+\sum_{y\in S_{cs}}[log(1-D(y))]$
(12)

where $S_{ns}$ and $S_{cs}$ are the noisy and clear EEG signals, respectively.
$\alpha$ is a hyperparameter that essentially controls the aggressiveness of
noise reduction; the authors chose a value of $\alpha=0.0001$.

For this work, the generator does not sample from a latent space; rather, it
attempts to generate the clear signal from the noisy EEG signal input, see
Figure 10. The authors found that the NR-GAN is competitive with classical
frequency filters in terms of noise reduction. They also state that the
experimental conditions may favour the NR-GAN and list some limitations in
terms of the amount of noise NR-GAN can handle and the influence of $\alpha$.
However, this is a novel method for noise reduction in continuous sequential
data using GANs.

Figure 10: NR-GAN architecture with noisy EEG input ${S_{ns}}$, clean input
data ${S_{cs}}$.

####  4.3.1 Time GAN (Dec. 2019)

TimeGAN provides a framework that utilises both the conventional unsupervised
GAN training method and the more controllable supervised learning approach
[21]. By combining an unsupervised GAN network with a supervised
autoregressive model, the network aims to generate time series with preserved
temporal dynamics. The architecture of the TimeGAN framework is illustrated in
Figure 11. The input to the framework is considered to consist of two
elements, a static feature and a temporal feature. s represents a vector of
static features and x of temporal features at the input to the encoder. The
generator takes a tuple of static and temporal random feature vectors drawn
from a known distribution. The real and synthetic latent codes h and
$\hat{\textbf{h}}$ are used to calculate the supervised loss element of this
network. The discriminator receives the tuple of real and synthetic latent
codes and classifies them as either real ($y$) or synthetic ($\hat{y}$), the
$\tilde{}$ operator denotes the sample as either real or fake.

Figure 11: TimeGAN architecture.

The three losses used in TimeGAN are calculated as follows:

${L}_{reconstruction}=\mathbb{E}_{s,x_{1:T}\sim
p}[\|s-\tilde{s}\|_{2}+\sum_{t}\|x_{t}-\tilde{x}_{t}\|_{2}]$ (13)

${L}_{unsupervised}=\mathbb{E}_{s,x_{1:T}\sim
p}[log(y_{S})+\sum_{t}log(y_{t})]+\mathbb{E}_{s,x_{1:T}\sim{\hat{p}}}[log(1-\hat{y}_{S})+\sum_{t}log(1-\hat{y}_{t})]$
(14)

${L}_{supervised}=\mathbb{E}_{s,x_{1:T}\sim
p}[\sum_{t}\|h_{t}-g_{X}(h_{S},h_{t-1},z_{t})\|_{2}]$ (15)

The creators of TimeGAN conducted experiments on generating sine waves, stocks
(daily historical Google stocks data from 2004 to 2019), energy (UCI
Appliances energy prediction dataset) [33], and event (private lung cancer
pathways dataset) datasets. A batch size of 128 and Adam optimiser were used
for training, implementation details are available online444TimeGAN GitHub:
https://github.com/jsyoon0823/TimeGAN. The authors demonstrated improvements
over other state-of-the-art time series GANs such as RCGAN C-RNN-GAN and
WaveGAN.

####  4.3.2 Conditional Sig-Wasserstein GAN (SigCWGAN) (Jun. 2020)

A problem addressed by [53] is that long time series data streams can greatly
increase the dimensionality requirements of generative modelling, which may
render such approaches infeasible. To counter this problem, the authors
develop a metric named Signature Wasserstein-1 (Sig-$W_{1}$) that captures
time series models’ temporal dependency and uses it as a discriminator in a
time series GAN. It provides an abstract and universal description of complex
data streams and does not require costly computation like the Wasserstein
metric. A novel generator is also presented that is named conditional
autoregressive feed-forward neural network (AR-FNN) that captures the auto-
regressive nature of the time series. The generator is capable of mapping past
series and noise into future series. For a rigorous mathematical description
of their method, the interested reader should consult [53].

For the AR-FNN generator the idea is to obtain the step-q estimator
$\hat{X}^{(t)}_{t+1:t+q}$. The loss function for $D$ is defined as:

$L(\theta)=\sum_{t}|\mathbb{E}_{\mu}[S_{M}(X_{t+1:t+q})|X_{t-p+1:t}]-\mathbb{E}_{v}[S_{M}(\hat{X}_{t+1:t+q}^{(t)})|X_{t-p+1:t}]|$
(16)

Where $v$ and $\mu$ are the conditional distributions induced by the real data
and synthetic generator, respectively, further details of the author’s
algorithm can be found in the appendix of the original paper. The authors
state that SigCWGAN eliminates the problem of approximating a costly $D$ and
simplifies training. It is reported to achieve state-of-the-art results on
synthetic and empirical datasets compared to TimeGAN, RCGAN and Generative
Moment Matching Networks (GMMN) [54]. The empirical dataset consists of the
S&P 500 index (SPX) and Dow Jones index (DJI) and their realized volatility,
which is retrieved from the Oxford-Man Institute’s "realised library" [55]. A
batch size of 200 with the Adam optimiser was used for training555SigCWGAN
GitHub: https://github.com/SigCGANs/Conditional-Sig-Wasserstein-GANs/.

####  4.3.3 Decision Aware Time series conditional GAN (DAT-CGAN) (Sept. 2020)

This framework is designed to provide support for end-users decision
processes, specifically in financial portfolio choices. It uses a multi-
Wasserstein loss on structured decision-related quantities [56]. The
discriminator loss and generator loss are defined in Equations (17) and (18)
respectively. For further details on the loss functions, see Section 3 of the
original paper.

$\mathop{inf}_{\eta}\mathop{sup}_{\gamma_{k},\theta_{j,k}}\sum_{k=1}^{K}\omega_{k}(\mathbb{E}_{k}^{r}-\mathbb{E}_{k}^{G_{\eta}})+\sum_{k=1}^{K}\sum_{j=1}^{J}\lambda_{j,k}(\mathbb{E}_{j,k}^{f,R}-\mathbb{E}_{j,k}^{f,G_{\eta}})$
(17)

$\mathop{inf}_{\eta}-\sum_{k}\omega_{k}\mathbb{E}_{k}^{G_{\eta}}-\sum_{k,j}\lambda_{j,k}\mathbb{E}_{j,k}^{f,G_{\eta}}$
(18)

The generator is a two-layer feed-forward neural network for each input which
are assets in this case. $G$ outputs asset returns that are used to compute
decision-related quantities. These quantities are fed into $D$, which is also
a two-layer feed-forward NN. Further details about the architecture can be
found in the appendix of [56]. The dataset used is daily price data for each
of four U.S. Exchange-traded fund (ETFs), i.e. Material (XLB), Energy (XLE),
Financial (XLF) and Industrial (XLI) ETFs, from 1999 to 2016. The authors
found this model capable of high-fidelity time series generation that supports
decision processes by end-users due to incorporating a decision-aware loss
function. However, this approach’s limitation is that the computational
complexity of this model is vast and requires one month of training time for a
single generative model.

###  4.4 Synthetic biomedical Signals GAN (SynSigGAN) (Dec. 2020)

SynSigGAN is designed to generate different kinds of continuous
physiological/biomedical signal data [57]. It is capable of generating
electrocardiogram (ECG), electroencephalogram (EEG), electromyography (EMG),
and photoplethysmography (PPG) from MIT-BIH Arrhythmia database [58], Siena
Scalp EEG database [59] and BIDMC PPG and Respiration dataset [60]. A novel
GAN architecture is proposed here that uses a bidirectional grid long short
term memory (BiGridLSTM) for the generator (Figure 12) and a CNN for the
discriminator. The BiGridLSTM is a combination of a double GridLSTM (a version
of LSTM that can represent the LSTMs in a multidimensional grid) with two
directions that can combat the gradient phenomenon from two dimensions and
found to work well in time sequence problems. The authors used the value
function defined previously in Equation (1).

Figure 12: Architecture of BiGridLSTM with LSTM blocks for the time and depth
dimension, $`$ symbol indicates reverse in the figure as in [61].

SynSigGAN is capable of capturing the different physiological characteristics
associated with each of these signal types and has demonstrated an ability to
generate biomedical time series data with a max sequence length of 191 data
points. The authors also present a preprocessing stage to clean and refine the
biomedical signals in this paper. They compare their architecture to several
variants (BiLSTM-GRU, BiLSTM-CNN GAN, RNN-AE GAN, Bi-RNN, LSTM-AE, BiLSTM-MLP,
LSTM-VAE GAN, and RNN-VAE GAN) and found the BiGrid-LSTM as the best
performing model.

##  5 Applications

###  5.1 Data Augmentation

It is common knowledge in the deep learning community that GANs are among the
methods of choice when discussing data augmentation. Reasons for augmenting
datasets range from increasing the size/variety of small and imbalanced
datasets [62, 63, 64, 65] to reproducing restricted datasets for
dissemination.

A well-defined solution to the data shortage problem is transfer learning, and
it works well in domain adaptation which has led to advancements in
classification and recognition problems [66]. However, it has been found that
augmenting datasets with GANs can lead to further improvements in certain
classification and recognition tasks [67]. Data synthesised by a GAN can
adhere to stricter privacy measures discussed in Section 7. This further
demonstrates the advantages of augmenting your training dataset with GANs over
implementing transfer learning with a pre-trained model from a different
domain on a smaller dataset.

Many researchers find that accessing datasets for their deep learning research
and models to be time-consuming, laborious work, particularly when the
research is concerned with personal sensitive data. Often medical and clinical
data are presented as continuous sequential data that can only be accessed by
a small contingent of researchers who are not at liberty to disseminate their
research openly. This, in turn, may lead to stagnation in the research
progress in these domains.

Fortunately, we are beginning to see the uptake of GANs applied to time series
with these types of medical and physiological data [23, 57, 68, 69, 70]. With
[71] showing dependent multivariate continuous high-fidelity physiological
signal generation is capable via GANs, demonstrating the impressive capability
of these networks. See Figure 13 for an example of the real input and
synthetic generated data.

Figure 13: An example of dependent multichannel ECG data (left) and generated
ECG from a multivariate GAN (right) [71]. NSR indicates the training dataset
which is the Normal Sinus Rhythm. The generated data is produced by a GAN
named by the authors as LSGAN-DTW.

Of course, this is not comprehensive coverage of the research using time
series GANs for data synthesis and augmentation. GANs have been applied to
time series data for a plethora of use cases.

Audio generation (both music and speech) and text-to-speech (TTS) [72] has
been a popular area for researchers to explore with GANs. The C-RNN-GAN
described in Section 4.2.1 was one of the seminal works to apply GANs to
generating continuous sequential data in the form of music.

In the financial sector, GANs have been implemented to generate data and
predict/forecast values. Wiese et al. implemented a GAN to approximate
financial time series in discrete-time [46]. In [53], the authors designed a
decision-aware GAN that generates synthetic data and supports decision
processes to financial portfolio selection of end-users.

Other time series generation/prediction methods range from estimating soil
temperature [73] to predicting medicine expenditure based on the current state
of patients [74].

###  5.2 Imputation

In real-world datasets, missing or corrupt data is an all too common problem
that leads to downstream problems. These issues manifest themselves in further
analytics of the dataset and can induce biases in the data. Common methods of
dealing with missing or corrupted data in the past have been the deletion of
data streams containing the missing segments, statistical modelling of the
data, or machine learning imputation approaches. Looking at the latter, we
review the work in imputing these data using GANs. Guo et al. designed a GAN
based approach for multivariate time series imputation [75], see Figure 14 for
an example of imputed data from a toy experiment [70].

Figure 14: An example of the incomplete corrupted time series (top) and
imputed signal (bottom).

###  5.3 Denoising

Artefacts induced in time series data often manifest themselves as noise in
the signals. This has become an ever-present challenge in further processing
and analytical applications. Corrupted data can cause biases in the datasets
or lead to degradation in the performance of critical systems such as those
used for health applications. Common methods for dealing with noise include
the use of adaptive linear filtering. Another approach recently explored in
[52] used GANs as a noise-reduction technique in EEG data. Their experiments
showed that their proposed NR-GAN (Section 4.3) was capable of competitive
noise reduction performance compared to more traditional frequency filters.

###  5.4 Anomaly Detection

Detecting outliers or anomalies in time series data is an important part of
many real-world systems and sectors. Whether it is detecting unusual patterns
in physiological data that may be a precursor to some more malicious condition
or detecting irregular trading patterns on the stock exchange, anomaly
detecting can be vital to keeping us informed on important information.
Statistical measures of non-stationary time series signals may achieve good
performance on the surface, but they might also miss some important outliers
present in deeper features. They may also struggle in exploiting large
unlabelled datasets; this is where the unsupervised deep learning approaches
can outperform the conventional methods. Zhu et al. designed a GAN algorithm
for anomaly detection in time series data (ECG and taxi dataset) with LSTMs
and GANs, which achieved superior performance compared to conventional, more
shallow approaches. Similar approaches have been applied to detect
cardiovascular diseases [76], in cyber-physical systems to detect nefarious
players [77] and even irregular behaviours such as stock manipulation on the
stock markets [78].

###  5.5 Other Applications

Some works have utilised image-based GANs for time series and sequential data
generation by first converting their sequences to images via some
transformation function and training the GAN on these images. Once the GAN
converges, similar images can be generated; then, a sequence can be retrieved
using the inverse of the original transformation function. For example, this
approach has been implemented in audio generation with waveforms [79, 80, 81],
anomaly detection [82] and physiological time series generation [70].

##  6 Evaluation Metrics

As mentioned in Section 3 GANs can be difficult to evaluate, and researchers
are yet to agree on what metrics reflect the GANs performance best. There has
been plenty of metrics proposed in the literature [2] with most of them suited
to the computer vision domain. Work is still ongoing to suitably evaluate time
series GANs. We can break down evaluation metrics into two categories:
qualitative and quantitative. Qualitative evaluation is another term for human
visual assessment via the inspection of generated samples from the GAN.
However, this cannot be deemed a full evaluation of GAN performance due to the
lack of a suitable objective evaluation metric. The quantitative evaluation
includes the use of metrics associated with statistical measures used for time
series analytics and similarity measures such as; Pearson Correlation
Coefficient (PCC), percent root mean square difference (PRD), (Root) Mean
Squared Error MSE and RMSE, Mean Relative Error (MRE) and Mean Absolute Error
(MAE). These metrics are among the most commonly used for time series
evaluation and, as such, used as a suitable GAN performance metric as they can
reflect the stability between the training data and synthetic generated data.

Several metrics have become well-established choices in evaluating image-based
GANs, and some of these have permeated through to the sequential and time
series GANs such as Inception Score (IS) [17], Fréchet (Inception) Distance
(FD and FID) [18]. Structural Similarity Index (SSIM) is a measure of
similarity between two images. However, [83] use this with time series data as
SSIM does not exclude itself from comparing aligned sequences of fixed length.
Of course, some of these metrics are measures of similarities/dissimilarities
between two probability distributions, suitable for many types of data,
particularly the maximum mean discrepancy (MMD) [84] is very suitable to this
task across domains. Another metric that generalises well to the sequential
data case is the Sliced-Wasserstein Distance approximating the Wasserstein
distance by computing Wasserstein distances between all 1d-projections of two
distributions.

The data generated from GANs have been used in downstream classification
tasks. Using the generated data together with the training data has lead to
the Train on Synthetic, Test on Real (TSTR) and Train on Real, Test on
Synthetic (TRTS) evaluation methods, first proposed by Esteban et al. [23]. In
scoring downstream classification applications that use both real and
generated data, studies have adopted the precision, recall, and F1 scores to
determine the classifier’s quality and, in turn, the quality of the generated
data. Other accuracy measures of classifier performance include the weighted
accuracy (WA) and unweighted average recall (UAR).

Often used distance and similarity measures in time series data are the
Euclidean Distance (ED) and Dynamic Time Warping (DTW) algorithms.
Multivariate (in)dependent DTW (MVDTW), implemented in [71], can determine
similarity measures across both dependent and independent multichannel time
series signals.

Other metrics used across different applications include:

  * •

Financial Sector; autocorrelation function (ACF) score, DY metric.

  * •

Temperature Estimation; Nash-Sutcliffe model efficiency coefficient (NS),
Willmott index of agreement (WI) and the Legates and McCabe index (LMI).

  * •

Audio Generation; Normalised Source-to-Distortion Ratio (NSDR), Source-to
Interference Ratio (SIR), Source-to-artifact ratio (SAR) and t-SNE [19].

For a full list of GAN architectures reviewed in this work, their
applications, evaluation metrics, and datasets used in their respective
experiments, see Table 2. Results for the sine wave and ECG generation using
variants of GAN architectures can be found in Tables 3 and 4, respectively.

Table 2: A list of GAN architectures, their applications, and datasets used in their experiments and evaluation metrics used to judge the quality of the respective GANs. For novel approaches, the GAN name is given as they have been covered already in Section 4 Application |  GAN Architecture(s) |  Dataset(s) |  Evaluation Metrics
---|---|---|---
Medical/Physiological Generation |  LSTM-LSTM, [23] [62] [63], [64], [85], [50] LSTM-CNN, [71] [68] BiLSTM-CNN, [69] BiGridLSTM-CNN, [57] CNN-CNN, [86], [87] AE-CNN, [88] FCNN [89] |  EEG, ECG, EHRs, PPG, EMG, Speech, NAF, MNIST, Synthetic sets |  TSTR, MMD, Reconstruction error, DTW, PCC, IS, FID, ED, S-WD, RMSE, MAE, FD, PRD, Averaging Samples, WA, UAR, MV-DTW
Financial time series generation/prediction |  TimeGAN [21] SigCWGAN [53] DAT-GAN [56] QuantGAN [46] |  S&P500 index (SPX), Dow Jones Index (DJI), ETFs |  Marginal Distributions, Dependencies, TSTR, Wasserstein Distance, EM distance, DY Metric, ACF score, leverage effect score, discriminative score, predictive score
Time series Estimation/Prediction |  LSTM-NN [73] LSTM-CNN [74] LSTM-MLP [74] |  Meteorological data, Truven MarketScan dataset |  RMSE, MAE, NS, WI, LMI
Audio Generation |  C-RNN-GAN [48] TGAN (variant) [81] RNN-FCN [90] DCGAN (variant) [80] CNN-CNN [72] |  Nottingham dataset, Midi music files, MIR-1K, TheSession, Speech |  Human perception, Polyphony, Scale Consistency, Tone Span, Repetitions, NSDR, SIR, SAR, FD, t-SNE, Distribution of notes
Time series Imputation/Repairing |  MTS-GAN [75] CNN-CNN [91] DCGAN variant [92] AE-GRUI [93] RGAN [94] FCN-FCN [95] GRUI-GRUI [96] |  TEP, Point Machine, Wind Turbin data, PeMS, PhysioNet Challenge 2012, KDD CUP 2018, Parking lot data, |  Visually, MMD, MAE, MSE, RMSE, MRE, Spatial Similarity, AUC score
Anomaly Detection |  LSTM-LSTM [78] LSTM-(LSTM&CNN) [97] LSTM-LSTM (MAD-GAN) [77] |  SET50, NYC Taxi data, ECG, SWaT, WADI |  Manipulated data used as a test set, ROC Curve, Precision, Recall, F1, Accuracy
Other time series generation |  VAE-CNN [83] |  Fixed length time series ’vehicle and engine speed’ |  DTW, SSIM
Table 3: Experimental results comparing the performance of time series GANs for sinewave generation Architecture | Loss Function | Toy Sine Dataset
---|---|---
MMD | DTW | MSE
LSTM-LSTM | BCE | – | – | –
MSE | 0.007805404 | 54.16447988 | 0.148043822
BiLSTM-LSTM | BCE | 0.121597451 | 428.43108 | 3.070050828
MSE | 0.951539381 | 79.56070154 | 0.236244962
LSTM-CNN | BCE | 0.006319945 | 55.36204204 | 0.31545636
MSE | 0.575763914 | 86.73579748 | 0.564357309
BiLSTM-CNN | BCE | 1.12958E-05 | 129.9257789 | 0.919376137
MSE | 0.489165394 | 43.26942611 | 0.186902029
GRU-CNN | BCE | 0.024468314 | 37.1630491 | 0.230310881
MSE | 0.372734849 | 42.7348549 | 0.228260543
FC-CNN | BCE | 0.003933644 | 58.35650673 | 0.304803267
MSE | 0.011720286 | 43.36115622 | 0.297288744
Table 4: Experimental results comparing the performance of time series GANs for ECG generation on MIT-BIH Dataset Architecture | Loss Function | MIT-BIH Arrhythmia Dataset
---|---|---
MMD | DTW | MSE
LSTM-LSTM | BCE | 0.993149132 | 30.18161173 | 0.086768344
MSE | 0.8842204 | 44.45535914 | 0.138977587
BiLSTM-LSTM | BCE | 0.991669978 | 22.86345584 | 0.069901973
MSE | 0.973720273 | 23.55338573 | 0.080641843
LSTM-CNN | BCE | 0.551958508 | 13.0158286 | 0.015101118
MSE | 0.0005273 | 24.73064185 | 0.045702711
BiLSTM-CNN | BCE | 0.924689511 | 117.3994295 | 0.227219626
MSE | 0.068707491 | 22.67402118 | 0.058676694
GRU-CNN | BCE | 0.005577049 | 20.48459161 | 0.033519309
MSE | 0.770406707 | 108.4124982 | 0.194816291
FC-CNN | BCE | 0.206880394 | 23.99100792 | 0.030977119
MSE | 0.308249889 | 18.23405152 | 0.021292498

##  7 Privacy

As well as evaluating the quality of the data, a wide range of methods have
been used to evaluate and mitigate the privacy risk associated with synthetic
data created by GANs.

###  7.1 Differential Privacy

The goal of Differential Privacy is to preserve the underlying privacy of a
database. An algorithm or, more specifically, a GAN achieves differential
privacy if, by looking at the generated samples, we cannot identify whether
the samples were included in the training set. As GANs attempts to model the
training dataset, the problem of privacy lies in capturing and generating
useful information about the training set population without the possibility
of linkage from generated sample to an individuals data [98].

As we have previously addressed, one of the main goals of GANs is to augment
existing under-resourced datasets for use in further downstream applications
such as upskilling of clinicians where healthcare data is involved. These
personal sensitive data must contain privacy guarantees, and the rigorous
mathematical definition of DP [99] offers this assurance.

Work is ongoing to develop machine learning methods with privacy-preserving
mechanisms such as differential privacy (DP). Abadi et al. demonstrate the
ability to train deep neural networks with DP and implement a mechanism for
tracking privacy loss [100]. Xie et al. proposed a differentially private GAN
(DPGAN) that achieved differential privacy by adding noise gradients to the
optimiser during the training phase [101].

###  7.2 Decentralised/Federated Learning

Distributed or decentralised learning is another method for limiting the
privacy risk associated with personal and personal sensitive data in machine
learning. Standard approaches to machine learning require that all training
data be kept on one server. Decentralised/distributed approaches to GAN
algorithms require large communication bandwidth to ensure convergence [102,
103] and are also subject to strict privacy constraints. A new method that
enables communication efficient collaborative learning on a shared model while
keeping all the training data decentralised is known as Federated learning
[104]. Rasouli et al. applied federated learning algorithm to a GAN for
communication efficient distributed learning and proved the convergence of
their federated learning GAN (FedGAN) [105]. However, it should be noted that
they did not experiment with differential privacy in this study but note it as
an avenue of future work.

Combining the above techniques of federated learning and differential privacy
in developing new GAN algorithms would lead to a fully decentralised private
GAN capable of generating data without leakage of private information to the
source data. This is clearly an open research avenue for the community.

###  7.3 Assessment of Privacy Preservation

We can also assess how well the generative model was able to protect our
privacy through tests known as attribute and presence disclosure [22]. The
latter test is more commonly known in the machine learning space as a
membership inference attack. This has become a quantitative assessment of how
machine learning models leak information about the individual data records on
which they were trained [106].

Hayes et al. carried out membership inference attacks on synthetic images and
concluded that for acceptable levels of privacy in the GAN, the quality of the
data generated is sacrificed [107]. Conversely, others have followed this
approach and found that DP networks can successfully generate data that
adheres to differential privacy and resists membership inference attacks
without too much degradation in the quality of the generated data [23, 68,
71].

##  8 Discussion

We have presented a survey of time series GAN-variants that have made
significant progress in addressing the primary challenges identified in
Section 3.2. These GANs introduced the idea of both discrete and continuous
sequential data generation and have made incremental improvements over one
another via an architecture variant or a modified objective function capable
of capturing the spatio-temporal dependencies present in these data types. The
loss functions implemented in these works for some architectures will not
necessarily generalise to others; hence they become architecture-specific. The
architecture choices of the time series GANs affect both the quality and
diversity of the data. However, there remain open problems in terms of the
practical implementation of the generated data and GANs in real-world
applications, particularly in health applications where the performance of
these models can directly affect patients’ quality of care/treatment.

The ’best’ GAN architecture and objective function is still yet to be
determined. As it stands, GANs tend to application-specific, that is, perform
well for their intended purpose but do not generalise well beyond their
original domain. A major limitation of time series GANs is the restrictions
placed on the length of the sequence specified that the architecture can
manage; documented experiments validating how well a time series GAN can adapt
to varying data lengths are notably absent at the time of writing.

##  9 Conclusion

This paper reviews a niche but growing use of Generative Adversarial Networks
for time series data based mainly around architectural evolution and loss
function variants. We see that each GAN provides application-specific
performance and doesn’t necessarily generalise well to other applications,
e.g. a GAN for generating high-quality physiological time series may not
produce high-fidelity audio due to some limitation imposed by the architecture
or loss function. A detailed review of the applications of time series GANs to
real-world problems has been provided, along with their datasets and the
evaluation metrics used for each domain. As stated in [4], GAN-related
research for time series lags that of computer vision both in terms of
performance and defined rules for generalisation of models. In conclusion,
this review has highlighted the open challenges in this area and offers
directions for future work and technological innovation, particularly for
those GAN aspects related to evaluation, privacy, and decentralised learning.

## Acknowledgements

This work is funded by Science Foundation Ireland under grant numbers 17/RC-
PhD/3482 and SFI/12/RC/2289_P2.

## References

  * [1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in Neural Information Processing Systems_ , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, Eds., vol. 27. Montréal, Canada: Curran Associates, Inc., 2014. [Online]. Available: https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
  * [2] A. Borji, “Pros and cons of gan evaluation measures,” 2018.
  * [3] G. Dorffner, “Neural networks for time series processing,” _Neural Network World_ , vol. 6, pp. 447–468, 1996.
  * [4] Z. Wang, Q. She, and T. E. Ward, “Generative adversarial networks in computer vision: A survey and taxonomy,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 2, pp. 1–38, 2021.
  * [5] J. Gui, Z. Sun, Y. Wen, D. Tao, and J. Ye, “A review on generative adversarial networks: Algorithms, theory, and applications,” 2020.
  * [6] L. Gonog and Y. Zhou, “A review: Generative adversarial networks,” in _2019 14th IEEE Conference on Industrial Electronics and Applications (ICIEA)_. Xi’an, China: IEEE, Jun. 2019, pp. 505–510. [Online]. Available: https://ieeexplore.ieee.org/document/8833686/
  * [7] H. Alqahtani, M. Kavakli-Thorne, and G. Kumar, “Applications of Generative Adversarial Networks (GANs): An Updated Review,” _Archives of Computational Methods in Engineering_ , vol. 28, pp. 525–552, Dec. 2019. [Online]. Available: http://link.springer.com/10.1007/s11831-019-09388-y
  * [8] C. Yinka-Banjo and O.-A. Ugot, “A review of generative adversarial networks and its application in cybersecurity,” _Artificial Intelligence Review_ , vol. 53, no. 3, pp. 1721–1736, Mar. 2020. [Online]. Available: http://link.springer.com/10.1007/s10462-019-09717-4
  * [9] X. Yi, E. Walia, and P. Babyn, “Generative adversarial network in medical imaging: A review,” _Medical Image Analysis_ , vol. 58, p. 101552, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841518308430
  * [10] A. Radford, L. Metz, and S. Chintala, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks,” 2015. [Online]. Available: http://arxiv.org/abs/1511.06434
  * [11] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein GAN,” 2017. [Online]. Available: http://arxiv.org/abs/1701.07875
  * [12] A. Borji, “Pros and cons of GAN evaluation measures,” _Computer Vision and Image Understanding_ , vol. 179, pp. 41–65, 2019.
  * [13] ——, “Pros and cons of gan evaluation measures: New developments,” 2021.
  * [14] Z. Wang, Q. She, A. F. Smeaton, T. E. Ward, and G. Healy, “Synthetic-neuroscore: Using a neuro-ai interface for evaluating generative adversarial networks,” _Neurocomputing_ , vol. 405, pp. 26–36, 2020.
  * [15] Z. Wang, G. Healy, A. F. Smeaton, and T. E. Ward, “Use of neural signals to evaluate the quality of generative adversarial network performance in facial image generation,” _Cognitive Computation_ , vol. 12, no. 1, pp. 13–24, 2020\.
  * [16] D. J. Sutherland, H.-Y. Tung, H. Strathmann, S. De, A. Ramdas, A. Smola, and A. Gretton, “Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy,” 2016. [Online]. Available: http://arxiv.org/abs/1611.04488
  * [17] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, and X. Chen, “Improved techniques for training gans,” in _Advances in Neural Information Processing Systems_ , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29. Barcelona, Spain: Curran Associates, Inc., 2016, pp. 2234–2242. [Online]. Available: https://proceedings.neurips.cc/paper/2016/file/8a3363abe792db2d8761d6403605aeb7-Paper.pdf
  * [18] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” 2018.
  * [19] L. van der Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of Machine Learning Research_ , vol. 9, no. 86, pp. 2579–2605, 2008\. [Online]. Available: http://jmlr.org/papers/v9/vandermaaten08a.html
  * [20] F. B. Bryant and P. R. Yarnold, “Principal-components analysis and exploratory and confirmatory factor analysis.” in _Reading and understanding multivariate statistics._ Washington, DC, US: American Psychological Association, 1995, pp. 99–136.
  * [21] J. Yoon, D. Jarrett, and M. van der Schaar, “Time-series generative adversarial networks,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Vancouver, Canada: Curran Associates, Inc., 2019, pp. 5508–5518. [Online]. Available: https://proceedings.neurips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf
  * [22] E. Choi, S. Biswal, B. Malin, J. Duke, W. F. Stewart, and J. Sun, “Generating Multi-label Discrete Patient Records using Generative Adversarial Networks,” 2017. [Online]. Available: http://arxiv.org/abs/1703.06490
  * [23] C. Esteban, S. L. Hyland, and G. Rätsch, “Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs,” 2017. [Online]. Available: http://arxiv.org/abs/1706.02633
  * [24] K. El Emam, E. Jonker, L. Arbuckle, and B. Malin, “A systematic review of re-identification attacks on health data,” _PLoS ONE_ , vol. 6, 2011.
  * [25] B. Malin and L. Sweeney, “Re-identification of DNA through an automated linkage process,” _Proceedings. AMIA Symposium_ , vol. 2001, pp. 423–427, 2001, publisher: American Medical Informatics Association. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/11825223
  * [26] C. Culnane, B. I. P. Rubinstein, and V. Teague, “Health Data in an Open World,” 2017. [Online]. Available: http://arxiv.org/abs/1712.05627
  * [27] B. P. Hejblum, G. M. Weber, K. P. Liao, N. P. Palmer, S. Churchill, N. A. Shadick, P. Szolovits, S. N. Murphy, I. S. Kohane, and T. Cai, “Probabilistic record linkage of de-identified research datasets with discrepancies using diagnosis codes,” _Scientific Data_ , vol. 6, 2019.
  * [28] European Union, “Data protection act 2018 (section36(2)),” 2018,
http://www.irishstatutebook.ie/eli/2018/si/314/made/en/pdf.

  * [29] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” University of Toronto, Toronto, Ontario, Tech. Rep. 0, 2009.
  * [30] Y. LeCun and C. Cortes, “MNIST handwritten digit database,” http://yann.lecun.com/exdb/mnist/, 2010. [Online]. Available: http://yann.lecun.com/exdb/mnist/
  * [31] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_. Miami, FL, USA: IEEE, 2009, pp. 248–255.
  * [32] H. A. Dau, E. Keogh, K. Kamgar, C.-C. M. Yeh, Y. Zhu, S. Gharghabi, C. A. Ratanamahatana, Yanping, B. Hu, N. Begum, A. Bagnall, A. Mueen, G. Batista, and Hexagon-ML, “The ucr time series classification archive,” October 2018, https://www.cs.ucr.edu/~eamonn/time_series_data_2018/.
  * [33] D. Dua and C. Graff, “UCI machine learning repository,” 2017. [Online]. Available: http://archive.ics.uci.edu/ml
  * [34] L. Yu, W. Zhang, J. Wang, and Y. Yu, “Seqgan: Sequence generative adversarial nets with policy gradient,” in _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , ser. AAAI’17. San Francisco, California, USA: AAAI Press, 2017, p. 2852–2858.
  * [35] R. D. Hjelm, A. P. Jacob, T. Che, A. Trischler, K. Cho, and Y. Bengio, “Boundary-seeking generative adversarial networks,” 2018.
  * [36] I. Goodfellow, “Generative adversarial networks for text,” 2016. [Online]. Available: https://www.reddit.com/r/MachineLearning/comments/40ldq6/generative_adversarial_networks_for_text/
  * [37] J. T. Guibas, T. S. Virdi, and P. S. Li, “Synthetic Medical Images from Dual Generative Adversarial Networks,” 2017. [Online]. Available: http://arxiv.org/abs/1709.01872
  * [38] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” _Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017_ , vol. 2017-January, pp. 105–114, 2017.
  * [39] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” _33rd International Conference on Machine Learning, ICML 2016_ , vol. 3, pp. 1681–1690, 2016.
  * [40] D. J. Im, C. D. Kim, H. Jiang, and R. Memisevic, “Generating images with recurrent adversarial networks,” 2016. [Online]. Available: http://arxiv.org/abs/1602.05110
  * [41] S. Hochreiter and J. Urgen Schmidhuber, “Lstm,” _Neural Computation_ , vol. 9, no. 8, pp. 1735–1780, 1997. [Online]. Available: http://www7.informatik.tu-muenchen.de/{~}hochreit{%}0Ahttp://www.idsia.ch/{~}juergen
  * [42] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,” 2015.
  * [43] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, Jul. 2002, pp. 311–318. [Online]. Available: https://www.aclweb.org/anthology/P02-1040
  * [44] M. Lapata, “Emnlp14,” http://homepages.inf.ed.ac.uk/mlap/Data/EMNLP14/, 2015, accessed: 2021-04-30.
  * [45] S. Winiger, “Obama political speech generator - recurrent neural network,” https://github.com/samim23/obama-rnn, 2015, accessed: 2021-04-30.
  * [46] M. Wiese, R. Knobloch, R. Korn, and P. Kretschmer, “Quant GANs: Deep Generation of Financial Time Series,” _Quantitative Finance_ , vol. 20, no. 9, pp. 1419–1440, Sep. 2020, arXiv: 1907.06673. [Online]. Available: http://arxiv.org/abs/1907.06673
  * [47] T. Bollerslev, “Generalized autoregressive conditional heteroskedasticity,” _Journal of Econometrics_ , vol. 31, no. 3, pp. 307–327, 1986. [Online]. Available: https://EconPapers.repec.org/RePEc:eee:econom:v:31:y:1986:i:3:p:307-327
  * [48] O. Mogren, “C-rnn-gan: Continuous recurrent neural networks with adversarial training,” 2016.
  * [49] T. J. Pollard, A. E. W. Johnson, J. D. Raffa, L. A. Celi, R. G. Mark, and O. Badawi, “The eICU Collaborative Research Database, a freely available multi-center database for critical care research,” _Scientific Data_ , vol. 5, no. 1, p. 180178, Sep. 2018. [Online]. Available: https://doi.org/10.1038/sdata.2018.178
  * [50] L. Wang, W. Zhang, and X. He, “Continuous patient-centric sequence generation via sequentially coupled adversarial learning,” in _Database Systems for Advanced Applications_ , G. Li, J. Yang, J. Gama, J. Natwichai, and Y. Tong, Eds. Cham: Springer International Publishing, 2019, pp. 36–52.
  * [51] A. E. Johnson, T. J. Pollard, L. Shen, L. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark, “Mimic-iii, a freely accessible critical care database,” _Scientific data_ , vol. 3, p. 160035, 2016.
  * [52] Y. Sumiya, K. Horie, H. Shiokawa, and H. Kitagawa, “NR-GAN: Noise Reduction GAN for Mice Electroencephalogram Signals,” in _Proceedings of the 2019 4th International Conference on Biomedical Imaging, Signal Processing_. Nagoya Japan: ACM, Oct. 2019, pp. 94–101. [Online]. Available: http://dl.acm.org/doi/10.1145/3366174.3366186
  * [53] H. Ni, L. Szpruch, M. Wiese, S. Liao, and B. Xiao, “Conditional sig-wasserstein gans for time series generation,” 2020.
  * [54] Y. Li, K. Swersky, and R. Zemel, “Generative moment matching networks,” 2015.
  * [55] O.-M. Institute, “Oxford-man institute of quantitative finance realized library,” https://realized.oxford-man.ox.ac.uk, 2021, accessed: 2021-04-30.
  * [56] H. Sun, Z. Deng, H. Chen, and D. C. Parkes, “Decision-aware conditional gans for time series data,” 2020.
  * [57] D. Hazra and Y.-C. Byun, “SynSigGAN: Generative Adversarial Networks for Synthetic Biomedical Signal Generation,” _Biology_ , vol. 9, no. 12, p. 441, Dec. 2020. [Online]. Available: https://www.mdpi.com/2079-7737/9/12/441
  * [58] G. Moody and R. Mark, “The impact of the mit-bih arrhythmia database,” _IEEE Engineering in Medicine and Biology Magazine_ , vol. 20, no. 3, pp. 45–50, 2001.
  * [59] P. Detti, G. Vatti, and G. Zabalo Manrique de Lara, “Eeg synchronization analysis for seizure prediction: A study on data of noninvasive recordings,” _Processes_ , vol. 8, no. 7, 2020. [Online]. Available: https://www.mdpi.com/2227-9717/8/7/846
  * [60] M. A. F. Pimentel, A. E. W. Johnson, P. H. Charlton, D. Birrenkott, P. J. Watkinson, L. Tarassenko, and D. A. Clifton, “Toward a robust estimation of respiratory rate from pulse oximeters,” _IEEE Transactions on Biomedical Engineering_ , vol. 64, no. 8, pp. 1914–1923, 2017.
  * [61] H. Fei and F. Tan, “Bidirectional grid long short-term memory (bigridlstm): A method to address context-sensitivity and vanishing gradient,” _Algorithms_ , vol. 11, no. 11, 2018. [Online]. Available: https://www.mdpi.com/1999-4893/11/11/172
  * [62] K. Nikolaidis, S. Kristiansen, V. Goebel, T. Plagemann, K. Liestøl, and M. Kankanhalli, “Augmenting physiological time series data: A case study for sleep apnea detection,” 2019.
  * [63] S. M. Abdelfattah, G. M. Abdelrahman, and M. Wang, “Augmenting The Size of EEG datasets Using Generative Adversarial Networks,” in _2018 International Joint Conference on Neural Networks (IJCNN)_. Rio de Janeiro: IEEE, Jul. 2018, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/document/8489727/
  * [64] S. Harada, H. Hayashi, and S. Uchida, “Biosignal Data Augmentation Based on Generative Adversarial Networks,” in _2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. Honolulu, HI: IEEE, Jul. 2018, pp. 368–371. [Online]. Available: https://ieeexplore.ieee.org/document/8512396/
  * [65] D. Kiyasseh, G. A. Tadesse, L. N. T. Nhan, L. Van Tan, L. Thwaites, T. Zhu, and D. Clifton, “PlethAugment: GAN-Based PPG Augmentation for Medical Diagnosis in Low-Resource Settings,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 24, no. 11, pp. 3226–3235, Nov. 2020. [Online]. Available: https://ieeexplore.ieee.org/document/9078801/
  * [66] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 22, no. 10, pp. 1345–1359, 2010.
  * [67] Z. Zhang, J. Han, K. Qian, C. Janott, Y. Guo, and B. Schuller, “Snore-gans: Improving automatic snore sound classification with synthesized data,” 2019.
  * [68] A. M. Delaney, E. Brophy, and T. E. Ward, “Synthesis of Realistic ECG using Generative Adversarial Networks,” 2019. [Online]. Available: https://arxiv.org/abs/1909.09150
  * [69] F. Zhu, F. Ye, Y. Fu, Q. Liu, and B. Shen, “Electrocardiogram generation with a bidirectional LSTM-CNN generative adversarial network,” _Scientific Reports_ , vol. 9, no. 1, 2019.
  * [70] E. Brophy, Z. Wang, and T. E. Ward, “Quick and easy time series generation with established image-based gans,” 2019.
  * [71] E. Brophy, “Synthesis of dependent multichannel ecg using generative adversarial networks,” in _Proceedings of the 29th ACM International Conference on Information &; Knowledge Management_, ser. CIKM ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 3229–3232. [Online]. Available: https://doi.org/10.1145/3340531.3418509
  * [72] L. Juvela, B. Bollepalli, J. Yamagishi, and P. Alku, “Waveform Generation for Text-to-speech Synthesis Using Pitch-synchronous Multi-scale Generative Adversarial Networks,” in _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. Brighton, United Kingdom: IEEE, May 2019, pp. 6915–6919. [Online]. Available: https://ieeexplore.ieee.org/document/8683271/
  * [73] Q. Li, H. Hao, Y. Zhao, Q. Geng, G. Liu, Y. Zhang, and F. Yu, “GANs-LSTM Model for Soil Temperature Estimation From Meteorological: A New Approach,” _IEEE Access_ , vol. 8, pp. 59 427–59 443, 2020\. [Online]. Available: https://ieeexplore.ieee.org/document/9045947/
  * [74] S. Kaushik, A. Choudhury, S. Natarajan, L. A. Pickett, and V. Dutt, “Medicine Expenditure Prediction via a Variance- Based Generative Adversarial Network,” _IEEE Access_ , vol. 8, pp. 110 947–110 958, 2020\. [Online]. Available: https://ieeexplore.ieee.org/document/9116991/
  * [75] Z. Guo, Y. Wan, and H. Ye, “A data imputation method for multivariate time series based on generative adversarial network,” _Neurocomputing_ , vol. 360, pp. 185–197, Sep. 2019\. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0925231219308306
  * [76] F. Luer, D. Mautz, and C. Bohm, “Anomaly Detection in Time Series using Generative Adversarial Networks,” in _2019 International Conference on Data Mining Workshops (ICDMW)_. Beijing, China: IEEE, Nov. 2019, pp. 1047–1048. [Online]. Available: https://ieeexplore.ieee.org/document/8955499/
  * [77] D. Li, D. Chen, B. Jin, L. Shi, J. Goh, and S.-K. Ng, “MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks,” in _Artificial Neural Networks and Machine Learning – ICANN 2019: Text and Time Series_ , I. V. Tetko, V. Kůrková, P. Karpov, and F. Theis, Eds. Cham: Springer International Publishing, 2019, vol. 11730, pp. 703–716, series Title: Lecture Notes in Computer Science. [Online]. Available: http://link.springer.com/10.1007/978-3-030-30490-4_56
  * [78] T. Leangarun, P. Tangamchit, and S. Thajchayapong, “Stock Price Manipulation Detection using Generative Adversarial Networks,” in _2018 IEEE Symposium Series on Computational Intelligence (SSCI)_. Bangalore, India: IEEE, Nov. 2018, pp. 2104–2111. [Online]. Available: https://ieeexplore.ieee.org/document/8628777/
  * [79] C. Donahue, J. McAuley, and M. Puckette, “Adversarial audio synthesis,” 2019.
  * [80] A. Kolokolova, M. Billard, R. Bishop, M. Elsisy, Z. Northcott, L. Graves, V. Nagisetty, and H. Patey, “Gans & reels: Creating irish music using a generative adversarial network,” 2020.
  * [81] P.-S. Cheng, C.-Y. Lai, C.-C. Chang, S.-F. Chiou, and Y.-C. Yang, “A Variant Model of TGAN for Music Generation,” in _Proceedings of the 2020 Asia Service Sciences and Software Engineering Conference_. Nagoya Japan: ACM, May 2020, pp. 40–45. [Online]. Available: https://dl.acm.org/doi/10.1145/3399871.3399888
  * [82] Y. Choi, H. Lim, H. Choi, and I.-J. Kim, “GAN-Based Anomaly Detection and Localization of Multivariate Time Series Data for Power Plant,” in _2020 IEEE International Conference on Big Data and Smart Computing (BigComp)_. Busan, Korea (South): IEEE, Feb. 2020, pp. 71–74. [Online]. Available: https://ieeexplore.ieee.org/document/9070362/
  * [83] D. Parthasarathy, K. Bäckström, J. Henriksson, and S. Einarsdóttir, “Controlled time series generation for automotive software-in-the-loop testing using gans,” 2020.
  * [84] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A kernel two-sample test,” _J. Mach. Learn. Res._ , vol. 13, no. null, p. 723–773, Mar. 2012.
  * [85] S. Harada, H. Hayashi, and S. Uchida, “Biosignal Generation and Latent Variable Analysis With Recurrent Generative Adversarial Networks,” _IEEE Access_ , vol. 7, pp. 144 292–144 302, 2019. [Online]. Available: https://ieeexplore.ieee.org/document/8794813/
  * [86] K. G. Hartmann, R. T. Schirrmeister, and T. Ball, “EEG-GAN: Generative adversarial networks for electroencephalograhic (EEG) brain signals,” 2018. [Online]. Available: http://arxiv.org/abs/1806.01875
  * [87] F. Fahimi, Z. Zhang, W. B. Goh, K. K. Ang, and C. Guan, “Towards EEG Generation Using GANs for BCI Applications,” in _2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI)_. Chicago, IL, USA: IEEE, May 2019, pp. 1–4. [Online]. Available: https://ieeexplore.ieee.org/document/8834503/
  * [88] D. Pascual, A. Amirshahi, A. Aminifar, D. Atienza, P. Ryvlin, and R. Wattenhofer, “Epilepsygan: Synthetic epileptic brain activities with privacy preservation,” _IEEE Transactions on Biomedical Engineering_ , vol. 67, pp. 1–1, 2020.
  * [89] L. Yi and M. Mak, “Adversarial data augmentation network for speech emotion recognition,” in _2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. Lanzhou, China: IEEE, 2019, pp. 529–534.
  * [90] H. Zhang, N. Xiao, P. Liu, Z. Wang, and R. Tang, “G-RNN-GAN for Singing Voice Separation,” in _Proceedings of the 2020 5th International Conference on Multimedia Systems and Signal Processing_. Chengdu China: ACM, May 2020, pp. 69–73. [Online]. Available: https://dl.acm.org/doi/10.1145/3404716.3404718
  * [91] F. Qu, J. Liu, Y. Ma, D. Zang, and M. Fu, “A novel wind turbine data imputation method with multiple optimizations based on GANs,” _Mechanical Systems and Signal Processing_ , vol. 139, p. 106610, May 2020. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0888327019308313
  * [92] L. Han, K. Zheng, L. Zhao, X. Wang, and H. Wen, “Content-Aware Traffic Data Completion in ITS Based on Generative Adversarial Nets,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 10, pp. 11 950–11 962, Oct. 2020. [Online]. Available: https://ieeexplore.ieee.org/document/9133309/
  * [93] Y. Luo, Y. Zhang, X. Cai, and X. Yuan, “E2GAN: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_. Macao, China: International Joint Conferences on Artificial Intelligence Organization, Aug. 2019, pp. 3094–3100. [Online]. Available: https://www.ijcai.org/proceedings/2019/429
  * [94] Y. Sun, L. Peng, H. Li, and M. Sun, “Exploration on Spatiotemporal Data Repairing of Parking Lots Based on Recurrent GANs,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. Maui, HI: IEEE, Nov. 2018, pp. 467–472. [Online]. Available: https://ieeexplore.ieee.org/document/8569319/
  * [95] Y. Chen, Y. Lv, and F.-Y. Wang, “Traffic Flow Imputation Using Parallel Data and Generative Adversarial Networks,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 4, pp. 1624–1630, Apr. 2020. [Online]. Available: https://ieeexplore.ieee.org/document/8699108/
  * [96] Y. Luo, X. Cai, Y. ZHANG, J. Xu, and Y. xiaojie, “Multivariate time series imputation with generative adversarial networks,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Montréal, Canada: Curran Associates, Inc., 2018, pp. 1596–1607. [Online]. Available: https://proceedings.neurips.cc/paper/2018/file/96b9bff013acedfb1d140579e2fbeb63-Paper.pdf
  * [97] G. Zhu, H. Zhao, H. Liu, and H. Sun, “A Novel LSTM-GAN Algorithm for Time Series Anomaly Detection,” in _2019 Prognostics and System Health Management Conference (PHM-Qingdao)_. Qingdao, China: IEEE, Oct. 2019, pp. 1–6.
  * [98] C. Dwork, “Differential privacy,” in _Proceedings of the 33rd International Conference on Automata, Languages and Programming - Volume Part II_ , ser. ICALP’06. Berlin, Heidelberg: Springer-Verlag, 2006, p. 1–12. [Online]. Available: https://doi.org/10.1007/11787006_1
  * [99] C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,” _Found. Trends Theor. Comput. Sci._ , vol. 9, no. 3–4, p. 211–407, aug 2014. [Online]. Available: https://doi.org/10.1561/0400000042
  * [100] M. Abadi, H. B. McMahan, A. Chu, I. Mironov, L. Zhang, I. Goodfellow, and K. Talwar, “Deep learning with differential privacy,” _Proceedings of the ACM Conference on Computer and Communications Security_ , vol. 24-28-October-2016, no. Ccs, pp. 308–318, 2016.
  * [101] L. Xie, K. Lin, S. Wang, F. Wang, and J. Zhou, “Differentially Private Generative Adversarial Network,” 2018. [Online]. Available: http://arxiv.org/abs/1802.06739
  * [102] S. Augenstein, H. B. McMahan, D. Ramage, S. Ramaswamy, P. Kairouz, M. Chen, R. Mathews, and B. A. y Arcas, “Generative models for effective ml on private, decentralized datasets,” 2020.
  * [103] C. Hardy, E. L. Merrer, and B. Sericola, “Md-gan: Multi-discriminator generative adversarial networks for distributed datasets,” 2019.
  * [104] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” 2017.
  * [105] M. Rasouli, T. Sun, and R. Rajagopal, “Fedgan: Federated generative adversarial networks for distributed data,” 2020.
  * [106] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” 2017.
  * [107] J. Hayes, L. Melis, G. Danezis, and E. De Cristofaro, “LOGAN: Membership Inference Attacks Against Generative Models,” _Proceedings on Privacy Enhancing Technologies_ , vol. 2019, no. 1, pp. 133–152, 2019.