# Kernel-U-Net: Hierarchical and Symmetrical Framework for Multivariate Time
Series Forecasting

Jiang YOU1,2,3 René NATOWICZ2 Arben CELA2 Jacob OUANOUNOU3 Patrick SIARRY1
1Université Paris-Est Créteil
2ESIEE Paris
3HN-Services, Paris
{jiang.you, rene.natowicz<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Time series forecasting task predicts future trends based on historical
information. Recent U-Net-based methods have demonstrated superior performance
in predicting real-world datasets. However, the performance of these models is
lower than patch-based models or linear models. In this work, we propose a
symmetric and hierarchical framework, Kernel-U-Net, which cuts the input
sequence into slices at each layer of the network and then computes them using
kernels. Furthermore, it generalizes the concept of convolutional kernels in
classic U-Net to accept custom kernels that follow the same design pattern.
Compared to the existing linear or transformer-based solution, our model
contains 3 advantages: 1) A small number of parameters: the parameters size is
$O(log(L)^{2})$ where $L$ is the look-back window size, 2) Flexibility: its
kernels can be customized and fitted to the datasets, 3) Computation
efficiency: the computation complexity of transformer modules is reduced to
$O(log(L)^{2})$ if they are placed close to the latent vector. Kernel-U-Net
accuracy was greater than or equal to the state-of-the-art model on six (out
of seven) real-world datasets.

##  1 Introduction

Time series forecasting predicts future trends based on recent historical
information. It allows experts to track the incoming situation and react
timely in critical cases. Its applications range from different domains such
as predicting the road occupancy rates from different sensors in the city Lai
et al. (2018), monitoring influenza-like illness weekly patient cases Wu et
al. (2021), monitoring electricity transformer temperature in the electric
power long-term deployment Zhou et al. (2021) or forecasting temperature,
pressure and humidity in weather station Liu et al. (2022b) etc.

Over the past few decades, time series forecasting solutions have evolved from
traditional statistical methods (e.g., ARIMA Khandelwal et al. (2015) and
machine learning techniques (e.g., GBRT Persson et al. (2017)) to deep
learning-based solutions, such as RNN recurrent neural networks Tokgöz and
Ünal (2018), LSTM Kong et al. (2017), Temporal Convolutional Network (TCN)
Hewage et al. (2020) and Transformer-based model Li et al. (2019).

Large-scale time series forecasting addresses the problem of predicting a
longer future horizon with a larger look-back window. With the increasing size
of data, the cost of learning deep models increases crucially. Moreover,
models such as LSTM or RNN suffer gradient vanishing problems in learning long
sequences. To balance the quality of prediction and efficiency of learning
deep models, researchers proposed hierarchical structure in Duan et al.
(2022), pyramidal structure in Pyraformer Liu et al. (2022a), AutoEncoders
Asres et al. (2022), U-Net-like model in Yformer Madhusudhanan et al. (2023)
or patches in PatchTST Nie et al. (2023). Meanwhile, there is an increasing
trend in combining different components for learning particular time series.
For example in time series classification, researchers studied hybrid models
such as combing GRU Elsayed et al. (2019), LSTM Lin et al. (2020); Khan et al.
(2021) or Transformer Huang et al. (2020b) with linear or CNN layers Hewage et
al. (2020).

Among the Transformer models applying to time series data, Informer Zhou et
al. (2021), Autoformer Wu et al. (2021), and FEDformer Zhou et al. (2022) are
the best variants that improve the quality of prediction. Meanwhile, a recent
paper Zeng et al. (2023) gives a simple linear model NLinear which outperforms
pure transformer-based models on a variety of common benchmarks and challenges
the Transformer models for time series forecasting. Later, the authors in
PatchTST Nie et al. (2023) argued that applying transformer modules on a
point-wise sequence degrades the performance. By adding a linear patch layer,
they successfully relieved the overfitting problem of transformer modules and
they reached state-of-the-art result on several large datasets.

We remark that models are particularly sensible to different types of
datasets. For example, NLinear is more robust against overfitting in small-
size datasets such as Etth and EttmZeng et al. (2023) and PatchTST is more
expressive in large-size datasets such as Weather and Traffic etc.Nie et al.
(2023). At the same time, it is also important to notice the parameter size in
Linear models such as NLinear is in $O(L^{2})$ and the computation complexity
is $O(L)$. Meanwhile, the parameter size in a transformer block is $O(1)$ but
the computational complexity is $O(L^{2})$ in general Transformer modules. It
is reduced to $O(Llog(L))$ using Propsparse and to
$O(\frac{L}{S}log(\frac{L}{S}))$ using patches, where $S$ is the size of the
patch. Based on previous analysis, there is a motivation to propose a more
flexible framework for combining the advantages of different modules in high-
paced development and testing.

As a classical and highly expressive model in medical image segmentation,
Convolutional U-net has an elegant symmetrical encoder and decoder
structureRonneberger et al. (2015). It coincides well with the time series
forecasting problem because the inputs and outputs are drawn from the same
distribution. The first U-Net-like model for time series forecasting was
YformerMadhusudhanan et al. (2023), which used Transformers in the encoder and
decoder. As mentioned before, applying a Transformer on point-wise data can
lead to overfitting. Therefore, we are interested in answering the question of
whether there is a U-net-like structure that performs well for the time series
forecasting task.

Figure 1: Structure of Kernel U-Net

We propose a general hierarchical and symmetrical framework, Kernel-U-Net,
inspired by convolutional U-net Ronneberger et al. (2015) and Yformer
Madhusudhanan et al. (2023) for time series forecasting. Kernel-U-Net offers
$O(log(L)^{2})$ parameters size for processing a time series of size $L$.
Meanwhile, it provides convenience for composing particular models with custom
kernels that follow the same design pattern. Since the hierarchical structure
reduces the length of the sequence at each level exponentially, the complexity
of learning such sequences is simultaneously reduced. In particular, the
computation complexity of Transformer and Long Short-term Memory(LSTM) modules
is reduced to $O(log(L)^{2})$ and $O(log(L))$ if they are only employed in the
layers close to the latent vector. Therefore, Kernel-U-Net helps reduce the
computation cost of the Transformer and relieves the gradient vanishing
problem in LSTM in learning long sequences.

To fully study the performance and efficiency of Kernel-U-Net, we conduct
experiments for time series forecasting tasks on several widely used benchmark
datasets. Our results show that in time series forecasting, Kernel-U-Net meets
or outperforms baseline models, such as NLinear and PatchTST, in the majority
cases.

In summary, the contributions of this work include:

  * •

We propose Kernel-U-Net, a general hierarchical and symmetrical framework that
partitions input sequences into small slices and learns them at each layer of
the network.

  * •

Kernel-U-Net generalizes the concept of kernels and provides convenience for
composing particular models with custom kernels that follow the same design
pattern. The parameter size is guaranteed in $O(log(L)^{2})$.

  * •

The computation complexity of Transformer module is $O(Llog(L))$ and can be
reduced to $O(log(L)^{2})$ if it is only employed in the layers close to the
latent vector.

  * •

Kernel-U-Net achieves results with MSE close to or better than the state-of-
the-art methods on 6 (out of 7) time series data sets.

We conclude that Kernel-U-Net is a highly potential candidate for large-scale
time series forecasting tasks. It is a general hierarchical and symmetrical
framework with low complexity in terms of parameters. It reaches state-of-the-
art results in the majority cases. With flexible customized kernels, it adapts
well in high-paced development and test context.

##  2 Related works

Transformer
Transformer Vaswani et al. (2017) was initially introduced in the field of
Natural Language Processing(NLP) on language translation tasks. It contains a
layer of positional encoding, blocks that are composed of layers of multiple
head attentions, and a linear layer with SoftMax activation. As it
demonstrated outstanding performance on NLP tasks, many researchers follow
this technique route. Bidirectional Encoder Representations from Transformers
(BERT)Devlin et al. (2019) is designed to pre-train deep bidirectional
representations from the unlabeled text by joint conditioning on both left and
right context in all layers. Generative Pre-trained Transformer 3(GPT-3) pre-
trained a massive transformer-based model using 175 billion parameters Brown
et al. (2020) and achieved strong performance on different types of downstream
natural language tasks without requiring any fine-tuning.

Later, Vision Transformers (ViTs) Dosovitskiy et al. (2021) ViT applies a pure
transformer directly to sequences of image patches to classify the full image
and outperformed CNN based method on ImageNetDeng et al. (2009). Swin
Transformer Liu et al. (2021) proposed a hierarchical Transformer whose
representation is computed with shifted windows. As a shifted window brings
greater efficiency by limiting self-attention computation to non-overlapping
local windows, it also allows cross-window connection. This hierarchical
architecture has the flexibility to model at various scales and has linear
computational complexity concerning image size.

In time series forecasting, the researchers were also attracted by
transformer-based models. LogTrans Li et al. (2019) proposed convolutional
self-attention by employing causal convolutions to produce queries and keys in
the self-attention layer. It proposes also a LogSparse Transformer with only
$O(L(logL)^{2})$ space complexity to break the memory bottleneck. Informer
Zhou et al. (2021) has an encoder-decoder architecture that validates the
Transformer-like model’s potential value to capture individual long-range
dependency between long sequences. It proposes a ProbSparse self-attention
mechanism to efficiently replace the canonical self-attention. It achieves the
$O(LlogL)$ time complexity and $O(LlogL)$ memory usage on dependency
alignments. PyraformerLiu et al. (2022a) simultaneously captures temporal
dependencies of different ranges in a compact multi-resolution fashion.
Theoretically, by choosing parameters appropriately, it achieves concurrently
the maximum path length of $O(1)$ and the time and space complexity of $O(L)$
in forward pass.

Meanwhile, another family of transformer-based models combines the transformer
with the traditional method in time series processing. Autoformer Wu et al.
(2021) introduces an Auto-Correlation mechanism in place of self-attention,
which discovers the sub-series similarity based on the series periodicity and
aggregates similar sub-series from underlying periods. Frequency Enhanced
Decomposed Transformer (FEDformer)Zhou et al. (2022) captures global
properties of time series with seasonal-trend decomposition. The authors
proposed Fourier-enhanced blocks and Wavelet-enhanced blocks in the
Transformer structure to capture important time series patterns through
frequency domain mapping.

Recently, YFormer Madhusudhanan et al. (2023) proposed a U-Net-like
architecture that combines coarse and fine-grained features to capture long-
range effects on all scale levels. PatchTST Nie et al. (2023) adopted the
patches method from ViTs and proposed a time series transformer with a linear
patch layer. It reaches state-of-the-art results on many time series
forecasting benchmarks. The author claims Reduction in time and space
complexity of learning transformer block from $O(L^{2})$ to
$O((\frac{L}{S})^{2})$, where $L$ is sequence size and $S$ is stride size.

U-Net
U-Net is a neural network architecture designed primarily for medical image
segmentation Ronneberger et al. (2015). U-net is composed of an encoder and a
decoder. At the encoder phase, a long sequence is reshaped into small slices
and reduced by kernels step by step into a latent vector. At the decoder
stage, the vector is developed by kernels for generating an output with the
same shape as the input. With the help of skip-connection between the encoder
and decoder, U-Net can also capture and merge low-level and high-level
information easily.

With such a neat structural design, U-Net has achieved great success in a
variety of applications such as medical image segmentation Ronneberger et al.
(2015), biomedical 3D-Image segmentation Çiçek et al. (2016), time series
segmentation Perslev et al. (2019), image super-resolution Han et al. (2022)
and image denoising Zhang et al. (2023). Its techniques evolve from basic
2D-U-Net to 3D U-Net Çiçek et al. (2016), 1D-U-Net Azar et al. (2022),
Weighted Res-UNet Xiao et al. (2018) with residual structure and weighted
attention mechanism, U-Net++ Zhou et al. (2018) with nested U-net inside each
layer, UNet3+ Huang et al. (2020a) that allows full-scale skip connections
incorporating low-level details with high-level semantics from feature maps in
different scales and Swin-U-Net Cao et al. (2023) that only use transformer
blocks at each layer of U-Net. Modification on U-Net layers has been studied
by Selective Kernel U-Net Byra et al. (2020) which merges information from
different sizes of convolutional kernels and Large Kernel U-Net Jia et al.
(2022) which combines TransMorph layers for better performance in medical
image segmentation.

In time series processing, U-time Perslev et al. (2019) is a U-Net composed of
convolutional layers for time series segmentation tasks, and YFormer
Madhusudhanan et al. (2023) is the first U-Net based model for time series
forecasting task. In particular, YFormer applied transformer blocks on each
layer of U-Net and capitalized on multi-resolution feature maps effectively.

Hierarchical and hybrid model
In time series processing, learning sequences of length $L$ requires $O(L)$
computations for recurrent models such as RNN, LSTM, GRU, and $O(L^{2})$
computations for transformer-based models. With ProbSparse Attention, the
complexity can be reduced to $O(Llog(L))$. To balance the expressivity of
complex modules and computational efficiency, researchers investigated
hierarchical architecture and hybrid methods that merge different modules into
the network.

For example, authors in Du et al. (2015) investigated a tree structure model
made of bidirectional RNN layers and concatenation layers for skeleton-based
action recognition, authors in Xiao and Liu (2016) stacked RNN and LSTM with
Attention mechanism for semantic relation classification of text, authors in
Kowsari et al. (2017) applied hierarchical LSTM and GRU was used for document
classification. In time series processing, the authors in Hong and Yoon (2017)
combine Deep Belief Network (DBN) and LSTM for sleep signal pattern
classification, authors in LogTrans Li et al. (2019), Informer Zhou et al.
(2021)) and Pyraformer Liu et al. (2022a) studied hierarchical or pyramidal
transformers for time series forecasting.

Kernel-U-Net is a general hierarchical and symmetrical framework with low
complexity in terms of parameter size. By replacing the convolutional kernel
in classical U-Net, our model allows users to define custom kernels such as
Attention, LSTM, or simple linear multiplication with a non-linear activation
function for extracting features on complex time series. There are 3
advantages in Kernel-U-Net: 1) A small number of parameters: the total
parameter of model is in order of $O(log(L)^{2})$, 2) Flexibility: the kernels
can be replaced easily by arbitrary function that follows the same design mode
so that non-linear kernels can be easily combined into it, 3) Computation
efficiency: the computation complexity of transformer modules is reduce to
$O(log(L)^{2})$ if they placed close to latent vector.

Algorithm 1 Kernel Wrapper

Input: kernel $f_{i}$, input segments $x$, input length $L_{in}$ and dimension
$M_{in}$, output length $L_{out}$ , and dimension $M_{out}$

Output: Instance of Kernel Wrapper

1: # Initiation of main kernel function $f$ is

2: $f=$ $f_{i}(L_{in},M_{in},L_{out},M_{out})$

3: # Holds variables for global operation

4: _unet_skip_output = None

5: _unet_skip_input = None

6:

7: # Define the forward function as :

8: Reshape $x$ into (-1, $L_{in}$,$M_{in}$)

9: if _unet_skip_input  is not None then

10: $x=x+\\_unet\\_skip\\_input$

11: end if

12: # Apply $f$ on $x$

13: $y=f_{i}(x)$

14: _unet_skip_output = $y$

15: return $y$

##  3 Method

###  3.1 Problem Formulation

Let us note by $x\in R^{M\times N}$ the matrix which represents the
multivariate time series dataset, where the first dimension $N$ represents the
sampling time and the second dimension $M$ is the size of the feature. Let L
be the length of memory or the look-back window, so
$(x_{t+1,1},...,x_{t+L,M})$ (or for short, $(x_{t+1},...,x_{t+L}))$ is a slice
of length $L$ of all features. It contains historical information about the
system at instant $t$. We call $X\in R^{N\times M\times L}$ the trajectory
matrix which represents the sliced historical data for each instant
$t\in[0,N-L-1]$.

Time series forecasting.
In the case of time series forecasting, the dataset is composed of a time
series of characteristics $x$ and future series $\hat{x}$. Let $x_{t}$ be the
feature at the time step $t$ and $L$ the length of the look-back window. Given
a historical data series $(x_{t+1},...,x_{t+L})$ of length $L$, time series
forecasting task is to predict the value
$(\hat{x}_{t+L+1},...,\hat{x}_{t+L+T})$ in the future T time steps. Then we
can define the basic time series forecasting problem:
$(\hat{x}_{t+L+1},...,\hat{x}_{t+L+T})=f(x_{t+1},...,x_{t+L})$ , where $f$ is
the function that predicts the value $(\hat{x}_{t+L+1},...,\hat{x}_{t+L+T})$
in the future T time steps based on a series $(x_{t+1},...,x_{t+L})$.

Algorithm 2 Kernel U-Net Encoder

Input: $L_{1},M_{1}$, $[,L_{2},...,L_{n},$ $L_{n+2},...,L_{n+m}]$,
$[M_{2},...,M_{n},$ $M_{n+2},...,M_{n+m}]$, $[f_{1},f_{2},...,f_{n},$
$f_{n+2},...,f_{n+m}]$, $L_{h}$, $M_{h}$
Output: Instance of Kernel U-Net Encoder

1: $L=[L_{1},L_{2},...,L_{n},$ $L_{n+2},...,L_{n+m}]$

2: # Create first layer

3: layers = [KernelWrapper($f_{1},L_{1},M_{1},1,M_{2})$]

4: # Create intermediate layers

5: for $i$ in $[2,...,n,$ $n+2,...,n+m-1]$ do

6: layers.append(KernelWrapper($f_{i},L_{i},M_{i},1,M_{i+1}$))

7: end for

8: # Create last layer

9: layers.append(KernelWrapper($f_{n+m},L_{n+m},M_{n+m},$ $L_{h},M_{h}$))

10: encoder = nn.Sequential (*layers)

###  3.2 Kernel U-Net

Kernel U-Net follows the hierarchical structure of convolutional U-Net. It
accepts a custom kernel to replace the linear kernel inside the original
convolutional U-Net. More precisely, it repeatedly applies custom kernels on
small slices of large-scale input data. In this context, Kernel U-Net avoids
problems in processing long sequences such as gradient vanishing in recurrent
models and quadradic complexity in transformers-based models. In the following
paragraphs, we describe methods such as the hierarchical partition of the
input sequence, and the creation of kernel wrapper, encoder, decoder, and
U-net.

Hierarchical Partition of the input sequence.
Let us consider an input trajectory matrix $X_{t}\in R^{L\times M}$, where $L$
is lookback window size, $M$ is feature size and $t$ is the time step. Given a
list of multiples $(L_{1},...,L_{n})$ for look-back window and
$(M_{1},...,M_{m})$ for feature such that $L=\prod_{1}^{n}L_{i}$ and
$M=\prod_{1}^{m}M_{i}$ . We reshape $X_{t}$ as small slices $X_{t,i,j}\in
R^{L_{1}\times M_{1}}$ such that the concatenation of all slices is equal to
its original trajectory matrix:

$X_{t}=Concat([[X_{t,i,j}\forall i\in[1,\frac{L}{L_{1}}]]\forall
j\in[1,\frac{M}{M_{1}}]])$.

The partitioned sequences are processed by kernels in the encoder. Since the
operations in the decoder are symmetrical to those in the encoder, there will
also be output $\hat{X}_{t+L}$ composed of generated segments
$\hat{X}_{t+L,i,j}$.

Hierarchical processing with kernels in the encoder and decoder
Let us consider $X$ as a batch of trajectory matrix $X\in R^{B\times L\times
M}$, where $B$ is the batch size. The hierarchical processing of Kernel U-Net
consists of compressing an input sequence at the encoding stage and generating
an output sequence at the decoding stage. By default, kernels reduce the
dimension of input at each layer in an encoder and increase the dimension in a
decoder.

At the stage of the encoder, the shape of input $X$ is $(B,L,M)$. Let us
assume that $M_{hidden}$ is the unique dimension of the hidden vector at each
intermediate layer to simplify the problem. We reshape $X$ into
$(B,\prod_{2}^{n}L_{i},L_{1},\prod_{2}^{m}M_{i},M_{1})$ then transpose it into
$(B,\prod_{2}^{m}M_{i},\prod_{2}^{n}L_{i},L_{1},M_{1})$ and reshape it into
$(B*\prod_{2}^{m}M_{i}*\prod_{2}^{n}L_{i},L_{1},M_{1})$ . The kernel
$f_{L_{1}}$ can now process the reshaped $X$ as a large batch of slices and
output a vector $Z_{L_{1}}$ in shape of
$(B*\prod_{2}^{m}*M_{i}*\prod_{2}^{n}L_{i},1,M_{hidden})$. After this
operation, we reshape the output $Z_{L_{1}}$ into
$(B*\prod_{2}^{m}M_{i}*\prod_{3}^{n}L_{i},L_{2},M_{hidden})$ for next layer.
Iteratively, the encoder of Kernel U-Net processes all the multiples in the
list $(M_{m},...,M_{2},L_{n},...,L_{2})$ in reverse order, and gives the final
latent vector $Z$ in the shape of $(B,1,M_{hidden})$ as a batch of the latent
vector.

At the stage of the decoder, the operations are reversed. We send a latent
vector Z of shape $(B,1,M_{hidden})$ to decoder $f_{M_{m}}$ and get an output
$Z_{M_{m}}$ in shape of $(B,M_{m},M_{hidden})$, then we reshape it into
$(B*M_{m},1,M_{hidden})$ for the next kernels. At the end of this iteration
over multiples in the list $(M_{m-1},...,M_{2},L_{n},...,L_{2},L_{1})$, we
have finally a vector in the shape of
$(B*\prod_{2}^{m}M_{i}*\prod_{2}^{n}L_{i},L_{1},M_{1})$. The last operations
are reshaping it into $(B,\prod_{2}^{m}M_{i},\prod_{2}^{n}L_{i},L_{1},M_{1})$
, transposing it into the vector in shape
$(B,\prod_{2}^{n}L_{i},L_{1},\prod_{2}^{m}M_{i},M_{1})$ and reshaping it to
shape $(B,L,M)$.

The algorithm is optimized in implementation because the for loop is expensive
in the forward function of a PyTorch module. In our implementation, we nest
the reshape and transpose operations in the kernel wrapper to avoid iteration
over kernels. The complete algorithm is described in the next section.

Algorithm 3 Kernel U-Net Decoder

Input: $L_{h}$, $M_{h}$, $[L_{n},...,L_{2}]$, $[L_{n+m},...,L_{n+2}]$,
$L_{1}$, $M_{1}$, $[f_{n+m},...,f_{n+2},$ $f_{n},...,f_{2}]$,
$[M_{n+m},...,M_{n+2},$ $M_{n},...,M_{2}]$, $L_{1}$, $M_{1}$.

Output: Instance of Kernel U-Net Decoder

1: # Create a list of multiples

2: L=$[L_{n+m},...,L_{n+2}$ $,L_{n},...,L_{2},L_{1}]$

3: # Create first layer

4: layers = [KernelWrapper($f_{n+m},$ $1,$ $M_{h},$ $L_{n+m},$ $M_{n+m}$)]

5: # Create intermediate layers

6: for $i$ in $[n+m-1,...,n+2,n,...,2]$ do

7: layers.append(KernelWrapper($f_{i},1,M_{i+1},L_{i},M_{i})$)

8: end for

9: # Create last layer

10: layers.append(KernelWrapper($f_{2},1,$ $M_{2},$ $L_{1},$ $M_{1})$)

11: decoder = nn.Sequential (*layers)

Kernel Wrapper As described previously, Kernel U-Net accepts custom kernels
that follow predefined design patterns. A kernel wrapper is an interface that
executes the given kernel inside the encoder and decoder. It requires
necessary parameters such as model kernel, input dimension, input length,
output dimension, and output length and holds global variables such as skip
connection vector. By default, a kernel reduces the length of a segment from
$L_{i}$ to $1$ in the encoder and develops the length of a segment from $1$ to
$L_{i}$ in the decoder. The kernel wrapper initiates an instance of a given
kernel and calls it in the forward function.

Algorithm 4 Kernel U-Net

Input: $L_{1},M_{1}$, $[,L_{2},...,L_{n},$ $L_{n+2},...,L_{n+m}]$,
$[M_{2},...,M_{n},$ $M_{n+2},...,M_{n+m}]$, $[f_{1},f_{2},...,f_{n},$
$f_{n+2},...,f_{n+m}]$, $L_{h}$, $M_{h}$

Output: Instance of Kernel U-Net

1: # Initiation of encoder and decoder in K-U-N.

2: encoder = KernelUNetEncoder(inputs)

3: decoder = KernelUNetDecoder(inputs)

4:

5: # Define the forward function as

6: z = encoder(x)

7: for $i$ in $[1,...,n]$ do

8: output = encoder.layers[i]._unet_skip_output

9: decoder.layers[$n-i+1$]. _unet_skip_input = output

10: end for

11: y= decoder(z)

12: return y

Creation of Kernel U-Net
Let us note input length $L$, input dimension $M$, Lists of multiples of look-
back window $[L_{2},$ $...,$ $L_{n}]$ and feature $[L_{n+2},$ $...,$
$L_{n+m}]$ , size of segment-unit $L_{1}$ and feature-unit $M_{1}$ , a list of
hidden dimension $[M_{2},...,M_{n},$ $M_{n+2},...,M_{n+m}]$, a list of kernels
$[f_{1},f_{2},...,f_{n},$ $f_{n+2},...,f_{n+m}]$, latent vector length $L_{h}$
and dimension $M_{h}$.

Remark that the hidden dimension of the model is the size of intermediate
output vectors within layers. It corresponds to channel size in a
convolutional network. This dimension can be augmented for a larger passage of
information. The kernel wrapper is described in algorithm 1. The creation of
Kernel U-Net Encoder, Decoder, and complete U-Net are described in algorithms
2, 3, 4.

Methods |  | K-U-Net |  | PatchTST |  | Nlinear |  | Dlinear |  | FEDformer |  | Autoformer |  | Informer |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Metric |  | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
ETTh1 | 96 | 0.369 | 0.404 | 0.37 | 0.4 | 0.374 | 0.394 | 0.375 | 0.399 | 0.376 | 0.419 | 0.449 | 0.459 | 0.865 | 0.713
| 192 | 0.397 | 0.426 | 0.413 | 0.429 | 0.408 | 0.415 | 0.405 | 0.416 | 0.42 | 0.448 | 0.5 | 0.482 | 1.008 | 0.792
| 336 | 0.429 | 0.428 | 0.422 | 0.44 | 0.429 | 0.427 | 0.439 | 0.443 | 0.459 | 0.465 | 0.521 | 0.496 | 1.107 | 0.809
| 720 | 0.433 | 0.45 | 0.447 | 0.468 | 0.44 | 0.453 | 0.472 | 0.49 | 0.506 | 0.507 | 0.514 | 0.512 | 1.181 | 0.865
ETTh2 | 96 | 0.271 | 0.336 | 0.274 | 0.337 | 0.277 | 0.338 | 0.289 | 0.353 | 0.346 | 0.388 | 0.358 | 0.397 | 3.755 | 1.525
| 192 | 0.332 | 0.376 | 0.339 | 0.379 | 0.344 | 0.381 | 0.383 | 0.418 | 0.429 | 0.439 | 0.456 | 0.452 | 5.602 | 1.931
| 336 | 0.357 | 0.4 | 0.329 | 0.384 | 0.357 | 0.4 | 0.448 | 0.465 | 0.496 | 0.487 | 0.482 | 0.486 | 4.721 | 1.835
| 720 | 0.39 | 0.438 | 0.379 | 0.422 | 0.394 | 0.436 | 0.605 | 0.551 | 0.463 | 0.474 | 0.515 | 0.511 | 3.647 | 1.625
ETTm1 | 96 | 0.285 | 0.337 | 0.29 | 0.342 | 0.306 | 0.348 | 0.299 | 0.343 | 0.379 | 0.419 | 0.505 | 0.475 | 0.672 | 0.571
| 192 | 0.327 | 0.367 | 0.332 | 0.369 | 0.349 | 0.375 | 0.335 | 0.365 | 0.426 | 0.441 | 0.553 | 0.496 | 0.795 | 0.669
| 336 | 0.357 | 0.392 | 0.366 | 0.392 | 0.375 | 0.388 | 0.369 | 0.386 | 0.445 | 0.459 | 0.621 | 0.537 | 1.212 | 0.871
| 720 | 0.408 | 0.422 | 0.416 | 0.42 | 0.433 | 0.422 | 0.425 | 0.421 | 0.543 | 0.49 | 0.671 | 0.561 | 1.166 | 0.823
ETTm2 | 96 | 0.16 | 0.245 | 0.165 | 0.255 | 0.167 | 0.255 | 0.167 | 0.26 | 0.203 | 0.287 | 0.255 | 0.339 | 0.365 | 0.453
| 192 | 0.215 | 0.215 | 0.22 | 0.292 | 0.221 | 0.293 | 0.224 | 0.303 | 0.269 | 0.328 | 0.281 | 0.34 | 0.533 | 0.563
| 336 | 0.268 | 0.326 | 0.274 | 0.329 | 0.274 | 0.327 | 0.281 | 0.342 | 0.325 | 0.366 | 0.339 | 0.372 | 1.363 | 0.887
| 720 | 0.348 | 0.378 | 0.362 | 0.385 | 0.368 | 0.384 | 0.397 | 0.421 | 0.421 | 0.415 | 0.433 | 0.432 | 3.379 | 1.338
Electricity | 96 | 0.129 | 0.225 | 0.129 | 0.222 | 0.141 | 0.237 | 0.14 | 0.237 | 0.193 | 0.308 | 0.201 | 0.317 | 0.274 | 0.368
| 192 | 0.148 | 0.244 | 0.147 | 0.24 | 0.154 | 0.248 | 0.153 | 0.249 | 0.201 | 0.315 | 0.222 | 0.334 | 0.296 | 0.386
| 336 | 0.168 | 0.263 | 0.163 | 0.259 | 0.171 | 0.265 | 0.169 | 0.267 | 0.214 | 0.329 | 0.231 | 0.338 | 0.3 | 0.394
| 720 | 0.205 | 0.299 | 0.197 | 0.29 | 0.21 | 0.297 | 0.203 | 0.301 | 0.246 | 0.355 | 0.254 | 0.361 | 0.373 | 0.439
Traffic | 96 | 0.360 | 0.252 | 0.36 | 0.249 | 0.41 | 0.279 | 0.41 | 0.282 | 0.587 | 0.366 | 0.613 | 0.388 | 0.719 | 0.391
| 192 | 0.377 | 0.263 | 0.379 | 0.25 | 0.423 | 0.284 | 0.423 | 0.287 | 0.604 | 0.373 | 0.616 | 0.382 | 0.696 | 0.379
| 336 | 0.389 | 0.270 | 0.392 | 0.264 | 0.435 | 0.29 | 0.436 | 0.296 | 0.621 | 0.383 | 0.622 | 0.337 | 0.777 | 0.42
| 720 | 0.427 | 0.290 | 0.432 | 0.286 | 0.464 | 0.307 | 0.466 | 0.315 | 0.626 | 0.382 | 0.66 | 0.408 | 0.864 | 0.472
Weather | 96 | 0.146 | 0.200 | 0.149 | 0.198 | 0.182 | 0.232 | 0.176 | 0.237 | 0.217 | 0.296 | 0.266 | 0.336 | 0.3 | 0.384
| 192 | 0.192 | 0.244 | 0.194 | 0.241 | 0.225 | 0.269 | 0.22 | 0.282 | 0.276 | 0.336 | 0.307 | 0.367 | 0.598 | 0.544
| 336 | 0.241 | 0.287 | 0.245 | 0.282 | 0.271 | 0.301 | 0.265 | 0.319 | 0.339 | 0.38 | 0.359 | 0.395 | 0.578 | 0.523
| 720 | 0.310 | 0.340 | 0.314 | 0.334 | 0.338 | 0.348 | 0.323 | 0.362 | 0.403 | 0.428 | 0.419 | 0.428 | 1.059 | 0.741
Table 1: Multivariate time series forecasting results with Kernel U-Net. The
prediction lengths $T$ are in {96, 192, 336, 720} for all datasets. We note
the best results in bold and the second best results in underlined.

###  3.3 Custom Kernels

Linear kernel
The linear kernel is a simple matrix multiplication. Given $X$ of shape
$(B,L_{i},M_{i})$, linear kernel $f_{i}$ reshape it into $(B,L_{i}*M_{i})$ and
process it as follow:

$Z_{i}=f_{i}(X)=X*w+b$

, where $Z_{i}$ is output of kernel, $w\in R^{L_{i}M_{i}\times
L_{i+1}M_{i+1}}$ is weight matrix and $b\in R^{L_{i+1}M_{i+1}}$ is bias
vector. Remark that linear Kernel U-Net is an equivalent version to
convolutional U-Net whose stride size is $L_{i}$. We also test a non-linear
kernel that takes a non-linear activation function such as Tanh in the
intermediate hidden layers.

Transformer kernel
The Vanilla Transformer is made of a layer of positional encoding, blocks that
are composed of layers of multiple head attentions, and a linear layer with
SoftMax activation.Wen et al. (2023)

Positional encoding (PE).
In a vanilla Transformer, the input sequence is added by a positional encoding
vector given by:

$PE(t)=\left\\{\begin{array}[]{ll}sin(\omega it)&ifi\%2=0\\\ cos(\omega
it)&ifi\%2=1\\\ \end{array}\right.$

, where $\omega i$ is the frequency of waves generated at each dimension. By
adding positional information to the input, the transformer can consider the
same patterns that occur in different time steps as different patterns.

Multiple head attentions.
With the Query-Key-Value $(QKV)$ model, the scaled dot-product attention used
by Transformer is given by

$Attention(Q,K,V)=Softmax(\frac{QK^{T}}{\sqrt{D_{k}}}))V$

, where queries $Q\in R^{N\times D_{k}}$ , keys $K\in R^{M\times D_{k}}$ ,
values $V\in R^{M\times D_{v}}$ , $N,M$ denote the lengths of queries and keys
(or values), and $D_{k},D_{v}$ denote the dimensions of keys (or queries) and
values. Remark that in the context of self-attention, $Q=K=V=inputdata$.

The multiple attention block contains repeated heads and can be given by

$MultiHeadAttn(Q,K,V)=Concat(h_{1},...,h_{H})W^{O}$

, where $h_{i}=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}).$

LSTM kernel
The Long Short-term memory contains 3 gates: Forget Gate $f_{t}$, Input Gate
$i_{t}$, Output Gate $o_{t}$. Each gate has a weight matrix $W$ and a bias
vector $b$ for the current input $x_{t}$ and the previous hidden state
$h_{t-1}$. The next cell state $C_{t}$ is updated by the current activated
cell state $C^{\prime}_{t}$. The next hidden state $h_{t}$ is updated by the
output gate and the next activated cell state. The formulation of LSTM is as
follows:

$f_{t}=\sigma(W_{f}[h_{t-1},x_{t}]+b_{f})$
---
$i_{t}=\sigma(W_{i}[h_{t-1},x_{t}]+b_{i})$
$C^{\prime}_{t}=tanh(W_{C}[h_{t-1},x_{t}]+b_{C})$
$C_{t}=f_{t}C_{t-1}+i_{t}C^{\prime}_{t}$
$o_{t}=\sigma(W_{o}[h_{t-1},x_{t}]+b_{o})$
$h_{t}=o_{t}tanh(C_{t})$

Methods |  | K-U-Net |  | PatchTST |  | Nlinear |  | Dlinear |  | FEDformer |  | Autoformer |  | LogTrans |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Metric |  | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |  |
ETTh1 | 96 | 0.052 | 0.176 | 0.055 | 0.179 | 0.053 | 0.177 | 0.056 | 0.18 | 0.079 | 0.215 | 0.071 | 0.206 | 0.283 | 0.468 |  |  |
| 192 | 0.067 | 0.202 | 0.071 | 0.215 | 0.069 | 0.204 | 0.071 | 0.204 | 0.104 | 0.245 | 0.114 | 0.262 | 0.234 | 0.409 |  |  |
| 336 | 0.074 | 0.22 | 0.076 | 0.22 | 0.081 | 0.226 | 0.098 | 0.244 | 0.119 | 0.27 | 0.107 | 0.258 | 0.386 | 0.546 |  |  |
| 720 | 0.074 | 0.223 | 0.087 | 0.232 | 0.08 | 0.226 | 0.189 | 0.359 | 0.127 | 0.28 | 0.126 | 0.283 | 0.475 | 0.629 |  |  |
ETTh2 | 96 | 0.114 | 0.262 | 0.129 | 0.282 | 0.129 | 0.278 | 0.131 | 0.279 | 0.128 | 0.271 | 0.153 | 0.306 | 0.217 | 0.379 |  |  |
| 192 | 0.149 | 0.304 | 0.168 | 0.328 | 0.169 | 0.324 | 0.176 | 0.329 | 0.185 | 0.33 | 0.204 | 0.351 | 0.281 | 0.429 |  |  |
| 336 | 0.158 | 0.319 | 0.171 | 0.336 | 0.194 | 0.355 | 0.209 | 0.367 | 0.231 | 0.378 | 0.246 | 0.389 | 0.293 | 0.437 |  |  |
| 720 | 0.178 | 0.34 | 0.223 | 0.38 | 0.225 | 0.381 | 0.276 | 0.426 | 0.278 | 0.42 | 0.268 | 0.409 | 0.218 | 0.387 |  |  |
ETTm1 | 96 | 0.026 | 0.122 | 0.026 | 0.121 | 0.026 | 0.122 | 0.028 | 0.123 | 0.033 | 0.14 | 0.056 | 0.183 | 0.049 | 0.171 |  |  |
| 192 | 0.039 | 0.15 | 0.039 | 0.15 | 0.039 | 0.149 | 0.045 | 0.156 | 0.058 | 0.186 | 0.081 | 0.216 | 0.157 | 0.317 |  |  |
| 336 | 0.051 | 0.173 | 0.053 | 0.173 | 0.052 | 0.172 | 0.061 | 0.182 | 0.071 | 0.209 | 0.076 | 0.218 | 0.289 | 0.459 |  |  |
| 720 | 0.068 | 0.202 | 0.073 | 0.206 | 0.073 | 0.207 | 0.08 | 0.21 | 0.102 | 0.25 | 0.11 | 0.267 | 0.43 | 0.579 |  |  |
ETTm2 | 96 | 0.061 | 0.181 | 0.065 | 0.187 | 0.063 | 0.182 | 0.063 | 0.183 | 0.063 | 0.189 | 0.065 | 0.189 | 0.075 | 0.208 |  |  |
| 192 | 0.085 | 0.22 | 0.093 | 0.231 | 0.09 | 0.223 | 0.092 | 0.227 | 0.102 | 0.245 | 0.118 | 0.256 | 0.129 | 0.275 |  |  |
| 336 | 0.104 | 0.25 | 0.121 | 0.266 | 0.117 | 0.259 | 0.119 | 0.261 | 0.13 | 0.279 | 0.154 | 0.305 | 0.154 | 0.302 |  |  |
| 720 | 0.149 | 0.303 | 0.171 | 0.322 | 0.17 | 0.318 | 0.175 | 0.32 | 0.178 | 0.325 | 0.182 | 0.335 | 0.16 | 0.321 |  |  |
Table 2: Univariate time series forecasting results with Kernel U-Net. The
prediction lengths $T$ are in {96, 192, 336, 720} for all datasets. We note
the best results in bold and the second best results in underlined.

The LSTM Kernel used in our experiment contains an LSTM cell and a linear
layer that takes all hidden states of a sequence as input and gives a vector
as output. The experimental result is shown in the section 4

##  4 Experiments and Results

Datasets.
We conducted experiments on our proposed Kernel U-Net on 7 public datasets,
including 4 ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2), Weather, Traffic and
Electricity. These datasets have been benchmarked and publicly available onWu
et al. (2021) and the description of the dataset is available in Zeng et al.
(2023)).

Here, we followed the experiment setting in Zeng et al. (2023) and partitioned
the data into 12, 4, and 4 months for training, validation, and testing
respectively for the ETT dataset. The data is split into [0.7,0.1,0.2] for
training, validation, and testing for dataset Weather, Traffic, and
Electricity.

Baselines and Experimental Settings.
In the time series forecasting task, we follow the experiment setting in
NLinear Zeng et al. (2023), which take $L={336,720}$ step historical data as
input and forecast $T\in{96,192,336,720}$ step value in the future. We replace
the last value normalization with mean value normalization for univariate
forecasting in the ETT dataset. We include five recent methods: PatchTSTNie et
al. (2023), NLinear, DLinearZeng et al. (2023), FEDformer Zhou et al. (2022),
Autoformer Wu et al. (2021), Informer Zhou et al. (2021)and LogTrans Li et al.
(2019). We merge the result reported in Nie et al. (2023) and Zeng et al.
(2023) for taking their best in a supervised setting.

Following previous works Wu et al. (2021), we use Mean Squared Error (MSE) and
Mean Absolute Error (MAE) as the core metrics to compare performance for
Forecasting problems. We take a look back at Windows 336 and 720 in
experiments. The list of multiples is respectively [3, 4, 4, 5] and [4, 4, 7]
and the segment-unit input length is 3. The input dimension is 1 as we follow
the channel-independent setting in Zeng et al. (2023). The hidden dimensions
are 128 or 256 for multivariate tasks and 5 or 30 for univariate tasks
depending on of dataset. We use the Random Erase Zhong et al. (2020) from
image processing for data augmentation. The learning rate is selected in
[0.00005, 0.001] depending on the dataset. The training epoch is 300 and the
patience is 30 in general.

Instance Normalization and Mean Normalization Instance normalization Ulyanov
et al. (2017) Kim et al. (2021) contains a step of substruction with the
average value of the input sequence and a division of its standard deviation.
It computes reversely the multiplication of its standard deviation and the
addition of the average value after the output of the model. Mean
normalization removes the division and multiplication step of the standard
deviation of instance normalization to avoid introducing noises in the small
dataset. We followed the mean normalization as the preprocessing step and
compared it with instance normalization for selected models.

Model Variants.
We propose 4 kernels and 5 versions of kernel U-Net for experiments. The
kernels tested are Linear, Linear with 1 hidden layer and non-linear
activation, Transformer, and LSTM.

  * •

Linear U-Net is a baseline method that contains a list of linear kernels.

  * •

Linear-1-Hidden U-Net replaces 2 kernels close to latent vector to kernels
with hidden layer and nonlinear activation.

  * •

Linear-5-Hidden U-Net replaces all linear kernels with non-linear kernels.

  * •

Transformer U-Net replaces 2 kernels close to latent vector to transformer
kernels and keeps the rest as linear kernels.

  * •

LSTM U-Net replaces 2 kernels close to latent vector to LSTM kernels and keeps
the rest as linear kernels.

The result shows that the performance of Kernel U-Net reaches or outperforms
the state-of-art methods in univariate setting (Table 2) and reaches similar
or better results in multivariate setting (Table 1).

##  5 Conclusion

In this paper, we propose Kernel U-Net, a symmetric and hierarchical framework
that can effectively process long temporal dependencies with $O(log(L)^{2})$
number of parameters. Specifically, Kernel U-Net cut input sequences into
small slices at each layer in the network for applying customized kernels.

In the future, we hope to develop more kernels that adapt well to particular
datasets. We also hope that Kernel U-Net can be useful for other time series
tasks such as classification or anomaly detection.

## References

  * Asres et al. [2022] Mulugeta Weldezgina Asres, Grace Cummings, Aleko Khukhunaishvili, Pavel Parygin, Seth I. Cooper, David Yu, Jay Dittmann, and Christian W. Omlin.  Long horizon anomaly prediction in multivariate time series with causal autoencoders.  PHM Society European Conference, 7(1):21–31, 2022.
  * Azar et al. [2022] Joseph Azar, Gaby Bou Tayeh, Abdallah Makhoul, and Raphaël Couturier.  Efficient lossy compression for iot using sz and reconstruction with 1d u-net.  Mob. Netw. Appl., 27(3):984–996, jun 2022.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * Byra et al. [2020] Michal Byra, Piotr Jarosik, Aleksandra Szubert, Michael Galperin, Haydee Ojeda-Fournier, Linda Olson, Mary O’Boyle, Christopher Comstock, and Michael Andre.  Breast mass segmentation in ultrasound with selective kernel u-net convolutional neural network.  Biomedical Signal Processing and Control, 61:102027, 2020.
  * Cao et al. [2023] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  In Leonid Karlinsky, Tomer Michaeli, and Ko Nishino, editors, Computer Vision – ECCV 2022 Workshops, pages 205–218. Springer Nature Switzerland, 2023.
  * Deng et al. [2009] Jia Deng, R. Socher, Li Fei-Fei, Wei Dong, Kai Li, and Li-Jia Li.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), volume 00, pages 248–255, 06 2009.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * Du et al. [2015] Yong Du, Wei Wang, and Liang Wang.  Hierarchical recurrent neural network for skeleton based action recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1110–1118, 2015.
  * Duan et al. [2022] Ziheng Duan, Haoyan Xu, Yueyang Wang, Yida Huang, Anni Ren, Zhongbin Xu, Yizhou Sun, and Wei Wang.  Multivariate time-series classification with hierarchical variational graph pooling.  Neural Networks, 154:481–490, 2022.
  * Elsayed et al. [2019] Nelly Elsayed, Anthony S. Maida, and Magdy Bayoumi.  Deep gated recurrent and convolutional network hybrid model for univariate time series classification.  International Journal of Advanced Computer Science and Applications, 10(5), 2019.
  * Han et al. [2022] Ning Han, Li Zhou, Zhengmao Xie, Jingli Zheng, and Liuxin Zhang.  Multi-level u-net network for image super-resolution reconstruction.  Displays, 73:102192, 2022.
  * Hewage et al. [2020] Pradeep Hewage, Ardhendu Behera, Marcello Trovati, Ella Pereira, Morteza Ghahremani, Francesco Palmieri, and Yonghuai Liu.  Temporal convolutional neural (TCN) network for an effective weather forecasting using time-series data from the local weather station.  Soft Computing, 24(21):16453–16482, 2020.
  * Hong and Yoon [2017] Jeonghan Hong and Junho Yoon.  Multivariate time-series classification of sleep patterns using a hybrid deep learning architecture.  In 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom), pages 1–6, 2017.
  * Huang et al. [2020a] Huimin Huang, Lanfen Lin, Ruofeng Tong, Hongjie Hu, Qiaowei Zhang, Yutaro Iwamoto, Xianhua Han, Yen-Wei Chen, and Jian Wu.  UNet 3+: A full-scale connected UNet for medical image segmentation.  In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1055–1059, 2020.  ISSN: 2379-190X.
  * Huang et al. [2020b] Shaohan Huang, Yi Liu, Carol Fung, Rong He, Yining Zhao, Hailong Yang, and Zhongzhi Luan.  HitAnomaly: Hierarchical transformers for anomaly detection in system log.  IEEE Transactions on Network and Service Management, 17(4):2064–2076, 2020.  Conference Name: IEEE Transactions on Network and Service Management.
  * Jia et al. [2022] Xi Jia, Joseph Bartlett, Tianyang Zhang, Wenqi Lu, Zhaowen Qiu, and Jinming Duan.  U-net vs transformer: Is u-net outdated in medical image registration?  In Chunfeng Lian, Xiaohuan Cao, Islem Rekik, Xuanang Xu, and Zhiming Cui, editors, Machine Learning in Medical Imaging, pages 151–160. Springer Nature Switzerland, 2022.
  * Khan et al. [2021] Mehak Khan, Hongzhi Wang, Adnan Riaz, Aya Elfatyany, and Sajida Karim.  Bidirectional LSTM-RNN-based hybrid deep learning frameworks for univariate time series classification.  The Journal of Supercomputing, 77(7):7021–7045, 2021.
  * Khandelwal et al. [2015] Ina Khandelwal, Ratnadip Adhikari, and Ghanshyam Verma.  Time series forecasting using hybrid ARIMA and ANN models based on DWT decomposition.  Procedia Computer Science, 48:173–179, 2015.
  * Kim et al. [2021] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo.  Reversible instance normalization for accurate time-series forecasting against distribution shift.  In International Conference on Learning Representations, 2021.
  * Kong et al. [2017] Weicong Kong, Z.Y. Dong, Youwei Jia, David Hill, Yan Xu, and Yuan Zhang.  Short-term residential load forecasting based on LSTM recurrent neural network.  IEEE Transactions on Smart Grid, PP:1–1, 2017.
  * Kowsari et al. [2017] Kamran Kowsari, Donald E. Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S. Gerber, and Laura E. Barnes.  HDLTex: Hierarchical deep learning for text classification.  In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 364–371, 2017.
  * Lai et al. [2018] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu.  Modeling long- and short-term temporal patterns with deep neural networks.  In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 95–104, New York, NY, USA, 2018. Association for Computing Machinery.
  * Li et al. [2019] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan.  Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting, page 11.  Curran Associates Inc., Red Hook, NY, USA, 2019.
  * Lin et al. [2020] Shuyu Lin, Ronald Clark, Robert Birke, Sandro Schonborn, Niki Trigoni, and Stephen Roberts.  Anomaly detection for time series using VAE-LSTM hybrid model.  In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4322–4326. IEEE, 2020.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9992–10002. IEEE, 2021.
  * Liu et al. [2022a] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar.  Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting.  In International Conference on Learning Representations, 2022.
  * Liu et al. [2022b] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long.  Non-stationary transformers: Exploring the stationarity in time series forecasting.  Advances in Neural Information Processing Systems, 2022.
  * Madhusudhanan et al. [2023] Kiran Madhusudhanan, Johannes Burchert, Nghia Duong-Trung, Stefan Born, and Lars Schmidt-Thieme.  U-net inspired transformer architecture for far horizon time series forecasting.  In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part VI, page 36–52, Berlin, Heidelberg, 2023. Springer-Verlag.
  * Nie et al. [2023] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam.  A time series is worth 64 words: Long-term forecasting with transformers.  International Conference on Learning Representations, 2023.
  * Perslev et al. [2019] Mathias Perslev, Michael Jensen, Sune Darkner, Poul Jø rgen Jennum, and Christian Igel.  U-time: A fully convolutional network for time series segmentation applied to sleep staging.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * Persson et al. [2017] Caroline Persson, Peder Bacher, Takahiro Shiga, and Henrik Madsen.  Multi-site solar power forecasting using gradient boosted regression trees.  Solar Energy, 150:423–436, 2017.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241. Springer International Publishing, 2015.
  * Tokgöz and Ünal [2018] Alper Tokgöz and Gözde Ünal.  A RNN based time series approach for forecasting turkish electricity load.  In 2018 26th Signal Processing and Communications Applications Conference (SIU), pages 1–4, 2018.
  * Ulyanov et al. [2017] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Instance normalization: The missing ingredient for fast stylization, 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * Wen et al. [2023] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun.  Transformers in time series: A survey.  In International Joint Conference on Artificial Intelligence(IJCAI), 2023.
  * Wu et al. [2021] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long.  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 22419–22430. Curran Associates, Inc., 2021.
  * Xiao and Liu [2016] Minguang Xiao and Cong Liu.  Semantic relation classification via hierarchical recurrent neural network with attention.  In Yuji Matsumoto and Rashmi Prasad, editors, Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1254–1263. The COLING 2016 Organizing Committee, 2016.
  * Xiao et al. [2018] Xiao Xiao, Shen Lian, Zhiming Luo, and Shaozi Li.  Weighted res-UNet for high-quality retina vessel segmentation.  In 2018 9th International Conference on Information Technology in Medicine and Education (ITME), pages 327–331, 2018.  ISSN: 2474-3828.
  * Zeng et al. [2023] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu.  Are transformers effective for time series forecasting?  In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
  * Zhang et al. [2023] Kai Zhang, Yawei Li, Jingyun Liang, Jiezhang Cao, Yulun Zhang, Hao Tang, Deng-Ping Fan, Radu Timofte, and Luc Van Gool.  Practical blind image denoising via swin-conv-UNet and data synthesis.  Mach. Intell. Res., 2023.
  * Zhong et al. [2020] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
  * Zhou et al. [2018] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang.  UNet++: A nested u-net architecture for medical image segmentation.  In Danail Stoyanov, Zeike Taylor, Gustavo Carneiro, Tanveer Syeda-Mahmood, Anne Martel, Lena Maier-Hein, João Manuel R.S. Tavares, Andrew Bradley, João Paulo Papa, Vasileios Belagiannis, Jacinto C. Nascimento, Zhi Lu, Sailesh Conjeti, Mehdi Moradi, Hayit Greenspan, and Anant Madabhushi, editors, Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Lecture Notes in Computer Science, pages 3–11. Springer International Publishing, 2018.
  * Zhou et al. [2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11106–11115. AAAI Press, 2021.
  * Zhou et al. [2022] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin.  FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting.  In Proceedings of the 39th International Conference on Machine Learning, pages 27268–27286. PMLR, 2022.  ISSN: 2640-3498.
  * Çiçek et al. [2016] Özgün Çiçek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: Learning dense volumetric segmentation from sparse annotation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2016.

## Ablation study

Ablation study Ablation study


# Rayleigh EigenDirections (REDs):
GAN latent space traversals for multidimensional features

Guha Balakrishnan
Rice University
<EMAIL_ADDRESS>Raghudeep Gadde
Amazon
<EMAIL_ADDRESS>Aleix Martinez
Amazon
<EMAIL_ADDRESS>Pietro Perona
Amazon Web Services
<EMAIL_ADDRESS>
###### Abstract

We present a method for finding paths in a deep generative model’s latent
space that can maximally vary one set of image features while holding others
constant. Crucially, unlike past traversal approaches, ours can manipulate
multidimensional features of an image such as facial identity and pixels
within a specified region. Our method is principled and conceptually simple:
optimal traversal directions are chosen by maximizing differential changes to
one feature set such that changes to another set are negligible. We show that
this problem is nearly equivalent to one of Rayleigh quotient maximization,
and provide a closed-form solution to it based on solving a generalized
eigenvalue equation. We use repeated computations of the corresponding optimal
directions, which we call Rayleigh EigenDirections (REDs), to generate
appropriately curved paths in latent space. We empirically evaluate our method
using StyleGAN2 on two image domains: faces and living rooms. We show that our
method is capable of controlling various multidimensional features out of the
scope of previous latent space traversal methods: face identity, spatial
frequency bands, pixels within a region, and the appearance and position of an
object. Our work suggests that a wealth of opportunities lies in the local
analysis of the geometry and semantics of latent spaces.

Figure 1: Method and examples. (A) Our method traverses the local latent space
around a ‘seed point’ $\mathbf{z}_{0}$ along optimally chosen paths to
synthesize images that share the same high-dimensional attribute value
$\mathbf{y}_{0}$ (e.g., identity), and vary as much as possible across other
image attributes (e.g., lighting, expression, age, hairstyle). (B) Sample of
an identity-preserving face traversal and (C) a living room traversal with a
fixed object (lamp in red frame of seed image) in the latent space of
StyleGAN2 generators.

##  1 Introduction

Latent spaces of deep generative networks like generative adversarial networks
(GANs) [12, 16, 17, 28] and variational autoencoders (VAEs) [18] are known to
organize semantic attributes into disentangled subspaces without supervision
[13, 15, 28, 36, 38]. This property is the basis of several _traversal_
algorithms proposed in the literature that can modify specific image
attributes while holding others constant by moving along carefully-chosen
latent space directions [4, 11, 27, 30, 41]. Traversal methods have many
potential applications including dataset creation/augmentation, image editing,
entertainment and graphic design.

Virtually all existing traversal methods assume _scalar_ attributes of
interest that may be modeled well with global linear functions, e.g., a linear
regressor or a support vector machine, in the latent space. This approach
works well for attributes like gender, hair color and smile of faces [4, 30]
and image transformations like translation, color change and camera movements
[15, 27]. But these approaches cannot be easily extended to work with
attributes like ‘style of a couch’ and ‘face identity’ which are best
described with high-dimensional vectors.111There is no individual person
behind a GAN-generated portrait and therefore there is no physical ‘identity’
ground truth. However, human observers or face recognition algorithms can
respond to the question “Is this the same person?” and can produce consistent
judgments. Therefore ‘identity’ here denotes ‘perceptual identity’. For
example, to find a latent space traversal that preserves identity in our
experiments, we need a representation that can compute the similarity between
two $512$-dimensional embeddings returned by a face recognition model [9]. In
addition, faces with the same identity or rooms with the same furniture layout
(see Fig. 1C) tend to be tightly clustered in latent space, requiring methods
tuned to local latent space geometry unlike the common global linear models
used for scalar attributes.

We propose a method to tackle this broader class of traversal problems. Given
a point in latent space, we aim to generate many traversals, or sequences of
images, such that we vary one multidimensional feature ($\mathbf{x}$) in as
many ways as possible subject to other multidimensional features
($\mathbf{y}$) being held approximately constant. We formalize the task of
finding local latent directions that fulfill these criteria as a constrained
optimization problem. By using differential approximations of the feature
functions, we recast the problem into an instance of Rayleigh quotient
maximization, which has a well-known closed-form solution (Sec. 3.1). The
principal directions that solve this problem, which we call Rayleigh
EigenDirections(REDs), span the local latent subspace containing good paths.
Using REDs, we propose a fast linear and more accurate iterative nonlinear
projection traversal algorithm (Sec. 3.3) to produce arbitrary-length paths.
Our approach is agnostic to network architecture, scene content, and choice of
attribute embedding functions.

We evaluate our method using StyleGAN2 [16, 17] generators. We consider a
number of challenging applications outside the scope of previous GAN traversal
algorithms: face traversals that preserve identity (Fig. 3) while changing
hairstyle and facial geometries, face traversals that preserve/change content
from specific spatial frequency bands (Fig. 5), and living room traversals
that preserve the appearance and location of selected pieces of furniture
(Fig. 4). We provide a number of qualitative results demonstrating the
perceptual quality of our generated image sequences, and quantitatively
demonstrate the necessity for nonlinear traversal strategies in these
applications. Finally, we also compare our method against well-known global
linear model baselines [4, 30] for scalar attributes and perform comparably,
though with some failure cases that we discuss in Sec. 5.1.

Our main contributions are: (a) REDs, a local method for synthesizing a
diverse set of images that share a chosen set of multidimensional attribute.
The method is principled, simple, and versatile – applicable to pretrained
generators, any image type, and to both low-level and semantically meaningful
features. (b) A nonlinear technique for long-distance traversals in latent
space; (c) Qualitative and quantitative validation experiments on a number of
challenging synthesis tasks in two different image domains.

Figure 2: Comparison of _Linear_ and _Projection_ traversal. We show a
_Linear_ and _Projection_ traversal originating from the same latent seed code
(left-most face), and top RED vector at the seed. $f(\cdot)$ measures identity
and $c(\cdot)$ measures raw face pixel values. We also plot squared pixel
distance versus squared identity distance. _Projection_ and _Linear_ change
pixels by roughly the same amount, but _Projection_ is better at preserving
identity (lower distance values).

##  2 Related Work

Several studies focus on finding interpretable directions in GAN latent spaces
for editing and synthesizing images. Most propose finding global linear
directions correlated with scalar attributes of interest [4, 13, 11, 27, 30,
37, 41]. One popular technique is to train a linear predictor, e.g., SVM, from
latent codes and corresponding attribute labels, and use the norm of the
learned hyperplane as the traversal direction [4, 30]. We find that
multidimensional features like face identity and hairstyle lie on complex
manifolds in latent space rather than on simple linear ones, requiring
locally-varying, nonlinear traversals. We propose an algorithm (see Sec. 3.3)
that forms nonlinear paths by sequentially taking locally optimal linear
steps. A few nonlinear traversal strategies do exist in the literature,
typically based on training nonlinear neural networks to map latent codes to
features [15, 35, 42]. Our method is complementary to these – ours requires no
additional training, but also does not leverage global latent space structure
as theirs presumably can. Finally, our focus on a local rather than global
view of the latent space may also complement various theoretical studies on
understanding GAN latent space structure [3, 5, 7, 20, 29, 38].

A more explicit way to control GAN outputs is to train the generator using
attribute values as inputs. Many of these so-called “conditional GANs” have
been proposed, particularly for altering face attributes [2, 6, 8, 14, 19, 21,
22, 23, 24, 25, 33, 34, 40, 43], controlling face identity [6, 31, 32, 33],
and conditioning on semantic maps [26, 39]. Our approach is complementary to
all of these in that offers the benefit of not needing to design and train a
GAN from scratch with apriori-known attribute controls. Working with a
general-purpose black-box GAN has the advantage of keeping all control
objectives open and not committing to a specific goal, e.g. preserving
identity, from the beginning.

##  3 Method

Given a point $\mathbf{z}_{0}\in\mathcal{R}^{d}$ in latent space defining an
image, we want to generate a set of images that holds fixed the
multidimensional features $\mathbf{y}_{0}\in\mathcal{R}^{n}$ while maximally
changing the features $\mathbf{x}_{0}\in\mathcal{R}^{m}$. For ease of
explanation, we assume $\mathbf{y}_{0}$ and $\mathbf{x}_{0}$ each define a
single multidimensional feature like facial identity or hairstyle, though our
method easily handles features from multiple semantic attributes as explained
in Sec. 3.2.

We denote the function that computes the fixed features
$f(\cdot):\mathbf{z}\rightarrow\mathbf{y}\in\mathcal{R}^{n}$, and the function
that computes the changing features
$c(\cdot):\mathbf{z}\rightarrow\mathbf{x}\in\mathcal{R}^{m}$. For example, in
one of our experiments with faces, $f(\cdot)$ is the concatenation of two
functions: the GAN generator on the input latent vector, and a face
recognition embedding model on the synthesized face. $c(\cdot)$ may be the
generator itself (i.e., $\mathbf{x}$ are the raw pixels of the image) or the
concatenation of the generator with learning models computing various image
attributes.

Starting at $\mathbf{z}_{0}$, our method traverses different paths in latent
space to generate latent code sequences. For each such trajectory $t$ of
length $L$, $\mathbf{z}_{0},\mathbf{z}^{t}_{1}\cdots,\mathbf{z}^{t}_{L}$, we
want $\mathbf{y}^{t}_{i}\approx\mathbf{y}_{0}$ for all $i$ and
$\mathbf{x}_{0},\mathbf{x}^{t}_{1},\cdots,\mathbf{x}^{t}_{L}$ to progressively
change such that
$\|\mathbf{x}^{t}_{i}-\mathbf{x}^{t}_{i+1}\|<\|\mathbf{x}^{t}_{i}-\mathbf{x}^{t}_{i+2}\|$,
where $\|\cdot\|$ is a norm. We return all points from all sequences.

The key intuition behind our approach is that there exists a manifold on which
$\mathbf{y}$ does not change around $\mathbf{z}_{0}$ (see Fig. 1). This is
true whenever $d>n$ (and thus the iso-$\mathbf{y}$ manifold has dimension
$n-d$) and the generator function is continuous (which, by inspection, it is,
apart from a zero-size set). When $d\leq n$, our approach naturally
transitions to a “soft” constraint $\mathbf{y}_{i}\approx\mathbf{y}_{0}$ as
will become clear below. We find directions, which we call Rayleigh
EigenDirections (REDs), that maximally change $\mathbf{x}$ within this
subspace. This procedure is described in Sec. 3.1. We propose two traversal
strategies using REDs in Sec. 3.3: a linear method which simply extrapolates
the local REDs throughout the latent space, and a nonlinear method
(_Projection_) which updates traversal directions based on local latent space
geometry.

###  3.1 Rayleigh EigenDirections (REDs)

Let $\mathbf{z}$ be a generic point in the generator’s latent space with fixed
and changing features $\mathbf{y}=f(\mathbf{z})$ and
$\mathbf{x}=c(\mathbf{z})$. Given a displacement $\delta\mathbf{z}$, the
displacements to $\mathbf{y}$ and $\mathbf{x}$ are:

$\displaystyle\delta\mathbf{y}$
$\displaystyle=f(\mathbf{z}+\delta\mathbf{z})-f(\mathbf{z})$ (1)
$\displaystyle\delta\mathbf{x}$
$\displaystyle=c(\mathbf{z}+\delta\mathbf{z})-c(\mathbf{z}).$ (2)

We aim to find the displacement $\delta\mathbf{z}^{*}$ that maximizes
$\delta\mathbf{x}$ with insignificant changes to $\delta\mathbf{y}$:

$\displaystyle\delta\mathbf{z}^{*}=$
$\displaystyle\operatorname*{argmax}_{\delta\mathbf{z}:\|\delta\mathbf{z}\|=\epsilon}\|\delta\mathbf{x}(\mathbf{z},\delta\mathbf{z})\|^{2}$
(3) $\displaystyle\text{ s.t.
}\|\delta\mathbf{y}(\mathbf{z},\delta\mathbf{z})\|^{2}\approx 0,$ (4)

where we write $\delta\mathbf{x}$ and $\delta\mathbf{y}$ as functions of
$\mathbf{z}$ and $\delta\mathbf{z}$, and $\epsilon$ is a small, fixed
constant. For sufficiently small $\epsilon$, we can approximate
$\delta\mathbf{y}$ and $\delta\mathbf{x}$ with local linear expansions:
$\delta\mathbf{y}\approx J_{f}(\mathbf{z})\delta\mathbf{z}$ and
$\delta\mathbf{x}\approx J_{c}(\mathbf{z})\delta\mathbf{z}$, where
$J_{f}\in\mathcal{R}^{n\times d}$ and $J_{c}\in\mathcal{R}^{m\times d}$ are
Jacobian matrices. Letting
$A_{f}(\mathbf{z})=J^{T}_{f}(\mathbf{z})J_{f}(\mathbf{z})$ and
$A_{c}(\mathbf{z})=J^{T}_{c}(\mathbf{z})J_{c}(\mathbf{z})$, we get:

$\displaystyle\delta\mathbf{z}^{*}=$
$\displaystyle\operatorname*{argmax}_{\delta\mathbf{z}:\|\delta\mathbf{z}\|=\epsilon}\delta\mathbf{z}^{T}A_{c}(\mathbf{z})\delta\mathbf{z}$
(5) $\displaystyle\text{ s.t.
}\delta\mathbf{z}^{T}A_{f}(\mathbf{z})\delta\mathbf{z}\approx 0$ (6)

This optimization is similar to one of finding the $\delta\mathbf{z}$ that
maximizes the Rayleigh quotient
$\left(\delta\mathbf{z}^{T}A_{c}(\mathbf{z})\delta\mathbf{z}\right)/\left(\delta\mathbf{z}^{T}A_{f}(\mathbf{z})\delta\mathbf{z}\right)$,
known to be the solution of the generalized eigenvalue problem
$A_{c}\delta\mathbf{x}=\lambda A_{f}\delta\mathbf{x}$, or the principal
eigenvector of $A_{f}^{-1}A_{c}$ (see Supplementary). The main point of
difference is that in our applications $A_{f}$ is often singular ($n<d$) and
therefore not invertible. Put another way, $f(\cdot)$ is constant in a
subspace $\text{null}(A_{f})$ around $\mathbf{z}$ and any $\delta\mathbf{z}$
in that subspace will exactly satisfy constraint (6). We instead first project
$A_{c}$ onto $\text{null}(A_{f})$, and then find the principal eigenvectors of
the resulting matrix (Alg. 1) [10]. We return the top eigenvectors (REDs) in
matrix $R\in\mathcal{R}^{d\times s}$, where $s$ is some integer, to define the
local subspace of good traversal directions.

For some high-dimensional features, the rank of $\text{null}(A_{f})$ may be
too small (or even 0 when $d<n$), yielding little to no diversity of
$\mathbf{x}$ in the generated trajectories. To address this, we introduce
hyperparameter $\beta_{f}$ in Alg. 1 that lets users smoothly control the
approximation of $A_{f}$’s rank based on explained variance. We also introduce
$\beta_{c}$ to control the rank of the REDs matrix $R$.

The main computational cost of finding REDs is in calculating the Jacobian
matrices $J_{f}$ and $J_{c}$. We compute them using two-sided finite
difference approximations with step size $\epsilon$, which requires $2d+1$
forward evaluations of $f(\cdot)$ and $c(\cdot)$.

Input: $A_{f}$, $A_{c}$, $\beta_{f}$, $\beta_{c}$

Output: $R$

$A_{f},A_{c}\leftarrow A_{f}/\|A_{f}\|_{2},A_{c}/\|A_{c}\|_{2}$

$\textbf{u}_{f},V_{f}\leftarrow$ eig$(A_{f})$

$\text{rank}_{f}\leftarrow\text{smallest }k\text{ s.t.
}\sum_{i=0}^{k}{\textbf{u}_{f}^{2}(i)}\geq\beta_{f}\|\textbf{u}_{f}\|^{2}$

$\text{null}(A_{f})\leftarrow V_{f}[:,\text{rank}_{f}:d]$

$\tilde{\textbf{u}}_{c},\tilde{V}_{c}\leftarrow$
eig$(\text{null}(A_{f})^{T}A_{c}\text{ null}(A_{f}))$

$\text{rank}_{c}\leftarrow\text{smallest }k\text{ s.t.
}\sum_{i=0}^{k}{\tilde{\textbf{u}}_{c}(i)}\geq\beta_{c}\|\tilde{\textbf{u}}_{c}\|^{2}$

$R\leftarrow\text{null}(A_{f})\tilde{V}_{c}[:,0:\text{rank}_{c}]$

Algorithm 1 Compute local REDs (solves optimization problem (5)-(6))

###  3.2 Fixing multiple attributes

In practice, we often want to fix multiple attributes simultaneously. One way
to do so is to simply concatenate them together into $\mathbf{y}$. However,
this approach offers limited individual control over each feature’s
variability.

Instead, given multiple features $\mathbf{y}^{1},\cdots,\mathbf{y}^{n_{f}}$,
we replace (6) with multiple constraints:
$\delta\mathbf{z}^{T}A^{i}_{f}(\mathbf{z})\delta\mathbf{z}\approx 0,i=1\cdots
n_{f}$, and introduce a separate $\beta^{i}_{f}$ for computing the rank of
each $A^{i}_{f}$. We compute REDs by projecting $A_{c}$ onto
$\cap_{i=1}^{n_{f}}\text{null}(A^{i}_{f})$ – the intersection of the fixed
attribute nullspaces – and returning the top eigenvectors of the resulting
matrix as before.

###  3.3 Traversal Algorithms

We propose two traversal algorithms using REDs. The first is a simple _Linear_
traversal (see Supplementary for algorithm). We randomly select a direction in
the span of $R_{0}$ (the REDs of $\mathbf{z}_{0}$), and generate a sequence of
latent codes $\mathbf{z}_{1},\cdots,\mathbf{z}_{K}$ by moving in that
direction starting from $\mathbf{z}_{0}$ with step size $s$. In the likely
case that the constant-$\mathbf{y}$ manifold is curved, the linear traversal
is expected to diverge quadratically from $\|\delta\mathbf{y}\|=0$ as a
function of $\|\delta\mathbf{z}\|$.

Our second algorithm, _Projection_ (see Supplementary for algorithm),
addresses this shortcoming by recomputing the space of local REDs along the
traversal path. We again start by selecting a random direction in $R_{0}$.
However, at each step $i$ (of length $s$), we project the previous direction,
$\delta\mathbf{z}_{i-1}$, onto $R_{i}$. This results in a path that more
faithfully adheres to the local geometries of $f(\cdot)$ and $c(\cdot)$ in
latent space.

A visual example of a _Linear_ and _Projection_ traversal for the same initial
latent code is shown in Fig. 2, where $f(\cdot)$ measures identity and
$c(\cdot)$ measures raw face pixels. _Projection_ is better than _Linear_ at
preserving identity for long trajectories (right plot), while achieving
similar levels of image change (left plot).

##  4 Experiments

We evaluate our method on two image domains: faces and living rooms. We use
StyleGAN2 [17] with _config-f_ configuration for both domains. For faces, we
use the public model from NVIDIA trained on the Flickr Faces HQ (FFHQ) dataset
222https://github.com/NVlabs/stylegan2. For living rooms, we train the GAN
from scratch on an in-house dataset of 100K $1024\times 1024$ living room
scenes from the web. We use StyleGAN2’s “style” space,
$\mathbf{w}\in\mathcal{R}^{512}$, as our latent space for both applications.

Figure 3: Results for traversals controlled by identity, facial landmarks, and
hairstyle. (Top) Results using our method (REDs + projection) for 5 seed faces
and two experiments: changing hair while fixing identity and landmarks
(columns 2-4) and changing landmarks while fixing identity and hair (columns
5-7). We selected three samples per seed from different trajectories to
illustrate the perceptual diversity of faces generated by our method while
adhering to the fixed attribute constraints. (Bottom) Quantitative comparison
of traversal methods. We generated $5$ traversals with $L=5$ steps for each
method for 50 random seeds. We plot changes to hair (left) and landmarks
(right) versus changes to identity in log-log scale, where each dot in the
plot is the average value for each step over all examples. _Leftward and
higher values are better_. Our method using linear traversal (REDs-lin)
outperforms the baselines also using linear traversal. Our method with
projection traversal, REDs-proj, outperforms REDs-lin by reducing identity
changes with no impact to hair or landmarks. Figure 4: Object-preserving
living room traversals. We used REDs with _Projection_ traversal, with
$f(\cdot)$ and $c(\cdot)$ encoding raw pixel values inside and outside a
bounding box on a piece of furniture (red box on seed image at left). The
object within the box often stays fixed, but can undergo stylistic changes and
movements (examples in rows 1, 4, 6) due to feature correlations in latent
space. There are diverse changes to the rooms outside of the boxes, including
new furniture (rows 1, 3, 4, 6), wall and window properties/decorations (all
rows), and house plants (rows 4, 5). Figure 5: Samples from traversals
controlled by spatial frequency bands. (Columns 2-4) The embedding function
$f(\cdot)$ returns the raw pixels of the low-pass filtered image and
$c(\cdot)$ the high-pass one. High-pass modifications change physiognomies,
age and expressions, as well as hair and accessory texture, while the
silhouette, lighting and color scheme are preserved. (Columns 5-7) $f(\cdot)$
and $c(\cdot)$ are inverted; low-pass modifications change colors, lighting
and shading while mostly preserving identity, hair and textures. Figure 6:
Samples from traversals controlled by scalar semantic attributes. On scalar
attributes one may compare our method to a baseline of using a global linear
model (SVM or ridge regressor) in latent space [4, 30] (the global method is
not defined and cannot handle multi-dimensional attributes). We change one
attribute (age, pose, smile, gender) at a time while fixing the other three.
Both methods are comparable for many cases. REDs sometimes fails (red-boxed
images), particularly for gender (see Sec. 5.1 for further discussion).

###  4.1 Identity, hairstyle and landmark face traversals

We first demonstrate our method on controlling three multidimensional facial
features: identity, hairstyle, and 3D facial landmark positions.

We use ArcFace [9], a popular open-source face identification model that
encodes identity with a $512$-dimensional vector. To encode hairstyle, we run
a public face segmentation model 333https://github.com/zllrunning/face-
parsing.PyTorch on each image, set pixels outside of the hair region to $0$,
and flatten all pixels into a $256\times 256\times 3=196,608$-dimensional
vector. We encode 3D landmarks using the MediaPipe mesh model [1], which
predicts 468 landmarks around the face. This results in a $468\times
3=1404$-dimensional vector.

We performed two experiments: changing hair while keeping identity and
landmarks fixed, and changing landmarks while keeping identity and hair fixed.
Fig. 3 presents sample results for five test seed points using REDs and
_Projection_ traversal. In both experiments, we set $\beta$s for fixed
attributes to 0.99, and $\beta$ of the changing attribute to 0.999. We set
both the Jacobian finite difference step and path step $s$ to $1$. We set path
length $L=5$. Along with changing the input images along the intended
features, our method is able to produce a _wide variety_ of different samples
from different paths.

We quantitatively evaluated REDs against three baseline direction-finding
approaches: choosing directions at random (Random), choosing the most
significant eigenvectors of $A_{c}$, thereby maximizing changes to
$\mathbf{x}$ ($\text{Max-}\Delta\mathbf{x}$), and choosing the least
significant eigenvectors of $A_{f}$, thereby minimizing changes to
$\mathbf{y}$ ($\text{Min-}\Delta\mathbf{y}$).

The plots in Fig. 3 present our results. When using _Linear_ traversal, REDs
outperforms the three baseline direction-finding approaches.
$\text{Max-}\Delta\mathbf{x}$ finds directions that significantly change
hairstyle/landmarks and identity, $\text{Min-}\Delta\mathbf{y}$ preserves
identity but also minimally changes hairstyle/landmarks, and Random performs
worst of all. The figure also shows that when using REDs, _Projection_
outperforms _Linear_. See Fig. 2 for a visual sample of this comparison and
Supplementary for complete traversals.

###  4.2 Frequency band face traversals

Our method can handle arbitrary low-level image representations. We
demonstrate this by controlling specific spatial frequency bands in Fig. 5. We
let $f(\cdot)$ and $c(\cdot)$ encode the raw pixels of low-pass and high-pass
filtered versions of the input image (and vice versa). High-pass modifications
change physiognomies, expressions and accessory textures. Low-pass
modifications mainly change colors, lighting and shading.

###  4.3 Object-preserving living room traversals

We next apply our method to living room scenes. We aim to keep selected
furniture fixed while changing other parts of the scene. We generated
furniture bounding boxes with an object detector. We let $f(\cdot)$ encode the
raw pixels within the bounding box, and let $c(\cdot)$ encode all remaining
pixels in the scene. We set $\beta=0.99$ for both features, a Jacobian finite
difference step of $0.75$, path step $s=0.25$, and a path length $L=10$.

Fig. 4 shows several sample sequences. See caption for a detailed description.
In Supplementary, we show sample strips of full traversals. We observe two
notable degradations in these strips the farther we move away from the seed
image. First, the ‘fixed’ object often moves slightly at each step. Second,
artifacts become more prominent because we rapidly advance to low-probability
regions of the latent space.

###  4.4 Scalar attribute face traversals

For scalar attributes, we can compare our method against a baseline that uses
global linear directions [4, 30]. These methods train a linear model per
attribute (regressor for a continuous attribute or an SVM for a binary
attribute) to predict the attribute value from the latent code. We change an
attribute by moving along the hyperplane’s normal direction. To fix other
attributes, we orthogonalize the changing attribute’s direction with respect
to the other attribute directions.

Fig. 6 presents our results for four attributes: age, pose, smile, and gender.
Overall, REDS-proj achieves similar qualitative performance to the baseline
for most samples, but also has more failures cases when changing an attribute
like gender, which often does not have a large local gradient in latent space.
We discuss this more in 5.1.

##  5 Discussion

Our experiments demonstrate the effectiveness of REDs at finding locally
optimal orientations. By contrast, selecting random traversal directions or
local directions that prioritize only one of Eq. (5) or (6) do not work well
due to the high dimensionality of the latent space (see plots in Fig. 3).

The superiority of _Projection_ over _Linear_ traversal (Fig. 3) also
demonstrates the need for localized approximations of latent space geometry
for complex image features. This is in contrast to past traversal studies [4,
13, 15, 27, 30] that found global linear directions to suffice for simple
scalar attributes.

A consideration in all image synthesis works is the balance between perceptual
quality based on human judgment, and quantitative optimization and analysis.
In the application of faces, the user may have his/her own internal tradeoff
curve between identity preservation and image diversity. Our method offers a
principled way to explore different points on this curve by tuning the $\beta$
parameters (see Supplementary). Image perception also factors into the
embedding functions used to measure image changes.

GAN latent spaces are not all alike, and each requires different
considerations. Faces are easier to model than living rooms, because the
latter are a composition of many discrete objects interacting with one
another. As a result, we found the face latent space and traversals to be
smoother. Our living room traversals often exhibit large perceptual “jumps”
due to discontinuities in latent space (see Supplementary). The complexity of
a distribution also affects the degree of correlation between attributes. As
Fig. 4 shows, it is not always possible to exactly fix a particular region of
a living room while obtaining enough diversity elsewhere due to entangled
features. Different regions of the latent space are also not alike. We found
that high-likelihood regions produce the most realistic images and diverse
traversals. Thus, the biases of the generative model have a direct effect on
how well our method performs for a given image (see Sec. 5.1 for further
discussion).

###  5.1 Limitations

Our method takes a local view of the latent space to identify good traversal
directions. However, as our results in Sec. 4.4 suggest, there are benefits to
taking a global view. Global linear models are likely better for attributes
that are discrete, such as ‘wearing eyeglasses,’ or approximately discrete for
a large majority of samples like gender. For such attributes, local gradients
in latent space can be near zero and swamped by noise. Another limitation of a
local view is that gradients are undefined near sharp discontinuities in the
latent space. We did not find this to be a decisive issue for faces, but did
notice perceptual ‘jumps’ in the living room scenes during traversals (see
Supplementary for traversal strips). However, we note that our framework could
be extended to use both global and local directions per traversal step, which
we leave for future work.

Though our method can theoretically work with any deep generative model latent
space, we used StyleGAN2 in all our experiments. Further experiments using
other GAN or VAE architectures can give a more complete picture of our
method’s benefits and limitations.

###  5.2 Ethics

Fairness: As in past work [4] we observed bias in StyleGAN’s face
distribution: Caucasian faces are most likely to be generated. This bias also
affects trajectory quality, with light-skinned seed faces producing more
diverse trajectories than dark-skinned ones. Biases in fixed and changing
functions that use learning models also affect results. One example are face
recognition models, like the one we used in our experiments to fix identity,
which are known to have gender and ethnicity biases. To reduce bias one will
want to train GANs and any learned models on rich and diverse datasets.

Fake portrayals: GANs could be used to generate fake images of individuals
under different conditions. This could include the case where the image of the
face of a real person is projected onto the GAN latent space and then
manipulated.

##  6 Conclusion

We presented a simple, principled and versatile method designed to explore a
generative model’s latent space to produce sets of synthetic samples where one
group of multidimensional features is held constant while another is varied as
much as possible. We demonstrated traversal results on several features that
previous works are not capable of handling: landmark locations, pixels within
regions, frequency information, and facial identity as measured by a deep
neural network. Our experiments show the need for modeling local geometry of
latent spaces for high-dimensional features. Understanding the complex nature
and geometry of the latent space of image generators is a fascinating question
which we have only started to explore.

## References

  * [1] Mediapipe.  https://github.com/google/mediapipe.
  * [2] Grigory Antipov, Moez Baccouche, and Jean-Luc Dugelay.  Face aging with conditional generative adversarial networks.  In 2017 IEEE international conference on image processing (ICIP), pages 2089–2093. IEEE, 2017.
  * [3] Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg.  Latent space oddity: on the curvature of deep generative models.  arXiv preprint arXiv:1710.11379, 2017.
  * [4] Guha Balakrishnan, Yuanjun Xiong, Wei Xia, and Pietro Perona.  Towards causal benchmarking of bias in face analysis algorithms.  In European Conference on Computer Vision, pages 547–563. Springer, 2020.
  * [5] Randall Balestriero, Sebastien Paris, and Richard Baraniuk.  Max-affine spline insights into deep generative networks.  arXiv preprint arXiv:2002.11912, 2020.
  * [6] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua.  Towards open-set identity preserving face synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6713–6722, 2018.
  * [7] Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt.  Metrics for deep generative models.  In International Conference on Artificial Intelligence and Statistics, pages 1540–1550. PMLR, 2018.
  * [8] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789–8797, 2018.
  * [9] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2019.
  * [10] Benyamin Ghojogh, Fakhri Karray, and Mark Crowley.  Eigenvalue and generalized eigenvalue problems: Tutorial.  arXiv preprint arXiv:1903.11240, 2019.
  * [11] Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola.  Ganalyze: Toward visual definitions of cognitive image properties.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5744–5753, 2019.
  * [12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, volume 27, 2014.
  * [13] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  In Advances in Neural Information Processing Systems, pages 9841–9850, 2020.
  * [14] Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen.  Attgan: Facial attribute editing by only changing what you want.  IEEE Transactions on Image Processing, 28(11):5464–5478, 2019.
  * [15] Ali Jahanian*, Lucy Chai*, and Phillip Isola.  On the ”steerability” of generative adversarial networks.  In International Conference on Learning Representations, 2020.
  * [16] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [17] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  arXiv preprint arXiv:1912.04958, 2019.
  * [18] Diederik P. Kingma and Max Welling.  Auto-Encoding Variational Bayes.  In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  * [19] Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath.  Causalgan: Learning causal implicit generative models with adversarial training.  In International Conference on Learning Representations, 2018.
  * [20] Line Kuhnel, Tom Fletcher, Sarang Joshi, and Stefan Sommer.  Latent space non-linear statistics.  arXiv preprint arXiv:1805.07632, 2018.
  * [21] Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc Ranzato.  Fader networks: Manipulating images by sliding attributes.  In 31st Conference on Neural Information Processing Systems (NIPS 2017), pages 5969–5978, 2017.
  * [22] Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, and Shilei Wen.  Stgan: A unified selective transfer network for arbitrary image attribute editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3673–3682, 2019.
  * [23] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [24] Augustus Odena, Christopher Olah, and Jonathon Shlens.  Conditional image synthesis with auxiliary classifier gans.  In International conference on machine learning, pages 2642–2651. PMLR, 2017.
  * [25] Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher-Shlizerman.  Lifespan age transformation synthesis.  In European Conference on Computer Vision, pages 739–755. Springer, 2020.
  * [26] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2337–2346, 2019.
  * [27] Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot.  Controlling generative models with continuous factors of variations.  In International Conference on Learning Representations, 2020.
  * [28] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [29] Hang Shao, Abhishek Kumar, and P Thomas Fletcher.  The riemannian geometry of deep generative models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 315–323, 2018.
  * [30] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  arXiv preprint arXiv:1907.10786, 2019.
  * [31] Yujun Shen, Ping Luo, Junjie Yan, Xiaogang Wang, and Xiaoou Tang.  Faceid-gan: Learning a symmetry three-player gan for identity-preserving face synthesis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 821–830, 2018.
  * [32] Yujun Shen, Bolei Zhou, Ping Luo, and Xiaoou Tang.  Facefeat-gan: a two-stage approach for identity-preserving face synthesis.  arXiv preprint arXiv:1812.01288, 2018.
  * [33] Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, and Gerard Medioni.  Gan-control: Explicitly controllable gans.  arXiv preprint arXiv:2101.02477, 2021.
  * [34] Luan Tran, Xi Yin, and Xiaoming Liu.  Disentangled representation learning gan for pose-invariant face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1415–1424, 2017.
  * [35] Christos Tzelepis, Georgios Tzimiropoulos, and Ioannis Patras.  Warpedganspace: Finding non-linear rbf paths in gan latent space.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6393–6402, 2021.
  * [36] Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger.  Deep feature interpolation for image content changes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7064–7073, 2017.
  * [37] Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the gan latent space.  In International Conference on Machine Learning, pages 9786–9796. PMLR, 2020.
  * [38] Binxu Wang and Carlos R Ponce.  A geometric analysis of deep generative image models and its applications.  In International Conference on Learning Representations, 2021.
  * [39] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.
  * [40] Taihong Xiao, Jiapeng Hong, and Jinwen Ma.  Elegant: Exchanging latent encodings with gan for transferring multiple face attributes.  In Proceedings of the European conference on computer vision (ECCV), pages 168–184, 2018.
  * [41] Ceyuan Yang, Yujun Shen, and Bolei Zhou.  Semantic hierarchy emerges in deep generative representations for scene synthesis.  International Journal of Computer Vision, pages 1–16, 2021.
  * [42] Huiting Yang, Liangyu Chai, Qiang Wen, Shuang Zhao, Zixun Sun, and Shengfeng He.  Discovering interpretable latent space directions of gans beyond binary attributes.  In CVPR, pages 12177–12185, 2021.
  * [43] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker.  Towards large-pose face frontalization in the wild.  In Proceedings of the IEEE international conference on computer vision, pages 3990–3999, 2017.