# Proportionate gradient updates with PercentDelta

Sami Abu-El-Haija
Google Research
Mountain View, CA
<EMAIL_ADDRESS>

###### Abstract

Deep Neural Networks are generally trained using iterative gradient updates.
Magnitudes of gradients are affected by many factors, including choice of
activation functions and initialization. More importantly, gradient magnitudes
can greatly differ across layers, with some layers receiving much smaller
gradients than others. causing some layers to train slower than others and
therefore slowing down the overall convergence. We analytically explain this
disproportionality. Then we propose to explicitly train all layers at the same
speed, by scaling the gradient w.r.t. every trainable tensor to be
proportional to its current value. In particular, at every batch, we want to
update all trainable tensors, such that the relative change of the L1-norm of
the tensors is the same, across all layers of the network, throughout training
time. Experiments on MNIST show that our method appropriately scales
gradients, such that the relative change in trainable tensors is approximately
equal across layers. In addition, measuring the test accuracy with training
time, shows that our method trains faster than other methods, giving higher
test accuracy given same budget of training steps.

##  1 Introduction

Weights of Deep Neural Networks (DNNs) are commonly randomly initialized. The
$j$-th layer’s weight matrix $W_{j}$ is initialized from a distribution that
is generally conditioned on the shape of $W_{j}$.

Feed-forward DNNs generally consist of a series of matrix multiplications and
element-wise activation functions. For example, input vector $\mathbf{x_{0}}$
is transformed to the output vector $\mathbf{x}_{L}$ by an $L$-layer neural
network $h_{\theta}(\mathbf{x_{0}})=\mathbf{x}_{L}$, depicted as:

$\mathbf{x}_{0}$$W_{0}$$\times$$\sigma$$W_{1}$$\times$$\sigma$…$W_{L-1}$$\times$$\sigma$$\mathbf{x}_{L}$

Where $\times$ is a matrix-multply operator, $\sigma$ is an element-wise
activation (e.g. logistic or ReLu). The DNN parameters
$\theta=\\{W_{0},W_{1},\dots,W_{L-1}\\}$ are optimized by a training
algorithm, such as Stochastic Gradient Descent (SGD), which iteratively
applies gradient updates:

$W_{j}^{(t+1)}:=W_{j}^{(t)}-\eta\ \gamma(t)\ \frac{\partial J}{\partial
W_{j}},$ (1)

where $W_{j}^{(t)}$ is the value of the $j$-th layer weight matrix $W_{j}$ at
timestep $t$, $\eta\in\mathbb{R}$ is the learning rate, the decay function
$\gamma:\mathbb{R}\rightarrow[0,1]$, generally decreases with time $t$, and
$\frac{\partial J}{\partial W_{j}}$ is the partial gradient of training
objective $J$ w.r.t. $W_{j}$, evaluated at time $t$ for a batch of training
examples. For notational convenience, we define the delta SGD as:
$\Delta_{\text{SGD}}W=\eta\ \gamma(t)\ \frac{\partial J}{\partial W}$

Under usual circumstances, the magnitudes of gradients can widely vary across
layers of a DNN, causing some trainable tensors to change much slower than
others, slowing down overall training. Several proposed methods metigate this
problem, including utilizing: per-parameter adaptive learning rate such as
AdaGrad (Duchi et al.,, 2011) or Adam (Ba and Kingma,, 2015); normalization
operators such as BatchNorm (Ioffe and Szegedy,, 2015), WeightNorm (Salimans
and Kingma,, 2016), or LayerNorm (Ba et al.,, 2016); and intelligent
initialization schemes such as xavier’s (Glorot and Bengio,, 2010). These
methods heuristically attack the disproportionate training problem, which we
justify in section 2. In this paper, we propose to directly enforce
proportionate training of layers. Specifically, We propose a gradient update
rule that moves every weight tensor in the direction of the gradient, but with
a magnitude that changes the tensor value by a relative amount. We use the
same relative amount across all layers, to train them all at the same speed.

The remainder of the paper is organized as follows. In Section 2 we illustrate
the disproportionate training phenomena on a (toy) hypothetical 4-layer neural
network, by expanding gradient terms using BackProp Rumelhart et al., (1986).
In Section 3, we summarize related work. Then, we introduce our algorithm in
Section 4. We show experimental results on MNIST in Section 5. In Section 6,
we discuss where PercentDelta could be useful and potential future direction.
Finally, we conclude our findings in Section 7.

##  2 Disproprtionate Training

We use a toy example to illustrate disproportionate training across layers.
Assume a 4-layer network with trainable weight matrices
$\\{W_{0},W_{1},W_{2},W_{3}\\}$ and output vector $\mathbf{x}_{4}$. The
gradient of the objective w.r.t. output vector $\mathbf{x}_{4}$ can be
directly calculated from the data e.g. using cross-entropy loss. We write-down
the gradient of the objective $J$ w.r.t. the last layer’s weight matrix
$W_{3}$ as:

$\frac{\partial J}{\partial
W_{3}}=\left[\sigma^{\prime}(W_{3}\times\mathbf{x}_{3})\circ\frac{\partial
J}{\partial\mathbf{x}_{4}}\right]\times\mathbf{x}_{3}^{T},$ (2)

where $\sigma^{\prime}()$ is the derivative of $\sigma()$ w.r.t. its input,
and $\circ$ is the Hadamard product.

We also write-down the gradient w.r.t. $W_{1}$ and $W_{2}$, then expand the
expressions using the Back Propagation Algorithm (Rumelhart et al.,, 1986):

$\displaystyle\frac{\partial J}{\partial W_{2}}$
$\displaystyle=\left[\sigma^{\prime}(W_{2}\times\mathbf{x}_{2})\circ\underbrace{\frac{\partial
J}{\partial\mathbf{x}_{3}}}_{=\leavevmode\hbox to0pt{\vbox
to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\right]\times\mathbf{x}_{2}^{T}$
$\displaystyle=\left[\sigma^{\prime}(W_{2}\times\mathbf{x}_{2})\circ\overbrace{\left[W_{3}^{T}\times\left[\sigma^{\prime}(W_{3}\times\mathbf{x}_{3})\circ\frac{\partial
J}{\partial\mathbf{x}_{4}}\right]\right]}^{\leavevmode\hbox to0pt{\vbox
to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\right]\times\mathbf{x}_{2}^{T}$
$\displaystyle\frac{\partial J}{\partial W_{1}}$
$\displaystyle=\left[\sigma^{\prime}(W_{1}\times\mathbf{x}_{1})\circ\underbrace{\frac{\partial
J}{\partial\mathbf{x}_{2}}}_{=\leavevmode\hbox to0pt{\vbox
to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\right]\times\mathbf{x}_{1}^{T}$
$\displaystyle=\left[\sigma^{\prime}(W_{1}\times\mathbf{x}_{1})\circ\overbrace{\left[W_{2}^{T}\times\left[\sigma^{\prime}(W_{2}\times\mathbf{x}_{2})\circ\underbrace{\left[W_{3}^{T}\times\left[\sigma^{\prime}(W_{3}\times\mathbf{x}_{3})\circ\frac{\partial
J}{\partial\mathbf{x}_{4}}\right]\right]}_{=\partial
J/\partial\mathbf{x}_{3}}\right]\right]}^{\leavevmode\hbox to0pt{\vbox
to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\right]\times\mathbf{x}_{1}^{T}$

Note the following in the above equations: First, the derivatives
$\frac{\partial J}{\partial W_{3}},\frac{\partial J}{\partial
W_{2}},\frac{\partial J}{\partial W_{1}}$ look similar. In fact, they are
almost sub-expressions of one another, with the exception of the right-most
row-vector $\mathbf{x}_{j}^{T}$. Second, all quantities in brackets are
column-vectors. The gradient matrix $\frac{\partial J}{\partial W_{j}}$ is
determined by an outer product of a column-vector (in brackets) times a row-
vector $\mathbf{x}_{j}^{T}$.

What can we conclude about the magnitudes (e.g. the L1 norm) of
$\frac{\partial J}{\partial W_{3}},\frac{\partial J}{\partial
W_{2}},\frac{\partial J}{\partial W_{1}}$? Each $\frac{\partial J}{\partial
W_{j}}$ is calculated using three types of multiplicands:
$\sigma^{\prime}(.)$, $W_{.}^{T}$, and $\mathbf{x}_{j}^{T}$. Therefore, the
magnitudes of $\frac{\partial J}{\partial W_{j}}$ are affected by:

  1. 1.

The value of the derivative of $\sigma^{\prime}()$, which is evaluated
element-wise. If $\sigma^{\prime}()$ is generally less than 1, then we can
expect the gradient to be smaller for earlier layers than later ones, since
they are multiplied by $\sigma^{\prime}()$ more times. If it is generally
greater than 1, then we can expect the gradient to be larger for earlier
layers. For very deep networks (recurrent or otherwise), the former situation
can cause the gradients to vanish, while the latter can cause the gradients to
explode. ReLu mitigates this problem, as its derivative $=1$, in locations
where input is positive.

  2. 2.

The magnitude of $W$’s. In practice, all $W$’s are initialized from the same
distribution. If the L1-norms of rows in $W_{j}^{T}$ are less than 1, then we
should expect
$||W_{j}^{T}\times\sigma^{\prime}(\mathbf{z})||<||\sigma^{\prime}(\mathbf{z})||$,
yielding smaller gradients for earlier layers. Otherwise, if the row L1 norms
are greater than 1, then earleir layers should receive larger gradient
magnitudes. Weight normalization (Salimans and Kingma,, 2016) and intelligent
initialization schemes (e.g. Glorot and Bengio,, 2010) can mitigate this
problem.

  3. 3.

The norm of the row-vector $\mathbf{x}_{j}$. If in the forward pass, the
activations consistently grow (e.g. unbounded activation) with subsequent
layers, then later layers will receive larger gradients. BatchNorm (Ioffe and
Szegedy,, 2015) and LayerNorm (Ba et al.,, 2016) mitigate this problem.

##  3 Related Work

###  3.1 Adaptive Gradient

Duchi et al., (2011) proposed AdaGrad. A training algorithm that keeps a
cumulative sum-of-squared gradients (i.e. second moment of gradients):

$S^{(t)}_{j}=S^{(t-1)}_{j}+\left(\frac{\partial J}{\partial
W_{j}}\circ\frac{\partial J}{\partial W_{j}}\right),$ (3)

Then divides (element-wise) the gradient by the square-root of the sum:

$W_{j}^{(t+1)}:=W_{j}^{(t)}-\eta\ \gamma(t)\ \left(\frac{\partial J}{\partial
W_{j}}\circ\left(S^{(t)}_{j}\right)^{-1/2}\right),$ (4)

or equivalently $\Delta_{\textrm{AdaGrad}}W_{j}=\eta\ \gamma(t)\
\frac{\partial J}{\partial W_{j}}\circ\left(S^{(t)}_{j}\right)^{-1/2}$, where
the $(.)^{-1/2}$ power operator is applied element-wise. In essense, if some
layer receives large gradients, then they will be normalized to smaller
gradients through the division. However, a weakness in AdaGrad is that at some
point, the $S$ will grow too large, effectively making $(S)^{-1/2}\approx 0$
and therefore slowing or halting training.

Ba and Kingma, (2015) propose Adam, which keeps exponential decaying sums, of
gradients and of square gradients, respectively known as first and second
moments. Adam offers two benefits over AdaGrad. First, its decaying sums
should not grow to infinity and therefore training should not halt. Second,
the exponential-decay averaging was shown to speed up training (Momentum,
Sutskever et al.,, 2013). For details, we point readers to the Adam paper (Ba
and Kingma,, 2015).

###  3.2 LARS

You et al., (2017) propose to normalize every layer’s gradients by the ratio
of L2 norms of the parameter and the gradient. Namely, they propose the
gradient update rule:

$\Delta_{\textrm{LARS}}W_{j}=\eta\ \gamma(t)\
\left(\frac{\left|\left|W_{j}^{(t)}\right|\right|_{2}}{\ \left|\left|(\partial
J/\partial W_{j})\right|\right|_{2}}\right)\ \frac{\partial J}{\partial
W_{j}},$ (5)

which effectively normalizes the gradient $\frac{\partial J}{\partial W_{j}}$
to be unit-norm. This setup is very similar to ours, with two differences:
First, our norm operator is L1 rather than L2. Second, our norm operator is
applied outside the division (i.e. our division is element-wise). Our proposed
algorithm, PercentDelta, was used to train our work (Abu-El-Haija et al.,,
2017), before we where aware of the work of (You et al.,, 2017). In (Abu-El-
Haija et al.,, 2017), PercentDelta gave us $1\%$ improvement over Adam, over
all datasets. Nonetheless, we only discuss MNIST experiments in this paper,
and leave graph embedding experiments for follow-up work.

##  4 PercentDelta

For every weight matrix $W_{j}$, and similarily bias vectors, we propose the
gradient update rule:

$\Delta_{\textrm{PercentDelta}}W_{j}=\eta\ \gamma(t)\
\left(\frac{\textrm{size}(W_{j})}{\ \left|\left|\frac{(\partial J/\partial
W_{j})}{W_{j}^{(t)}}\right|\right|_{1}}\right)\ \frac{\partial J}{\partial
W_{j}},$ (6)

where scalar $\textrm{size}(W)\in\mathbb{Z}^{+}$ is the number of
entries111$\textrm{size}(W)$ is the product of $W$’s dimensions. For a 2-D
matrix, it is equal to # rows $\times$ # columns, for a vector, it is equal to
its length, etc. We also define $\textrm{size(scalar)}=1$. in $W$, and the
scalar:

$\frac{\textrm{size}(W_{j})}{\ \left|\left|\frac{(\partial J/\partial
W_{j})}{W_{j}^{(t)}}\right|\right|_{1}}$ (7)

normalizes the gradient $\frac{\partial J}{\partial W_{j}}$ of the $j$-th
layer, so that it is more proportional to its current parameter value
$W_{j}^{(t)}$, and $||.||_{1}$ is the L1-norm. We avoid division-by-zero
errors by adding an epsilon to the denominator222In practice, rather than
converting $\frac{a}{b}$ to $\frac{a}{b+\epsilon}$, we use
$\frac{a}{b+\epsilon\cdot sign(b)}$, as our goal is to push $b$ slightly away
from zero while keeping its sign.. The divide operator within the L1 norm is
applied element-wise and the outer divide operator is scalar. Note that the
“gradient multiplier” fraction (Equation 7) is $\in\mathbb{R}^{+}$. Therefore,
we only changes the gardient’s magnitude, but not its direction.

Now, we justify Equation 6. Assume that $W_{s}$ is scalar. Therefore,
$\textrm{size}(W_{s})=1$ and the L1-norm becomes an absolute value. Equation 6
simplifies to:

$\displaystyle\begin{split}W_{s}^{(t+1)}&:=W_{s}^{(t)}-\eta\ \gamma(t)\ \
\left|\frac{W_{s}^{(t)}}{(\partial J/\partial W_{s})}\right|\ \frac{\partial
J}{\partial W_{s}}\\\ &:=W_{s}^{(t)}-\eta\ \gamma(t)\ \
\left|W_{s}^{(t)}\right|\ \textrm{sign}\left(\frac{\partial J}{\partial
W_{s}}\right),\end{split}$ (8)

where $\textrm{sign}(z)=1$ if $z>0$ and $=-1$ if $z<0$. Therefore, $W_{s}$
will change with a quantity proportional to its current value. In particular,
the scalar $\eta\ \gamma(t)$ determines the percentage at which $W$ changes at
timestep $t$, giving rise to the name: PercentDelta. For example, if we setup
a decay schedule on $\gamma(t)$, such that $\eta\gamma(t)$ changes during
training from $10\%$ to $0.01\%$, then the network parameters will change at a
rate of $10\%$ in early mini-batches, and gradually decrease their
PercentDelta updates to $0.01\%$ towards end of training, consistently across
all layers, regardless of the current parameter value or the gradient value,
which are influenced by choice of activation function, initialization, and
network architecture.

##  5 Experiments

We run experiments on MNIST. We use the same model for all experiments and fix
the batch size to 500. Our model code is copied from the TensorFlow
tutorial333https://www.tensorflow.org/get_started/mnist/pros, and contains 4
trainable layers: 2D Convolution, Max-pooling, 2D Convolution, Max-pooling,
Fully-connected, Fully-connected. The convolutional layers contain trainable
tensors with dimensions: (5, 5, 1, 32), (5, 5, 32, 64), and bias vectors. The
Fully-connected layers contain trainable tensors with dimensions: (3136,
1024), (1024, 10), and bias vectors. The model is trained with Softmax loss,
uses ReLu for hidden activations, and does not use BatchNorm.

###  5.1 MNIST Network Gradient Magnitudes

We record the gradient magnitudes for our 4-layer MNIST network throughout
training. We compare the relative magnitude that $W$ changes, under vanilla
SGD versus under PercentDelta. We plot
$\left(\left|\left|\Delta_{\textrm{SGD}}W\right|\right|_{1}/\left|\left|W\right|\right|_{1}\right)$
and
$\left(\left|\left|\Delta_{\textrm{PercentDelta}}W\right|\right|_{1}/\left|\left|W\right|\right|_{1}\right)$
for tranable tensors $W$ of convolutional and fully-connected layers. We
notice that vanilla SGD proposes gradients that are not proportional to
current weights. For example, using some learning rate, some layers can
completely diverge (e.g. all entries switching signs) while other layers would
change at a rate of <1%. In this case, to prevent divergence in any layer
tensors, the learning rate would be lowered. However, using PercentDelta,
relative magnitude of parameter updates is almost equal for all layers,
showing that all layers are training at the same speed, consistently
throughout the duration of training.

Figure 1: Magnitude of relative deltas for every layer, at training timestamps
(batch at step 0, step 15, …). Y-axis is the relative change of the L1 norm of
every layer’s trainable tensor. For Stochastic Gradient Descent (SGD) and
PercentDelta, respectively, we measure it as
$\left(\left|\left|\Delta_{\textrm{SGD}}W\right|\right|_{1}/\left|\left|W\right|\right|_{1}\right)$
and
$\left(\left|\left|\Delta_{\textrm{PercentDelta}}W\right|\right|_{1}/\left|\left|W\right|\right|_{1}\right)$.
Disproportionate training is played down by the log-scale. Without log-scale,
some SGD bars wont be visible. Curves are scaled by $\eta\ \gamma(t)$.

###  5.2 MNIST Test Accuracy Curves

We want to measure how fast can PercentDelta train MNIST. We compare training
speed with other algorithms, including per-parameter adaptive learning rate
algorithms, AdaGrad (Duchi et al.,, 2011) and Adam (Ba and Kingma,, 2015), as
well as a recent algorithm with similar spirit, LARS (You et al.,, 2017),
which also normalizes the gradient w.r.t. a weight tensor, by the current
value of the weight tensor.

Figure 2 In our experiments, we fix the learning rate of PercentDelta to
$0.03$, use momentum, and set $\gamma(t)$ to Equation 10. For Adagrad and
Adam, we sweep the learning rate but we fix $\gamma(t)=1$, as they implement
their own decay. We do not use Momentum for Adagrad or Adam, as the later
applies its own momentum scheme. For LARS, we use momentum, vary the learning
rate, and set $\gamma(t)$ to Equation 10 with $m=0.01$. We note the following:

  1. 1.

In early stages, MNIST test accuracy climbs up the fastest with our algorithm.

  2. 2.

The final test accuracy produced by our algorithm is higher, given the
training budget of 5000 steps.

|  |
---|---|---
|  |
|  |
|  |
|  |
Figure 2: MNIST experiments – Accuracy on Test set VS number of training
steps. Each training step processes a batch size of 500 examples. The test
accuracy is recorded after every 5-th steps. Figure contains subplots
organized in 3 columns and 5 rows. Columns indicate the training stage. Left
column shows early training (initial 200 steps) with y-axis starting at 0.5.
Middle and right column show entire training curve, respectively showing raw
values and smoothed values. Rows indicate algorithms. Each line color is
consistent across the row and refers to the same hyper-parameter setting. We
color the best performer with red. Top-to-bottom: PercentDelta (our
algorithm), AdaGrad (Duchi et al.,, 2011), Adam (Ba and Kingma,, 2015), LARS
(You et al.,, 2017), and the last row shows the best performer from all
algorithms.

##  6 Discussion

###  6.1 Situations where PercentDelta is useful

While PercentDelta has outperforms other training algorithms on 4-layer MNIST,
the space of models-datasets is enormous we leave it as future work to try
PercentDelta under various models and datasets. Nonetheless, we speculate that
PercentDelta (and similarily, LARS, You et al., (2017)) would be very useful
in the following scenarios:

  1. 1.

Learning Embeddings. Consider the common setup of feeding word embeddings (or
graph embeddings, Abu-El-Haija et al.,, 2017) into a shared Neural Network and
jointly learning the embeddings and Neural Network for an upstream objective.
In this setup, a certain embedding vector is only affected by a fraction of
training examples, while the shared network parameters are affected by all
training examples. The sum of gradients w.r.t. the shared network parameters
over all training examples can be disproportionately larger than embedding
gradients. PercentDelta ensures that the shared network is not being updated
much faster than the emebddings.

  2. 2.

Soft-Attention Models on Bag-of-Words. It is common to convert from variable-
length bag-of-words $(\mathbf{x}_{j}[1],\mathbf{x}_{j}[2],\dots)$ into fixed-
length representation by a convex combination:
$\mathbf{x}_{j+1}:=\sum_{i}\alpha_{i}\mathbf{x}_{j}[i]$, which can then be
used for an upstream objective (e.g. event detection in videos, Ramanathan et
al.,, 2016). Here, $\alpha_{i}$ can be the $i$-th position of the softmax over
all Words. The parameters of the softmax model would receive gradients from
all words. PercentDelta ensures that, the otherwise disproportionately large,
gradient updates of the softmax model are proportional to the remainder of the
network.

  3. 3.

Matrix Factorization Models. For example, Koren et al., (2009) propose to
factorize a user-movie rating matrix $R\in\mathbb{R}^{u\times m}$ into:

$R\approx W_{U}\times
W_{M}+\mathbf{b}_{U}\times\vec{\mathbf{1}}^{T}+\vec{\mathbf{1}}\times\mathbf{b}_{M}+b,$

where $W_{U}\in\mathbb{R}^{u\times d}$ and $W_{M}\in\mathbb{R}^{d\times m}$
are the user and movie embedding matrix; $d$ is the size of the latent-space;
$\mathbf{b}_{U}\in\mathbb{R}^{u}$ and $\mathbf{b}_{M}\in\mathbb{R}^{m}$ are
the user and movie bias vectors, and $b\in\mathbb{R}$ is the global bias
scalar. In this setup, $b$ would receive very large sum-of-gradients, and
PercentDelta can ensure that all parameters are training at the same speed.

###  6.2 Hyperparamters and Decay Function

It seems that PercentDelta has many knobs to tune. However, we can fix $\eta$
to some value and only change $\gamma(t)$ as their product determines the
effective rate of change across all layers’ trainable tensors. We can set
$\gamma(t)$ to constant decay:

$\gamma(t)=1-t\times m,$ (9)

where $0<m<<1$ determines the decay slope. In addition, we can ensure that
$\gamma(t)>0$ to allow training continue indefinitely, by modifying Equation 9
to:

$\gamma(t)=\max(\beta,1-t\times m),$ (10)

where $\beta$ can be set to a small positive value, such as 0.01. In this
case, if we fix $\eta=0.03$, then we are effectively changing each trainable
tensor by $3\%$ for every training batch initially, then gradually annealing
this change-rate to $0.03\%$ after $\frac{1}{m}$ steps.

More importantly, we feel that $\eta$ and $\gamma(t)$ are a function of the
dataset, and not the model. Experimentally, we observe the algorithm is
insensitive to the choices of $m$ and $\eta$ as long as they are “reasonable”
(i.e. removing diverging setups that can be quickly detected). However, we do
not yet have a formula to automatically set them. Nonetheless, with a wide
range of $\eta$ and $m$, we experimentally show on MNIST that PercentDelta
beats all training algorithms, given the same budget of training steps.

##  7 Conclusion

We propose an algorithm that trains layers of a neural network, all at the
same speed. Our algorithm, PercentDelta, is a simple modification over
standard Gradient Descent. It divides the gradient w.r.t. a trainable tensor
over the mean of $||\textrm{gradient / tensor}||_{1}$. The division over mean
L1-norm is scalar, and only changes the gradient’s magnitude but not its
direction. Effectively, this updates the L1 norm of trainable layers, all at
the same rate. We recommend a linear decaying change-rate schedule. Our
modified gradients can be passed through a standard momentum accumulator
(Sutskever et al.,, 2013). Overall, we show experimentally that our algorithm
puts an upper envelop on all training algorithms, reaching higher test
accuracy with fewer steps.

## References

  * Abu-El-Haija et al., (2017) Abu-El-Haija, S., Perozzi, B., and Al-Rfou, R. (2017).  Learning edge representations via low-rank asymmetric projections.  In ACM International Conference on Information and Knowledge Management (CIKM).
  * Ba and Kingma, (2015) Ba, J. and Kingma, D. (2015).  Adam: A method for stochastic optimization.  In International Conference on Learning Representations.
  * Ba et al., (2016) Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016).  Layer normalization.  In arxiv.
  * Duchi et al., (2011) Duchi, J., Hazan, E., and Singer, Y. (2011).  Adaptive subgradient methods for online learning and stochastic optimization.  In Journal of Machine Learning Research.
  * Glorot and Bengio, (2010) Glorot, X. and Bengio, Y. (2010).  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS’10).
  * Ioffe and Szegedy, (2015) Ioffe, S. and Szegedy, C. (2015).  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Journal of Machine Learning Research (JMLR).
  * Koren et al., (2009) Koren, Y., Bell, R. M., and Volinsky, C. (2009).  Matrix factorization techniques for recommender systems.  In IEEE Computer.
  * Ramanathan et al., (2016) Ramanathan, V., Huang, J., Abu-El-Haija, S., Gorban, A., Murphy, K., and Fei-Fei, L. (2016).  Detecting events and key actors in multi-person videos.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Rumelhart et al., (1986) Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986).  Learning representations by back-propagating errors.  In Nature.
  * Salimans and Kingma, (2016) Salimans, T. and Kingma, D. P. (2016).  Weight normalization: A simple reparameterization to accelerate training of deep neural networks.  In Neural Information Processing Systems.
  * Sutskever et al., (2013) Sutskever, I., Martens, J., Dahl, G. E., and Hinton, G. E. (2013).  On the importance of initialization and momentum in deep learning.  In International Conference on Machine Learning.
  * You et al., (2017) You, Y., Gitman, I., and Ginsburg, B. (2017).  Scaling sgd batch size to 32k for imagenet training.  In UC Berkeley Technical Report.


# Proca equation and vector field quantization in rotating system

Tian Xu  Physics Department, Beihang University, 37 Xueyuan Rd, Beijing
100191, China  Yin Jiang  Physics Department, Beihang University, 37 Xueyuan
Rd, Beijing 100191, China

(August 27, 2024)

###### Abstract

A strong background field will change the vacuum structure and the proper
basis of a system drastically in both classical and quantum mechanics, e.g.
the Landau levels in a background magnetic field. The situation is the same
for the rotating case. In such a system the usual set of plane-wave states
would no longer be suitable as a starting point of perturbation. Alternatively
and straightforwardly in a rapidly and globally rotating system, it is better
to reformulate the perturbation computation in principle. In this work we will
complete the first step for the spin-1 field, which includes solving the Proca
equation in present of a background rotation and complete its canonical
quantization. It will be shown that because of the symmetry the eigen states
are actually the same as the ones of Maxwell equations in cylindrical
coordinate. The propagator as well as the near-central approximation will be
obtained by considering the vorticity areas are so small in the relativistic
QGP.

###### pacs:

12.38.Aw, 12.38.Mh

##  I Introduction

As the measurements at the Relativistic Heavy Ion Collider(RHIC) and the Large
Hadron Collider(LHC) processingBedangadasMohantyfortheALICE:2017xgh ;
Zhou:2019lun ; Mohanty:2020bqq , the polarization behaviors of vector mesons
are more and more non-trivial and difficult to understand recently. These
measurements which were motivated by explorations of large background field in
the fire ball of Quark Gluon Plasma(QGP) produced in the relativistic heavy
ion collisions, includes the magnetic and vorticity fields, have led the
studies of the QGP phenomenological properties into a more complicated and
mysterious situation. For the magnetic field, although the mechanism of the
chiral magnetic effect(CME) is quite clear theoreticallyFukushima:2008xe ;
Kharzeev:2013ffa ; Sadofyev:2010pr , its detection appears to be quite
difficult because of large non-relevant fluctuationsWang:2012qs ; Aziz:2020nia
; Haque:2018jht ; Li:2020dwr . While for the vorticity, after a inspiring
qualitative agreements of the simulations of the $\Lambda$ polarization
dependence on the collision energy and centrality with the STAR
measurementsSTAR:2017ckg ; Xie:2017upb , it indicated that the vorticity
distribution and evolution in QGP may be much more out of ones’ expectation by
both the quantitative analysis on the global polarization and the qualitative
mismatch of the local polarization profilesShapoval:2017jej ; Li:2017dan ;
Becattini:2020ngo .

The dependence of the vorticity amplitude on the collision energy and
centrality has been studied in Wei:2018zfb  for the initial state of QGP and
further simulated in Jiang:2016woz  with a multi-phase transport model(AMPT)
for the whole period of QGP evolution. As the qualitative predictions is quite
reasonable, ones believe that the traditional understanding on the vorticity
should not be completely wrong. Thus from the aspects of kinetic theory,
quantum Wigner functions and hydrodynamics, various attemptsKharzeev:2013ffa ;
Li:2017dan ; Karpenko:2021wdm ; Gao:2019znl ; Weickgenannt:2019dks ;
DelZanna:2013eua ; Karpenko:2013wva  have been made to mend the usual theory
which is basing on the straightforward scenario of the parton collisions, such
as different vorticity definitions by considering the thermal
environmentBecattini:2020ngo ; Becattini:2015ska  and novel vacuum structures
by the strongly rotating systemFang:2016uds . In most of these works, quraks,
which serve as the visible spin-carries of the final-state hadrons, have
attracted almost all the interestsYang:2017sdk ; Sheng:2019kmk . The gluons,
which carrying double spins and thus suffered double polarization effects, are
neglected because of the technical problems in most cases. As it is closely
related with the quantum chromodynamics(QCD)’s fundamental problem, in this
work we will only take our first step towards the gluon’s part, namely to
study the vector field in the present of a background rotation field.

A strong background field will change the vacuum structure of a system
drastically in both classical and quantum mechanicsFukushima:2018grm . The
Landau level is the most famous example when the system in a background
magnetic field. In such a case the usual set of plane-wave states would no
longer be suitable as a starting point of perturbation. However the standard
quantum field theory on textbooks roots in choosing the in and out states are
plane-wave which is natural in high energy collisions. Therefore, in a rapidly
and globally rotating system, a difficult but straightforward alternative is
to reformulate the perturbation computation in principle. In this work we will
complete the first step, which includes solving the Proca equation in present
of a background rotation and complete its canonical quantization. It will be
shown that because of the symmetry the eigen states are actually the same as
the ones of Maxwell equations in cylindrical coordinateDai:2012bc ;
Tobar:2018arx ; Chernodub:2018era . The propagator as well as the near-central
approximation will be obtained by considering the vorticity areas are so small
in the relativistic QGP. We will discuss the subtle zero-mass and gauge-
symmetry topics in such a curved space-time system in our further works.

In this work the Latin and Greek index represent components of vectors/tensors
in the local rest/inertial frame and curved frame respectively, such as
$X^{a}$ and $X^{\mu}$. The $X^{a}$ respect the usual Lorentz transformation,
while the $X^{\mu}$s obey the coordinate transformation in its curved space-
time. For any vector these two kinds of components are connected by tetrads
$e^{a}_{\ \mu}$ as

$\displaystyle X^{a}=e^{a}_{\ \mu}X^{\mu}$ (1)

##  II Rotating Proca field

We consider the rotating system in a local rest/inertial frame, up to a local
Lorentz transformation tetrads $e_{a}^{\ \mu}$ and $e^{a}_{\ \mu}$ which
bridge the vectors in curved and flat space-time can be chosen as

$\displaystyle e_{\ \mu}^{a}=\left(\begin{array}[]{cccc}1&0&0&0\\\
v_{1}&1&0&0\\\ v_{2}&0&1&0\\\ v_{3}&0&0&1\\\ \end{array}\right),e^{\
\mu}_{a}=\left(\begin{array}[]{cccc}1&-v_{1}&-v_{2}&-v_{3}\\\ 0&1&0&0\\\
0&0&1&0\\\ 0&0&0&1\\\ \end{array}\right)$ (10)

By definition they are connected with the global metrics and should satisfy
the following relations which roughly means defining a local Minkowski frame
by absorbing the space-time curvature into local coordinates.

$\displaystyle\eta_{ab}=g_{\mu\nu}e_{a}^{\ \mu}e_{b}^{\ \nu}$ (11)
$\displaystyle e^{a}_{\ \mu}e^{\ \mu}_{b}=\delta^{a}_{b}$ (12) $\displaystyle
e^{\ \mu}_{a}e^{a}_{\ \nu}=\delta^{\mu}_{\nu}$ (13)

where $\eta_{ab}=\\{+,-,-,-\\}$ and the rotating metric is

$\displaystyle
g_{\mu\nu}=\left(\begin{array}[]{cccc}1-v_{1}^{2}-v_{2}^{2}-v_{3}^{2}&-v_{1}&-v_{2}&-v_{3}\\\
-v_{1}&-1&0&0\\\ -v_{2}&0&-1&0\\\ -v_{3}&0&0&-1\\\ \end{array}\right)$ (18)

Obviously there are infinite equivalent choices of tetrads satisfying the
relation 11,12,13 up to a local Lorentz transformation. Here we consider the
uniformly rotation case which means the linear velocity is
$\vec{v}=\vec{\omega}\times\vec{x}$, where $\vec{\omega}$, the angular
velocity, is a constant. For such a globally rotating system the boundary
condition is necessary because of the light-speed limit. As the sharp cut
boundary will only result in a discrete set of radial wave function thus
import no structural changes to the following formalism, we will not do this
discussion explicitly in this work and keep the summation over $k_{t}$ as
integration. We should keep in mind for different boundary condition this will
be discretized into different series. And realistically in the QGP the
rotation areas or volumes are so small that the light-speed limit problem is
far from importantJiang:2016woz .

We will extract rotation effects by construct the Proca Lagrangian density
with the field in the local inertial frame. According to the general
relativistic principle it should be the same as the flat one except all the
quantities are in the curved space-time. By using the relation 11 the vector
Lagrangian density could be written as

$\displaystyle\mathcal{L}_{v}=-\frac{1}{4}F^{\mu\nu}F_{\mu\nu}+\frac{1}{2}m^{2}A^{2}=-\frac{1}{4}F^{ab}F_{ab}+\frac{1}{2}m^{2}A^{2}$
(19)

where the covariant derivative in the flat space-time is

$\displaystyle D_{a}A_{b}=e_{a}^{\ \mu}D_{\mu}A_{b}=e_{a}^{\
\mu}(\partial_{\mu}A_{b}+\Gamma_{\mu bc}A^{c})$ (20)

and the tensor of field strength is given by $F_{ab}=D_{a}A_{b}-D_{b}A_{a}$.
The reason we need Eq.20 is that we are trying to study the system in an
inertial frame which means what we observe or measure are the quantities
constructed by the field in the flat space-time. The operation $D_{a}$ can be
defined via the $D_{\mu}$ which operates on the quantities as functions of
global coordinates, i.e. Eq.20. By this space-time shifts we are now sitting
in a local-rest frame(up to a local Lorentz transformation) and the rotation
effects whose effect is bending or twisting the global space-time is felt as
some background interactions appearing in the local Lagrangian density as an
additional term, e.g. the explicit polarization effect term for the fermionic
case. Here one should note that the connection $\Gamma_{\mu bc}$ is not the
usual Christoffel connection. It has included derivatives of the local
inertial basis along the world line of the curved space-time. It is expressed
as

$\displaystyle\Gamma_{\mu ab}=\eta_{ac}[e^{c}_{\ \nu}e^{\
\lambda}_{b}G^{\nu}_{\ \mu\lambda}-e^{\ \lambda}_{b}\partial_{\mu}e^{c}_{\
\lambda}]$ $\displaystyle=\eta_{ac}(\delta^{c}_{\ \nu}+\xi^{c}_{\
\nu})(\delta^{\ \lambda}_{b}+\xi^{\
\lambda}_{b})\frac{1}{2}g^{\nu\alpha}(\partial_{\mu}h_{\alpha\lambda}+\partial_{\lambda}h_{\alpha\mu}-\partial_{\alpha}h_{\mu\lambda})$
$\displaystyle\ \ \ \ -\eta_{ac}(\delta^{\ \lambda}_{b}+\xi^{\
\lambda}_{b})\partial_{\mu}(\delta^{c}_{\ \lambda}+\xi^{c}_{\ \lambda})$
$\displaystyle=\eta_{ac}(\delta^{c}_{\ \nu})(\delta^{\
\lambda}_{b})\frac{1}{2}\eta^{\nu\alpha}(\partial_{\mu}h_{\alpha\lambda}+\partial_{\lambda}h_{\alpha\mu}-\partial_{\alpha}h_{\mu\lambda})$
$\displaystyle\ \ \ \ -\eta_{ac}(\delta^{\
\lambda}_{b})\partial_{\mu}(\xi^{c}_{\ \lambda})$
$\displaystyle=\frac{1}{2}(\partial_{\mu}h_{ab}+\partial_{b}h_{a\mu}-\partial_{a}h_{\mu
b})-\partial_{\mu}\xi_{ab}$ (21)

where $G^{\nu}_{\ \mu\lambda}$ is the usual Christoffel connection in the
curved space-time. Its nonzero elements are

$\displaystyle\Gamma_{0ij}=\frac{1}{2}(-\partial_{j}v_{i}+\partial_{i}v_{j})=\epsilon_{ijm}\omega_{m}$
$\displaystyle\Gamma_{i0j}=-\frac{1}{2}(\partial_{j}v_{i}+\partial_{i}v_{j})$
$\displaystyle\Gamma_{ij0}=\frac{1}{2}(\partial_{j}v_{i}+\partial_{i}v_{j})$
(22)

Obviously the last two equations are actually zero in the uniform rotation
case.

Substituting the covariant derivation into the Lagrangian density, the
straightforward calculation shows in the local flat frame the field strength
tensor is modified by the rotation as

$\displaystyle
F_{0i}=-F^{0i}=(\partial_{0}A_{i}-\partial_{i}A_{0})-v_{j}\partial_{j}A_{i}-\epsilon_{ijk}\omega_{k}A_{j}$
$\displaystyle F_{ij}=F^{ij}=\partial_{i}A_{j}-\partial_{j}A_{i}$ (23)

With our choice of the tetrads only the ”electric” parts is changed while the
”magnetic” parts are invariant. Obviously this conclusion is tetrads-dependent
because the ”electric” and ”magnetic” part are transferable under a local
Lorentz transformation. Hence it is easy to compute the rotation-modified
Lagrangian density and split it into the free and rotation-polarization parts
as

$\displaystyle\mathcal{L}_{v}(\vec{\omega})=\mathcal{L}_{v}(\vec{\omega}=0)+\delta\mathcal{L}_{v}(\vec{\omega})$
(24)
$\displaystyle=\mathcal{L}_{v}(\vec{\omega}=0)-v_{j}f_{ji}f_{0i}+\frac{1}{2}(v_{j}\partial_{j}A_{i}+\epsilon_{ijm}\omega_{m}A_{j})^{2}$

where
$\mathcal{L}_{v}(\vec{\omega}=0)=-\frac{1}{2}(\sum_{i}-f_{0i}^{2}+\sum_{i<j}f_{ij}^{2})$
and $f_{ab}=\partial_{a}A_{b}-\partial_{b}A_{a}$. Here we have lowered all of
the indices. It is easy to discover that the $\mathcal{O}(\omega)$ term is
polarization form
$\vec{v}\cdot(\vec{E}\times\vec{B})=\vec{\omega}\cdot\vec{J}$, where angular
momentum is $\vec{J}=\vec{r}\times\vec{P}$ and $\vec{P}=\vec{E}\times\vec{B}$.
However different from the fermionic case the $\mathcal{O}(\omega^{2})$ terms
are non-negligible because of the eigen equations of the vector field are
second order, i.e. $\partial_{0}^{2}$. The $\mathcal{O}(\omega^{2})$ terms are
actually corresponding to the $\mathcal{O}(\omega)$ corrections to the eigen
energies. This is shown as follows.

The corresponding equations of motion are obtained as

$\displaystyle\partial_{i}f_{i0}-m^{2}A_{0}=\Delta A_{0}-m^{2}A_{0}=0$ (25)
$\displaystyle\partial^{2}_{0}A_{i}-\Delta
A_{i}-2v_{j}\partial_{0}\partial_{j}A_{i}+(\partial_{j}v_{i}-\partial_{i}v_{j})\partial_{0}A_{j}$
$\displaystyle+v_{j}\partial_{j}(v_{n}\partial_{n}A_{i}+2\epsilon_{inm}\omega_{m}A_{n})-(\omega^{2}A_{i}-\omega_{n}\omega_{i}A_{n})$
$\displaystyle+m^{2}A_{i}=0$ (26)

There are three polarization components for the massive vector field. We could
first adopt the transverse constraint $\nabla\cdot\vec{A}=0$ to extract the
two transverse ones. Thus the $\partial_{i}f_{i0}$ is reduced to $\Delta
A_{0}$. Considering the operator $\Delta=\nabla^{2}$ is semi-positive, the
first equation gives $A_{0}=0$. By choosing the angular velocity direction as
the z axis, i.e. $\vec{\omega}=\omega e_{z}$ in the cylindrical coordinate the
second one degenerates to the ordinary form in the $\omega=0$ case
$\partial^{2}_{0}A_{i}-\Delta A_{i}=0$. The solutions are the well-known
cylindrical waves which could be found in any electrodynamics textbook.
Adopting the same symbols as a recent literature Chernodub:2018era  in the
cylindrical system the solutions are

$\displaystyle A_{TE}=\left(\begin{array}[]{c}A_{0}\\\ A_{\rho}^{TE}\\\
A_{\phi}^{TE}\\\ A_{z}^{TE}\\\ \end{array}\right)=\left(\begin{array}[]{c}0\\\
\frac{n}{k_{t}\rho}J_{n}(k_{t}\rho)\\\
\frac{i}{k_{t}}\partial_{\rho}J_{n}(k_{t}\rho)\\\ 0\\\
\end{array}\right)e^{i(n\phi+k_{z}z-E_{nk_{t}k_{z}}t)}$ (35) (36)
$\displaystyle A_{TM}=\left(\begin{array}[]{c}A_{0}\\\ A_{\rho}^{TM}\\\
A_{\phi}^{TM}\\\ A_{z}^{TM}\\\ \end{array}\right)=\left(\begin{array}[]{c}0\\\
\frac{k_{z}}{E_{k}k_{t}}\partial_{\rho}J_{n}(k_{t}\rho)\\\
i\frac{nk_{z}}{E_{k}k_{t}\rho}J_{n}(k_{t}\rho)\\\
-i\frac{k_{t}}{E_{k}}J_{n}(k_{t}\rho)\\\
\end{array}\right)e^{i(n\phi+k_{z}z-E_{nk_{t}k_{z}}t)}$ (45)

where $E_{k}=\sqrt{k_{t}^{2}+k_{z}^{2}}$. In the static case the eigen
energies are $E_{nk_{t}k_{z}}=E_{k}$. For the longitudinal part we choose
$A_{\rho}=A_{\phi}=0$ and the equation gives the solution for the third
polarization direction

$\displaystyle A_{L}=\left(\begin{array}[]{c}\frac{k_{z}}{E_{kt}}\\\ 0\\\ 0\\\
-\frac{E_{k}}{E_{kt}}\\\
\end{array}\right)J_{n}(k_{t}\rho)e^{in\phi}e^{ik_{z}z}e^{-iE_{nk_{t}k_{z}}t}$
(50)

where $E_{kt}=\sqrt{k_{t}^{2}+m^{2}}$.

As the uniform rotation preserves the cylindrical symmetry we expect these
solutions are eigen states of the rotation case as well with a modified
energy. By substituting them into Eq.II we find the equations are satisfied if
the energy is solved from

$\displaystyle(-E_{nk_{t}k_{z}}^{2}+E_{k}^{2}-2E_{nk_{t}k_{z}}\omega
n-\omega^{2}n^{2})\vec{A}=0$ (51)

This equation gives the new eigen energies are $E_{nk_{t}k_{z}}=E_{k}-n\omega$
which as our expectation have obtained $n\omega$ polarization corrections for
the spin-1 field on top of the rotation background.

The nontrivial difference between these solutions and the usual plane-wave
case is the following properties are satisfied

$\displaystyle\nabla\times\vec{A}_{TE}=E_{k}\vec{A}_{TM}$ (52)
$\displaystyle\nabla\times\vec{A}_{TM}=E_{k}\vec{A}_{TE}$ (53)

As comparison we list the similar relations for the plane-wave
$\vec{A}^{pw}_{1,2}$ as follows

$\displaystyle\nabla\times\vec{A}_{1}^{pw}=i|\vec{k}||\vec{A}^{pw}_{1}|/|\vec{A}^{pw}_{2}|\vec{A}^{pw}_{1}$
(55)
$\displaystyle\nabla\times\vec{A}_{2}^{pw}=-i|\vec{k}||\vec{A}^{pw}_{2}|/|\vec{A}^{pw}_{1}|\vec{A}^{pw}_{2}$
(56)

The same signs in Eq.52 and 53 allow us to construct two new modes, i.e.
$\vec{A}_{\pm}=\vec{A}_{TM}\pm\vec{A}_{TE}$, whose $\vec{E}\cdot\vec{B}\neq
0$. Obviously they satisfy

$\displaystyle\nabla\times\vec{A}_{+}=E_{k}\vec{A}_{+}$ (57)
$\displaystyle\nabla\times\vec{A}_{-}=-E_{k}\vec{A}_{-}$ (58)

To see this clearly we list the $\vec{E}$ and $\vec{B}$ as follows

$\displaystyle\vec{E}_{+}=\partial_{0}A^{+}_{i}=-iE_{nk_{t}k_{z}}A^{+}_{i}$
(60)
$\displaystyle\vec{B}_{+}=-\epsilon_{ijk}\partial_{j}A^{+}_{k}=-E_{k}A^{+}_{i}$
(61)

and

$\displaystyle\vec{E}_{-}=\partial_{0}A^{-}_{i}=-iE_{nk_{t}k_{z}}A^{-}_{i}$
(62)
$\displaystyle\vec{B}_{-}=-\epsilon_{ijk}\partial_{j}A^{-}_{k}=E_{k}A^{-}_{i}$
(63)

Of course this will not cause problems because of
$\vec{E}_{+}\cdot\vec{B}_{+}+\vec{E}_{-}\cdot\vec{B}_{-}=0$.

##  III Propagator

In this section we will complete the canonical quantization for the spin-1
vector field. First we check the orthogonality of the three eigen states.
Obviously the $A_{TM}$ has mixed part of $A_{L}$. In order to construct the
propagator of vector field, we first orthogonalize it as

$\displaystyle B_{L}=A_{L}-\alpha A_{TM}$ (64) $\displaystyle B_{L}\cdot
A_{TM}=0$ (65)

The orthogonalization constraint gives
$\alpha=-ik_{t}\sqrt{k^{2}+m^{2}}/(k\sqrt{k^{2}_{t}+m^{2}})$. In the following
we will rename $B_{L}$ as $A_{L}$. And all the three eigen states are
normalized as

$\displaystyle\int\rho d\rho d\phi dzA^{\mu}_{TE,TM,L}A_{\mu,TE,TM,L}$
$\displaystyle=-(2\pi)^{2}\delta_{mn}\delta(k_{z}-p_{z})\frac{1}{k_{t}}\delta(k_{t}-p_{t})$
(66)

The canonical quantization gives the vector field as

$\displaystyle
A_{\mu}=\sum_{n,\lambda}\int\frac{dk_{z}dk_{t}}{2\pi\sqrt{2E_{k}}}e^{-iE_{n}t}e^{ik_{z}z}e^{in\phi}a_{nk_{t}k_{z}}A_{\mu,\lambda}+h.c.$
(67)

where $\lambda\in\\{L,TM,TE\\}$ is the polarization index. Starting from the
quantization assumption $[A_{\mu}(x),\Pi_{\nu}(y)]=g_{\mu\nu}\delta^{4}(x-y)$,
the commutation relation of $a$s and $a^{\dagger}$ are derived as

$\displaystyle[a_{nk_{t}k_{z}},a^{\dagger}_{mp_{t}p_{z}}]=\delta_{mn}\delta(k_{z}-p_{z})k_{t}\delta(k_{t}-p_{t})$
(68)

Therefore the propagator from $x=(t,\vec{x})=(t,\rho,\theta,\phi,z)$ to
$y=(s,\vec{y})=(s,r,\theta,\zeta)$ is defined as

$\displaystyle D_{\mu\nu}(x,y)=\langle 0|TA_{\mu}(x)A_{\nu}(y)|0\rangle$ (69)
$\displaystyle=\theta(t-s)\langle 0|A_{\mu}A_{\nu}|0\rangle+\theta(s-t)\langle
0|A_{\nu}A_{\mu}|0\rangle$

where the $T$ is the time-ordering operator. Substituting the field operators
into it and changing the summation over $n$ into $-n$ in one of the two terms,
the propagator is reduced as

$\displaystyle D_{\mu\nu}(x,y)=\frac{i}{(2\pi)^{3}}\sum_{n,\lambda}\int
dk_{0}dk_{z}k_{t}dk_{t}e^{ik_{z}(z-\zeta)}e^{in(\phi-\theta)}$
$\displaystyle\times
e^{-i(k_{0}-n\omega)(t-s)}\frac{A_{\mu,\lambda}(k_{t},n,k_{z};\rho)A^{*}_{\nu,\lambda}(k_{t},n,k_{z};r)}{k_{0}^{2}-E_{k}^{2}+i\eta}$
$\displaystyle=\frac{i}{(2\pi)^{3}}\sum_{n,\lambda}\int
dk_{0}dk_{z}k_{t}dk_{t}e^{ik_{z}(z-\zeta)}e^{in(\phi-\theta)}$
$\displaystyle\times
e^{-ik_{0}(t-s)}\frac{A_{\mu,\lambda}(k_{t},n,k_{z};\rho)A^{*}_{\nu,\lambda}(k_{t},n,k_{z};r)}{(k_{0}-n\omega)^{2}-E_{k}^{2}+i\eta}$
(70)

which is similar to the result in flat space-time. At the second equation we
have shifted the integration over $k_{0}$ to $k_{0}-n\omega$ to make the poles
locate at $E_{k}-n\omega$ which corresponding to the eigen energies of the
vector field. Once again we find the rotation effects serve as an additional
chemical potential rather than a straightforward energy shift as
$k_{0}^{2}-E_{nk_{t}k_{z}}^{2}$. This is the same as the fermionic case. And
the straightforward computation gives the Lorentz structure part as

$\displaystyle
D^{n}_{\mu\nu}(k_{t},k_{z};\rho,r)=\sum_{\lambda}A_{\mu,\lambda}(k_{t},n,k_{z};\rho)A^{*}_{\nu,\lambda}(k_{t},n,k_{z};r)$
$\displaystyle=\frac{1}{4}\left(\begin{array}[]{cccc}0&0&0&0\\\
0&M_{n}^{++}&-iM_{n}^{+-}&0\\\ 0&iM_{n}^{-+}&M_{n}^{--}&0\\\ 0&0&0&0\\\
\end{array}\right)$ (75)
$\displaystyle+\frac{E_{kt}^{2}}{4m^{2}}\left(\begin{array}[]{cccc}0&0&0&0\\\
0&M_{n}^{--}&-iM_{n}^{-+}&-2i\frac{k_{t}k_{z}}{E_{kt}^{2}}N_{n}^{1-}\\\
0&iM_{n}^{+-}&M_{n}^{++}&2\frac{k_{t}k_{z}}{E_{kt}^{2}}N_{n}^{1+}\\\
0&2i\frac{k_{t}k_{z}}{E_{kt}^{2}}N_{n}^{2-}&2\frac{k_{t}k_{z}}{E_{kt}^{2}}N_{n}^{2+}&4\frac{E_{z}^{2}}{E_{kt}^{2}}\Pi_{n}\\\
\end{array}\right)$ (80)
$\displaystyle+\frac{k_{t}E_{kt}}{2m^{2}}\left(\begin{array}[]{cccc}\frac{2k^{2}}{k_{t}E_{k}}\Pi_{n}&-iN_{n}^{2-}&-N_{n}^{2+}&-2\frac{k_{z}}{k_{t}}\Pi_{n}\\\
iN_{n}^{1-}&0&0&0\\\ -N_{n}^{1+}&0&0&0\\\
-2\frac{k_{z}}{k_{t}}\Pi_{n}&0&0&0\\\ \end{array}\right)$ (85)

In order to make the expression more compact we have used symbols

$\displaystyle M_{n}^{++}=Z^{+}_{n}(\rho,\phi)Z^{+}_{n}(r,\theta)$
$\displaystyle M_{n}^{+-}=Z^{+}_{n}(\rho,\phi)Z^{-}_{n}(r,\theta)$
$\displaystyle M_{n}^{-+}=Z^{-}_{n}(\rho,\phi)Z^{+}_{n}(r,\theta)$
$\displaystyle M_{n}^{--}=Z^{-}_{n}(\rho,\phi)Z^{-}_{n}(r,\theta),$ (86)
$\displaystyle N_{n}^{1+}=Z^{+}_{n}(\rho,\phi)J_{n}(r)$ $\displaystyle
N_{n}^{1-}=Z^{-}_{n}(\rho,\phi)J_{n}(r)$ $\displaystyle
N_{n}^{2+}=J_{n}(\rho)Z^{+}_{n}(r,\theta)$ $\displaystyle
N_{n}^{2-}=J_{n}(\rho)Z^{-}_{n}(r,\theta)$ (87)

and

$\displaystyle\Pi_{n}=J_{n}(\rho)J_{n}(r)$ (88)

where $Z^{+}_{n}(\rho,\phi)=J_{n-1}(\rho)e^{-i\phi}+J_{n+1}(\rho)e^{i\phi}$
and $Z^{-}_{n}(\rho,\phi)=J_{n-1}(\rho)e^{-i\phi}-J_{n+1}(\rho)e^{i\phi}$ and
write the $J_{n}(k_{t}\rho)$ as $J_{n}(\rho)$ for simplicity. Beware here the
propagator is in the Cartesian coordinate, e.g. the $D^{n}_{01}=D^{n}_{tx}$
and so on. Obviously the propagator is not invariant under translation on the
$X-Y$ plane. If we set the rotation speed $\omega=0$ it will be reduced to the
usual one $D_{\mu\nu}(x-y)$ in flat space-time. This can be checked by a
straightforward computation. During the checking process the following
relations are helpful

$\displaystyle\sum_{n}e^{in(\phi-\phi^{\prime})}J_{n}(k_{t}\rho)J_{n}(k_{t}\rho^{\prime})=J_{0}(k_{t}|\vec{\rho}-\vec{\rho}^{\prime}|)$
(89)

$\displaystyle\int_{0}^{2\pi}d\xi e^{ikrcos\xi}cos(\xi+\alpha)=2\pi icos\alpha
J_{1}(kr)$ $\displaystyle\int_{0}^{2\pi}d\xi e^{ikrcos\xi}sin(\xi+\alpha)=2\pi
isin\alpha J_{1}(kr)$ (90)

$\displaystyle\int_{0}^{2\pi}d\xi e^{ikrcos\xi}cos(\xi+\alpha)cos(\xi+\alpha)$
$\displaystyle=\pi J_{0}(kr)-\pi cos2\alpha J_{2}(kr)$
$\displaystyle\int_{0}^{2\pi}d\xi e^{ikrcos\xi}sin(\xi+\alpha)sin(\xi+\alpha)$
$\displaystyle=\pi J_{0}(kr)+\pi cos2\alpha J_{2}(kr)$
$\displaystyle\int_{0}^{2\pi}d\xi e^{ikrcos\xi}cos(\xi+\alpha)sin(\xi+\alpha)$
$\displaystyle=-\pi sin2\alpha J_{2}(kr)$ (91)

##  IV Near central expansion

For a globally rotating system, the size is limited, especially for the QGP
which is believed rotating very rapidly fireball product in relativistic heavy
ion collisions. AMPT’s simulation indicates the vorticity profile is more like
a set of vorticity spots rather than a rotating lemon. Therefore we consider
the propagator in the near central case and set one of the space-time point as
the origin, i.e. $r=0$ and $\rho\rightarrow 0$. This will reduce it into three
non-zero parts corresponding to $n=0,\pm 1$ because of only $J_{0}(0)\neq 0$.
The explicit results are

$\displaystyle
D^{n=-1}_{\mu\nu}(k_{t},k_{z},\rho)=\frac{1}{4}\left(\begin{array}[]{cccc}0&-2i\frac{k_{t}E_{k}}{m^{2}}J_{1}&2\frac{k_{t}E_{k}}{m^{2}}J_{1}&0\\\
0&2J_{0}e^{i\phi}-\frac{k_{t}^{2}}{m^{2}}Z^{-}_{-1}(\rho,\phi)&2iJ_{0}e^{i\phi}-i\frac{k_{t}^{2}}{m^{2}}Z^{-}_{-1}(\rho,\phi)&0\\\
0&-2iJ_{0}e^{i\phi}-i\frac{k_{t}^{2}}{m^{2}}Z^{+}_{-1}(\rho,\phi)&2J_{0}e^{i\phi}+\frac{k_{t}^{2}}{m^{2}}Z^{+}_{-1}(\rho,\phi)&0\\\
0&2i\frac{k_{t}k_{z}}{m^{2}}J_{1}&-2\frac{k_{t}k_{z}}{m^{2}}J_{1}&0\\\
\end{array}\right)$ (96)

$\displaystyle
D^{n=0}_{\mu\nu}(k_{t},k_{z},\rho)=\left(\begin{array}[]{cccc}\frac{k^{2}}{m^{2}}J_{0}&0&0&-\frac{E_{k}k_{z}}{m^{2}}J_{0}\\\
i\frac{k_{t}E_{k}}{2m^{2}}Z^{-}_{0}(\rho,\phi)&0&0&-i\frac{k_{t}E_{k}}{2m^{2}}Z^{-}_{0}(\rho,\phi)\\\
-\frac{k_{t}E_{k}}{2m^{2}}Z^{+}_{0}(\rho,\phi)&0&0&\frac{k_{t}E_{k}}{2m^{2}}Z^{+}_{0}(\rho,\phi)\\\
-\frac{E_{k}k_{z}}{m^{2}}J_{0}&0&0&\frac{E^{2}_{z}}{m^{2}}J_{0}\\\
\end{array}\right)$ (101)

$\displaystyle
D^{n=1}_{\mu\nu}(k_{t},k_{z},\rho)=\frac{1}{4}\left(\begin{array}[]{cccc}0&-2i\frac{k_{t}E_{k}}{m^{2}}J_{1}&-2\frac{k_{t}E_{k}}{m^{2}}J_{1}&0\\\
0&2J_{0}e^{-i\phi}+\frac{k_{t}^{2}}{m^{2}}Z^{-}_{1}(\rho,\phi)&-2iJ_{0}e^{-i\phi}-i\frac{k_{t}^{2}}{m^{2}}Z^{-}_{1}(\rho,\phi)&0\\\
0&2iJ_{0}e^{-i\phi}+i\frac{k_{t}^{2}}{m^{2}}Z^{+}_{1}(\rho,\phi)&2J_{0}e^{-i\phi}+\frac{k_{t}^{2}}{m^{2}}Z^{+}_{1}(\rho,\phi)&0\\\
0&2i\frac{k_{t}k_{z}}{m^{2}}J_{1}&2\frac{k_{t}k_{z}}{m^{2}}J_{1}&0\\\
\end{array}\right)$ (106)

##  V Summary and Outlook

We studied the massive vector field with the Proca equation in the present of
rotation background by introducing the curved metrics in the equation.
Different from the fermionic case besides a explicit polarization term
$\vec{\omega}\cdot\vec{J}$ there are $\mathcal{O}(\omega^{2})$ terms which are
non-negligible as well. As our expectation because of the cylindrical symmetry
in the rotating system the eigen states are the same as the free Proca
equation in this coordinates. And the polarization effects are also obtained
as the energy shift in the eigen energies as $\Delta E=n\omega$. With
canonical quantization the propagator is also computed. Its poles indicate
that the rotation serves as an additional chemical potential rather than a
trivial energy modification. This is the same as that in the fermionic case.
It is also shown that the translation symmetry has been broken in the
cylindrical coordinate and thus the propagator depends on two space-time
points explicitly. Such a behavior will obviously make the further loop
computation much more complicated. As a potential simplified case we studied
the near-central result briefly because of the size of rotating area/volume is
usual limited. We have checked in zero-rotation limit it can be reduced to the
usual form explicitly by using the summation relations of Bessel functions.

This work is a preliminary study on the complete framework of solving the
quantum field problems defined in a curved coordinate. In such a case the
translation symmetry will be broken and so will the usual energy-momentum
conservation at interaction vertices because of the momentum are no longer
good quantum numbers when the space is not flat. In this work we have shown
that although the computation is much more complicated the standard process is
straightforward for the interaction-free case. But it could be expected that
the interaction terms will induce more fundamental problems especially for the
p-wave cases, e.g. $A^{2}\partial A$ which will suffer not only the no-flat
eigen states modification but also additional terms from the covariant
derivation. Obviously this will appear in the non-abelian gauge field model
such as QCD. Therefore before face the interaction problem we should also do
more studies on the gauge symmetry in such coordinates. We will continue to
discuss these topics in our future works.

Acknowledgments. The work of this research is supported by the National
Natural Science Foundation of China, Grant Nos. 11875002(YJ) and the Zhuobai
Program of Beihang University.

## References

  * (1) B. Mohanty [ALICE], EPJ Web Conf. 171, 16008 (2018) doi:10.1051/epjconf/201817116008 [arXiv:1711.02018 [nucl-ex]].
  * (2) C. Zhou, Nucl. Phys. A 982, 559-562 (2019) doi:10.1016/j.nuclphysa.2018.09.009
  * (3) B. Mohanty [ALICE], PoS ICHEP2020, 555 (2021) doi:10.22323/1.390.0555 [arXiv:2012.04167 [nucl-ex]].
  * (4) K. Fukushima, D. E. Kharzeev and H. J. Warringa, Phys. Rev. D 78, 074033 (2008) doi:10.1103/PhysRevD.78.074033 [arXiv:0808.3382 [hep-ph]].
  * (5) D. E. Kharzeev, Prog. Part. Nucl. Phys. 75, 133-151 (2014) doi:10.1016/j.ppnp.2014.01.002 [arXiv:1312.3348 [hep-ph]].
  * (6) A. V. Sadofyev and M. V. Isachenkov, Phys. Lett. B 697, 404-406 (2011) doi:10.1016/j.physletb.2011.02.041 [arXiv:1010.1550 [hep-th]].
  * (7) G. Wang [STAR], Nucl. Phys. A 904-905, 248c-255c (2013) doi:10.1016/j.nuclphysa.2013.01.069 [arXiv:1210.5498 [nucl-ex]].
  * (8) S. Aziz [ALICE], Nucl. Phys. A 1005, 121817 (2021) doi:10.1016/j.nuclphysa.2020.121817 [arXiv:2005.06177 [hep-ex]].
  * (9) M. R. Haque [ALICE], Nucl. Phys. A 982, 543-546 (2019) doi:10.1016/j.nuclphysa.2018.11.008 [arXiv:1807.09360 [hep-ex]].
  * (10) W. Li and G. Wang, Ann. Rev. Nucl. Part. Sci. 70, 293-321 (2020) doi:10.1146/annurev-nucl-030220-065203 [arXiv:2002.10397 [nucl-ex]].
  * (11) L. Adamczyk et al. [STAR], Nature 548, 62-65 (2017) doi:10.1038/nature23004 [arXiv:1701.06657 [nucl-ex]].
  * (12) Y. Xie, D. Wang and L. P. Csernai, Phys. Rev. C 95, no.3, 031901 (2017) doi:10.1103/PhysRevC.95.031901 [arXiv:1703.03770 [nucl-th]].
  * (13) V. M. Shapoval, P. Braun-Munzinger and Y. M. Sinyukov, Nucl. Phys. A 968, 391-402 (2017) doi:10.1016/j.nuclphysa.2017.09.002 [arXiv:1707.06753 [hep-ph]].
  * (14) H. Li, H. Petersen, L. G. Pang, Q. Wang, X. L. Xia and X. N. Wang, Nucl. Phys. A 967, 772-775 (2017) doi:10.1016/j.nuclphysa.2017.04.008 [arXiv:1704.03569 [nucl-th]].
  * (15) F. Becattini and M. A. Lisa, Ann. Rev. Nucl. Part. Sci. 70, 395-423 (2020) doi:10.1146/annurev-nucl-021920-095245 [arXiv:2003.03640 [nucl-ex]].
  * (16) D. X. Wei, W. T. Deng and X. G. Huang, Phys. Rev. C 99, no.1, 014905 (2019) doi:10.1103/PhysRevC.99.014905 [arXiv:1810.00151 [nucl-th]].
  * (17) Y. Jiang, Z. W. Lin and J. Liao, Phys. Rev. C 94, no.4, 044910 (2016) [erratum: Phys. Rev. C 95, no.4, 049904 (2017)] doi:10.1103/PhysRevC.94.044910 [arXiv:1602.06580 [hep-ph]].
  * (18) I. Karpenko, [arXiv:2101.04963 [nucl-th]].
  * (19) J. H. Gao and Z. T. Liang, Phys. Rev. D 100, no.5, 056021 (2019) doi:10.1103/PhysRevD.100.056021 [arXiv:1902.06510 [hep-ph]].
  * (20) N. Weickgenannt, X. L. Sheng, E. Speranza, Q. Wang and D. H. Rischke, Phys. Rev. D 100, no.5, 056018 (2019) doi:10.1103/PhysRevD.100.056018 [arXiv:1902.06513 [hep-ph]].
  * (21) L. Del Zanna, V. Chandra, G. Inghirami, V. Rolando, A. Beraudo, A. De Pace, G. Pagliara, A. Drago and F. Becattini, Eur. Phys. J. C 73, 2524 (2013) doi:10.1140/epjc/s10052-013-2524-5 [arXiv:1305.7052 [nucl-th]].
  * (22) I. Karpenko, P. Huovinen and M. Bleicher, Comput. Phys. Commun. 185, 3016-3027 (2014) doi:10.1016/j.cpc.2014.07.010 [arXiv:1312.4160 [nucl-th]].
  * (23) F. Becattini, G. Inghirami, V. Rolando, A. Beraudo, L. Del Zanna, A. De Pace, M. Nardi, G. Pagliara and V. Chandra, Eur. Phys. J. C 75, no.9, 406 (2015) [erratum: Eur. Phys. J. C 78, no.5, 354 (2018)] doi:10.1140/epjc/s10052-015-3624-1 [arXiv:1501.04468 [nucl-th]].
  * (24) R. h. Fang, J. y. Pang, Q. Wang and X. n. Wang, Phys. Rev. D 95, no.1, 014032 (2017) doi:10.1103/PhysRevD.95.014032 [arXiv:1611.04670 [nucl-th]].
  * (25) Y. G. Yang, R. H. Fang, Q. Wang and X. N. Wang, Phys. Rev. C 97, no.3, 034917 (2018) doi:10.1103/PhysRevC.97.034917 [arXiv:1711.06008 [nucl-th]].
  * (26) X. L. Sheng, L. Oliva and Q. Wang, Phys. Rev. D 101, no.9, 096005 (2020) doi:10.1103/PhysRevD.101.096005 [arXiv:1910.13684 [nucl-th]].
  * (27) K. Fukushima, Prog. Part. Nucl. Phys. 107, 167-199 (2019) doi:10.1016/j.ppnp.2019.04.001 [arXiv:1812.08886 [hep-ph]].
  * (28) L. Dai, M. Kamionkowski and D. Jeong, Phys. Rev. D 86, 125013 (2012) doi:10.1103/PhysRevD.86.125013 [arXiv:1209.0761 [astro-ph.CO]].
  * (29) M. E. Tobar, B. T. McAllister and M. Goryachev, Phys. Dark Univ. 26, 100339 (2019) doi:10.1016/j.dark.2019.100339 [arXiv:1809.01654 [hep-ph]].
  * (30) M. N. Chernodub, A. Cortijo and K. Landsteiner, Phys. Rev. D 98, no. 6, 065016 (2018) doi:10.1103/PhysRevD.98.065016 [arXiv:1807.10705 [hep-th]].


# Enhancing Indoor Temperature Forecasting through Synthetic Data in Low-Data
Environments

Zachari Thiry1, Massimiliano Ruocco12, Alessandro Nocente1, Michail Spitieris1
1SINTEF AS, Strindvegen 4, 7034 Trondheim, Norway2Department of Computer
Science, Norwegian University of Science and Technology, Sem Sælandsvei 9,
7034 Trondheim, Norway

###### Abstract

Forecasting indoor temperatures is of paramount importance to achieve
efficient control of HVAC systems. In this task, the limited data availability
presents a challenge as most of the available data is acquired during standard
operation where extreme scenarios and transitory regimes such as major
temperature increases or decreases are de-facto excluded. Acquisition of such
data requires significant energy consumption and a dedicated facility,
hindering the quantity and diversity of available data. To acquire such data,
we make use of such a facility referred to as the Test-cell. Cost related
constraints however do not allow for continuous year-around acquisition. To
address this, we investigate the efficacy of data augmentation techniques,
particularly leveraging state-of-the-art AI-based methods for synthetic data
generation. Inspired by practical and experimental motivations, we explore
fusion strategies of real and synthetic data to improve forecasting models.
This approach alleviates the need for continuously acquiring extensive time
series data, especially in contexts involving repetitive heating and cooling
cycles in buildings. Our evaluation methodology for synthetic data synthesis
involves a dual-focused approach: firstly, we assess the performance of
synthetic data generators independently, particularly focusing on SoTA AI-
based methods; secondly, we measure the utility of incorporating synthetically
augmented data in a subsequent downstream tasks (forecasting). In the
forecasting tasks, we employ a simple model in two distinct scenarios: 1) we
first examine an augmentation technique that combines real and synthetically
generated data to expand the training dataset, 2) Second, we delve into
utilizing synthetic data to tackle dataset imbalances. Our results highlight
the potential of synthetic data augmentation in enhancing forecasting accuracy
while mitigating training variance. Through empirical experiments, we show
significant improvements achievable by integrating synthetic data, thereby
paving the way for more robust forecasting models in low-data regime.

##  I Introduction

Indoor temperature forecasting predicts future temperature values in the
different rooms of a building, leveraging historical data and environmental
factors for proactive Heating, Ventilation and Air Conditioning (HVAC) system
management and comfort optimization. The European Union emphasizes the
importance of efficient building energy management systems to achieve
sustainability goals, given that buildings contribute to 40% of energy
consumption and 36% of CO2 emissions in the EU [14]. HVAC systems, responsible
for the majority of energy consumption in buildings, significantly influence
both household comfort and environmental impact.

Typically, heating and cooling systems in buildings are controlled by a
schedule. This type of regulation does not take into account factors such as
outdoor weather, solar radiation, and changes in occupancy, and therefore can
lead to excessively heating (or cooling) of a room thus creating discomfort
for the occupants. A solution that, instead, makes use of the predicted room
temperature as an input, can lead to a better comfort for the occupants while
achieving consistent savings in energy use.

Machine learning models have demonstrated superiority over traditional
physics-based methods in indoor temperature forecasting [15, 2, 3]. Currently,
Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM)
networks, remain a practical choice for such forecasting tasks [5, 1, 6]. Our
approach differs from those aforementioned as we do not focus on finding the
best forecaster. Instead, we seek to enhance forecasting as a whole, in
particular in low-data environments.

Synthetic data generation is a rapidly growing field [13, 17, 8, 18, 15, 22]
within the realm of data augmentation, predominantly relying on variations of
Generative Adversarial Networks (GANs). Its applications span across diverse
domains, from medicine to maintenance tasks [9, 7, 21, 10]. However, the
impact of synthetic data augmentation on temperature forecasting in low-data
environments remains relatively unexplored. This study endeavors to address
this gap by augmenting forecasters with synthetic data and evaluating their
performance in subsequent tasks. Through this investigation, we aim to uncover
the key effects and potential benefits of synthetic data augmentation in
enhancing temperature forecasting accuracy amidst data scarcity.

Roadmap: We begin by reviewing the state-of-the-art and existing synthesizers.
Next, we look into our methodology, including the fusion of synthetic and real
samples and addressing class imbalance. Finally, we present our experimental
results.

##  II State of the Art

The literature surveys, as evidenced by [8], [13], and [17], commonly
delineate modern approaches to time series data augmentation into three broad
categories: traditional methods, GAN-based techniques, and Auto-Encoder-based
techniques. Traditional methods such as homogeneous scaling and rotation are
noted for their cost-effectiveness and simplicity. However, they often disrupt
temporal relationships within the time series. Consequently, contemporary
approaches lean towards generative models to better preserve temporal
dynamics.

For instance, [8] discusses 23 methods, including 9 Variational Auto-Encoder
based (VAE) and 14 GAN-based techniques. Despite the widespread use of GANs,
they are prone to convergence issues, particularly in low-data contexts,
although they yield more diverse data. The over representation of GANs in the
generic synthetic generation literature is furthermore confirmed by the
reports from [13].

Synthetic data augmentation has demonstrated its effectiveness in enhancing
forecasting performance for time series data, spanning various domains. For
example, [18] showcases successful synthetic data generation in the context of
renewable power plant energy forecasting, leveraging physical models and
generic weather prediction systems.

In addition, Machine Learning approaches have been explored extensively in
this domain. Notably, [22] successfully predicts emerging technologies within
a year by augmenting synthetic patent data using GANs. Then, [16] proposes a
traditional ML method, K-means, for synthetic data generation, albeit with
mixed results in forecasting methods involving deep learning architectures
like LSTMs.

In summary, the literature offers a variety of methods for time series data
augmentation, ranging from traditional to advanced machine learning
techniques. Given the diverse landscape of options, we opt to focus on deep
learning methods for data synthesis in our work. Specifically, we choose to
study GANs for their ability to generate diverse data and VAEs for their ease
of training, especially in low-data scenarios. We highlight three notable
models from the literature [20, 12, 11], and detail their mechanisms in the
subsequent section.

##  III Background theory

###  III-A TimeGAN

Initially proposed in [20], it represents a pioneering effort in exploring the
capabilities of generative adversarial network architectures for time series
data. It incorporates serveral strategies to enhance efficiency: firstly, it
employs both an adversarial loss and a supervised loss, combining the control
provided by supervised learning with the flexibility inherent in unsupervised
GAN models. Additionally, TimeGAN utilizes a dimension reduction technique
involving embedding and recovery networks. These networks map features to
latent representations, effectively reducing the dimensionality of the time
series data. This approach capitalizes on the fact that the temporal dynamics
of time series data can often be captured in a lower-dimensional space
relative to the length of the series, thereby simplifying the tasks performed
by the GAN. Furthermore, the generator and discriminator operate within the
latent space. TimeGAN adopts a joint training approach for the embedding and
the generative network. This strategy facilitates the learning of temporal
relationships by the generator.

###  III-B DoppleGANger

Developed as a versatile network-time series synthesizer, DoppleGANger was
designed to address fidelity problems between measurements and their
associated data, to better capture long-term correlations within time series
data, and to mitigate issues such as mode collapse in generative models.

To address fidelity concerns, DoppleGANger introduces an auxiliary
discriminator dedicated to metadata generation ; although this aspect is not
utilized in the context of this article. To mitigate mode collapse, it
implements a strategy that constrains generation to randomized min-max values
at each iteration, which are later scaled back to realistic ranges. This
technique ensures diversity in generated samples and effectively combats mode
collapse. Moreover, DoppleGANger modifies the canonical GAN framework by
integrating LSTM cells to better capture temporal dependencies. To mitigate
memory loss associated with RNN cells, DoppleGANger introduces the concept of
batched generation, enabling the simultaneous generation of multiple records
at each cell pass instead of the traditional single-step generation approach.
This enhancement significantly improves the efficiency and memorisation
effectiveness of the generative process.

###  III-C TimeVQVAE

TimeVQVAE pioneers the application of Vector Quantization techniques to tackle
the time series generation challenge, introducing several novel features:
firstly, they employ Vector Quantization of the latent space with VQVAE [11],
a type of variational Auto-Encoder that leverages vector quantization to
discretize the latent space while learning the prior distribution. This
approach ensures that VQ-VAE avoids posterior collapse by learning a quantized
latent space instead of constraining it, for example, by a Gaussian
distribution.

Moreover, TimeVQVAE adopts a modified MaskGIT [4] prior learning process for
the sampling phase, which is asserted to not only accelerate the process but
also enhance the quality and diversity of generated samples.

Additionally, TimeVQVAE operates on a modified space: initially, time series
data is shifted to a time-frequency space using Discrete Fourier Transform.
Subsequently, separate sets of VQ-VAEs are trained for both low-frequency and
high-frequency generation tasks, not only facilitating the learning process,
but also ensuring the preservation of key features in both components.

##  IV Methods

###  IV-A Methods for data acquisition and processing

####  IV-A1 Description of the dataset

The data in question has been acquired over a dedicated test facility111Link
to the laboratory used for acquisition: https://www.sintef.no/en/all-
laboratories/zeb-test-cell-laboratory/ and is stored under a tabular time
series format of size ($N=59,040$, $D=81$), acquired at a rate of $1\text{
min}^{-1}$ and following the principles detailed in the section below.

Regarding the dimensions, we define a series as a vector of shape $(240,D)$,
constituting a sequence of 240 consecutive rows from the dataset, starting
from the beginning. In essence, each series is D-dimensional, encapsulates
four hours of data acquisition, and it is uniquely characterized by a phase
and a step. See Figure 1 for visual examples of a series.

The dataset is acquired over four distinct phases, each yielding a subset
denoted as RICO<X>, where X represents the acquisition number.

  * •

RICO1: collected between July and August 2023 spanning over 17 days,
encompassing 102 series, equivalent to 24,480 rows. RICO1 exhibits some
inconsistencies due to sub-optimal tuning of the acquisition facility.
Although flawed, most data points exhibit "normal" behaviour.

  * •

RICO2: collected in October 2023 spanning over 10 days, encompassing 60
series, equivalent to 14,400 rows. In this acquisition, we introduced one hour
of downtime, or "free fall", at the end of each four-hour point recording: in
essence, the constraints are stopped and ’natural’ heat exchanges are the only
ones that remain.

  * •

RICO3: collected in January 2024, spanning over 4 days, encompassing 24
series, equivalent to 5,760 rows. RICO3 was acquired with sequences of sixteen
hours of constraint for each series instead of four, with its last four hours
left as free fall. To ensure compatibility with other datasets, only the first
four hours of each series will be utilized, for a total of 6 useful series.

  * •

RICO4: collected in February 2024, spanning over 10 days, encompassing 60
series, equivalent to 14,400 rows. This acquisition followed a similar
protocol to RICO1, featuring fixed actuators tuning and no free fall. As of
today, RICO4 stands as the phase that produced the highest quality samples.

The features of the dataset can be divided into five categories:

  * •

Identifiers: These include Phase, Step, and Flag. Phase and Step serve to
uniquely identify a point, while Flag enables the flagging of points we opt
not to utilize.

  * •

Setpoints: These variables (EC3, SB43, B46 and SB47) represent the setpoints
of the four HVAC systems within the test cell (two heaters and two coolers)
that we can control. They are randomly adjusted every four hours to introduce
diversity into the dataset.

  * •

Features of Interest: These entail variables like internal air temperature,
which constitute the focal points of our predictive experiments.

  * •

Environmental Variables: This category encompasses external temperatures, wind
direction, sun radiation, dew point, and other weather related metrics.

  * •

Control Features: Examples include JP40_head, and pid.EC3.enabled. They
primarily serve to verify data integrity but hold marginal relevance in
machine learning contexts.

In each phase, the acquisition process unfolds as follows: a random
combination of set points is generated based on predefined permissible values.
These values reflect the temperatures reached by the HVAC actuators themselves
and not those reached by the room temperature. For example, for heaters, these
values may include [off, 20°C, 40°C, 60°C]. Every four hours, this combination
of set points is dispatched to various system actuators. Each unique
combination generates a single series.

####  IV-A2 Feeding to the models

All acquired series are utilized for our analysis; however, certain points are
manually excluded based on specific criteria:

  * •

Series from RICO1 exhibiting unexpected behaviors are identified and excluded
from the analysis, totaling 19 series.

  * •

Series from RICO2, acquired over a duration of (3h constrained + 1h free
fall), are excluded as they differed from the intended format of 4h
constrained.

  * •

A subset of series from RICO3 (only 6 out of the total 24 series) is utilized,
while others are excluded from the analysis.

  * •

Series from RICO4 with missing values are identified and excluded from the
analysis.

Each series earmarked for exclusion from our analysis is labeled with a tag
point, denoted as 1 for inclusion and 0 for exclusion.

Regarding transformations, we first apply standard scaling to the data.
Subsequently, the data is restructured from shape (N,D) to (N,L,C), where L
represents the sequence length (240 here), and C represents the channels. In
our case, C equals 1, focusing solely on one dimension (B.RTD3), which
represents the temperature at the center of the room.

Ultimately, the dataset is partitioned into distinct training and testing
sets. A fraction of 0.2 of the series from each phase is reserved for the test
set, with the remaining series allocated to the training set. Consequently,
the training set train_real comprises 116 series, while the testing set
test_real contains 31 series.

####  IV-A3 Data labeling methodology

To label our dataset systematically, we follow these steps:

  1. 1.

Subset Selection: We focus on the initial 3 hours of each series to capture
relevant data, excluding the last hour, which often represents a stable regime
distinct from the rest of the data.

  2. 2.

Smoothing: Applying a 5-large moving average with edge-repetition padding to
smooth the data, reducing noise.

  3. 3.

Derivative Calculation: Local derivatives are computed to capture trends in
value changes.

  4. 4.

Label Classes:

     * •

Monotonic Positive (0): Showcases consistent increase in trends.

     * •

Monotonic Negative (1): Showcases consistent decrease in trend.

     * •

Non-Monotonic (2): Exhibits fluctuations or irregularities in trend.

These classes enable our synthesizers to effectively capture diverse trends
present in the data.

Figure 1: Visual observations from the dataset: four dimensions from eight
random series.

###  IV-B Comparing the synthetisers

####  IV-B1 Methods for evaluating the synthesizers

In this study, we evaluate the performance of TimeVQVAE under various
scenarios. We compare its results and synthetic capabilities with two other
synthesizers: TimeGAN, serving as a baseline despite expected limitations, and
the DoppleGANger model.

Our code base builds upon the implementations from [11] for TimeVQVAE, the
gretel-ai python library for DoppleGANger, and J.Yoon’s official
implementation of TimeGAN [20].

We initially train on 116 series extracted from all RICO phases as detailed in
IV-A2. Training specifics for each model are as follows:

  * •

TimeGAN: Various settings were experimented with, but no satisfactory results
were achieved.

  * •

DoppleGANger: Sequence length of 240, batch size 8, and 1000 epochs.

  * •

TimeVQVAE: After manual hyper-parameter tuning, the base parameters proved
optimal: 2000 epochs for training the VQVAE and 10000 epochs for prior
learning.

In evaluating the synthesizers, we employ a mix of traditional metrics
alongside a utility metric which, in downstream tasks inspired by [12] and
[20]. The traditional algorithms include:

  * •

t-SNE (t-stochastic Neighborhood Embedding): is a technique for dimensionality
reduction, particularly effective for visualizing high-dimensional datasets in
lower-dimensional spaces. It captures the local structure of the data,
offering insights into its dependencies and feature relationships.

  * •

PCA (Principal Component Analysis): identifies the principal components of a
dataset, reducing its dimension while retaining as much variance as possible.
It aids in understanding the underlying structure and dominant patterns within
the data.

In addition to these traditional metrics, we will also rely on visual
observation of the samples to complement the quantitative analyses. This
holistic approach ensures a comprehensive evaluation of the synthesizers’
performance.

Lastly, our incorporation of the utility metric, provides an alternative
perspective on the synthesizers’ performance, highlighting their practical
utility in forecasting tasks. The next chapter dives into the specifics of
this metric and its implications for our study.

###  IV-C Forecasting Utility

####  IV-C1 Utility metric

This utility metric enables us to gauge the effectiveness of our synthesized
samples in real-world forecasting tasks. Specifically, we conduct controlled
experiments where a simple forecasting model is trained on a baseline dataset.
Subsequently, we reduce or augment the dataset with synthetic samples and
analyze the outcomes of these experiments.

In our study, we employ a straightforward one-layer LSTM model followed by a
fully connected layer as our forecasting model. The hyper parameters of this
model are tuned manually using a training set identical to the remainder of
our experiments. Predictions are made for the subsequent thirty minutes, with
the data being sub-sampled by a factor of 10 before being fed into the model.

####  IV-C2 Experiment: General data augmentation

In our first experiment, we start by training an instance of the chosen
synthesizer on train_real. From the synthesizer, we sample 256 points denoted
synth from the trained synthesizer. We utilise synth to construct training
sets for the three following strategies:

  * •

TRTR: or "Train Real, Test Real" where the train set consists of train_real
and the test set consists in test_real. This corresponds to a normal control
experiment.

  * •

TSTR: or "Train Synthetic, Test Real" where the train set consists of synth
and the test set consists in test_real.

  * •

TRSTR: or "Train Real and Synthetic, Test Real" where the train set consists
in both train_real and synth, and the test set consists in test_real.

We will employ a TRTR strategy as the baseline approach to establish a
reference point for performance evaluation. Additionally, we will explore the
alternative strategies TRSTR and TSTR, which involve the integration of
synthetic data. By comparing the performance across these strategies, we aim
to assess the effectiveness of incorporating synthetic data in our forecasting
models.

####  IV-C3 Experiment: Class imbalance

In our second experiment, we tackle the question of class imbalance. Class
imbalance is a typical Machine Learning problem where if one class in under-
represented within the training set, the model’s performance can be hindered
in deployment for points belonging to that class. To see if synthetic data
points can solve this issue, we experiment by artificially under-sampling a
class from the training set and after, leveraging the conditional generation
capabilities of our synthesizer to over-sample from this class, thus restoring
balance in the forecaster training set.

We construct imbalanced training sets as a subset of our main training set:
let $i$ be the class index, $n_{i}^{init}$ represent the number of samples
initially within that class, and $n_{i}^{ablated}$ denote the number of
samples remaining after ablation. We define the ablation ratio $r$ as:

$r=\frac{n_{i}^{ablated}}{n_{i}^{init}}$

We then create a new training set denoted $Set_{i,r}$ where we remove
$n_{missing}=n_{i}^{init}-n_{i}^{ablated}$ samples from class $i$. Initially,
we train a series of synthesizers on the $Set_{0,r}\text{ where }r\in
R=\\{0.25,0.5,0.75,1.0\\}$. This process is repeated for all three classes,
resulting in the training of 12 synthesizers, each tailored to a specific type
of training data, and denoted $\Sigma_{i,r}$.

Subsequently, for each of these 12 scenarios, we conduct the following
training procedures:

  * •

We train a set of ’baseline’ LSTMs on $Set_{i,r}$

  * •

We train a set of ’test’ LSTMs on $Set_{i,r}$ to which are appended a unique
set of $n_{missing}$ points samples from $\Sigma_{i,r}$ for each training
instance.

Results are discussed in the Results section V.

###  IV-D Metrics used

Our experiments evaluation will be based on four different standard metrics
for time series forecasting [17]:

  * •

MSE is a standard metric for measuring losses in continuous regression
problems. It measures the average of the squared differences between
forecasted and actual values, emphasizing large errors due to squaring terms.
While it provides insight into the average squared deviation, MSE is sensitive
to outliers and offers limited interpretability.

$MSE=\frac{1}{n}\sum_{t=1}^{n}(y_{t}-\hat{y}_{t})^{2}$ (1)

  * •

MAE computes the average of the absolute differences between forecasted and
actual values. It is simple, easy to interpret, has symmetric penalisation and
is robust to outliers. However, MAE does not emphasize large errors as much as
MSE does.

$MAE=\frac{1}{n}\sum_{t=1}^{n}|y_{t}-\hat{y}_{t}|$ (2)

  * •

MAPE calculates the average of the absolute percentage errors between actual
and forecasted values. It offers easy interpretation in percentage terms and
reflects the relative error size. However, closer to 0 and due the the non
linearity of the inverse function, MAPE can be either undefined or easily
influenced by outliers. ($y_{t}$ close to zero and $\hat{y}_{t}\neq y-t$)

$MAPE=\frac{1}{n}\sum_{t=1}^{n}\left|\frac{y_{t}-\hat{y}_{t}}{y_{t}}\right|$
(3)

  * •

MASE provides a standardized measure of forecast accuracy by comparing a
model’s performance to that of a naïve forecast. Its calculation involves the
mean of the absolute errors divided by the mean absolute error of a naïve
forecast. MASE is scale independent, symmetric with respect to over and under
predictions, but depends on a naive forecaster and is thus different for every
dataset encountered.

In most cases, $n$ is chosen as the seasonality. However, due to the absence
of seasonality in our data, we opted to use n as the forecasting output window
length.

$MASE=\frac{1}{n}\sum_{t=1}^{n}\frac{|y_{t}-\hat{y}_{t}|}{\frac{1}{n-1}\sum_{t=2}^{n}|y_{t}-y_{t-1}|}$
(4)

##  V Results and Discussion

###  V-A Synthetiser performance

The training of our forecasters show the following results: Upon visual
inspection of the generated samples in Figure 2, it appears clearly, and
despite our best efforts, that TimeGAN (last row) was not able to converge and
is thus disconsidered from the downstream tasks. Concerning the DoppleGANger
model (first row), it demonstrates an ability to capture the general trend
from the training set, but introduces a high frequency parasite component,
which resembles samples with high noise to signal ratio (see for example,
samples 201 and 254). Lastly, TimeVQVAE appears to show some more diversity
than its counterpart, and is free of the noisy artifacts present in
DoppleGANger.

Figure 2: Example sample series from the trained synthetizers. From top to
bottom: DoppleGANger, TimeVQVAE, TimeGAN.

In Figure 3, which presents the PCA results from all synthesizers, one can
observe the diversity of some unconditionally generated samples. Disregarding
results from TimeGAN, the following observations emerge:

  * •

DoppleGANger (left) is capable of generating plausible samples; however, it
seems to struggle with generalization, as it only generates from limited
regions of the underlying data distribution.

  * •

Conversely, TimeVQVAE (center) not only produces plausible samples but also
appears to cover the entirety of the data space in the two-dimensional PCA
mapping.

Figure 3: PCA Analysis (first two components) of the generated samples from
all synthesizers, compared to the training samples.

These observations are furthermore confirmed upon observation of the t-SNE of
the generated samples shown in Figure 4. While none of the studied models
exhibit perfect overlap between image and data space, we observe that
DoppleGANger exhibits correlations groups scattered apart from each other
nonexistent in the real data space, and TimeVQVAE, though there is no
scattering, appears to extend beyond the boundaries set by the original data
t-SNE. It is however challenging to analyse the distances in t-SNE since they
do not necessarily reflect actual distances between points [19].

Figure 4: t-SNE Anslysis (first two components) of the generated samples from
all synthesizers, compared to the training samples.

###  V-B Experimental study 1: synthetic augmentation

We conduct an analysis comparing the effectiveness of the three strategies
outlined in Section IV-C2. Initially, we train 100 forecasters using a TRTR
strategy (IV-C), on train_real. Subsequently, we train:

  * •

100 forecasters using a TSTR strategy, where each forecaster is given a unique
set of synthetic series generated by the synthesizer

  * •

100 forecasters using TRSTS strategy, where each forecaster is also given a
new set of series.

Figures 5 and 6 present histograms of the MAE test losses (lower is better)
where we observe the following: overall performance improves on average with
increased training data samples. These are the cases where synthetically
generated samples have been introduced in the training process. Specifically,
a significant performance improvement is observed with the use of only
synthetic samples for training (TSTR scenario), and further improved when
combining both synthetic and real samples (TRSTR) for training.

TABLE I: Experiment 1 results - aggregated means Type | test_mse | test_mase | test_mae | test_mape
---|---|---|---|---
trtr | 0.003119 | 2.390434 | 0.037563 | 0.251342
tstr | 0.001791 | 2.027180 | 0.030242 | 0.199215
trstr | 0.001714 | 1.854287 | 0.028266 | 0.162576
TABLE II: Experiment 1 results - aggregated standard deviations Type | test_mse | test_mase | test_mae | test_mape
---|---|---|---|---
trtr | 0.000579 | 0.290714 | 0.003708 | 0.012571
tstr | 0.000570 | 0.346175 | 0.005047 | 0.044246
trstr | 0.000756 | 0.392574 | 0.005974 | 0.032716

We observe the following key points:

  * •

Table I demonstrates an overall performance improvement with increased data
volumes. Significant enhancement is observed with the addition of synthetic
samples (TSTR scenario), further augmented when combining both sythetic and
real samples (TRSTR).

  * •

In Table II, the variance remains consistent across all strategies, with a
general trend favouring the baseline. We believe that this increase in
variance is due to the inherent variability introduced by the inclusion of
newly generated samples in the TRSTR and TSTR strategies.

The behaviour observed on the MAE test losses is consistent across all four
metrics employed.

Figure 5: Histogram: MAE loss - forecast over 100 iterations for different
strategies, excluding the outmost 5% outliers. Figure 6: Histogram: MASE loss
- forecast over 100 iterations for different strategies, excluding the outmost
5% outliers.

###  V-C Experimental study 2: Class balancing

In this experiment, we conduct another analysis to evaluate the effectiveness
of augmenting imbalanced datasets with synthetic samples for prediction tasks.
Figure 7 illustrates one such scenario, where we selectively remove 75% of the
samples from class 0.

Tables III and IV present aggregated results for the mean and variance,
respectively, across 100 runs of the test metrics. Interestingly, our analysis
reveals no significant discernible improvement or deterioration in
performance, as evidenced by the overlapping likelihood distributions of both
the baseline and the augmented scenarios. However, regarding variance, our
observations vary depending on the metric utilized and the ratio, with
fluctuations ranging from a minor 0.14% decrease to a more substantial 55%
increase. We hypothesise that such an increase might be caused by the nature
of the datasets which remain strictly identical in all the baseline runs but
is unique to each test run.

Despite these insights, our experiments alone do not offer conclusive
explanations for the observed behavior. Further investigation is necessary to
fully understand the underlying mechanisms driving these findings.

Figure 7: Histogram: Analysis of imbalanced vs augmented forecasting - Ablation ratio of .25 - Results for 100 runs. Figure 8: Histogram: Analysis of imbalanced vs augmented forecasting - Ablation ratio of .75 - Results for 100 runs. TABLE III: Imbalancing experiment on class 0 results - aggregated means Type | Ratio | 0.25 | 0.50 | 0.75 | 1.00
---|---|---|---|---|---
Baseline | test_mae | 0.04665 | 0.04671 | 0.03457 | 0.03140
Baseline | test_mape | 0.31032 | 0.29151 | 0.23888 | 0.25104
Baseline | test_mase | 3.17415 | 3.18327 | 2.20503 | 2.10880
Baseline | test_mse | 0.00425 | 0.00418 | 0.00277 | 0.00217
Test run | test_mae | 0.04677 | 0.04547 | 0.03447 | 0.03243
Test run | test_mape | 0.31002 | 0.28877 | 0.23835 | 0.25472
Test run | test_mase | 3.17502 | 3.07901 | 2.19738 | 2.18728
Test run | test_mse | 0.00438 | 0.00399 | 0.00277 | 0.00229
TABLE IV: Imbalancing experiment on class 0 results - aggreated standard deviations Type | Ratio | 0.25 | 0.50 | 0.75 | 1.00
---|---|---|---|---|---
Baseline | test_mae | 0.01074 | 0.00918 | 0.00463 | 0.00535
Baseline | test_mape | 0.03297 | 0.03034 | 0.02228 | 0.03348
Baseline | test_mase | 0.86050 | 0.75173 | 0.36545 | 0.42427
Baseline | test_mse | 0.00181 | 0.00137 | 0.00047 | 0.00052
Test run | test_mae | 0.01213 | 0.00857 | 0.00431 | 0.00520
Test run | test_mape | 0.03597 | 0.02768 | 0.02324 | 0.03209
Test run | test_mase | 0.94686 | 0.70215 | 0.33682 | 0.42085
Test run | test_mse | 0.00226 | 0.00126 | 0.00044 | 0.00056

##  VI Conclusions and Further Work

Our experiments highlight the superior performance of a VQVAE-based model
compared to some state-of-the-art GAN models for synthesising uni-variate time
series in low data environments. Additionally, we discover that in relatively
simple datasets, augmenting the dataset with synthetic samples can lead to
enhanced forecasting accuracy in subsequent tasks particularly in cases of
data scarcity, albeit the expanse of training variance. This latter problem
requires further investigation as explained in Section V-C. We also find out
that using synthesizers to balance out class distribution neither particularly
increases or decreases overall performance. However, it’s worth noting the
imbalance present in our testing set, which might affect the results. Further
investigation and experiments are required on the test set to better
understand this behaviour. Notably, we deem necessary to run similar
experiments on time series datasets from other domains to ensure validity of
our conclusions.

## Acknowledgement

This research was carried out with the support of the ML4ITS project (312062)
and RICO project (329730), funded by the Norwegian Research Council (NFR). We
also thank Odne Andreas Oksavik for his technical support in data collection
of the RICO dataset.

## References

  * [1] Sadi Alawadi, David Mera, Manuel Fernández-Delgado, Fahed Alkhabbas, Carl Magnus Olsson, and Paul Davidsson.  A comparison of machine learning algorithms for forecasting indoor temperature in smart buildings.  Energy Systems, 13(3):689–705, 2022.
  * [2] Nivine Attoue, Isam Shahrour, and Rafic Younes.  Smart building: Use of the artificial neural network approach for indoor temperature forecasting.  Energies, 11(2), 2018.
  * [3] Andrea Bellagarda, Silvia Cesari, Alessandro Aliberti, Francesca Ugliotti, Lorenzo Bottaccioli, Enrico Macii, and Edoardo Patti.  Effectiveness of neural networks and transfer learning for indoor air-temperature forecasting.  Automation in Construction, 140:104314, 2022.
  * [4] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman.  Maskgit: Masked generative image transformer.  Google Research, 2022.  arXiv preprint arXiv:xxxx.xxxxx.
  * [5] Ali Deihimi and Hemen Showkati.  Application of echo state networks in short-term electric load forecasting.  Energy, 39(1):327–340, 2012.  Sustainable Energy and Environmental Protection 2010.
  * [6] Zhen Fang, Nicolas Crimier, Lisa Scanu, Alphanie Midelet, Amr Alyafi, and Benoit Delinchant.  Multi-zone indoor temperature prediction with lstm-based sequence to sequence model.  Energy and Buildings, 245:111053, 2021.
  * [7] Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan.  Synthetic data augmentation using gan for improved liver lesion classification.  In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 289–293, 2018.
  * [8] Guillermo Iglesias, Edgar Talavera, Ángel González-Prieto, Alberto Mozo, and Sandra Gómez-Canaval.  Data augmentation techniques in time series domain: a survey and taxonomy.  Neural Computing and Applications, 35(14):10123–10145, March 2023.
  * [9] Asif Khan, Hyunho Hwang, and Heung Soo Kim.  Synthetic data augmentation and deep learning for the fault diagnosis of rotating machines.  Mathematics, 9(18):2336, 2021.
  * [10] Jun-Hyung Kim and Youngbae Hwang.  Gan-based synthetic data augmentation for infrared small target detection.  IEEE Transactions on Geoscience and Remote Sensing, 60:1–12, 2022.
  * [11] Daesoo Lee, Sara Malacarne, and Erlend Aune.  Vector quantized time series generation with a bidirectional prior model, 2023.
  * [12] Zinan Lin, Alankar Jain, Chen Wang, Giulia Fanti, and Vyas Sekar.  Generating high-fidelity, synthetic time series datasets with doppelganger.  CoRR, abs/1909.13403, 2019.
  * [13] Alhassan Mumuni, Fuseini Mumuni, and Nana Kobina Gerrar.  A survey of synthetic data augmentation methods in machine vision.  Machine Intelligence Research, TY(JOUR), 2024.
  * [14] European Parliament.  Directive 2010/31/eu of the european parliament and of the council of 19 may 2010 on the energy performance of buildings.  Off. J. Eur. Union, 2010(3):124–146, 2010.  [Google Scholar].
  * [15] Lara Ramadan, Isam Shahrour, Hussein Mroueh, and Fadi Hage Chehade.  Use of machine learning methods for indoor temperature forecasting.  Future Internet, 13(10), 2021.
  * [16] Tiantian Tang, Donglai Jiao, Tao Chen, and Guan Gui.  Medium- and long-term precipitation forecasting method based on data augmentation and machine learning algorithms.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15:1000–1011, 2022.
  * [17] José F. Torres, Dalil Hadjout, Abderrazak Sebaa, Francisco Martínez-Álvarez, and Alicia Troncoso.  Deep learning for time series forecasting: A survey.  Big Data, 9(1):3–21, February 2021.  Published online ahead of print: December 3, 2020.
  * [18] Stephan Vogt, Jens Schreiber, and Bernhard Sick.  Synthetic photovoltaic and wind power forecasting data, 2022.
  * [19] Martin Wattenberg, Fernanda Viégas, and Ian Johnson.  How to use t-sne effectively.  Distill, 2016.
  * [20] Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar.  Time-series generative adversarial networks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [21] Guanghao Zhai, Yasutaka Narazaki, Shuo Wang, Shaik Althaf V S, and Billie Jr.  Synthetic data augmentation for pixel-wise steel fatigue crack identification using fully convolutional networks.  SMART STRUCTURES AND SYSTEMS, 29:237–250, 01 2022.
  * [22] Yuan Zhou, Fang Dong, Yufei Liu, Zhaofu Li, JunFei Du, and Li Zhang.  Forecasting emerging technologies using data augmentation and deep learning.  Scientometrics, 123(1):1–29, 2020.