# Scalar Invariant Networks with Zero Bias

Chuqin Geng, Xiaojie Xu, Haolin Ye, Xujie Si
McGill University
{chuqin.geng, xiaojie.xu<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Just like weights, bias terms are the learnable parameters of many popular
machine learning models, including neural networks. Biases are believed to
effectively increase the representational power of neural networks to solve a
wide range of tasks in computer vision. However, we argue that if we consider
the intrinsic distribution of images in the input space as well as some
desired properties a model should have from the first principles, biases can
be completely ignored in addressing many image-related tasks, such as image
classification. Our observation indicates that zero-bias neural networks could
perform comparably to neural networks with bias at least on practical image
classification tasks. In addition, we prove that zero-bias neural networks
possess a nice property called scalar (multiplication) invariance, which has
great potential in learning and understanding images captured under poor
illumination conditions. We then extend scalar invariance to more general
cases that allow us to formally verify certain convex regions of the input
space. Our experimental results show that zero-bias models could outperform
the state-of-art models by a very large margin (over 60%) when predicting
images under a low illumination condition (multiplying a scalar of 0.01);
while achieving the same-level performance as normal models.

##  1 Introduction

Using bias terms in neural networks is a common practice. Its theoretical
foundation goes back to the invention of artificial neural networks, which are
loosely inspired by biological neurons. Biological neurons have some
thresholds to determine whether they should ”fire” (produce an output that
goes to other neurons)[23, 45, 15]. These thresholds are essentially the same
thing as bias terms. From the representation learning perspective, the bias
term is widely believed to increase the representational power of neural
networks and thus is always needed when designing neural networks to solve a
broad array of tasks in computer vision [43, 33, 2].

In this work, we challenge the commonly-held beliefs of the necessity of
including bias terms in neural networks to solve computer vision tasks. Our
geometric observations suggest the intrinsic distribution of images should
incorporate both _locality_ and _directionality_. With these two properties
holding, bias terms should not affect models’ representational power and
performance, even for large modern CNN models such as ResNets [16]. Our
thorough experimental results also support this argument.

Figure 1: Scalar invariant networks (without bias) and their counterparts with
bias share similar accuracies on CIFAR-100 when the multiplying scalar is 1,
i.e., the original images. As the scalar diminishes, the accuracies of normal
models drop quickly whereas that of models without bias achieve invariance.

In addition, we show that neural networks will possess an intriguing property
- scalar (multiplication) invariance after dropping bias terms. We then extend
scalar invariance to CNNs as well as ResNets. While removing biases may cause
gradient vanishing/exploding which hinders models’ learning, we mitigate this
issue by leveraging recent advances in normalization-free methods including
Fixup[47] and NF-ResNets [5, 6]. This property will allow us to make
robustness predictions on low-illuminated images without any data pre-
processing and augmentation techniques, which normal neural networks (with
biases) usually fail to do so, as illustrated in Figure 1. Based on the scalar
invariance property, we further derive more general robustness guarantees that
could verify even certain convex regions of input space. In contrast, such
guarantees hardly exist on normal neural networks due to their highly
combinatorial nature.

We summarize our contributions as follows: (1) We show basic building blocks
of neural networks are scalar multiplication associative if ignoring bias,
which in turn, assures the scalar invariant property of convolutional neural
networks. By adapting batch normalization-free methods, we can extend scalar
invariance to ResNets; (2) Derived from the scalar invariant property, we
propose two more robustness guarantees that can verify inputs on certain lines
and convex regions of the input space; (3) Our geometric observations suggest
the intrinsic distribution of images should incorporate both _locality_ and
_directionality_. Under these two properties, scalar invariant neural networks
111We use terms _scalar invariant, zero-bias, without bias_ interchangeably to
describe the same variant of neural network. should have the same
representational power as normal neural networks, thus delivering comparable
performances; (4) Our experiment suggests scalar invariant neural networks
could outperform normal models including state-of-the-art models by over 60%
on predicting images 100 times darker. In addition, we show that scalar
invariant networks share the same bias as humans when predicting the zero
image, i.e., an image with all pixel values being zero. We also empirically
validate the robustness merit of scalar invariant networks using visual
examples.

##  2 Related Work

###  2.1 Invariance in Neural Networks

Studying invariance in machine learning as well as neural networks has
attracted much attention as real-world data such as images often exhibit rich
invariant structures. Incorporating such invariance properties as prior
knowledge (inductive bias) could expand the expressive power of the network
without much increase in the number of parameters, which usually leads to
better performance. For instance, Convolutional Neural Networks have a
stronger geometric prior - translation invariance [7, 4]. In addition, Group
equivariant Convolutional Neural Networks (G-CNNs) adapt group convolution
layers to achieve great results on images generated by translations,
reflections, and rotations [9]. Similar work also focuses on studying the
invariance of neural network’s outputs under group actions on its inputs [27,
31, 3].

Given the scale invariant nature of images, there is also a line of work
studies how to improve the consistency of models’ prediction on varying scale
images [44, 12, 34, 48]. However, the most related invariance to our work is
illumination invariance which has great impact on many real-world
applications. For example, Ramaiah et al. uses convolutional neural networks
for face recognition under non-uniform illumination [35]. Maddern et al.
studies illumination invariant transform to improve visual localization,
mapping, and scene classification for autonomous road vehicles [32]. Huang et
al. leverages Retinex Decomposition Net and bottom-up attention to approach
person re-identification [21]. Despite absolute invariance being considered
hard to achieve and most works usually failing to guarantee it, our work shows
that absolute invariance under scalar multiplication can be achieved with
zero-bias neural networks.

###  2.2 Zero-bias neural networks

Although zero-bias neural networks do not appear as much as normal neural
networks in the machine-learning literature due to potential reductions in
models’ expressive capability, they have been used in some real-world
applications such as facial expression recognition[26], abnormal event
detection in IoT[29], identification of Internet-of-Things devices[30], RF
signal surveillance[28], and anomaly data detection[46]. There are several
reasons for choosing zero-bias neural networks over normal neural networks:
(1) Their incremental learning fashion and better decision fairness; (2)
Better interpretability without losing accuracy, which challenges the common
first impression of the weaker expressive capability of zero-bias models; (3)
More reliable and robust performance. Although these works achieve some
success with zero-bias neural networks, none of them dive deeper to analyze
these advantages formally. Our work explores zero-bias from an invariant
perspective for the first time, to our best knowledge, identifying scalar
multiplication invariance in zero-bias models, proving some rigorous robust
guarantees, and explaining their comparable accuracy based on geometric sights
of image distribution.

##  3 Scalar invariant neural networks

###  3.1 Preliminary

A neural network consists of an input layer, hidden layers, and an output
layer. For convolutional neural networks, some of the hidden layers are called
convolution layers which perform convolution operations on their input tensors
with convolution kernels. The outputted tensors are passed to an activation
function, commonly ReLU, before downsampling through pooling layers. After
that, the input tensor is flattened out so that a fully connected network can
process it and calculate the final prediction. For classification tasks, the
final prediction is represented by a probability distribution over all classes
using some activation functions such as Softmax. To further investigate the
scalar invariant property, we formally denote the input tensor as $X$ and a
convolutional neural network as $\mathcal{N}$. Then $\mathcal{N}$ is composed
of convolutional layers $\mathcal{F}_{i}$, pooling layers $\mathcal{P}_{i}$,
and fully connected layers $\mathcal{L}_{j}$, where $i,j\in\mathbb{N}$. And we
denote the final activation function as $\mathcal{A}$ and ReLU as
$\mathcal{R}$. We think of layers and activation functions as transformations
on the input $X$, then the output of the network before the final activation
function $\mathcal{A}$ is represented by:

$\mathcal{O}(X)=\underbrace{\mathcal{L}_{j}\circ\mathcal{R}\circ...\circ\mathcal{L}_{1}}_{\text{j
linear
layers}}\circ\underbrace{\mathcal{P}_{i}\circ\mathcal{R}\circ\mathcal{F}_{i}...\circ\mathcal{P}_{1}\circ\mathcal{R}\circ\mathcal{F}_{1}}_{\text{i
convolutional layers}}\circ X$

And the final prediction class is determined by the one with the highest
probability over all classes $\mathcal{C}$, that is:

$\mathcal{N}(X)=\operatorname*{argmax}_{c\in\mathcal{C}}\mathcal{\\{A\circ
O}(X)\\}$

###  3.2 Scalar associative transformations

We consider the operation inside a convolution layer $\mathcal{F}$ with a
kernel $\mathcal{K}$, it is easy to show the associative property with scalar
multiplication hold for convolution operations. More formally, let $s$ be a
positive scalar s.t. $s\in\mathbb{R}^{+}$, then we have:

$\displaystyle\mathcal{F}\circ(sX)=$
$\displaystyle\sum_{m}\sum_{n}sX(i+m,j+n)\mathcal{K}(m,n)$ $\displaystyle=$
$\displaystyle s\sum_{m}\sum_{n}X(i+m,j+n)\mathcal{K}(m,n)=s(\mathcal{F}\circ
X)$

In addition, the above property also holds for pooling layers $\mathcal{P}$,
including max pooling and average pooling. Since both the max and average
operation should preserve the scalar multiplication. The same argument also
applies to the ReLU function. So we have:

$\mathcal{P}\circ(sX)=s(\mathcal{P}\circ X)$
$\mathcal{R}\circ(sX)=s(\mathcal{R}\circ X)$

Finally, passing the input $X$ to a fully connected layer $\mathcal{L}$ can be
thought of as applying a linear transformation ($\mathcal{W,B}$) on $X$. If we
set the bias term $\mathcal{B}$ to $\mathbf{0}$. We will have the scalar
associative property. That is:

$\mathcal{L}\circ(sX)=(sX)\mathcal{W}^{T}=sX\mathcal{W}^{T}=s(\mathcal{L}\circ
X)$

Note our proofs also use the commutative property which generally holds for
matrix and vector multiplication with a scalar. Put together, by setting
biases to zeros, we have the scalar (multiplication) associative property
holds for the output function, i.e., ($\mathcal{O}(sX)=s\mathcal{O}(X)$).

###  3.3 Scalar invariant convolutional neural networks

Now we consider how to calculate the final prediction of the network
$\mathcal{N}$. For classification tasks, the last activation function
$\mathcal{A}$ is usually Softmax. If we multiply the input $X$ with a scalar
$s$ ( $s\in\mathbb{R}^{+}$ ) and pass the product to Softmax, it is equivalent
to changing the temperature of the distribution. Note that the rank of
candidate classes remains the same despite the change in the shape of the
distribution. Or in other words, the predicted class by the network
$\mathcal{N}$ is scalar (multiplication) invariant:

$\operatorname*{argmax}_{c}\frac{e^{s\mathcal{O}(X)_{c}}}{\displaystyle\sum_{c\in\mathcal{C}}e^{s\mathcal{O}(X)_{c}}}=\operatorname*{argmax}_{c}\frac{e^{\mathcal{O}(X)_{c}}}{\displaystyle\sum_{c\in\mathcal{C}}e^{\mathcal{O}(X)_{c}}}$

Put together with the scalar associative property of the output function
$\mathcal{O}(\cdot)$, we have a scalar invariant neural network:

$\displaystyle\mathcal{N}(sX)=$
$\displaystyle\operatorname*{argmax}_{c}\mathcal{\\{A}\circ\mathcal{O}(sX)\\}$
$\displaystyle=$
$\displaystyle\operatorname*{argmax}_{c}\mathcal{\\{A}\circ\mathcal{O}(X)\\}=\mathcal{N}(X)$

The concept of scalar invariant neural networks generalizes beyond just
convolutional neural networks. In fact, as long as hidden layers perform
scalar associative (and commutative) transformations and the last activation
function preserves the highest probable candidate under scalar multiplication,
the neural network will be scalar invariant.

###  3.4 Scalar invariant ResNet

We briefly discussed the most simple architecture of convolutional neural
networks in the previous section. However, in addition to those basic layers
we mention before, modern powerful CNNs also employ extra layers and
techniques to address over-fitting and gradient exploding/vanishing issues.
For example, ResNet adopts _Dropout_ [38], _additive skip connections_ [16]
and _Batch Normalization_ [22] which contributes enormously to its success.
First, as dropout layers are disabled during the inference phase, it has no
impact on the scalar invariant property. Second, it is trivial to show skip
connection is also scalar multiplication associative if the corresponding
residual branch $\mathcal{G}$ is also scalar multiplication associative.

$sX+\mathcal{G}(sX)=s(X+\mathcal{G}(X))\text{ }\forall s\in\mathbb{R}^{+}$

Lastly, we consider batch normalization, which is performed through a
normalization transformation that fixes the means and variances of inputs to
each layer. Let us use $X_{\mathcal{B}}$ to denote a mini-batch of the entire
training set. Then we have the batch normalization transformation as follows:

$\mathcal{Y}=\gamma{{\hat{X}_{\mathcal{B}}}}+\beta$

where $\gamma$ and $\beta$ are learnable parameters, and
$\hat{X}_{\mathcal{B}}$ is the normalized input, represented by
${\hat{X}_{\mathcal{B}}}={\frac{X_{\mathcal{B}}-\mu_{\mathcal{B}}}{\sqrt{\left(\sigma_{\mathcal{B}}\right)^{2}+\epsilon}}}$,
$\epsilon$ is an arbitrarily small constant. Clearly, we observe that the
scalar associative/invariant property doesn’t hold for the normalization step,
because:

${(\widehat{sX})_{\mathcal{B}}}={\frac{(sX)_{\mathcal{B}}-s\mu_{\mathcal{B}}}{s\sqrt{\left(\sigma_{\mathcal{B}}\right)^{2}+\epsilon}}}={\hat{X}_{\mathcal{B}}}$
$\gamma{(\widehat{sX})_{\mathcal{B}}}+\beta=\gamma{\hat{X}_{\mathcal{B}}}+\beta\neq
s(\gamma{\hat{X}_{\mathcal{B}}}+\beta)$

Thus, in order to achieve scalar invariance, we need to consider some
alternatives to batch normalization. We mainly discuss two previous works on
exploring reliable and efficient residual learning without normalization:
Fixup and NF-ResNets. Both methods achieve state-of-art performance on a wide
collection of benchmarks.

Fixup enables training deep residual networks with comparable performance in
terms of convergence, generalization, etc, without normalization. More
specifically, this method rescales the standard initialization of residual
branches by taking the network architecture into account. The key steps of
Fixup initialization are described as follows:

  1. 1.

Initialize the last layer of each residual branch and the classification layer
to 0.

  2. 2.

Initialize other layers using a standard method [17], and scale only the
weight layers inside residual branches by $L^{-\frac{1}{2m-2}}$, where $L$ and
$m$ are the numbers of residual blocks and layers inside a residual branch
respectively.

  3. 3.

Add a scalar multiplier before each convolution, linear, and element-wise
activation layer in each residual branch, the multiplier is initialized at $1$
222We intentionally ignore the scalar bias (initialized at 0) presented in the
original paper to ensure scalar invariance. This, however, significantly
reduces the training performance, as we will show in Section 4. .

It is obvious that the above initialization steps perform some transformations
on the weights of neural networks instead of the input, and the scalar
multiplier is scalar associative which ensures the trained ResNet is scalar
invariant.

NF-ResNets aims to overcome the same challenge of developing ResNet variants
without normalization layers yet is comparable to batch-normalized ResNets in
many aspects. The effect of standard batch normalization operation within each
residual block can be summarized as: 1) downscales the input by a factor
proportional to its standard deviation; 2) increases the variance of the input
signal by an approximately constant factor. By mimicking the effect of batch
normalization, the residual blocks can be written in the form of
$X_{l+1}=X_{l}+\alpha\mathcal{G}_{l}(X_{l}/\beta_{l})$, where $X_{l}$ denotes
the input to the $l^{th}$ residual block and $\mathcal{G}_{l}(\cdot)$ denotes
the $l^{th}$ residual branch. Moreover, the network should be designed such
that:

  * •

$\mathcal{G}_{l}(\cdot)$ is parameterized to be able to preserve variance at
initialization, i.e., $Var(\mathcal{G}_{l}(z))=Var(z)$ for all $l$.

  * •

$\beta_{l}$ is a fixed scalar, set it to be $\sqrt{Var(X_{l})}$, the expected
empirical standard deviation of $X_{l}$ at initialization.

  * •

$\alpha$ is a hyperparameter that controls the growth rate of variance between
blocks.

Since both $\alpha$ and $\beta$ are fixed scalar during the inference phase.
The modified residual blocks are scalar associative since
$sX_{l}+\alpha\mathcal{G}_{l}(sX_{l}/\beta_{l})=s(X_{l}+\alpha\mathcal{G}_{l}(X_{l}/\beta_{l}))$.
We conclude the NF-ResNets method also ensures scalar invariance.

###  3.5 Robustness guarantees

Despite achieving great success in a wide range of tasks, neural networks have
proven not robust under even small perturbations to the input[8, 1], which
accelerates the study of neural network verification and adversarial attacks.
There are also many attempts focusing on improving the robustness of neural
networks such as data augmentation [37, 36, 41], however, these methods mostly
lack theoretical guarantees. In contrast, we show that scalar invariant neural
networks possess some robustness guarantees illustrated in Figure 2 without
any augmentation techniques.

Figure 2: Derived from the scalar invariant property, it is straightforward to
show two more robustness guarantees that can verify inputs on certain lines
and convex regions of the input space.

Direction robustness property An input $X$ specifies a direction in the input
space. From the origin, there are infinitely many points residing along that
direction, i.e., $\\{sX|s\in\mathbb{R}^{+}\\}$. From this point of view, the
scalar invariant property can be restated as the direction robustness
property. That is:

$\mathcal{N}(sX)=\mathcal{N}(X)\text{ }\text{ }\forall s\in\mathbb{R}^{+}$

With this property, we are able to verify any inputs along the direction
specified by the input $X$.

Interpolation robustness property We first introduce the notion of neural
activation pattern [11]. A _Neural Activation Pattern (NAP)_ of a neural
network $\mathcal{N}$ is a tuple
$\mathcal{P}_{\mathcal{N}}\mathbin{:=}(\mathcal{A},\mathcal{D})$, where
$\mathcal{A}$ and $\mathcal{D}$ are two disjoint set of neurons and
$\mathcal{A}\cup\mathcal{D}$ are all neurons. The notion of neural activation
patterns can be relaxed to consider only a subset of neurons, but that is
beyond the scope of our current discussion. We say that an input $X$ _follows_
a NAP $\mathcal{P}_{\mathcal{N}}$ if after computing $\mathcal{N}(X)$, the
neurons in $\mathcal{A}$ are all activated, and neurons in $\mathcal{D}$ are
all deactivated. We denote this as $\mathcal{P}_{\mathcal{N}}(X)=True$.

Now we consider any two inputs $X,Y$ that follow the same NAP
$\mathcal{P}_{\mathcal{N}}$ and output the same prediction by the
corresponding neural network $\mathcal{N}$, i.e.,
$\mathcal{P}_{\mathcal{N}}(X)=\mathcal{P}_{\mathcal{N}}(Y)=True$ and
$\mathcal{N}(X)=\mathcal{N}(Y)$. Then for $\lambda$ s.t. $\lambda\in[0,1]$, we
have:

$\mathcal{N}(\lambda X+(1-\lambda)Y)=\mathcal{N}(X)=\mathcal{N}(Y)$

The above property can be easily proved using the scalar invariant nature of
zero-bias neural networks. With this property, we are able to verify any
points interpolated between two reference points $X,Y$ within the same neural
activation pattern.

Convex region robustness property We can further extend the above property to
the multiple reference points setting. Suppose we have a collection of inputs
$\\{X_{i}|i\in\\{1,2,...,n\\}\\}$ s.t. they follow the same NAP
$\mathcal{P}_{\mathcal{N}}$ and output the same prediction by the
corresponding neural network $\mathcal{N}$. Let $\mathcal{M}$ be a convex
polygon whose vertices are $\\{X_{i}\\}$, then for any point $m$ lies inside
the polygon $\mathcal{M}$, we have:

$\mathcal{N}(m)=\mathcal{N}(X_{i})\text{ }\forall m\in\mathcal{M}\text{ and
}\forall i\in\\{1,2,...,n\\}$

As $m$ can always be represented by some linear combination of vertices
$\\{X_{i}\\}$, the convex region robustness property holds as the direct
result of the interpolation robustness property. With this property, we are
able to verify a whole region of input space which has proven to be
challenging in the field of neural network verification.

Figure 3: The intrinsic distribution of images incorporates both locality (The
neighborhood of a sample point should belong to the same class.) and
directionality (Any points along the same direction should belong to the same
class.)

(a) Linear activation regions of a simple scalar invariant neural network are
cones.

(b) Linear activation regions of a simple neural network are polytopes.

Figure 4: Scalar invariant neural networks perform poorly along direction 1
(only locality holds) compared to normal neural networks. They perform
comparably along direction 2 (both locality and directionality hold).

###  3.6 Geometric insights

Although we have shown that scalar invariant neural networks demonstrate some
nice properties in terms of robustness, one major concern that emerges along
with eliminating bias is the reduction in the representational power of neural
networks. Such a reduction may hurt neural networks’ performances in certain
categories of tasks, yet we argue that image classification tasks seem not to
be one of them. Let us consider the intrinsic distribution of images in the
high-dimensional input space, illustrated in Figure 3. This leads to two key
observations:

  * •

Locality: The neighborhood with a certain radius of a sample point should
belong to the same class.

  * •

Directionality: Any points along the direction specified by a sample point
should belong to the same class.

Since a neural network can be thought of as a piece-wise (linear) function
defined over many convex polytopes [13, 14], we plot linear regions of a
simple scalar invariant neural network and a simple normal neural network
trained on a simple 2D dataset to study their representational power in Figure
4. We consider two easy learning tasks whose input distributions are
characterized by directions 1 and 2. Since locality is implicitly embedded in
input distributions of almost every task, including directions 1 and 2 cases
(otherwise, generalization is impossible), we mainly discuss the impact of
directionality on learning outcomes. First, in the absence of directionality,
e.g., in the case of data labeled differently along direction 1, the scalar
invariant neural network may not fit data well using only linear function
within the corresponding cone, as shown in Figure 4(a). Whereas the normal
network could overcome this using a piece-wise linear function across multiple
convex regions along direction 1, illustrated in Figure 4(b).

However, direction 1 is not a serious concern as we observe that the intrinsic
distribution of images should incorporate directionality, portrayed by
direction 2. We investigate the possible gap in the representational power
between the two types of neural networks in the direction 2 case. Given the
directionality holds, the scalar invariant neural network could fit the data
along that direction 2 using a linear function (within a cone) on par with
using piece-wise linear functions by the normal neural networks. This suggests
they should deliver comparable performances, and we will provide more
experimental evidence to support this claim in Section 4. Following our above
discussion, we believe the directionality can be used as a strong geometric
prior in image-related tasks, sharing the spirit of adapting transnational
invariance prior introduced by CNN. Not only could scalar invariant neural
networks perform comparably to normal neural networks, but also they possess
some rigorous robustness properties. For tasks whose underlying distribution
doesn’t satisfy directionality, we may assume directionality holds for them
after applying some pre-processing/transformations. In this way, we can get
better robustness guarantees enabled by scalar invariant neural networks,
without much performance-wise concern.

| Scalar
---|---
| 1 | 0.25 | 0.15 | 0.125 | 0.1 | 0.075 | 0.05 | 0.025 | 0.01 | 0.001 | 0.0001
MNIST | FCN | w/ bias | 88.12 | 87.07 | 84.46 | 82.57 | 79.52 | 74.76 | 65.82 | 42.84 | 16.34 | 10.28 | 10.28
w/o bias | 88.27 | 88.27 | 88.26 | 88.27 | 88.27 | 88.26 | 88.27 | 88.27 | 88.27 | 88.27 | 88.27
Fashion-MNIST | CNN | w/ bias | 89.10 | 67.10 | 40.12 | 32.52 | 24.16 | 17.91 | 12.46 | 10.12 | 10.00 | 10.00 | 10.00
w/o bias | 89.02 | 89.02 | 89.02 | 89.02 | 89.02 | 89.02 | 89.02 | 89.02 | 89.02 | 89.02 | 89.02
Imagenette[18] | Fixup_ResNet32 | w/ bias | 64.87 | 19.31 | 11.80 | 10.83 | 10.32 | 10.19 | 10.17 | 10.17 | 10.17 | 10.17 | 10.17
w/o bias | 59.62 | 59.62 | 59.62 | 59.62 | 59.62 | 59.62 | 59.62 | 59.62 | 59.62 | 59.62 | 59.62
NF_ResNet34 | w/ bias | 75.77 | 72.59 | 72.19 | 70.52 | 65.52 | 60.36 | 54.65 | 33.10 | 20.20 | 9.94 | 9.94
w/o bias | 78.45 | 78.45 | 78.45 | 78.45 | 78.45 | 78.45 | 78.45 | 78.45 | 78.45 | 78.45 | 78.45
CIFAR-100 | Fixup_ResNet110 | w/ bias | 63.14 | 25.05 | 12.85 | 9.93 | 7.42 | 4.6 | 2.32 | 1.16 | 1.0 | 1.0 | 1.0
w/o bias | 53.75 | 53.75 | 53.75 | 53.75 | 53.75 | 53.75 | 53.75 | 53.75 | 53.75 | 53.75 | 53.75
NF_ResNet101 | w/ bias | 61.44 | 56.21 | 44.77 | 39.10 | 31.00 | 20.68 | 9.56 | 2.79 | 1.03 | 1.00 | 1.00
w/o bias | 62.51 | 62.51 | 62.51 | 62.51 | 62.51 | 62.51 | 62.51 | 62.51 | 62.51 | 62.51 | 62.51
EfficientNet [40] | w/ bias | 81.94 | 62.44 | 40.42 | 31.81 | 22.68 | 13.22 | 5.39 | 1.25 | 1.00 | 1.00 | 1.00
ViT [10] | w/ bias | 91.48 | 81.58 | 66.11 | 58.55 | 47.98 | 34.11 | 18.49 | 4.54 | 1.08 | 1.00 | 1.00

Table 1: Normal neural networks are generally not robust against scalar
multiplication (with the input image), whereas their scalar invariant
counterparts achieve absolute invariance as we expected.

##  4 Experiemnts

(a) Learning curves of FCNs with and without bias on MNIST.

(b) Learning curves of CNNs with and without bias on Fashion-MNIST.

(c) Learning curves of ResNet34 with and without bias on Imagenette.

(d) Learning curves of ResNet101 with and without bias on CIFAR-100.

Figure 5: Learning curves of normally trained neural networks, and their
scalar invariant counterparts are almost identical, which supports our
argument that removing bias doesn’t impact the expressive capability of models
on image classification tasks.

###  4.1 Scalar invariance evaluation

In this section, we conduct several experiments to investigate the robustness
of normal neural networks and their scalar invariant counterparts under scalar
multiplication with the input. We train the two types of neural networks using
the same configuration except for the option of using bias on some popular
image classification benchmarks. To further show the scalar invariant effect,
we choose the scalar varies in a wide range, from $1$ to $0.0001$; the results
are reported in Table 1. Normal neural networks are generally not robust
against scalar multiplication with the input image. We also notice that models
trained using Fixup are more vulnerable than those trained using NF-ResNets as
the scalar decreases. This may be due to scaling operations on weights
performed by Fixup, which causes the weights to be sensitive to the scale of
the input. Moreover, in the presence of bias, most models could only achieve a
fraction of their original accuracies when the multiplying scalar is $0.01$.
For instance, the state-of-art ViT model can only achieve $1.00\%$ when
multiplying $0.01$ to the input, an enormous decline from $91.48\%$, its
original performance on CIFAR-100. Whereas their scalar invariant counterparts
achieve absolute invariance as we expected. At the scalar of $0.01$, scalar
invariant models outperform SOTA models by around $60\%$, the gap could be
further extended by fine-tuning or other improvements in training. We plot
some visual examples in Figure 6. We observe that the prediction of normally
trained neural networks changes constantly as the scalar diminishes, whereas
that of scalar invariant networks remains unchanged despite the corresponding
probability also decreasing.

We report the learning curves of both scalar invariant networks (excluding
Fixup models due to training issues with scalar bias fixed) and their normally
trained counterparts in Figure 5. Highly overlapped training curves, in this
case, indicate both two types of models have comparable expressive
capabilities. This supports our argument on ignoring bias as a consequence of
considering the directionality of the image distribution.

Figure 6: W/ bias and w/o bias stand for the prediction of normal and scalar
invariant models respectively. Prediction of normally trained neural networks
changes constantly as the scalar decreases, whereas that of scalar invariant
networks remains unchanged. Despite the probability of the corresponding class
diminishing. models inherit scalar invariance from removing bias.

###  4.2 Model bias on the zero image

In this section, we study the models’ bias when predicting the zero image,
i.e. image with all pixel values being zero. From the humans’ perspective, the
zero image contains no information, thus maximizing the information entropy.
To be more specific, the zero image could equally likely be an instance of any
class, i.e. follows a uniform distribution. It is trivial to show that scalar
invariant neural networks share the same bias as humans, because:

(a) The predicted probability of ResNets and scalar invariant models on the
zero image.

(b) The predicted probability of Inception-v4[39] and scalar invariant models
on the zero image.

(c) The predicted probability of EfficientNet and scalar invariant models on
the zero image.

(d) The predicted probability of ViT and scalar invariant models on the zero
image.

Figure 7: SOTA models are biased when predicting the zero image, whereas
scalar invariant neural networks are unbiased like humans.

$\displaystyle\mathcal{N}(\mathbf{0})=$
$\displaystyle\operatorname*{argmax}_{c}\mathcal{\\{A\circ\mathcal{O}(\mathbf{0})\\}}$
$\displaystyle=$
$\displaystyle\operatorname*{argmax}_{c}\frac{e^{\mathbf{0}}}{\displaystyle\sum_{c\in\mathcal{C}}e^{\mathbf{0}}}=\operatorname*{argmax}_{c}\frac{1}{|\mathcal{C}|}$

However, for those neural networks trained with bias, even including most
state-of-the-art models, such property may not hold as the model may bias
towards certain classes. Figure 7 reports selective results on models’ bias
when predicting the zero image. It is interesting to see that all models have
some degree of bias toward certain classes. And deeper ResNets such as
ResNet-152 are less biased than shallower ones such as ResNet-34, suggesting
larger models have the potential to better adjust their posterior belief
(distribution) after observing data than smaller models. On the other hand,
scalar invariant neural networks have no bias toward certain classes, as we
expected. Therefore, one may consider incorporating zero bias as a strong
inductive bias in model design/selection.

###  4.3 Robustness evaluation

We use visual examples to demonstrate the robustness merit of scalar invariant
neural networks. We mainly investigate the interpolation robustness, i.e.
interpolation of two images of the same class still belongs to that class. We
show that scalar invariant neural networks could correctly predict some
synthesized inputs (interpolations of images) which normal models fails to do
so, as reported in Figure 8.

Figure 8:  The left and right images are from the original dataset, whereas
synthesized/interpolated images are in the middle. For instance, the middle
image in the first row is generated by adding $(\alpha=0.5)$ times the left
image to $(1-\alpha)$ times the right image.

The semantics of those synthesized images are clear to humans, yet labeled
incorrectly by a normally trained model. We observe scalar invariant neural
networks generally perform better on interpolated images than normal models
do, this can be explained by our robustness properties introduced in Section
3. However, visual examples seem not sufficient to formally prove the models’
robustness on interpolation or convex regions of the input space. To address
this, we need to resort to neural network verification tools which generate
machine-checkable proofs [24, 25, 42]. However, current works mainly focus on
verifying models’ robustness under perturbations with a specific norm (usually
$\l_{\infty}$) to the input [20, 19], which cannot be easily applied to our
case. Thus, we plan to explore more flexible neural network verification
specifications and methods in our future work.

##  5 Conclusion

In this paper, we study scalar multiplication invariance in the field of
neural networks. We prove that, by simply dropping bias terms, the prediction
of neural networks achieves absolute invariance under scalar multiplication
with the input image. Moreover, scalar invariant neural networks could usually
outperform state-of-art models by a large margin, e.g. above $60\%$ on
CIFAR-100. Although it is commonly believed that bias improves models’
performance and thus is always needed, we show that it can be completely
ignored in addressing many image-related tasks such as image classification.
This is because the intrinsic distribution of images incorporates
directionality, which favors zero bias as a strong inductive prior in model
selection. Our experimental results on both small networks, such as simple FCN
and CNN and large models such as ResNet-101, also confirm this argument.

In addition to comparable performances, we show that scalar invariant neural
networks tend to have stronger robustness guarantees than normal neural
networks. This is because scalar invariance allows us to formally verify
certain lines and convex regions of the input space, which is usually
impractical for normal neural networks. We then support this argument using
some visual examples. Driven by their simplicity and effectiveness, we
hypothesize there exist more interesting properties of scalar invariant
networks, such as interpretability, and plan to investigate them in future
work.

## References

  * [1] Naveed Akhtar and Ajmal S. Mian.  Threat of adversarial attacks on deep learning in computer vision: A survey.  IEEE Access, 6:14410–14430, 2018.
  * [2] Laith Alzubaidi, Jinglan Zhang, Amjad J. Humaidi, Ayad Al-dujaili, Ye Duan, Omran Al-Shamma, Jesus Santamaría, Mohammed Abdulraheem Fadhel, Muthana Al-Amidie, and Laith Farhan.  Review of deep learning: concepts, cnn architectures, challenges, applications, future directions.  Journal of Big Data, 8, 2021.
  * [3] Benjamin Bloem-Reddy and Yee Whye Teh.  Probabilistic symmetries and invariant neural networks.  J. Mach. Learn. Res., 21:90:1–90:61, 2020.
  * [4] Jake Bouvrie.  Notes on convolutional neural networks.  CoRR, 2006.
  * [5] Andrew Brock, Soham De, and Samuel L. Smith.  Characterizing signal propagation to close the performance gap in unnormalized resnets.  In ICLR. OpenReview.net, 2021.
  * [6] Andy Brock, Soham De, Samuel L. Smith, and Karen Simonyan.  High-performance large-scale image recognition without normalization.  In ICML, volume 139 of Proceedings of Machine Learning Research, pages 1059–1071. PMLR, 2021.
  * [7] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic.  Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.  CoRR, abs/2104.13478, 2021.
  * [8] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian J. Goodfellow, Aleksander Madry, and Alexey Kurakin.  On evaluating adversarial robustness.  CoRR, abs/1902.06705, 2019.
  * [9] Taco Cohen and Max Welling.  Group equivariant convolutional networks.  In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2990–2999, New York, New York, USA, 20–22 Jun 2016. PMLR.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  CoRR, abs/2010.11929, 2020.
  * [11] Chuqin Geng, Nham Le, Xiaojie Xu, Zhaoyue Wang, Arie Gurfinkel, and Xujie Si.  Toward reliable neural specifications, 2022.
  * [12] Rohan Ghosh and Anupam K. Gupta.  Scale steerable filters for locally scale-invariant convolutional neural networks.  CoRR, abs/1906.03861, 2019.
  * [13] Boris Hanin and David Rolnick.  Complexity of linear regions in deep networks.  In ICML, volume 97 of Proceedings of Machine Learning Research, pages 2596–2604. PMLR, 2019.
  * [14] Boris Hanin and David Rolnick.  Deep relu networks have surprisingly few activation patterns.  In NeurIPS, pages 359–368, 2019.
  * [15] Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick.  Neuroscience-inspired artificial intelligence.  Neuron, 95(2):245–258, 2017.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In ICCV, pages 1026–1034. IEEE Computer Society, 2015.
  * [18] Jeremy Howard.  Imagenette: A smaller subset of 10 easily classified classes from imagenet, March 2019.
  * [19] Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi.  A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability.  Computer Science Review, 37:100270, 2020.
  * [20] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu.  Safety verification of deep neural networks.  In Rupak Majumdar and Viktor Kunčak, editors, Computer Aided Verification, pages 3–29, Cham, 2017. Springer International Publishing.
  * [21] Yukun Huang, Zheng-Jun Zha, Xueyang Fu, and Wei Zhang.  Illumination-invariant person re-identification.  In ACM Multimedia, pages 365–373. ACM, 2019.
  * [22] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 448–456. JMLR.org, 2015.
  * [23] A.K. Jain, Jianchang Mao, and K.M. Mohiuddin.  Artificial neural networks: a tutorial.  Computer, 29(3):31–44, 1996.
  * [24] Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer.  Reluplex: An efficient smt solver for verifying deep neural networks.  In Rupak Majumdar and Viktor Kunčak, editors, Computer Aided Verification, pages 97–117, Cham, 2017. Springer International Publishing.
  * [25] Guy Katz, Derek A. Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zeljic, David L. Dill, Mykel J. Kochenderfer, and Clark W. Barrett.  The marabou framework for verification and analysis of deep neural networks.  In CAV (1), volume 11561 of Lecture Notes in Computer Science, pages 443–452. Springer, 2019.
  * [26] Pooya Khorrami, Thomas Paine, and Thomas Huang.  Do deep neural networks learn facial action units when doing expression recognition?  In Proceedings of the IEEE international conference on computer vision workshops, pages 19–27, 2015.
  * [27] Risi Kondor and Shubhendu Trivedi.  On the generalization of equivariance and convolution in neural networks to the action of compact groups.  In ICML, volume 80 of Proceedings of Machine Learning Research, pages 2752–2760. PMLR, 2018.
  * [28] Yongxin Liu, Yingjie Chen, Jian Wang, Shuteng Niu, Dahai Liu, and Houbing Song.  Zero-bias deep neural network for quickest rf signal surveillance.  In 2021 IEEE International Performance, Computing, and Communications Conference (IPCCC), pages 1–8, 2021.
  * [29] Yongxin Liu, Jian Wang, Jianqiang Li, Shuteng Niu, Lei Wu, and Houbing Song.  Zero-bias deep learning enabled quickest abnormal event detection in iot.  IEEE Internet of Things Journal, 2021.
  * [30] Yongxin Liu, Jian Wang, Jianqiang Li, Houbing Song, Thomas Yang, Shuteng Niu, and Zhong Ming.  Zero-bias deep learning for accurate identification of internet-of-things (iot) devices.  IEEE Internet of Things Journal, 8(4):2627–2634, 2021.
  * [31] Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy.  On the benefits of invariance in neural networks.  CoRR, abs/2005.00178, 2020.
  * [32] Will Maddern, Alex Stewart, Colin McManus, Ben Upcroft, Winston Churchill, and Paul Newman.  Illumination invariant imaging: Applications in robust vision-based localisation, mapping and classification for autonomous vehicles.  In Proceedings of the Visual Place Recognition in Changing Environments Workshop, IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, volume 2, page 5, 2014.
  * [33] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Methods for interpreting and understanding deep neural networks.  Digital Signal Processing, 73:1–15, 2018.
  * [34] S.J. Perantonis and P.J.G. Lisboa.  Translation, rotation, and scale invariant pattern recognition by high-order neural networks and moment classifiers.  IEEE Transactions on Neural Networks, 3(2):241–251, 1992.
  * [35] N. Pattabhi Ramaiah, Earnest Paul Ijjina, and C. Krishna Mohan.  Illumination invariant face recognition using convolutional neural networks.  In 2015 IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (SPICES), pages 1–4, 2015.
  * [36] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann.  Data augmentation can improve robustness, 2021.
  * [37] Connor Shorten and Taghi M. Khoshgoftaar.  A survey on image data augmentation for deep learning.  J. Big Data, 6:60, 2019.
  * [38] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  J. Mach. Learn. Res., 15(1):1929–1958, 2014.
  * [39] Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke.  Inception-v4, inception-resnet and the impact of residual connections on learning.  CoRR, abs/1602.07261, 2016.
  * [40] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  CoRR, abs/1905.11946, 2019.
  * [41] Florian Tramèr and Dan Boneh.  Adversarial training and robustness for multiple perturbations.  CoRR, abs/1904.13000, 2019.
  * [42] Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter.  Beta-CROWN: Efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification.  Advances in Neural Information Processing Systems, 34, 2021.
  * [43] Shengjie Wang, Tianyi Zhou, and Jeff Bilmes.  Bias also matters: Bias attribution for deep neural network explanation.  In International Conference on Machine Learning, pages 6659–6667. PMLR, 2019.
  * [44] Yichong Xu, Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, and Zheng Zhang.  Scale-invariant convolutional neural networks.  CoRR, abs/1411.6369, 2014.
  * [45] Guangyu Robert Yang and Xiao-Jing Wang.  Artificial neural networks for neuroscientists: A primer.  Neuron, 107(6):1048–1070, 2020.
  * [46] Bo Zhang, Qiang Zhang, Yong Xin Liu, and Ou Ye.  Anomaly data detection for ads- b based on zero-bias inception network.  In 2021 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), pages 1–6, 2021.
  * [47] Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma.  Fixup initialization: Residual learning without normalization.  In ICLR (Poster). OpenReview.net, 2019.
  * [48] Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li.  S3fd: Single shot scale-invariant face detector.  In Proceedings of the IEEE international conference on computer vision, pages 192–201, 2017.


11institutetext: S. Dasgupta1,2 22institutetext: 1Bredesen Center, University
of Tennessee, TN, USA
2Quantum Computational Science Group,
Oak Ridge National Laboratory, TN, USA
22email<EMAIL_ADDRESS>33institutetext: K. E. Hamilton 44institutetext:
Quantum Computational Science Group,
Oak Ridge National Laboratory, TN, USA
44email<EMAIL_ADDRESS>55institutetext: A. Banerjee 66institutetext:
Department of Physics and Astronomy,
Purdue University, IN, USA
66email<EMAIL_ADDRESS>
# Designing a NISQ reservoir with maximal memory capacity for volatility
forecasting ††thanks: This manuscript has been authored by UT-Battelle, LLC
under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The
United States Government retains and the publisher, by accepting the article
for publication, acknowledges that the United States Government retains a non-
exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the
published form of this manuscript, or allow others to do so, for United States
Government purposes. The Department of Energy will provide public access to
these results of federally sponsored research in accordance with the DOE
Public Access Plan. (http://energy.gov/downloads/doe-public-279 access-plan).

Samudra Dasgupta  Kathleen E. Hamilton  Arnab Banerjee

(Received: date / Accepted: date)

###### Abstract

Forecasting the CBOE volatility index (VIX) is a highly non-linear and memory-
intensive task. In this paper, we use quantum reservoir computing to forecast
the VIX using S&P500 (SPX) time-series. Our reservoir is a hybrid quantum-
classical system executed on IBM’s 53 qubit Rochester chip. We encode the SPX
values in the rotation angles and linearly combine the average spin of the
six-qubit register to predict the value of VIX at next time step. Our results
demonstrate a potential application of noisy intermediate-scale quantum (NISQ)
devices to complex, real-world applications.

###### Keywords:

Quantum Reservoir Computing Memory Capacity NISQ Financial Risk Management
Volatility

##  1 Introduction

Accurate forecasting of financial data is a difficult task: financial data is
massive and contains many correlated dimensions. Risk estimation needs to
strike a careful balance between avoiding catastrophic crises and avoiding
risk altogether. Risk in finance is typically measured in terms of volatility
of returns or close analogues like Value at Risk (McNeil et al (2015)). Risk
can be unconditional, for example the 30-day rolling standard deviation of the
S&P 500 Index (SPX) returns. It can also be conditional, for example Expected
Shortfall which is defined as the average loss given the loss has crossed a
certain threshold. The observed price of options in the markets can help
impute the implied volatility. Developing useful machine learning based models
for financial forecasting tasks requires memory characteristics that balance
long-term and short-term risk.

The field of reservoir computing (RC) (Gerstner et al (2014)) provides a
detailed but flexible road map towards using signal-driven dynamical systems
to process information with non-von Neumann architectures. RC models are
useful in providing alternatives to deep learning that can deliver comparable
performance yet are low energy, and computationally simple. They are capable
of both one-shot and continuous real-time learning and excel at non-linear
function approximation tasks. RC systems have been utilized in many different
applications and can be constructed from many different dynamical systems (see
recent reviews in (Dambre et al (2012)) and (Tanaka et al (2019a))).

Quantum reservoir computing (QRC) uses quantum ensembles for information
processing. In a recent work (Nakajima et al (2019)), quantum spin systems
were used to construct a quantum reservoir and used for predicting non-linear
time series. Reservoirs built using superconducting qubits are demonstrated in
(Chen and Nurdin (2019); Chen et al (2020)) and these studies have developed a
theoretical underpinning behind the ability to use dissipative quantum systems
as quantum counterpart to approximating non-linear input-output maps using
classical dynamical systems.

###  1.1 Related Works

Understanding the computational capacity of quantum reservoirs is an open
question. There have been several approaches to quantum reservoir designs and
numerical experiments show that quantum systems consisting of 5–7 qubits
possess computational capabilities comparable to conventional recurrent neural
networks of 100 to 500 nodes (Fujii and Nakajima (2017)). Additionally, small
quantum systems also demonstrate significant computational capacities (Govia
et al (2020)). A recent study (Kutvonen et al (2020)) has also focused on
optimizing quantum reservoirs for time series forecasting for financial data
(the S&P 500 index).

Our methods are comparable to Chen and Nurdin (2019) and (Chen et al (2020))
with several significant differences:

  * •

We are focused on hybrid quantum-classical reservoirs (which we refer to as
NISQ reservoirs) which incorporate quantum circuits and classical feedback
elements.

  * •

We implement systematic design considerations of these NISQ reservoirs as a
computing engine which should be useful for practitioners.

  * •

We address the question of evaluating the memory capacity of various reservoir
topologies and how to select the optimal one.

  * •

We handle the case of a ‘real-life signal’ that cannot be expressed by an
analytical deterministic equation. VIX (see Section 3.1) is intrinsically
related to market fluctuations and trader psychology.

###  1.2 Organization and contribution

In this paper we focus on the task of VIX forecasting, using the SPX return as
the independent variable. Given that $\Delta\mathrm{SPX}$ explains less than
$75\%$ of $\Delta$VIX we fully acknowledge that a more sophisticated
implementation would use more economic indicators such as the unemployment
rate, gross domestic product and federal funds rate. However the focus of this
paper is demonstrating the design and use of a NISQ reservoir for forecasting
purposes and not pushing the envelope on forecasting accuracy.

We characterize the memory capacity of a six-qubit NISQ reservoir in Section
2. This characterization determines the reservoir design used in Section 3 to
forecast the VIX index. In Section 3, we discuss the relevant properties of
the VIX index, the input encoding methodology, the NISQ reservoir circuit
construction, the use of post-processing and feedback and finally the results
of the forecasting task. Section 4 concludes with a summary of the
contributions of this paper.

##  2 Memory Capacity

Memory capacity (MC) quantifies the ability of the reservoir to forecast at
different time-scales. Before we can design our reservoir, we characterize the
MC of different possible configurations of the reservoir, following the
approach given in (Nakajima et al (2019)). The configuration with the highest
MC will then be used for the time-series prediction task in Section 3.

Let $u_{k}$ be the time-series one is trying to forecast (where $k$ denotes
the time index). Let $\hat{u}_{k-\tau}$ denote the forecast of $\hat{u}_{k}$
using information till time-step $k-\tau$. The correlation $r_{\tau}$ between
$\hat{u}_{k-\tau}$ and $u_{k}$ is a measure of how well the system is able to
do a $\tau$ step look-ahead prediction:

$r_{\tau}^{2}=\frac{\mathrm{COV}(u_{k-\tau},\hat{u}_{k-\tau})}{\sigma^{2}(u_{k-\tau})\sigma^{2}(\hat{u}_{k-\tau})},$
(1)

where COV(x,y) denotes the covariance between x and y and $\sigma(x)$ denotes
the standard deviation of x. Intuitively, one expects that the larger the
value of $\tau$, the lower is the value of $r_{\tau}$ (as higher the value of
$\tau$, more amount of recent data is ignored).

The MC is the sum of $r_{\tau}^{2}$ over different values of $\tau$:

$MC=\sum\limits_{\tau=1}^{\tau_{max}}r_{\tau}^{2}.$ (2)

As in Nakajima et al (2019), we use a random sequence $\in[0,1]$ for $u_{k}$
(where $k$ denotes the time index) and fix the maximum value of $\tau$ to be
$\tau_{max}=120$. This is done to ensure that the MC benchmark does not depend
on a specific time-lag or a specific signal structure.

Figure 1: Schematic of the hybrid quantum-classical reservoir (NISQ reservoir)
system which consists of classical inputs and outputs (grey boxes), classical
computational layers (grey cylinders) and quantum computational layers (white
cylinder).

The NISQ reservoirs used in this study are hybrid quantum-classical systems.
The demarcation between classical and quantum resources is shown in Fig. 1.
The firse classical layer transforms the input into a qubit angle encoding.
The quantum layer is used to generate an array of N-qubit spin values. The
final classical layer is used to compute the forecast, and the forecast error.
Both the forecast error and spin values are fed back into the first classical
layer.

We characterize the MC of a 6-qubit NISQ reservoir as a function of recurrent
connections using a sequence of $1+N+\frac{N(N-1)}{2}$ graphs in increasing
order of network connectivity (and hence complexity). The first term in the
sequence is an empty graph on $N$ vertices. The next $N$ terms in the sequence
are sequentially constructed by adding self-loops to each vertex. The next $N$
terms are sequentially constructed by connecting the $N$ vertices into a
simple cycle. Finally the remaining ($\frac{N(N-1)}{2}$) terms of the sequence
are constructed by sequentially connecting vertices until the final circuit is
a fully connected graph with $N$ self-loops. Note that an edge can be realized
between any two nodes of the reservoir if a two-qubit gate is placed between
the qubits in the quantum layer; or if the output of one qubit is fed to
another qubit during the classical pre-processing layer.

Figure 2: Sequence of reservoir complexity circuits: (a) The first term is
always an empty graph on $N$ qubits, (b) The first ($N$) circuits are
generated by adding self-loops, (c) The next ($N$) circuits are generated by
connecting the qubits into a simple cycle, (d) The remaining circuits are
generated by adding edges to fully connect all $N$ qubits.

For a 6 qubit system, 22 configurations are possible. This sequence is shown
in Fig. 2.

Figure 3: MC as a function of reservoir complexity for a 6-qubit reservoir
executed on ibmq_rochester. [Inset] The optimal reservoir topology with self-
loops on 5 qubits.

The MC of each reservoir was evaluated using IBM’s $52$ superconducting qubit
platform (ibmq_rochester) and is shown in Fig. 3. A peak in the MC (within the
bounds of statistical significance of the MC) is observed for reservoirs with
5 self-loops. This reservoir design is then chosen for the information
processing in Section 3.

Figure 4: MC as a function of reservoir complexity for a 6-qubit reservoir
simulated with noiseless qubits.

The same sequence of reservoir topologies were also simulated in IBM Qiskit
(Abraham et al (2019)). The results of the noiseless simulation are shown in
Fig. 4. Comparison with Fig. 3 reveals that the hardware noisiness translates
into higher MC (within the bounds of statistical significance) for circuits
with higher connectivity (leading to higher degree of non-linear dynamics). We
also observe a slower decay in MC for the NISQ reservoir with hardware noise.
This points to a beneficial impact of the noise in today’s NISQ devices.

##  3 VIX forecasting

In the previous section we found the optimal design of the NISQ reservoir
(based on maximal MC value). In this section we will first give more
background for the economic indicator that we are trying to predict. Then, we
will discuss the components of the NISQ reservoir as shown in Fig. 1, tailored
to the VIX forecasting task: (a) input encoding (Section 3.2), (b) a quantum
circuit (Section 3.3) , and (c) forecast and feedback generation (Section
3.4).

In Fig. 5 we show the computational graph associated with this design,
tailored for the VIX forecasting task. The input encoding consists of the
transformation of $\Delta r(t)\rightarrow u(t)$, the quantum circuit generates
the spin values $s_{i}(t)$ and the forecast is generated by the combination of
$s_{i}(t)$.

Figure 5: Computational graph of the 6 qubit reservoir with 5 self-loops:
$\Delta r(t)=\mathrm{SPX}(t)-\mathrm{SPX}(t-1)$, $u(t)$ is the incoming signal
post application of a non-linear transformation, $s_{i}(t)$ is the average
spin state of qubit [i], delta_v(t+1) is the actual value while
pred_delta_v(t+1) is the predicted value. The error residual is denoted by
err(t+1). The residual from time step $t$ is used as feedback to the
reservoir.

###  3.1 VIX index forecasting

The VIX index represents the market’s expectation of volatility in the near
future as implied by SPX index options data. It is disseminated by CBOE on a
real-time basis and modern finance practitioners prefer using VIX for risk
estimation. It’s value denotes the expected annualized change in the SPX 500
index over the following 30 days, the methodology is detailed in (CBOE
(2019a)). In short, it is calculated using the CBOE-traded SPX options (which
have a non-zero bid-ask) whose expiration falls within next 23 days and 37
days. Using the classical Black Scholes model assumes a time-independent
(constant) volatility. However, economists have confirmed that volatility
varies with time (hence the name Stochastic Volatility). Stochastic models
(like GARCH) significantly improve the prediction accuracy against values
observed in the market and are thus valuable in asset pricing (for traders)
and asset management (for risk managers).

Figure 6: (Top): The VIX Index plotted as a function of time from January 2,
1990 through March 24, 2020. The data corresponding to the 2008 recession is
highlighted in the grey shaded region.(Bottom): The SPX returns ($r(t)$)
plotted as a function of time.

In this study we develop our NISQ reservoir to forecast the VIX index using
the SPX index ($\\{r_{t}\\}$) as the independent variable. The entire dataset
spans January 2, 1990 through March 24, 2020 (see Fig. 6). The initial one-
third of the data (from January 1, 1990 to December 31, 1997) was flushed out
to allow the system to stabilize. In Fig. 7 we plot
($\Delta\mathrm{VIX}_{t}=\mathrm{VIX}_{t}-\mathrm{VIX}_{t-1}$) versus
($\Delta\mathrm{SPX}_{t}=\mathrm{SPX}_{t}-\mathrm{SPX}_{t-1}$).

Figure 7: Scatter plot between the daily percentage change in SPX and daily
percentage change in VIX. It should be evident that change in SPX is
correlated (negatively) with change in VIX. This is why we use SPX as the main
input to the reservoir for VIX forecasting in Section 3.1.

These are the relevant data properties, as shown in Figs. 6,7:

  * •

VIX is always positive. It is derived from option implied volatility which can
never go negative.

  * •

The mean value of the VIX series is approximately 19. It hit an all time peak
of 82.69 on March 16, 2020. The previous maximum value of 80.86 was reached on
Nov 20, 2008, at the peak of the mortgage crisis (about eight weeks after the
collapse of Lehman Brothers).

  * •

The change in VIX is highly correlated with the change in SPX. The correlation
coefficient is approximately $-0.74$ over the entire date range (though it is
much higher during times of crisis). See (CBOE (2019b)) and (Robinson (2018))
for details on why SPX is the primary driver of VIX.

  * •

VIX spikes more when SPX suffers a high negative shock compared to a positive
shock of same magnitude. This is referred to as asymmetric volatility in
literature and is driven by behavioral psychology.

  * •

VIX exhibits volatility clustering i.e. volatility is persistently high during
times of high uncertainty and persistently low during times of more certainty.

###  3.2 Input encoding

The reservoir predicts a value for VIX at time (t+1) using SPX data for the
last seven days $(r(t-6)\cdots r(t))$. Our forecasting task uses
($\\{r_{t}\\}$), the sequence of time-dependent S&P500 (SPX) log return values
(Hudson and Gregoriou (2015)):

$r_{t}=\log{\frac{\mathrm{SPX}_{t}}{\mathrm{SPX}_{t-1}}}.$ (3)

In the classical pre-processing layer, these SPX return values are converted
into a vector of rotation angles $\theta(t)$ which will be implemented in the
quantum circuit.

First, the SPX log return values $\\{r_{t}\\}$ are used to construct a
sequence of time difference values:

$\Delta r_{t}=r_{t}-r_{t-1}.$ (4)

A non-linear transformation is applied to $\\{\Delta r_{t}\\}$ to define

$u(t)=1-e^{-(a_{0}+a_{1}I_{t}\Delta r_{t})},$ (5)

where $I_{t}$ is an indicator function

$\begin{split}I_{t}=\begin{cases}1&\Delta r_{t}<0\\\ 0&\Delta
r_{t}>0.\end{cases}\end{split}$

The non-linear transformation (Eq. 5) captures the empirical observation that
when returns go negative, volatility spikes more than when they are positive.
This transformation is shown in Fig. 8.

Figure 8: Transformation applied to $\Delta r$ to account for volatility
asymmetry.

The full encoding of the input signal ($u(t)$) into a vector of rotation
values $\theta_{m}(t)$ uses a heuristic encoding that is dependent on the SPX
return ($u(t)$), prediction error $e_{t}$, qubit register element $m$, and the
average qubit spin $s_{m}(t)$ (see following section). The values of
$\theta_{m}(t)$ are constrained to the range $[0,\pi/2]$.

$\theta_{m}(t+1)=\begin{cases}\frac{\pi}{2}\left(\alpha
u_{m}(t)+\beta\frac{s_{m}(t)+1}{2}+\gamma e_{t}\right)&m\in[0,4]\\\
\frac{\pi}{2}(\alpha^{\prime}u_{m}(t)+\gamma^{\prime}e_{t})&m=5.\end{cases}$
(6)

For the $6$-qubit reservoir, the parameters in Eq. 6 are:
$\alpha=0.3,\beta=0.3,\gamma=0.4,\alpha^{\prime}=0.6,\gamma^{\prime}=0.4$.

###  3.3 Reservoir circuit

Our NISQ reservoir system consists of a quantum circuit with classical
feedback loops. In a classical reservoir the connections between oscillators
are not trained, likewise in our NISQ reservoir the connections between qubits
are not trained.

Figure 9: The 6 qubit quantum circuit executed on ibmq_rochester with
arbitrary rotation angles. The RY gates are shown as
$U3(\theta,\phi=0,\lambda=0)$ rotation gates.

The quantum circuit is shown in Fig. 9. It is constructed using only single
qubit gates and was executed on ibmq_rochester, IBM’s $53$ superconducting
qubit platform 111Retired October 31, 2020.. The six qubit register was
executed on a subset of hardware qubits selected based on the lowest error
rates at the time of job execution. Each circuit was sampled using $8192$
shots.

Using the vector of angles found from the classical pre-processing (Section
3.2), the vector element ($\theta(t)[i]$) is passed as the argument to the RY
gate on qubit [i]). The reservoir does not include any two-qubit gates. When
deployed on a NISQ device any interactions between the reservoir nodes are
induced by hardware noise (for example: shifts in the implemented angles,
cross-talk, and readout noise) and feedback of previous output signals as
input.

The output of the reservoir at time t is a vector of average spin values of
each qubit $\mathbf{s}(t)=[s_{0}(t),\cdots,s_{5}(t)]$. Fig. 10 shows the
steady state view of the average spin of the 6 qubits in the register.

Figure 10: Steady state view of the average spin of the 6 qubits in the
register. These signals are linearly combined by an optimized weight vector to
produce the forecast.

###  3.4 Post-processing

These six spin values are linearly combined in a classical post-processing
layer using a six-dimensional, real-valued weight vector ($\mathbf{w}(t)$) to
produce the VIX forecast.

The optimal readout weights are determined by minimizing the mean-square error
(MSE) of the VIX value predicted at time ($t$). Let $\sigma_{t+1}$ represent
the actual value of the VIX at time t and $\hat{\sigma}_{t+1}$ to represent
the value predicted by the NISQ reservoir. The residual error is calculated
using the MSE:

$\begin{split}\hat{\sigma}_{t+1}&=\mathbf{w}(t)\cdot\mathbf{s}(t),\\\
\varepsilon_{t+1}&=\sigma_{t+1}-\hat{\sigma}_{t+1},\\\
\mathrm{MSE}&=\frac{1}{T}\sum\limits_{t=1}^{T}\varepsilon_{t}^{2}.\end{split}$
(7)

The histogram of residual values are shown in Fig. 11, they are shown to have
no bias.

Figure 11: Histogram of the forecasting error. Note that it shows very little
bias i.e. it is centered around zero.

At each time step, ($\mathbf{w}(t)$) is updated using newly available
information. In other words, we find at each time step the $\mathbf{w}(t)$
that gives the closest approximation for the VIX forecast using the measured
spin values.

As noted in Eq. 6, the residual error (the MSE at time-step (t)) is fed back
into the reservoir and utilized for determining the qubit rotation angle in
next time-step. This provides a negative feedback to our spin-based dynamical
system to minimize the error in the output.

###  3.5 Results

In Fig. 12 we plot the one-step ahead forecasts for the 2008 recession. We
also plot the change in VIX in Fig. 13 because for effective risk management
what matters more is change in volatility.

Figure 12: One step ahead predictions for $\Delta$VIX during the 2008
recession using the NISQ reservoir (red, dashed) compared to the actual values
(black, solid).  Figure 13: One step ahead predictions of the VIX index value
during the 2008 recession data. Values generated by the quantum reservoir
(red) and the actual VIX (blue).

##  4 Conclusion

NISQ devices are noisy by definition. Examples of noise sources are: qubit
decoherence, gate errors and readout error. Such noise can be beneficial in
machine learning related information processing tasks akin to regularization
(Noh et al (2017)). Noise induced regularization helps NISQ reservoirs to be
‘well-behaved’ and avoid taking extreme values in forecasting related tasks.
In this work we are interested in understanding how hardware noise can affect
NISQ reservoir performance. The circuit design is shallow and uses only single
qubit rotation gates. Thus, any interaction between qubits must be mediated by
noise (i.e. cross-talk) or errors induced by the measurement gate. To reliably
utilize noise-induced correlations, the interactions must be significant and
also long-lived in time. Recent studies (Dasgupta and Humble (2020); Hamilton
et al (2020)) have begun to quantify these properties of near-term quantum
devices.

In this study we developed a NISQ reservoir for the task of stochastic
volatility forecasting in finance - a highly non-linear and memory intensive
temporal information processing task which is well-suited for RC (Tanaka et al
(2019b)). Our results show that that quantum reservoirs implemented with
shallow circuits can be used for regression-type analysis in empirical finance
and also adaptable for near-term quantum processors.

Promising avenues of future work include analyzing the performance for
$\tau$-step look ahead-predictor where $\tau>1$, tuning the MC of the
reservoir to remember historical signal patterns based on a user-defined
appetite (which will lead to a trade-off with forecast accuracy), evaluating
the efficacy of the reservoir in predicting other financial time-series data
and modeling the noisy quantum dynamics accurately to understand the sources
of non-linearity.

##  5 Acknowledgements

This research used quantum computing resources of the Oak Ridge Leadership
Computing Facility, which is a DOE Office of Science User Facility supported
under Contract DE-AC05-00OR22725. This work was partially supported as part of
the ASCR QCAT Program at Oak Ridge National Laboratory under FWP #ERKJ347.
Part of the support for SD and AB came from College of Science, Purdue
University.

## Conflict of interest

The authors declare that they have no conflict of interest.

## References

  * Abraham et al (2019) Abraham H, Akhalwaya IY, Aleksandrowicz G, Alexander T, Alexandrowics G, Arbel E, Asfaw A, Azaustre C, AzizNgoueya, Barkoutsos P, Barron G, Bello L, Ben-Haim Y, Bevenius D, Bishop LS, Bosch S, Bravyi S, Bucher D, Cabrera F, Calpin P, Capelluto L, Carballo J, Carrascal G, Chen A, Chen CF, Chen R, Chow JM, Claus C, Clauss C, Cross AJ, Cross AW, Cross S, Cruz-Benito J, Culver C, Córcoles-Gonzales AD, Dague S, Dandachi TE, Dartiailh M, DavideFrr, Davila AR, Ding D, Doi J, Drechsler E, Drew, Dumitrescu E, Dumon K, Duran I, EL-Safty K, Eastman E, Eendebak P, Egger D, Everitt M, Fernández PM, Ferrera AH, Frisch A, Fuhrer A, GEORGE M, Gacon J, Gadi, Gago BG, Gambetta JM, Gammanpila A, Garcia L, Garion S, Gomez-Mosquera J, de la Puente González S, Gould I, Greenberg D, Grinko D, Guan W, Gunnels JA, Haide I, Hamamura I, Havlicek V, Hellmers J, Herok Ł, Hillmich S, Horii H, Howington C, Hu S, Hu W, Imai H, Imamichi T, Ishizaki K, Iten R, Itoko T, Javadi-Abhari A, Jessica, Johns K, Kachmann T, Kanazawa N, Kang-Bae, Karazeev A, Kassebaum P, King S, Knabberjoe, Kovyrshin A, Krishnan V, Krsulich K, Kus G, LaRose R, Lambert R, Latone J, Lawrence S, Liu D, Liu P, Maeng Y, Malyshev A, Marecek J, Marques M, Mathews D, Matsuo A, McClure DT, McGarry C, McKay D, McPherson D, Meesala S, Mevissen M, Mezzacapo A, Midha R, Minev Z, Mitchell A, Moll N, Mooring MD, Morales R, Moran N, Murali P, Müggenburg J, Nadlinger D, Nannicini G, Nation P, Naveh Y, Neuweiler P, Niroula P, Norlen H, O’Riordan LJ, Ogunbayo O, Ollitrault P, Oud S, Padilha D, Paik H, Perriello S, Phan A, Pistoia M, Pozas-iKerstjens A, Prutyanov V, Puzzuoli D, Pérez J, Quintiii, Raymond R, Redondo RMC, Reuter M, Rice J, Rodríguez DM, Rossmannek M, Ryu M, SAPV T, SamFerracin, Sandberg M, Sathaye N, Schmitt B, Schnabel C, Schoenfeld Z, Scholten TL, Schoute E, Schwarm J, Sertage IF, Setia K, Shammah N, Shi Y, Silva A, Simonetto A, Singstock N, Siraichi Y, Sitdikov I, Sivarajah S, Sletfjerding MB, Smolin JA, Soeken M, Sokolov IO, SooluThomas, Steenken D, Stypulkoski M, Suen J, Takahashi H, Tavernelli I, Taylor C, Taylour P, Thomas S, Tillet M, Tod M, de la Torre E, Trabing K, Treinish M, TrishaPe, Turner W, Vaknin Y, Valcarce CR, Varchon F, Vazquez AC, Vogt-Lee D, Vuillot C, Weaver J, Wieczorek R, Wildstrom JA, Wille R, Winston E, Woehr JJ, Woerner S, Woo R, Wood CJ, Wood R, Wood S, Wootton J, Yeralin D, Young R, Yu J, Zachow C, Zdanski L, Zoufal C, Zoufalc, azulehner, bcamorrison, brandhsn, chlorophyll zz, dan1pal, dime10, drholmie, elfrocampeador, faisaldebouni, fanizzamarco, gruu, kanejess, klinvill, kurarrr, lerongil, ma5x, merav aharoni, ordmoj, sethmerkel, strickroman, sumitpuri, tigerjack, toural, vvilpas, welien, willhbang, yangluh, yelojakit, yotamvakninibm (2019) Qiskit: An open-source framework for quantum computing. DOI 10.5281/zenodo.2562110
  * CBOE (2019a) CBOE CGM (2019a) CBOE VIX whitepaper. Tech. rep., CBOE, Chicago, Illinois, URL https://www.cboe.com/micro/vix/vixwhite.pdf, accessed Feb 21, 2020
  * CBOE (2019b) CBOE CGM (2019b) The relationship of the SPX and the VIX index. URL www.cboe.com/products/vix-index-volatility/vix-options-and-futures/vix-index/, accessed Mar 27, 2020
  * Chen and Nurdin (2019) Chen J, Nurdin HI (2019) Learning nonlinear input–output maps with dissipative quantum systems. Quantum Information Processing 18(7):198
  * Chen et al (2020) Chen J, Nurdin HI, Yamamoto N (2020) Temporal information processing on noisy quantum computers. arXiv preprint arXiv:200109498
  * Dambre et al (2012) Dambre J, Verstraeten D, Schrauwen B, Massar S (2012) Information processing capacity of dynamical systems. Scientific reports 2(1):1–7
  * Dasgupta and Humble (2020) Dasgupta S, Humble TS (2020) Characterizing the stability of nisq devices. 2008.09612
  * Farkaš et al (2016) Farkaš I, Bosák R, Gergel’ P (2016) Computational analysis of memory capacity in echo state networks. Neural Networks 83:109–120
  * Fujii and Nakajima (2017) Fujii K, Nakajima K (2017) Harnessing disordered-ensemble quantum dynamics for machine learning. Physical Review Applied 8(2):024,030
  * Gerstner et al (2014) Gerstner W, Kistler WM, Naud R, Paninski L (2014) Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press
  * Ghosh et al (2019) Ghosh S, Opala A, Matuszewski M, Paterek T, Liew TC (2019) Quantum reservoir processing. npj Quantum Information 5(1):1–6
  * Govia et al (2020) Govia L, Ribeill G, Rowlands G, Krovi H, Ohki T (2020) Quantum reservoir computing with a single nonlinear oscillator. arXiv preprint arXiv:200414965
  * Hamilton et al (2020) Hamilton KE, Kharazi T, Morris T, McCaskey AJ, Bennink RS, Pooser RC (2020) Scalable quantum processor noise characterization. arXiv preprint arXiv:200601805
  * Hudson and Gregoriou (2015) Hudson RS, Gregoriou A (2015) Calculating and comparing security returns is harder than you think: A comparison between logarithmic and simple returns. International Review of Financial Analysis 38:151–162
  * Inubushi and Yoshimura (2017) Inubushi M, Yoshimura K (2017) Reservoir computing beyond memory-nonlinearity trade-off. Scientific reports 7(1):1–10
  * Kia et al (2017) Kia B, Lindner JF, Ditto WL (2017) Nonlinear dynamics as an engine of computation. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 375(2088):20160,222
  * Kutvonen et al (2020) Kutvonen A, Fujii K, Sagawa T (2020) Optimizing a quantum reservoir computer for time series prediction. Scientific reports 10(1):1–7
  * McNeil et al (2015) McNeil AJ, Frey R, Embrechts P (2015) Quantitative risk management: concepts, techniques and tools-revised edition. Princeton University Press
  * Nakajima et al (2019) Nakajima K, Fujii K, Negoro M, Mitarai K, Kitagawa M (2019) Boosting computational power through spatial multiplexing in quantum reservoir computing. Physical Review Applied 11(3):034,021
  * Noh et al (2017) Noh H, You T, Mun J, Han B (2017) Regularizing deep neural networks by noise: Its interpretation and optimization. In: Advances in Neural Information Processing Systems, pp 5109–5118
  * Robinson (2018) Robinson P (2018) A guide to SP500 VIX index. URL www.dailyfx.com/sp-500/guide-to-sp-500-vix-index.html, accessed Mar 27, 2020
  * Tanaka et al (2019a) Tanaka G, Yamane T, Héroux JB, Nakane R, Kanazawa N, Takeda S, Numata H, Nakano D, Hirose A (2019a) Recent advances in physical reservoir computing: A review. Neural Networks 115:100–123
  * Tanaka et al (2019b) Tanaka G, Yamane T, HÃ©roux JB, Nakane R, Kanazawa N, Takeda S, Numata H, Nakano D, Hirose A (2019b) Recent advances in physical reservoir computing: A review. Neural Networks 115:100 – 123, DOI https://doi.org/10.1016/j.neunet.2019.03.005, URL http://www.sciencedirect.com/science/article/pii/S0893608019300784

##  Appendix A Appendix

###  A.1 High level overview of reservoir computing

Classical reservoir computing (RC) relies on a reservoir of randomly connected
oscillators. The connections between the oscillators in the reservoir are not
trained. In this computational framework, inputs are mapped to a high
dimensional space and the output from the high dimensional state is trained to
predict the desired function using a simple method like linear regression. RC
using a simple readout is suited to low-cost real-time computing history
dependent dynamical responses to external inputs. Let $\mathbf{x}(n)$ denote
the reservoir state vector:

$\mathbf{x}(n)=\begin{bmatrix}x_{0}(n)\\\ x_{1}(n)\\\ \vdots\\\
x_{N-1}(n)\end{bmatrix}$ (8)

Here each $x_{i}$ represents the state of a node in the reservoir. This state
vector undergoes a non-linear evolution in time.

Quantum Reservoir Computing (QRC) is a new, alternative paradigm for
information processing using quantum physics. It exploits natural quantum
dynamics of ensemble systems for machine learning. The key is to find an
appropriate form of physics that exhibits rich dynamics, thereby allowing us
to outsource a part of the computation. There have been several applications
of QRC most notably time-dependent signal processing, speech recognition, NLP,
sequential motor control of robots, and stock market predictions. QRC does not
require any sophisticated quantum gate (natural dynamics is enough). Thus it
exhibits high feasibility. Numerical experiments show that quantum systems
consisting of 5–7 qubits possess computational capabilities comparable to
conventional recurrent neural networks of 100 to 500 nodes (Fujii and Nakajima
(2017)).

What are the sufficient criterion for non-von-Neumann architectures like the
brain-inspired reservoir computers? We do not know yet. Unlike traditional
neural networks, we do not understand the guiding principles of reservoir
design for high-performance information processing. Leveraging the work of
several researchers in this field, we give a brief overview here of the
considerations which seem to matter the most when using a reservoir computer
for time-series forecasting.

  1. 1.

Common Signal Induced Synchronization: If the reservoir has two different
initial state $s(t_{0})$ and $\hat{s}(t_{0})$, then, if provided with the same
input stimuli $\\{u(t)\\}_{t\geq t_{0}}$, it must satisfy,

$||s(t)-\hat{s}(t)||\rightarrow 0\textrm{ as }t\rightarrow\infty.$ (9)

Another way of stating this is that the reservoir must have fading memory
(also know as echo state property in literature): the outputs of the dynamical
system should stay close if the corresponding input are close in recent times
(Inubushi and Yoshimura (2017)). This can be viewed as a consistency or
convergence criterion, it ensures that any computation performed by the
reservoir is independent of its initial condition.

  2. 2.

Reservoir Dimensionality: A reservoir should have adequate (preferably
exponential in number of nodes) linearly independent internal variables. The
number of linearly independent variables of the NISQ reservoir (the Hilbert
space dimension) gives an upper limit on the computational capacity. As noted
in (Ghosh et al (2019)) prediction accuracy improves as you increase the
number of nodes in the system.

  3. 3.

Adequate Memory: A reservoir can have memory of past inputs (Farkaš et al
(2016)). Using a one qubit reservoir for simplicity, let’s understand how
memory manifests in a dynamical system. Suppose $u(t)$ and $\hat{u}(t)$ are
two identical time series, except for a small perturbation at $t=t_{0}-1$:

$\begin{split}&\hat{u}(t_{0}-1)=u(t_{0}-1)+\Delta\textrm{, for }t=t_{0}-1,\\\
&\hat{u}(t)=u(t)\textrm{, for all }t\neq t_{0}-1.\end{split}$

When we feed $u(t)$ or $\hat{u}(t)$ into the quantum circuit, we get the spin
time series $\\{s(t)\\}$ and $\\{\hat{s}(t)\\}$ respectively. If $\delta
s(t)=s(t)-\hat{s}(t)$ denotes the difference between the outputs $s(t)$ and
$\hat{s}(t)$, then we say the reservoir has memory when $\delta s(t)$ and
$\delta s(0)$ are related (i.e. $\delta s(t)$ can provide information about
$\delta s(0)$). Higher mutual information between $\delta s(t)$ and $\delta
s(0)$ implies higher MC. A formal proof is given in (Inubushi and Yoshimura
(2017)). A linear circuit has higher MC as $\delta s(t)$ is strongly
correlated with $\delta s(0)$. Thus high degree of linearity is more suitable
for forecasting tasks which need to recall historical patterns. This implies
that to introduce linear elements in the NISQ reservoir we will need to
introduce ‘self-loops’ in the spin-system.

  4. 4.

Response Separability: The separation property is the reservoir’s capability
to generate dynamics sufficiently rich that can can distinguish between any
two different input sequences. This is important because it is not enough that
the reservoir is excitable by the input sequence you care about. It should be
excitable by any distinguishable inputs and the (input history dependent)
response should be adequately distinguishable (Tanaka et al (2019b)).

  5. 5.

Adequate Non-linearity: Non-linearity is required for effective functioning of
reservoir computers to address the ’linearly inseparable problem’ (Kia et al
(2017)) A non-linear transformation is mandatory for tasks such as
classification by support vector machines. This property turns out to be
crucial for achieving universal computing. However, non-linearity also
degrades memory. Thus a careful trade-off is required between the linear and
non-linear elements of the circuit.

  6. 6.

Edge Density: Edge density is a system level metric (as opposed to node level
metric) that is an important driver of the predictive power achieved by a
hybrid reservoir. We quantitatively define edge density as the ratio of the
total number of edges present in the reservoir configuration to the total
number of possible edges. A discussion on how heightened non-linearity in the
system due to increased connectivity leads to MC degradation can be found in
(Inubushi and Yoshimura (2017)).

  7. 7.

Feedback Strength: To be an effective forecasting engine, the reservoir has to
strike a balance between two competing aims: memorizing past patterns (which
is related to over-fit reduction) and reducing mean square error (which is
related to fit accuracy). The former requirement asks for the ‘state signal’
to play a dominant role (as the reservoir memorizes through the time evolution
of its quantum spin state) while the latter pushes the ‘incoming signal
pattern’ to have more weighting. This tunable parameter can be used in the
system evolution specification.

  8. 8.

Noise induced regularization: It is well-known that it is possible to use
dissipative quantum systems as universal function approximators for temporal
information processing even in the presence of noise. Such noise can be
beneficial in machine learning related information processing tasks. It plays
a role akin to regularization (Noh et al (2017)). The phrase ‘to regularize’
means ‘to make more acceptable’. Function approximators become more acceptable
when they ‘train’ on ’noisy’ data and thereby avoid over-fitting. Thus noise
induced regularization helps NISQ reservoirs to be ‘well-behaved’ and avoid
taking extreme values in forecasting related tasks.

###  A.2 Results for NARMA benchmarking

The Non-linear Auto-regressive Moving Average (NARMA) series is a forecasting
task that is commonly employed as a performance benchmark. It has a high
degree of non-linearity and dependence on long time lags, leading to
significant memory requirements in the forecasting model. We use one step
ahead forecasting of the NARMA5 series to benchmark the performance of our
quantum reservoir construction. This benchmark was executed using simulated
noisy qubits with the noise modeling capabilities available in Qiskit (Abraham
et al (2019)). The NARMA5 series is a temporal sequence defined by:

$\begin{split}v_{t+1}=&\alpha v_{t}+\beta
v_{t}(v_{t}+v_{t-1}+v_{t-2}+v_{t-3}+v_{t-4})+\\\ &\gamma
s_{t-4}s_{t}+\delta,\\\ s_{t}=&\mu\left[sin\frac{2\pi f_{0}t}{T}sin\frac{2\pi
f_{1}t}{T}sin\frac{2\pi f_{2}t}{T}+1\right].\end{split}$ (10)

The parameters in Eq. 10 are:
$\alpha=0.30,\beta=0.05,\gamma=1.50,\delta=0.10,\mu=0.10$, and
$f_{0}=2.11,f_{1}=3.73,f_{2}=4.11,T=100$. These values were originally used in
Fujii and Nakajima (2017) to benchmark quantum reservoirs.

Figure 14: One-step ahead predictions for the NARMA-5 time-series with the
quantum reservoir executed with noisy simulation in Qiskit. Figure 15:
Histogram of normalized mean square error for the NARMA5 prediction task.

Fig. 14 shows the comparison of realized vs predicted time-series for the
NARMA5 task. Only a zoomed-in snapshot is shown of the 5000 point long
sequence. The initial one-third of the data was flushed out to allow the
system to stabilize. The same optimal configuration that was utilized for VIX
forecasting (as discussed in the main text), was also employed here. Our
hybrid reservoir achieved an NMSE of $6\times 10^{-4}$. One can compare this
to the NMSE obtained in (Fujii and Nakajima (2017)) which lied in the range
$[3\times 10^{-3},7.6\times 10^{-6}]$. Thus, the benchmark performance of our
hybrid reservoir is comparable to the benchmark performance found in (Fujii
and Nakajima (2017)). As in the VIX prediction task, we observe low bias in
the prediction error (see Fig. 15).

###  A.3 Memory capacity of larger reservoirs

In the main text we focused on reservoirs with $6$ qubits. We also tested the
performance for quantum registers of different sizes. As an example, the
memory capacity (MC) characterization described in Section 2 is repeated for
an 8 qubit hybrid reservoir. The sequence of edge densities follow the same
sequence as shown in Fig. 2 but for an 8 qubit reservoir there are now $36$
graphs. In Fig. 16 we again observe a peak in the MC that occurs for the
reservoir with $n-1=7$ self-loops.

Figure 16: Variation of Memory Capacity with reservoir complexity for a
8-qubit quantum register on ibmq_rochester.