# Towards Accurate Binary Convolutional Neural Network

Xiaofan Lin Cong Zhao Wei Pan*
DJI Innovations Inc, Shenzhen, China
{xiaofan.lin, cong.zhao<EMAIL_ADDRESS>
###### Abstract

We introduce a novel scheme to train binary convolutional neural networks
(CNNs) – CNNs with weights and activations constrained to {-1,+1} at run-time.
It has been known that using binary weights and activations drastically reduce
memory size and accesses, and can replace arithmetic operations with more
efficient bitwise operations, leading to much faster test-time inference and
lower power consumption. However, previous works on binarizing CNNs usually
result in severe prediction accuracy degradation. In this paper, we address
this issue with two major innovations: (1) approximating full-precision
weights with the linear combination of multiple binary weight bases; (2)
employing multiple binary activations to alleviate information loss. The
implementation of the resulting binary CNN, denoted as ABC-Net, is shown to
achieve much closer performance to its full-precision counterpart, and even
reach the comparable prediction accuracy on ImageNet and forest trail
datasets, given adequate binary weight bases and activations.

††footnotetext: $*$ indicates corresponding author.

##  1 Introduction

Convolutional neural networks (CNNs) have achieved state-of-the-art results on
real-world applications such as image classification (He et al., 2016) and
object detection (Ren et al., 2015), with the best results obtained with large
models and sufficient computation resources. Concurrent to these progresses,
the deployment of CNNs on mobile devices for consumer applications is gaining
more and more attention, due to the widespread commercial value and the
exciting prospect.

On mobile applications, it is typically assumed that training is performed on
the server and test or inference is executed on the mobile devices
(Courbariaux et al., 2016; Esser et al., 2016). In the training phase, GPUs
enabled substantial breakthroughs because of their greater computational
speed. In the test phase, however, GPUs are usually too expensive to deploy.
Thus improving the test-time performance and reducing hardware costs are
likely to be crucial for further progress, as mobile applications usually
require real-time, low power consumption and fully embeddable. As a result,
there is much interest in research and development of dedicated hardware for
deep neural networks (DNNs). Binary neural networks (BNNs) (Courbariaux et
al., 2016; Rastegari et al., 2016), i.e., neural networks with weights and
perhaps activations constrained to only two possible values (e.g., -1 or +1),
would bring great benefits to specialized DNN hardware for three major
reasons: (1) the binary weights/activations reduce memory usage and model size
32 times compared to single-precision version; (2) if weights are binary, then
most multiply-accumulate operations can be replaced by simple accumulations,
which is beneficial because multipliers are the most space and power-hungry
components of the digital implementation of neural networks; (3) furthermore,
if both activations and weights are binary, the multiply-accumulations can be
replaced by the bitwise operations: xnor and bitcount Courbariaux et al.
(2016). This could have a big impact on dedicated deep learning hardware. For
instance, a 32-bit floating point multiplier costs about 200 Xilinx FPGA
slices (Govindu et al., 2004), whereas a 1-bit xnor gate only costs a single
slice. Semiconductor manufacturers like IBM (Esser et al., 2016) and Intel
(Venkatesh et al., 2016) have been involved in the research and development of
related chips.

However, binarization usually cause severe prediction accuracy degradation,
especially on complex tasks such as classification on ImageNet dataset. To
take a closer look, Rastegari et al. (2016) shows that binarizing weights
causes the accuracy of Resnet-18 drops from 69.3% to 60.8% on ImageNet
dataset. If further binarize activations, the accuracy drops to 51.2%. Similar
phenomenon can also be found in literatures such as (Hubara et al., 2016).
Clearly there is a considerable gap between the accuracy of a full-precision
model and a binary model.

This paper proposes a novel scheme for binarizing CNNs, which aims to
alleviate, or even eliminate the accuracy degradation, while still
significantly reducing inference time, resource requirement and power
consumption. The paper makes the following major contributions.

  * •

We approximate full-precision weights with the linear combination of multiple
binary weight bases. The weights values of CNNs are constrained to
$\\{-1,+1\\}$, which means convolutions can be implemented by only addition
and subtraction (without multiplication), or bitwise operation when
activations are binary as well. We demonstrate that 3$\sim$5 binary weight
bases are adequate to well approximate the full-precision weights.

  * •

We introduce multiple binary activations. Previous works have shown that the
quantization of activations, especially binarization, is more difficult than
that of weights (Cai et al., 2017; Courbariaux et al., 2016). By employing
five binary activations, we have been able to reduce the Top-1 and Top-5
accuracy degradation caused by binarization to around 5% on ImageNet compared
to the full precision counterpart.

It is worth noting that the multiple binary weight bases/activations scheme is
preferable to the fixed-point quantization in previous works. In those fixed-
point quantized networks one still needs to employ arithmetic operations, such
as multiplication and addition, on fixed-point values. Even though faster than
floating point, they still require relatively complex logic and can consume a
lot of power. Detailed discussions can be found in Section 5.2.

Ideally, combining more binary weight bases and activations always leads to
better accuracy and will eventually get very close to that of full-precision
networks. We verify this on ImageNet using Resnet network topology. This is
the first time a binary neural network achieves prediction accuracy comparable
to its full-precision counterpart on ImageNet.

##  2 Related work

Quantized Neural Networks: High precision parameters are not very necessary to
reach high performance in deep neural networks. Recent research efforts (e.g.,
(Hubara et al., 2016)) have considerably reduced a large amounts of memory
requirement and computation complexity by using low bitwidth weights and
activations. Zhou et al. (2016) further generalized these schemes and proposed
to train CNNs with low bitwidth gradients. By performing the quantization
after network training or using the “straight-through estimator (STE)" (Bengio
et al., 2013), these works avoided the issues of non-differentiable
optimization. While some of these methods have produced good results on
datasets such as CIFAR-10 and SVHN, none has produced low precision networks
competitive with full-precision models on large-scale classification tasks,
such as ImageNet. In fact, (Zhou et al., 2016) and (Hubara et al., 2016)
experiment with different combinations of bitwidth for weights and
activations, and show that the performance of their highly quantized networks
deteriorates rapidly when the weights and activations are quantized to less
than 4-bit numbers. Cai et al. (2017) enhance the performance of a low
bitwidth model by addressing the gradient mismatch problem, nevertheless there
is still much room for improvement.

Binarized Neural Networks: The binary representation for deep models is not a
new topic. At the emergence of artificial neural networks, inspired
biologically, the unit step function has been used as the activation function
(Toms, 1990). It is known that binary activation can use spiking response for
event-based computation and communication (consuming energy only when
necessary) and therefore is energy-efficient (Esser et al., 2016). Recently,
Courbariaux et al. (2016) introduce Binarized-Neural-Networks (BNNs), neural
networks with binary weights and activations at run-time. Different from their
work, Rastegari et al. (2016) introduce simple, efficient, and accurate
approximations to CNNs by binarizing the weights and even the intermediate
representations in CNNs. All these works drastically reduce memory
consumption, and replace most arithmetic operations with bitwise operations,
which potentially lead to a substantial increase in power efficiency.

In all above mentioned works, binarization significantly reduces accuracy. Our
experimental results on ImageNet show that we are close to filling the gap
between the accuracy of a binary model and its full-precision counterpart. We
relied on the idea of finding the best approximation of full-precision
convolution using multiple binary operations, and employing multiple binary
activations to allow more information passing through.

##  3 Binarization methods

In this section, we detail our binarization method, which is termed ABC-Net
(Accurate-Binary-Convolutional) for convenience. Bear in mind that during
training, the real-valued weights are reserved and updated at every epoch,
while in test-time only binary weights are used in convolution.

###  3.1 Weight approximation

Consider a $L$-layer CNN architecture. Without loss of generality, we assume
the weights of each convolutional layer are tensors of dimension $(w,h,c_{\rm
in},c_{\rm out})$, which represents filter width, filter height, input-channel
and output-channel respectively. We propose two variations of binarization
method for weights at each layer: 1) approximate weights as a whole and 2)
approximate weights channel-wise.

####  3.1.1 Approximate weights as a whole

At each layer, in order to constrain a CNN to have binary weights, we estimate
the real-value weight filter ${\bm{W}}\in\mathbb{R}^{w\times h\times c_{\rm
in}\times c_{\rm out}}$ using the linear combination of $M$ binary filters
${\bm{B}}_{1},{\bm{B}}_{2},\cdots,{\bm{B}}_{M}\in\\{-1,+1\\}^{w\times h\times
c_{\rm in}\times c_{\rm out}}$ such that
${\bm{W}}\approx\alpha_{1}{\bm{B}}_{1}+\alpha_{2}{\bm{B}}_{2}+\cdots+\alpha_{M}{\bm{B}}_{M}$.
To find an optimal estimation, a straightforward way is to solve the following
optimization problem:

$\begin{split}\min\limits_{{\bm{\alpha}},{\bm{B}}}&J({\bm{\alpha}},{\bm{B}})=||{\bm{w}}-{\bm{B}}{\bm{\alpha}}||^{2},\quad\text{s.t.}\
{\bm{B}}_{ij}\in\\{-1,+1\\},\end{split}$ (1)

where ${\bm{B}}=[{\rm vec}({\bm{B}}_{1}),{\rm vec}({\bm{B}}_{2}),\cdots,{\rm
vec}({\bm{B}}_{M})]$, ${\bm{w}}={\rm vec}({\bm{W}})$ and
${\bm{\alpha}}=[\alpha_{1},\alpha_{2},\cdots,\alpha_{M}]^{\rm T}$. Here the
notation ${\rm vec}(\cdot)$ refers to vectorization.

Although a local minimum solution to (1) can be obtained by numerical methods,
one could not backpropagate through it to update the real-value weight filter
${\bm{W}}$. To address this issue, assuming the mean and standard deviation of
${\bm{W}}$ are ${\rm mean}({\bm{W}})$ and ${\rm std}({\bm{W}})$ respectively,
we fix ${\bm{B}}_{i}$’s as follows:

${\bm{B}}_{i}=F_{u_{i}}({\bm{W}}):={\rm sign}({\bm{\bar{W}}}+u_{i}{\rm
std}({\bm{W}})),i=1,2,\cdots,M,$ (2)

where ${\bm{\bar{W}}}={\bm{W}}-{\rm mean}({\bm{W}}),$ and $u_{i}$ is a shift
parameter. For example, one can choose $u_{i}$’s to be
$u_{i}=-1+(i-1)\frac{2}{M-1},i=1,2,\cdots,M,$ to shift evenly over the range
$\left[-{\rm std}({\bm{W}}),{\rm std}({\bm{W}})\right]$, or leave it to be
trained by the network. This is based on the observation that the full-
precision weights tend to have a symmetric, non-sparse distribution, which is
close to Gaussian. To gain more intuition and illustrate the approximation
effectiveness, an example is visualized in Section S2 of the supplementary
material.

With ${\bm{B}}_{i}$’s chosen, (1) becomes a linear regression problem

$\min\limits_{{\bm{\alpha}}}J({\bm{\alpha}})=||{\bm{w}}-{\bm{B}}{\bm{\alpha}}||^{2},$
(3)

in which ${\bm{B}}_{i}$’s serve as the bases in the design/dictionary matrix.
We can then back-propagate through ${\bm{B}}_{i}$’s using the “straight-
through estimator” (STE) (Bengio et al., 2013). Assume $c$ as the cost
function, ${\bm{A}}$ and ${\bm{O}}$ as the input and output tensor of a
convolution respectively, the forward and backward approach of an approximated
convolution during training can be computed as follows:

Forward:
$\displaystyle{\bm{B}}_{1},{\bm{B}}_{2},\cdots,{\bm{B}}_{M}=F_{u_{1}}({\bm{W}}),F_{u_{2}}({\bm{W}}),\cdots,F_{u_{M}}({\bm{W}}),$
(4) $\displaystyle\textrm{Solve \eqref{ls_problem_B-fixed} for
${\bm{\alpha}}$},$ (5)
$\displaystyle{\bm{O}}=\sum\limits_{m=1}^{M}\alpha_{m}{\rm
Conv}({\bm{B}}_{m},{\bm{A}}).$ (6) Backward: $\displaystyle\frac{\partial
c}{\partial{\bm{W}}}=\frac{\partial
c}{\partial{\bm{O}}}\left(\sum_{m=1}^{M}\alpha_{m}\frac{\partial{\bm{O}}}{\partial{\bm{B}}_{m}}\frac{\partial{\bm{B}}_{m}}{\partial{\bm{W}}}\right)\stackrel{{\scriptstyle\text{STE}}}{{=}}\frac{\partial
c}{\partial{\bm{O}}}\left(\sum_{m=1}^{M}\alpha_{m}\frac{\partial{\bm{O}}}{\partial{\bm{B}}_{m}}\right)=\sum_{m=1}^{M}\alpha_{m}\frac{\partial
c}{\partial{\bm{B}}_{m}}.$ (7)

In test-time, only (6) is required. The block structure of this approximated
convolution layer is shown on the left side in Figure 1. With suitable
hardwares and appropriate implementations, the convolution can be efficiently
computed. For example, since the weight values are binary, we can implement
the convolution with additions and subtractions (thus without
multiplications). Furthermore, if the input ${\bm{A}}$ is binary as well, we
can implement the convolution with bitwise operations: xnor and bitcount
(Rastegari et al., 2016). Note that the convolution with each binary filter
can be computed in parallel.

Figure 1: An example of the block structure of the convolution in ABC-Net.
$M=N=3$. On the left is the structure of the approximated convolution
(ApproxConv). ApproxConv is expected to approximate the conventional full-
precision convolution with linear combination of binary convolutions
(BinConv), i.e., convolution with binary and weights. On the right is the
overall block structure of the convolution in ABC-Net. The input is binarized
using different functions $H_{v_{1}},H_{v_{2}},H_{v_{3}}$, passed into the
corresponding ApproxConv’s and then summed up after multiplying their
corresponding $\beta_{n}$’s. With the input binarized, the BinConv’s can be
implemented with highly efficient bitwise operations. There are 9 BinConv’s in
this example and they can work in parallel.

####  3.1.2 Approximate weights channel-wise

Alternatively, we can estimate the real-value weight filter
${\bm{W}}_{i}\in\mathbb{R}^{w\times h\times c_{\rm in}}$ of each output
channel $i\in\\{1,2,\cdots,c_{\rm out}\\}$ using the linear combination of $M$
binary filters
${\bm{B}}_{i1},{\bm{B}}_{i2},\cdots,{\bm{B}}_{iM}\in\\{-1,+1\\}^{w\times
h\times c_{\rm in}}$ such that
${\bm{W}}_{i}\approx\alpha_{i1}{\bm{B}}_{i1}+\alpha_{i2}{\bm{B}}_{i2}+\cdots+\alpha_{iM}{\bm{B}}_{iM}$.
Again, to find an optimal estimation, we solve a linear regression problem
analogy to (3) for each output channel. After convolution, the results are
concatenated together along the output-channel dimension. If $M=1$, this
approach reduces to the Binary-Weights-Networks (BWN) proposed in (Rastegari
et al., 2016).

Compared to weights approximation as a whole, the channel-wise approach
approximates weights more elaborately, however no extra cost is needed during
inference. Since this approach requires more computational resources during
training, we leave it as a future work and focus on the former approximation
approach in this paper.

###  3.2 Multiple binary activations and bitwise convolution

As mentioned above, a convolution can be implemented without multiplications
when weights are binarized. However, to utilize the bitwise operation, the
activations must be binarized as well, as they are the inputs of convolutions.

Similar to the activation binarization procedure in (Zhou et al., 2016), we
binarize activations after passing it through a bounded activation function
$h$, which ensures $h(x)\in[0,1]$. We choose the bounded rectifier as $h$.
Formally, it can be defined as:

$h_{v}(x)={\rm clip}(x+v,0,1),$ (8)

where $v$ is a shift parameter. If $v=0$, then $h_{v}$ is the clip activation
function in (Zhou et al., 2016).

We constrain the binary activations to either 1 or -1. In order to transform
the real-valued activation ${\bm{R}}$ into binary activation, we use the
following binarization function:

$H_{v}({\bm{R}}):=2\mathbb{I}_{h_{v}({\bm{R}})\geq 0.5}-1,$ (9)

where $\mathbb{I}$ is the indicator function. The conventional forward and
backward approach of the activation can be given as follows:

Forward: $\displaystyle{\bm{A}}=H_{v}({\bm{R}}).$ (10) Backward:
$\displaystyle\frac{\partial c}{\partial{\bm{R}}}=\frac{\partial
c}{\partial{\bm{A}}}\circ\mathbb{I}_{0\leq{\bm{R}}-v\leq 1}.\quad{\rm(using\
STE)}$

Here $\circ$ denotes the Hadamard product. As can be expected, binaizing
activations as above is kind of crude and leads to non-trivial losses in
accuracy, as shown in Rastegari et al. (2016); Hubara et al. (2016). While it
is also possible to approximate activations with linear regression, as that of
weights, another critical challenge arises – unlike weights, the activations
always vary in test-time inference. Luckily, this difficulty can be avoided by
exploiting the statistical structure of the activations of deep networks.

Our scheme can be described as follows. First of all, to keep the distribution
of activations relatively stable, we resort to batch normalization (Ioffe and
Szegedy, 2015). This is a widely used normalization technique, which forces
the responses of each network layer to have zero mean and unit variance. We
apply this normalization before activation. Secondly, we estimate the real-
value activation ${\bm{R}}$ using the linear combination of $N$ binary
activations ${\bm{A}}_{1},{\bm{A}}_{2},\cdots,{\bm{A}}_{N}$ such that
${\bm{R}}\approx\beta_{1}{\bm{A}}_{1}+\beta_{2}{\bm{A}}_{2}+\cdots+\beta_{N}{\bm{A}}_{N}$,
where

${\bm{A}}_{1},{\bm{A}}_{2},\cdots,{\bm{A}}_{N}=H_{v_{1}}({\bm{R}}),H_{v_{2}}({\bm{R}}),\cdots,H_{v_{N}}({\bm{R}}).$
(11)

Different from that of weights, the parameters $\beta_{n}$’s and $v_{n}$’s
($n=1,\cdots,N$) here are both trainable, just like the scale and shift
parameters in batch normalization. Without the explicit linear regression
approach, $\beta_{n}$’s and $v_{n}$’s are tuned by the network itself during
training and fixed in test-time. They are expected to learn and utilize the
statistical features of full-precision activations.

The resulting network architecture outputs multiple binary activations
${\bm{A}}_{1},{\bm{A}}_{2},\cdots,{\bm{A}}_{N}$ and their corresponding
coefficients $\beta_{1},\beta_{2},\cdots,\beta_{N}$, which allows more
information passing through compared to the former one. Combining with the
weight approximation, the whole convolution scheme is given by:

${\rm Conv}({\bm{W}},{\bm{R}})\approx{\rm
Conv}\left(\sum_{m=1}^{M}\alpha_{m}{\bm{B}}_{m},\sum_{n=1}^{N}\beta_{n}{\bm{A}}_{n}\right)\\\
=\sum_{m=1}^{M}\sum_{n=1}^{N}\alpha_{m}\beta_{n}{\rm
Conv}\left({\bm{B}}_{m},{\bm{A}}_{n}\right),$ (12)

which suggests that it can be implemented by computing $M\times N$ bitwise
convolutions in parallel. An example of the whole convolution scheme is shown
in Figure 1.

###  3.3 Training algorithm

A typical block in CNN contains several different layers, which are usually in
the following order: (1) Convolution, (2) Batch Normalization, (3) Activation
and (4) Pooling. The batch normalization layer (Ioffe and Szegedy, 2015)
normalizes the input batch by its mean and variance. The activation is an
element-wise non-linear function (e.g., Sigmoid, ReLU). The pooling layer
applies any type of pooling (e.g., max,min or average) on the input batch. In
our experiment, we observe that applying max-pooling on binary input returns a
tensor that most of its elements are equal to +1, resulting in a noticeable
drop in accuracy. Similar phenomenon has been reported in Rastegari et al.
(2016) as well. Therefore, we put the max-pooling layer before the batch
normalization and activation.

Since our binarization scheme approximates full-precision weights, using the
full-precision pre-train model serves as a perfect initialization. However,
fine-tuning is always required for the weights to adapt to the new network
structure. The training procedure, i.e., ABC-Net, is summarized in Section S1
of the supplementary material.

It is worth noting that as $M$ increases, the shift parameters get closer and
the bases of the linear combination are more correlated, which sometimes lead
to rank deficiency when solving (3). This can be tackled with the $\ell_{2}$
regularization.

##  4 Experiment results

In this section, the proposed ABC-Net was evaluated on the ILSVRC12 ImageNet
classification dataset (Deng et al., 2009), and visual perception of forest
trails datasets for mobile robots (Giusti et al., 2016) in Section S6 of
supplementary material.

###  4.1 Experiment results on ImageNet dataset

The ImageNet dataset contains about 1.2 million high-resolution natural images
for training that spans 1000 categories of objects. The validation set
contains 50k images. We use Resnet ((He et al., 2016)) as network topology.
The images are resized to 224x224 before fed into the network. We report our
classification performance using Top-1 and Top-5 accuracies.

####  4.1.1 Effect of weight approximation

We first evaluate the weight approximation technique by examining the accuracy
improvement for a binary model. To eliminate variables, we leave the
activations being full-precision in this experiment. Table 1 shows the
prediction accuracy of ABC-Net on ImageNet with different choices of $M$. For
comparison, we add the results of Binary-Weights-Network (denoted ‘BWN’)
reported in Rastegari et al. (2016) and the full-precision network (denoted
‘FP’). The BWN binarizes weights and leaves the activations being full-
precision as we do. All results in this experiment use Resnet-18 as network
topology. It can be observed that as $M$ increases, the accuracy of ABC-Net
converges to its full-precision counterpart. The Top-1 gap between them
reduces to only 0.9 percentage point when $M=5$, which suggests that this
approach nearly eliminates the accuracy degradation caused by binarizing
weights.

Table 1: Top-1 (left) and Top-5 (right) accuracy of ABC-Net on ImageNet, using full-precision activation and different choices of the number of binary weight bases $M$. | ${\rm BWN}$ | $M=1$ | $M=2$ | $M=3$ | $M=5$ | ${\rm FP}$
---|---|---|---|---|---|---
Top-1 | 60.8% | 62.8% | 63.7% | 66.2% | 68.3% | 69.3%
Top-5 | 83.0% | 84.4% | 85.2% | 86.7% | 87.9% | 89.2%

For interested readers, Figure S4 in section S5 of the supplementary material
shows that the relationship between accuracy and $M$ appears to be linear.
Also, in Section S2 of the supplementary material, a visualization of the
approximated weights is provided.

####  4.1.2 Configuration space exploration

We explore the configuration space of combinations of number of weight bases
and activations. Table 2 presents the results of ABC-Net with different
configurations. The parameter settings for these experiments are provided in
Section S4 of the supplementary material.

Table 2:  Prediction accuracy (Top-1/Top-5) for ImageNet with different choices of $M$ and $N$ in a ABC-Net (approximate weights as a whole). “res18”, “res34” and “res50” are short for Resnet-18, Resnet-34 and Resnet-50 network topology respectively. $M$ and $N$ refer to the number of weight bases and activations respectively.  Network |  $M$-weight base |  $N$-activation base | Top-1 | Top-5 | Top-1 gap | Top-5 gap
---|---|---|---|---|---|---
res18 | 1 | 1 | 42.7% | 67.6% | 26.6% | 21.6%
res18 | 3 | 1 | 49.1% | 73.8% | 20.2% | 15.4%
res18 | 3 | 3 | 61.0% | 83.2% | 8.3% | 6.0%
res18 | 3 | 5 | 63.1% | 84.8% | 6.2% | 4.4%
res18 | 5 | 1 | 54.1% | 78.1% | 15.2% | 11.1%
res18 | 5 | 3 | 62.5% | 84.2% | 6.8% | 5.0%
res18 | 5 | 5 | 65.0% | 85.9% | 4.3% |  3.3%
res18 | Full Precision | 69.3% | 89.2% | - | -
res34 | 1 | 1 | 52.4% | 76.5% | 20.9% | 14.8%
res34 | 3 | 3 | 66.7% | 87.4% | 6.6% | 3.9%
res34 | 5 | 5 | 68.4% | 88.2% | 4.9% |  3.1%
res34 | Full Precision | 73.3% | 91.3% | - | -
res50 | 5 | 5 | 70.1% | 89.7% | 6.0% |  3.1%
res50 | Full Precision | 76.1% | 92.8% | - | -

As balancing between multiple factors like training time and inference time,
model size and accuracy is more a problem of practical trade-off, there will
be no definite conclusion as which combination of ($M,N$) one should choose.
In general, Table 2 shows that (1) the prediction accuracy of ABC-Net improves
greatly as the number of binary activations increases, which is analogous to
the weight approximation approach; (2) larger $M$ or $N$ gives better
accuracy; (3) when $M=N=5$, the Top-1 gap between the accuracy of a full-
precision model and a binary one reduces to around 5%. To gain a visual
understanding and show the possibility of extensions to other tasks such
object detection, we print the a sample of feature maps in Section S3 of
supplementary material.

####  4.1.3 Comparison with the state-of-the-art

Table 3: Classification test accuracy of CNNs trained on ImageNet with Resnet-18 network topology. ‘W’ and ‘A’ refer to the weight and activation bitwidth respectively. Model | W | A | Top-1 | Top-5
---|---|---|---|---
Full-Precision Resnet-18 [full-precision weights and activation] | 32 | 32 | 69.3% | 89.2%
BWN [full-precision activation] Rastegari et al. (2016) | 1 | 32 | 60.8% | 83.0%
DoReFa-Net [1-bit weight and 4-bit activation] Zhou et al. (2016) | 1 | 4 | 59.2% | 81.5%
XNOR-Net [binary weight and activation] Rastegari et al. (2016) | 1 | 1 | 51.2% | 73.2%
BNN [binary weight and activation] Courbariaux et al. (2016) | 1 | 1 | 42.2% | 67.1%
ABC-Net [5 binary weight bases, 5 binary activations] | 1 | 1 | 65.0% | 85.9%
ABC-Net [5 binary weight bases, full-precision activations] | 1 | 32 | 68.3% | 87.9%

Table 3 presents a comparison between ABC-Net and several other state-of-the-
art models, i.e., full-precision Resnet-18, BWN and XNOR-Net in (Rastegari et
al., 2016), DoReFa-Net in (Zhou et al., 2016) and BNN in (Courbariaux et al.,
2016)††Courbariaux et al. (2016) did not report their result on ImageNet. We
implemented and presented the result. respectively. All comparative models use
Resnet-18 as network topology. The full-precision Resnet-18 achieves 69.3%
Top-1 accuracy. Although Rastegari et al. (2016)’s BWN model and DeReFa-Net
perform well, it should be noted that they use full-precision and 4-bit
activation respectively. Models (XNOR-Net and BNN) that used both binary
weights and activations achieve much less satisfactory accuracy, and is
significantly outperformed by ABC-Net with multiple binary weight bases and
activations. It can be seen that ABC-Net has achieved state-of-the-art
performance as a binary model.

One might argue that 5-bit width quantization scheme could reach similar
accuracy as that of ABC-Net with 5 weight bases and 5 binary activations.
However, the former one is less efficient and requires distinctly more
hardware resource. More detailed discussions can be found in Section 5.2.

##  5 Discussion

###  5.1 Why adding a shift parameter works?

Intuitively, the multiple binarized weight bases/activations scheme works
because it allows more information passing through. Consider the case that a
real value, say 1.5, is passed to a binarized function $f(x)={\rm sign}(x)$,
where ${\rm sign}$ maps a positive $x$ to 1 and otherwise -1. In that case,
the outputs of $f(1.5)$ is 1, which suggests that the input value is positive.
Now imagine that we have two binarization function $f_{1}(x)={\rm sign}(x)$
and $f_{2}(x)={\rm sign}(x-2)$. In that case $f_{1}$ outputs 1 and $f_{2}$
outputs -1, which suggests that the input value is not only positive, but also
must be smaller than 2. Clearly we see that each function contributes
differently to represent the input and more information is gained from $f_{2}$
compared to the former case.

From another point of view, both coefficients ($\beta$’s) and shift parameters
are expected to learn and utilize the statistical features of full-precision
tensors, just like the scale and shift parameters in batch normalization. If
we have more binarized weight bases/activations, the network has the capacity
to approximate the full-precision one more precisely. Therefore, it can be
deduced that when $M$ or $N$ is large enough, the network learns to tune
itself so that the combination of $M$ weight bases or $N$ binarized
activations can act like the full-precision one.

###  5.2 Advantage over the fixed-point quantization scheme

It should be noted that there are key differences between the multiple
binarization scheme ($M$ binarized weight bases or $N$ binarized activations)
proposed in this paper and the fixed-point quantization scheme in the previous
works such as (Zhou et al., 2016; Hubara et al., 2016), though at first
thought $K$-bit width quantization seems to share the same memory requirement
with $K$ binarizations. Specifically, our $K$ binarized weight
bases/activations is preferable to the fixed K-bit width scheme for the
following reasons:

(1) The $K$ binarization scheme preserves binarization for bitwise operations.
One or several bitwise operations is known to be more efficient than a fixed-
point multiplication, which is a major reason that BNN/XNOR-Net was proposed.

(2) A $K$-bit width multiplier consumes more resources than $K$ 1-bit
multipliers in a digital chip: it requires more than $K$ bits to store and
compute, otherwise it could easily overflow/underflow. For example, if a real
number is quantized to a 2-bit number, a possible choice is in range
{0,1,2,4}. In this 2-bit multiplication, when both numbers are 4, it outputs
$4\times 4=16$, which is not within the range. In (Zhou et al., 2016), the
range of activations is constrained within [0,1], which seems to avoid this
situation. However, fractional numbers do not solve this problem, severe
precision deterioration will appear during the multiplication if there are no
extra resources. The fact that the complexity of a multiplier is proportional
to THE SQUARE of bit-widths can be found in literatures (e.g., sec 3.1.1. in
(Grabbe et al., 2003)). In contrast, our $K$ binarization scheme does not have
this issue – it always outputs within the range {-1,1}. The saved hardware
resources can be further used for parallel computing.

(3) A binary activation can use spiking response for event-based computation
and communication (consuming energy only when necessary) and therefore is
energy-efficient (Esser et al., 2016). This can be employed in our scheme, but
not in the fixed $K$-bit width scheme. Also, we have mentioned the fact that
$K$-bit width multiplier consumes more resources than $K$ 1-bit multipliers.
It is noteworthy that these resources include power.

To sum up, $K$-bit multipliers are the most space and power-hungry components
of the digital implementation of DNNs. Our scheme could bring great benefits
to specialized DNN hardware.

###  5.3 Further computation reduction in run-time

On specialized hardware, the following operations in our scheme can be
integrated with other operations in run-time and further reduce the
computation requirement.

(1) Shift operations. The existence of shift parameters seem to require extra
additions/subtractions (see (2) and (8)). However, the binarization operation
with a shift parameter can be implemented as a comparator where the shift
parameter is the number for comparison, e.g.,
$H_{v}({\bm{R}})=\left\\{\begin{matrix}{\bm{1}},&{\bm{R}}\geq 0.5-v;\\\
\bm{-1},&{\bm{R}}<0.5-v.\end{matrix}\right.$ ($0.5-v$ is a constant), so no
extra additions/subtractions are involved.

(2) Batch normalization. In run-time, a batch normalization is simply an
affine function, say, ${\rm BN}({\bm{R}})=a{\bm{R}}+b$, whose scale and shift
parameters $a,b$ are fixed and can be integrated with $v_{n}$’s. More
specifically, a batch normalization can be integrated into a binarization
operation as follow: $H_{v}({\rm
BN}({\bm{R}}))=\left\\{\begin{matrix}{\bm{1}},&a{\bm{R}}+b\geq 0.5-v;\\\
\bm{-1},&a{\bm{R}}+b<0.5-v.\end{matrix}\right.=\left\\{\begin{matrix}{\bm{1}},&{\bm{R}}\geq(0.5-v-b)/a;\\\
\bm{-1},&{\bm{R}}<(0.5-v-b)/a.\end{matrix}\right.$ Therefore, there will be no
extra cost for the batch normalization.

##  6 Conclusion and future work

We have introduced a novel binarization scheme for weights and activations
during forward and backward propagations called ABC-Net. We have shown that it
is possible to train a binary CNN with ABC-Net on ImageNet and achieve
accuracy close to its full-precision counterpart. The binarization scheme
proposed in this work is parallelizable and hardware friendly, and the impact
of such a method on specialized hardware implementations of CNNs could be
major, by replacing most multiplications in convolution with bitwise
operations. The potential to speed-up the test-time inference might be very
useful for real-time embedding systems. Future work includes the extension of
those results to other tasks such as object detection and other models such as
RNN. Also, it would be interesting to investigate using FPGA/ASIC or other
customized deep learning processor (Liu et al., 2016) to implement ABC-Net at
run-time.

##  7 Acknowledgement

We acknowledge Mr Jingyang Xu for helpful discussions.

## References

  * Bengio et al. (2013) Y. Bengio, N. Léonard, and A. Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ , 2013.
  * Cai et al. (2017) Z. Cai, X. He, J. Sun, and N. Vasconcelos.  Deep learning with low precision by half-wave gaussian quantization.  _arXiv preprint arXiv:1702.00953_ , 2017.
  * Courbariaux et al. (2016) M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio.  Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1.  _arXiv preprint arXiv:1602.02830_ , 2016.
  * Deng et al. (2009) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on_ , pages 248–255. IEEE, 2009.
  * Esser et al. (2016) S. K. Esser, P. A. Merolla, J. V. Arthur, A. S. Cassidy, R. Appuswamy, A. Andreopoulos, D. J. Berg, J. L. McKinstry, T. Melano, D. R. Barch, et al.  Convolutional networks for fast, energy-efficient neuromorphic computing.  _Proceedings of the National Academy of Sciences_ , page 201604850, 2016.
  * Giusti et al. (2016) A. Giusti, J. Guzzi, D. Ciresan, F.-L. He, J. P. Rodriguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. Di Caro, D. Scaramuzza, and L. Gambardella.  A machine learning approach to visual perception of forest trails for mobile robots.  _IEEE Robotics and Automation Letters_ , 2016.
  * Govindu et al. (2004) G. Govindu, L. Zhuo, S. Choi, and V. Prasanna.  Analysis of high-performance floating-point arithmetic on fpgas.  In _Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International_ , page 149. IEEE, 2004.
  * Grabbe et al. (2003) C. Grabbe, M. Bednara, J. Teich, J. von zur Gathen, and J. Shokrollahi.  Fpga designs of parallel high performance gf (2233) multipliers.  In _ISCAS (2)_ , pages 268–271. Citeseer, 2003.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2016.
  * Hubara et al. (2016) I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio.  Quantized neural networks: Training neural networks with low precision weights and activations.  _arXiv preprint arXiv:1609.07061_ , 2016.
  * Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  _arXiv preprint arXiv:1502.03167_ , 2015.
  * Kingma and Ba (2014) D. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Liu et al. (2016) S. Liu, Z. Du, J. Tao, D. Han, T. Luo, Y. Xie, Y. Chen, and T. Chen.  Cambricon: An instruction set architecture for neural networks.  In _Proceedings of the 43rd International Symposium on Computer Architecture_ , pages 393–405. IEEE Press, 2016.
  * Qian (1999) N. Qian.  On the momentum term in gradient descent learning algorithms.  _Neural networks_ , 12(1):145–151, 1999.
  * Rastegari et al. (2016) M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  In _European Conference on Computer Vision_ , pages 525–542. Springer, 2016.
  * Ren et al. (2015) S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in neural information processing systems_ , pages 91–99, 2015.
  * Toms (1990) D. Toms.  Training binary node feedforward neural networks by back propagation of error.  _Electronics letters_ , 26(21):1745–1746, 1990\.
  * Venkatesh et al. (2016) G. Venkatesh, E. Nurvitadhi, and D. Marr.  Accelerating deep convolutional networks using low-precision and sparsity.  _arXiv preprint arXiv:1610.00324_ , 2016.
  * Zhou et al. (2016) S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou.  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.  _arXiv preprint arXiv:1606.06160_ , 2016.

Supplementary Material

###  S1 Summary of training algorithm in Section 3.3

Algorithm 1 Training a $L$-layer ABC-Net. $c$ is the cost function for
minibatch, and $\lambda$ is the learning rate decay factor. $\circ$ indicates
element-wise multiplication. BatchNorm() specifies how to batch-normalize the
output of convolution and BackBatchNorm specifies how to backpropagate through
the normalization (Ioffe and Szegedy, 2015). Analogously, Conv() and
BackConv() respectively specify how to do convolution and how to backpropagate
through the convolution. Update() specifies how to update the parameters when
their gradients are known, such as ADAM (Kingma and Ba, 2014) and Momentum
(Qian, 1999))

0: a minibatch of inputs and targets, number of binary weight bases $M$,
number of binary activations $N$, previous weights ${\bm{W}}$, and learning
rate $\eta$.

0: updated weights ${\bm{W}}$ and updated learning rate $\eta$.

{1. Computing the parameters gradients:}

{1.1. Forward propagation:}

for $l=1$ to $L$ do

Compute $\alpha_{m}^{l},{\bm{B}}_{m}^{l},m=1,\cdots,M$ with (2) and (3))

$s^{l}\leftarrow{\rm Conv}({\bm{W}}^{l},{\bm{R}}^{l-1})$ using (12)

Optionally apply max-pooling

$a^{l}\leftarrow{\rm BatchNorm}(s^{l})$

if $l<L$ then

${\bm{A}}_{n}^{l}\leftarrow H_{v_{n}}(a^{l}),n=1,2,\cdots,N$ (using (11))

end if

end for

{1.2. Backward propagation:}

{Note that the gradients are full-precision.}

Compute $g_{a_{L}}=\frac{\partial c}{\partial a_{L}}$

for $l=L$ to $1$ do

if $l<L$ then

$g_{a^{l}}\leftarrow\sum_{n=1}^{N}\beta_{n}g_{{\bm{A}}_{n}^{l}}\circ\mathbb{I}_{0\leq
a^{l}-v_{n}\leq 1}$

end if

$g_{s_{l}}\leftarrow{\rm BackBatchNorm}(g_{a^{l}},s_{l})$

$g_{\beta_{m}},g_{{\bm{B}}_{m}^{l}}\leftarrow{\rm
BackConv}(g_{s_{l}},{\bm{B}}_{m}^{l-1},{\bm{A}}_{m}^{l-1})$

end for

{2. Accumulating the parameters gradients:}

for $l=1$ to $L$ do

With $g_{{\bm{B}}_{m}^{l}}$ known, compute $g_{{\bm{W}}^{l}}$ using (7)

${\bm{W}}^{l}\leftarrow{\rm Update}({\bm{W}}^{l},\eta,g_{{\bm{W}}^{l}})$

$\beta_{m}\leftarrow{\rm Update}(\beta_{m},\eta,g_{\beta_{m}}),m=1,2,\cdots,M$

$v_{m}\leftarrow{\rm Update}(v_{m},\eta,g_{v_{m}}),m=1,2,\cdots,M$

$\eta\leftarrow\lambda\eta$

end for

###  S2 Weight approximation

In this section we explore how well the weight approximation can achieved
given adequate binary weight bases (Section 3.1). To gain a visual intuition,
we randomly sample a slice of weight tensor from a full-precision Resnet-18
model pretrained on ImageNet. The sliced tensor is then vectorized, and we
approximate it with $M$ bases using linear regression (see (3)). The results
are presented in Figure S2, where the left subfigure shows the root mean
square (RMSE) for the estimated weights with increasing number of bases, and
the right one shows 5 fitting results, whose choice of $M$ are respectively 1
to 5 from top to bottom. The blue line in the right subfigure draws the
groundtruth weights from the full-precision pretrained model, and the red line
is the estimated one. It can be observed that $M=3$ is adequate to have a
rough fitting, and it gets almost perfect when $M=5$.

Figure S2: Fitting a section of weights of full-precision Resnet-18 trained on
ImageNet. On the left side the RMSE for the estimated weights with increasing
number of bases is shown, and on the right side 5 fitting results are shown,
whose choice of $M$ are respectively 1 to 5 from top to bottom (Blue line:
full-precision weights; Red line: estimated weights).

###  S3 Feature map

It is also possible to perform more complex tasks beyond classification using
ABC-Net, as long as the model is built upon a CNN, such as faster RCNN for
object detection, in which the classification model serves as a pre-train
model. Thus, one might be interested in whether ABC-Net learns similar feature
maps as its full-precision counterpart. Figure S3 shows several example image
and the corresponding feature maps of these two models, from which we see that
they are indeed similar. This shows the potential for ABC-Net to further
generalize on more complex tasks mentioned above.

Figure S3: Examples of feature maps. The feature maps from the first
convolution layer of ABC-Net (above) looks similar to that of its full-
precision counterpart (below). Settings for the ABC-Net: $M=5,N=3$, using
Resnet-18.

###  S4 Parameter settings for the experiment in Section 4.1.2

The parameters $u_{i}$’s, the initial values for $\beta_{n}$’s and $v_{n}$’s
can be treated as hyperparameters. At the beginning of our exploration we
randomly choose these initial values. Bit by bit we began to find certain
patterns to achieve good performance: for $u_{i}$’s, usually symmetric; for
$v_{n}$’s, maybe slightly shift towards the negative direction. These are
based on tunings and also the observation of the full-precision distribution
of weights/activations. Table S4 provides the parameter settings for the
experiment in Section 4.1.2. All ABC-Net models in the experiments are trained
using SGD with momentum, and the initial learning rate is set to 0.01.

Table S4: Parameter settings for the experiment in Section 4.1.2. “res18", “res34" and “res50" are short for Resnet-18, Resnet-34 and Resnet-50 network topology respectively. $M$ and $N$ refer to the number of weight bases and activations respectively.  Network | $M$ | $N$ | shift parameters ($u_{i}$’s) | shift parameters ($v_{i}$’s) |  $\beta$’s
---|---|---|---|---|---
res18 | 1 | 1 | 0 | 0.0 | 1.0
res18 | 3 | 1 | -1,0,1 | 0.0 | 1.0
res18 | 3 | 3 | -1,0,1 | -1.5, 0.0, 1.5 | 1.0, 1.0, 1.0
res18 | 3 | 5 | -1,0,1 | -3.5, -2.5, -1.5, 0.0, 2.5 | 1.0, 1.0, 1.0, 1.0, 1.0
res18 | 5 | 1 | -2,-1,0,1,2 | 0.0 | 1.0
res18 | 5 | 3 | -2,-1,0,1,2 | -0.9, 0.0 0.9 | 1.0,1.0,1.0
res18 | 5 | 5 | -1,-0.5,0,0.5,1 | -3.5, -2.5, -1.5, 0.0, 2.5 | 1.0, 1.0, 1.0, 1.0, 1.0
res34 | 3 | 3 | -1,0,1 | -3.0, 0.0, 3.0 | 1.0, 1.0, 1.0
res34 | 5 | 5 | -1,-0.5,0,0.5,1 | -3.5, -2.5, -1.5, 0.0, 2.5 | 1.0, 1.0, 1.0, 1.0, 1.0
res50 | 5 | 5 | -1,-0.5,0,0.5,1 | -3.5, -2.5, -1.5, 0.0, 2.5 | 1.0, 1.0, 1.0, 1.0, 1.0

###  S5 Relationship between accuracy and number of binary weight bases $M$

Figure S4 shows that the relationship between accuracy and the number of
binary weight bases $M$ appears to be linear. Note that we keep the
activations being full-precision in this experiment.

Figure S4:  Top-1 (left) and Top-5 (right) accuracy of ABC-Net on ImageNet,
using full-precision activation and different choices of the number of binary
weight bases $M$.

###  S6 Application on visual perception of forest trails

####  S6.1 VGG-like Network Topology

A VGG-like network topology is used for visual perception of forest trails as
illustrated in Figure S5.

Figure S5: The network topology for visual perception of forest trails.

####  S6.2 Experiment results on visual perception of forest trails dataset

Giusti et al. (2016) cast the forest or mountain trails perception problem for
mobile robots as a image classification task based on Deep Neural Networks.
The dataset is composed by 8 hours of $1920\times 1080$ 30fps video acquired
by a hiker equipped with three head-mounted cameras . Each image is labelled
in one of three classes: turn right, go straight, turn left. We evaluate ABC-
Net against its full precision counterpart using this dataset. The
classification result is shown in Table S5 by fixing both number of weight
bases $M$ and activation bases $N$ to be 5.

Table S5: Classification accuracy on Forest Trails dataset. ‘FP’ stands for ‘Full Precision’. Network | shift parameters ($u_{i}$’s) | shift parameters ($v_{i}$’s) |  $\beta$’s | ABC-Net | FP
---|---|---|---|---|---
VGG-like | -1,0.5,0.0,0.5,1 | 0,0,0,0,0 | 1,1,1,1,1 | 78.0% | 77.7%


# Artificial neural networks and time series of counts: A class of nonlinear
INGARCH models

Malte Jahn Department of Mathematics and Statistics, Helmut Schmidt
University, 22043 Hamburg, Germany. E-Mail<EMAIL_ADDRESS>ORCID:
0000-0002-7165-0428.

Abstract
Time series of counts are frequently analyzed using generalized integer-valued
autoregressive models with conditional heteroskedasticity (INGARCH). These
models employ response functions to map a vector of past observations and past
conditional expectations to the conditional expectation of the present
observation. In this paper, it is shown how INGARCH models can be combined
with artificial neural network (ANN) response functions to obtain a class of
nonlinear INGARCH models. The ANN framework allows for the interpretation of
many existing INGARCH models as a degenerate version of a corresponding neural
model. Details on maximum likelihood estimation, marginal effects and
confidence intervals are given. The empirical analysis of time series of
bounded and unbounded counts reveals that the neural INGARCH models are able
to outperform reasonable degenerate competitor models in terms of the
information loss.

Keywords: Neural networks; count time series; non-linear regression

##  1 Introduction

Artificial neural networks (ANN) are state-of-the-art tools for data analysis.
They are often applied in big data contexts but also for classical problems
such as non-linear regression and time series analysis (e.g. Tkacz, 2001; Kock
and Teräsvirta, 2014; Jahn, 2020), which is the relevant setting of this
paper. There are different types of ANN but the main reason to use them is
almost always their universal approximation property (cf. Kock and Teräsvirta,
2014, p. 616). In most applications, the ANN function is fitted (“trained”) to
data by minimizing the mean squared error. In regression analysis, this
(nonlinear) least squares approach is very appropriate in case of a normal
error distribution, but less appropriate for other distributions, in
particular skewed distributions. In this paper, time series of counts are
considered and the corresponding count distributions are often not well
approximated by a normal distribution. An important class of models for this
type of data are integer-valued autoregressive models with conditional
heteroskedasticity (INARCH). These models assume a conditional count
distribution at every time step where the conditional mean depends on past
time series values. Natural choices are the Poisson distribution (for
unbounded counts) or the binomial distribution (for bounded counts). Since,
for these count distributions, the conditional variance is directly related to
the conditional mean parameters, the models exhibit conditional
heteroskedasticity. INARCH models can be discussed in their more general
version known as INGARCH (Ferland et al., 2006; Fokianos et al., 2009). By
including past conditional means as additional regressors, past conditional
variances are included implicitly which makes the INGARCH model similar to the
traditional GARCH approach for continuous distributions.
A frequent issue with exactly linear INGARCH models is that the conditional
mean parameter may leave the support of the count distribution. One solution
is to impose quite strict parameter constraints such as positive coefficients
in exactly linear INGARCH models (Ferland et al., 2006, eq. 1). The downside
is that these parameter constraints also limit the ability to account for
negative autocorrelation. The alternative solution is to employ response
functions which map values (i.e. conditional mean parameters) outside of the
plausible range to the known domain while maintaining desirable properties,
such as differentiability. Depending on the probability distribution, choices
are a log-linear link leading to a multiplicative model (Fokianos and
Tjøstheim, 2011) or the logit link (Chen et al., 2020). We refer to a link
function as the inverse response function (if exists), such that, e.g. , a log
link corresponds to an exponential response and a logit link corresponds to a
logistic response. Recent approaches are the softplus response for unbounded
counts (Weiß et al., 2022) and the soft-clipping response for bounded counts
(Weiß & Jahn, 2022) which yield nearly linear models but may still account for
negative autocorrelation. In this paper, nonlinear INGARCH models are
considered where the response function corresponds to a single hidden layer
feedforward artificial neural network (neural INGARCH model).
In the following section, neural network architectures are discussed and it is
shown how the ANN concept of activation functions relates to the INGARCH
concept of response functions and why many existing INGARCH models can be
interpreted as a degenerate version of a corresponding neural INGARCH model. A
short section considers model selection strategies via information criteria
and diagnostic quantities derived from the (standardized) Pearson residuals.
The first data example then considers the number of (countries experiencing)
banking crises in a given year from 1800 to 2011. For the neural INGARCH
model, a time variable allowing for a non-linear trend proves useful to
represent a period between the 1940s and the 1960s when almost no banking
crisis occurred. The remaining overdispersion can be modeled by replacing the
conditional Poisson distribution by a generalized Poisson distribution. As a
second data example for time series of bounded counts, the number of votes in
favor of an adjustment of the interest rate by the members of the Monetary
Policy Council (MPC) of the National Bank of Poland between 2002 and 2013 are
considered. The previous results are confirmed, the neural INGARCH model also
outperforms the corresponding “degenerate” conventional INGARCH model in terms
of the information loss. The model can be further improved by including a
discrete policy regime variable. The final section concludes.

##  2 The neural INGARCH model

An INGARCH(p,q) model is defined via the conditional mean equation

$E[Y_{t}|\mathcal{F}_{t-1}]=\lambda_{t}=f(Y_{t-1},\dots,Y_{t-p},\lambda_{t-1},\dots,\lambda_{t-q}),$
(1)

for some response function $f$. The actual conditional distribution of $Y_{t}$
is an appropriate count distribution with mean $\lambda_{t}$. The right hand
side depends on $p$ past values of the series and on $q$ past conditional
means. A linear response function $f(x)=x$ would lead to the exactly linear
INGARCH model (Ferland et al., 2006, eq. 1). In this paper, response functions
from the class of artificial neural networks are considered. More precisely,
it is assumed that $f$ is a single hidden layer feedforward network (SLFN).
The mathematical formulation of the SLFN regression function is:

$f^{ANN}(w^{0},w^{1},x)=g_{1}\\!\left(\sum_{h=1}^{H}w^{1}_{h}\cdot
g_{0}\\!\left(\sum_{j=1}^{J}w^{0}_{jh}x_{j}\right)\right).$ (2)

The SFLN function depends on a set of parameters $w^{1}_{h}$ and $w^{0}_{jh}$.
The input vector $x$ consists of a constant and the respective lagged values,
so $x=(1,Y_{t-1},\dots,Y_{t-p},\lambda_{t-1},\dots,\lambda_{t-q})$ and its
number of elements is $J=p+q+1$. The index $h$ ($1\leq h\leq H$) refers to the
so-called hidden neurons. Their number $H$ determines the complexity of the
model. In practice, the optimal $H$ can be determined by information criteria,
which will be discussed later. The functions $g_{0}$ and $g_{1}$ are
activation functions which are required to be increasing and continuously
differentiable. Since $g_{0}$ and $g_{1}$ are generally nonlinear, it is
possible to give a basic interpretation of the corresponding neural INGARCH
model: The conditional expectation of $Y_{t}$ is a nonlinear combination of
nonlinear combinations of
$Y_{t-1},\dots,Y_{t-p},\lambda_{t-1},\dots,\lambda_{t-q}$.

Figure 1 shows a concrete specification of a neural INGARCH(2,2) model and
helps explaining the concept of hidden neurons and activation functions. The
SLFN consists of three layers: one input layer, one hidden layer and one
output layer. The neurons (circles) in the input layer correspond to the
regressor variables, the hidden layer is unobserved and a specification with 4
hidden neurons is chosen for illustration. The single neuron in the output
layer corresponds to the conditional mean of the current value of the time
series. Alternatively, instead of the output, an illustration based on the
target value $Y_{t}$ could be used. As indicated by the direction of the
arrows, it is assumed that information is only passed in the forward direction
from the input layer through the hidden layer to the output layer, which makes
the network a single hidden layer “feedforward” network.

Figure 1: Visual representation of a softplus Poisson neural INGARCH(2,2)
model with $H=4$ hidden neurons

The interesting elements for this paper are the activation functions $g_{0}$
and $g_{1}$ which are sketched at the bottom of Figure 1. The term
“activation” refers to the name-giving biological interpretation: A neuron
passes on information depending on its level of activation which is determined
by the information in the previous layer. For the activation of the hidden
layer, the logistic function is employed throughout the paper which introduces
(desired) non-linearity into the model, so $g_{0}(x)=\frac{1}{1+\exp(-x)}$.
This also implies that the values of the hidden neurons are always between 0
(completely inactive) and 1 (completely active). The activation of the output
layer requires a deeper consideration and one has to look at the data to make
an appropriate choice for $g_{1}$. The depicted function corresponds to the
softplus function, so $g_{1}(x)=\ln(1+\exp(x))$. The softplus function can be
seen as a smoothed version of the ReLU function. The image of the function are
the positive real numbers, so it makes sense to use it in the case of
unbounded counts. In general, the output activation function $g_{1}$ takes the
same role as the response function in various INGARCH models (e.g. Chen et
al., 2020; Weiß & Jahn, 2022; Weiß et al., 2022). Such “conventional” INGARCH
models can be interpreted as degenerate neural INGARCH models where the hidden
layer is missing and the output is directly activated from the inputs through
the function $g_{1}$. Conversely, when the starting point is a certain
conventional INGARCH model with response function $g$, it is possible to
construct a corresponding neural model by setting $g_{1}=g$ in many cases. One
can then use statistical methods such as information criteria to check whether
it is reasonable to “upgrade” that particular conventional model to a neural
model by adding the hidden layer.

Assuming the generic conditional Poisson distribution for unbounded counts,
the conditional log-likelihood function of an INGARCH model with response
function $f$ and parameter vector $w$ is

$\ell(w)=\sum_{t}Y_{t}\ln(f(w,x))-f(w,x)+c,$ (3)

which has to be maximized w.r.t. $w$. For practical modeling, it shall be
noted here that the numerical maximization speed can be significantly improved
in the case of a purely autoregressive model ($q=0$) because then, the
regressors are predetermined and can be collected as columns of a design
matrix $X$ as usual for standard regression models. On the other hand, if
$q>0$, the log-likelihood function (and the gradient) are defined recursively,
implying that an optimization algorithm needs to loop through the individual
time periods in each iteration. According to the chain rule, we have the
following relationship:

$\frac{\partial\ell}{\partial w}=\frac{\partial\ell}{\partial f}\frac{\partial
f}{\partial w}.$ (4)

Regarding the gradient, the outer derivative is simply the derivative of the
conditional Poisson log-likelihood with respect to the Poisson parameter
$f(w,x)$:

$\frac{\partial\ell}{\partial f}=\sum_{t}\left(\frac{Y_{t}}{f(w,x)}-1\right).$
(5)

The first-order condition derived from equation (5) implies that the relative
error (ratio between true and fitted values) is the relevant error for the
Poisson model. The partial derivatives $\frac{\partial f}{\partial w}$ can be
computed by further applications of the chain rule. In case of the ANN
response function $f=f^{ANN}$, this can be done efficiently by using the so-
called backpropagation procedure. The only difference to the conventional
backpropagation is that the error which is back-propagated is now the relative
error instead of the plain error which arises from the first-order condition
of the least squares approach. For an efficient calculation, it makes sense to
use activation functions with simple derivatives. In the present example, we
have for the softplus function $g_{1}^{\prime}=g_{0}$ (i.e. the derivative of
the softplus function is the logistic function) and for the logistic function
$g_{0}^{\prime}=g_{0}(1-g_{0})$.

##  3 Model selection and diagnostics

From the previous section, it became clear that the relevant quantities
regarding model selection in the neural INGARCH model are the model order
$(p,q)$ and the number of hidden neurons $H$. Likelihood-based information
criteria such as Akaike’s information criterion (AIC) or the Bayesian
information criterion (BIC) can be used to determine the optimal $(p,q)$ and
$H$. The proposed model selection strategy is to select $(p,q)$ first based on
the corresponding “degenerate” conventional INGARCH model. This establishes a
reasonable simple competitor for the neural model and the further analysis can
be focused on the question whether to “upgrade” this particular conventional
model to a neural model and which complexity $H$ is necessary. There are also
many rules of thumb for restricting $H$ in advance. Obviously, the number of
available time periods $T$ prevents us from using excessively large $H$. For
example, one may require 10 times more observations than parameters, meaning
$H\leq\frac{0.1T}{J+1}$ and assuming that the number of regressors $J$ has
already been fixed.
For model diagnostic, a simple and informative quantity are the (standardized)
Pearson residuals:

$r_{t}=\frac{Y_{t}-E[Y_{t}|\mathcal{F}_{t-1}]}{\sqrt{Var[Y_{t}|\mathcal{F}_{t-1}]}}.$
(6)

The empirical Pearson residuals can be computed easily after estimation, even
for the otherwise complex neural model because the formula only involves
predicted conditional means and variances. According to the analysis in Weiß
et al. (2020), the Pearson residuals can help to detect (remaining)
overdispersion and serial correlation. In a correctly specified model, the
Pearson residuals should have mean zero, variance one and no serial
correlation. Weiß et al. (2020) also discuss asymptotic results allowing for
formal tests under certain assumptions. In this paper, mainly the variance of
the Pearson residuals will be used as a rough indicator for whether the
dispersion behavior is represented correctly by a given model.

##  4 Modeling unbounded counts

As a first example, the annual number of countries experiencing a banking
crisis among a group of 21 countries in the time period 1800 to 2010 (211
years) is considered. This data set was originally discussed by Reinhart and
Rogoff (2009) but the main reference for this paper will be Dungey et al.
(2020). Following Dungey et al. (2020), the data are treated as unbounded
counts (although they are technically bounded). For the neural INGARCH model,
this implies that the softplus function is chosen as the output activation
function ($g_{1}(x)=\ln(1+\exp(x))$), in combination with a conditional
Poisson distribution. The model definition becomes

$Y_{t}\sim Poi(\lambda_{t})\mbox{ with
}\lambda_{t}=f^{ANN}(Y_{t-1},\dots,Y_{t-p},\lambda_{t-1},\dots,\lambda_{t-q}).$
(7)

According to the model selection procedure proposed in the previous section,
the first step is to determine the model order by looking at the corresponding
conventional INGARCH model. Here, the relevant conventional model is the
(nearly linear) softplus Poisson INGARCH model proposed by Weiß et al. (2022).
The regression function for this model is $f^{sp}(x,\beta)=g_{1}(x\beta)$. The
analysis of the banking crisis data by Dungey et al. (2020) suggests that
small model orders are sufficient. Indeed, Table 1 (top) shows that the
simplest model, the softplus Poisson INGARCH(1,0) is preferred over the larger
alternatives INGARCH(2,0) and INGARCH(1,1) in terms of the information loss
(AIC/BIC).
Once the model order is established, the final step in the specification of
the neural INGARCH model is to determine the complexity $H$. The rule of thumb
from section 3 gives $H\leq\frac{0.1T}{J+1}=\frac{21}{3}=7$. Figure 2 shows
that the AIC is minimal for $H=3$ and the BIC is minimal for $H=2$. Since the
minimum of the BIC is a little bit more pronounced, $H=2$ is selected for now.

Figure 2: Banking crises: Information loss of the neural Poisson INGARCH(1,0)
for different $H$

The precise AIC and BIC value for the neural Poisson INGARCH(1,0) with $H=2$
hidden neurons is displayed in Table 1. In comparison with the conventional
competitor model, the neural model has lower AIC but higher BIC, so there is
no clear evidence (yet) whether the higher likelihood is worth the additional
parameters. Regarding interpretation, the estimated parameter values $\hat{w}$
are of minor interest because they have no direct meaning. Therefore, the
initial step in the interpretation of the neural INGARCH model is to compare
its predictions to the corresponding conventional model. Figure 3 shows the
prediction for the two models, along with the true values. The biggest
challenge is to capture the long period (almost) without any banking crises
between the 1940s and the 1960s when the Bretton-Woods (BW) system of fixed
exchange rates was in place (cf. Dungey et al., 2020, p. 2, bottom).

Figure 3: Banking crises: Predicted conditional means by the neural
INGARCH(1,0) model and the nearly linear softplus INGARCH(1,0) model

The second step of the analysis is concerned with marginal effects. As usual
for non-linear regression models, there are different options to calculate
marginal effects and it is usually advocated to consider marginal effects “at
the mean”. For the present model, there is only one variable which facilitates
the illustration and interpretation. Figure 4 (a) shows plots of the predicted
conditional mean of $y_{t}$ given different values of $y_{t-1}$. It becomes
clear that, in the neural model, the marginal effect (i.e. the slope of the
“curve”) is not limited to a constant as in the (nearly) linear model.
Although the differences between the two models are not very pronounced, the
neural model indicates, for example, that the marginal effect of $y_{t-1}$
becomes very small for $y_{t-1}\geq 10$ which means that the neural model is
more likely to return to a lower number of banking crises immediately after a
year with a large number of banking crises. Regarding the ability to model the
BW period, the probability of staying in the zero-count state is the relevant
quantity and it can be concluded visually from both Figure 3 and Figure 4 that
this probability is slightly higher for the neural model. Since a conditional
Poisson distribution is assumed, this zero probability can be calculated
easily. We have $p_{0|0}^{sp}=\exp(-g_{1}(\hat{\beta}_{0}))=0.50$ and
$p_{0|0}^{ANN}=\exp(-f^{ANN}(\hat{w},x=(1,0))=0.58$.

For further inference, one might also want to calculate confidence intervals.
Since the parameters have no direct interpretation, sensible confidence
intervals should refer to the conditional mean $E[y|x]$. Corresponding
predicted conditional means have been considered in the previous step of the
analysis (Figure 4 (a)). Now, the focus is on the conditional variance of
$f(x,\hat{w})$. For non-linear regression models, we generally have
$Var(f(x,\hat{w}))\approx\nabla f(x)^{T}\mathcal{H}^{-1}\nabla f(x)$. The
notation is such that $\hat{w}$ is the estimated parameter vector, $\nabla
f(x)$ is the gradient of $f$ (w.r.t. $w$) evaluated at $x$ and $\mathcal{H}$
is the Hessian matrix of the model at $w=\hat{w}$. As noted earlier, the
gradient $\nabla f(x)$ can be calculated efficiently in the neural INGARCH
model due to the simple derivatives of the involved activation functions and
the backpropagation method. Depending on the employed algorithm, the Hessian
matrix $\mathcal{H}$ may be computed as part of the optimization procedure or
afterwards. Further information on confidence intervals for ANN regression
models in the usual least squares context can be found in Rivals and Personnaz
(2000).
Concerning the nearly linear softplus model with
$f^{sp}(x,\beta)=g_{1}(x\beta)$, we have
$Var(f^{sp}(x,\hat{\beta}))\approx\nabla f^{sp}(x)\mathcal{H}_{sp}^{-1}\nabla
f^{sp}(x)$. For the softplus response function, the gradient (w.r.t. $\beta$)
is $\nabla f^{sp}(x)=\frac{-x}{1+\exp(-x)}$. The denominator is almost unity
for arguments $x\geq 2$. This shows that the softplus model behaves very
similarly to an exactly linear Poisson INGARCH model when the time series is
further away from zero.
Figure 4 (b) displays approximate 90% pointwise confidence intervals for the
conditional means for both the neural and the softplus model. The highest
confidence is generally around the mean of the process which is just 1.55 in
the present example. Banking crises affecting many countries at the same time
are quite rare (cf. Figure 3) and therefore, the confidence is lower in that
region.

|
---|---
(a) | (b)
Figure 4: Banking crises: Marginal effect of $y_{t-1}$ for the neural
INGARCH(1,0) model and the nearly linear softplus INGARCH(1,0) model (a) and
corresponding confidence intervals (b)

In order to investigate in particular the dispersion behavior, the Pearson
residuals are analyzed next. Table 2 (top) shows the mean, variance and the
first two ACF values of the empirical Pearson residuals for the four models
that have been considered so far. The biggest concern are the variances which
are far away from unity, the value expected under a correctly specified model.
Fortunately, there are several ways to modify the neural INGARCH model.
The first one aims to capture the zero counts during the BW period more
accurately. Therefore, a time variable is added as an additional regressor, so
the input vector becomes $x=(1,Y_{t-1},t)$. Such a time variable can also be
added to the conventional nearly linear softplus INGARCH but it is clear in
advance that a nearly linear time trend will not be able to represent the zero
counts during the BW period which are only temporary. On the contrary, the
universal approximation property of the ANN regression function allows the
neural INGARCH model to represent an arbitrary time trend. The number of
hidden neurons is increased to $H=3$ which gives the model the required
complexity (model selection procedure not shown again). The results are
displayed in the middle section of Table 1. As expected, (only) the neural
model benefits significantly from the additional time variable regarding the
information loss. For interpretation and comparison, consider Figure 5.
Subfigure 5 (a) shows the predicted conditional means analogously to Figure 3.
The main difference is the BW period where the neural INGARCH model is now
able to produce much lower conditional means, as intended. Figure 5 (b) shows
the marginal effect of time on $E[y_{t}|x]$ where $y_{t-1}$ is fixed at its
mean value (marginal effect at the mean). The “reset” of the global banking
system in the 1940s is clearly visible. The number of banking crises tends to
increase again after the BW period. The BW period can be analyzed more closely
by looking at subfigure 5 (c) which displays the marginal effect of $y_{t-1}$
at time $t=1958$ (during the BW period). The previously discussed probability
of staying in the zero-count state is considered again (for that particular
year). This time, we have $p_{0|0}^{sp}=0.44$ and $p_{0|0}^{ANN}=0.94$ which
confirms the benefit of the time trend in the neural INGARCH model. Finally,
subfigure 5 (d) is analogous to 5 (c) only that the marginal effect of
$y_{t-1}$ is calculate at $t=2009$ and serves to illustrate that the neural
model now allows the marginal effect of $y_{t-1}$ to vary across time.

|
---|---
(a) Predicted cond. means | (b) Marginal effect of time (at the mean)
|
(c) Marginal effect of $y_{t-1}$ at $t=1958$ |  (d) Marginal effect of $y_{t-1}$ at $t=2009$
Figure 5: Banking crises: Illustrations for the neural INGARCH(1,0) model with
time trend

The Pearson residuals in Table 2 (middle) reveal that there is still some
overdispersion which the neural INGARCH(1,0) cannot account for, even with a
time trend. Therefore, as a final step of the analysis of the banking crises,
a generalized Poisson distribution is considered as the conditional
distribution. Following Famoye (1993), its probability function with
parameters $\lambda,\alpha>0$ is defined for non-negative integers $k$ as:

$P(X=k)=\left(\frac{\lambda}{1+\alpha\lambda}\right)^{k}\frac{(1+\alpha
k)^{k-1}}{k!}\exp\left(\frac{-\lambda(1+\alpha k)}{1+\alpha\lambda}\right).$
(8)

This particular definition of the generalized Poisson distribution is very
useful in a regression context because it has $E[X]=\lambda$ and therefore,
the regression equation of corresponding INGARCH models is still given by
equation 1. The additional parameter $\alpha$ only affects the variance which
is $Var(X)=\lambda(1+\alpha\lambda)^{2}$. Moreover, the gradient of the log-
likelihood function (cf. Famoye (1993), eq. 3.3 and 3.4) is simpler than that
of the negative binomial distribution which would be a theoretical
alternative. The simpler gradient improves the numerical ML estimation, in
particular for the neural INGARCH model. The results of the corresponding
estimations of the softplus INGARCH(1,0) and the neural INGARCH(1,0) are given
at the bottom of Table 1 without further illustrations. It is only noted that
$\hat{\alpha}=0.2526$ for the conventional and $\hat{\alpha}=0.1304$ for the
neural model, both significant. Finally, a look at the Pearson residuals
(Table 2, bottom) shows that an acceptable representation of the dispersion
structure is now achieved for the neural INGARCH model.

response | cond. distribution | time trend | order | AIC | BIC
---|---|---|---|---|---
softplus | Poisson | no | (1,0) | 713.66 | 720.36
softplus | Poisson | no | (2,0) | 714.20 | 724.23
softplus | Poisson | no | (1,1) | 715.66 | 725.70
neural | Poisson | no | (1,0) | 709.63 | 729.72
softplus | Poisson | yes | (1,0) | 712.39 | 722.43
neural | Poisson | yes | (1,0) | 634.75 | 674.91
softplus | gen. Poisson | yes | (1,0) | 652.26 | 665.65
neural | gen. Poisson | yes | (1,0) | 613.71 | 657.23
Table 1: Banking crises: Information loss for different models response | cond. distribution | time trend | order |  mean($r_{t}$) |  var($r_{t}$) |  ACF(1)($r_{t}$) |  ACF(2)($r_{t}$)
---|---|---|---|---|---|---|---
softplus | Poisson | no | (1,0) | -0.014 | 2.527 | -0.052 | -0.025
softplus | Poisson | no | (2,0) | -0.013 | 2.531 | -0.044 | -0.032
softplus | Poisson | no | (1,1) | -0.014 | 2.526 | -0.054 | -0.024
neural | Poisson | no | (1,0) | 0.001 | 2.620 | -0.060 | -0.016
softplus | Poisson | yes | (1,0) | -0.010 | 2.452 | -0.052 | -0.027
neural | Poisson | yes | (1,0) | 0.003 | 1.585 | 0.048 | -0.025
softplus | gen. Poisson | yes | (1,0) | -0.013 | 1.490 | -0.040 | -0.048
neural | gen. Poisson | yes | (1,0) | -0.010 | 1.099 | 0.011 | -0.044
Table 2: Banking crises: Properties of (empirical) Pearson residuals $r_{t}$
for different models

To summarize, it was demonstrated that the neural INGARCH model constitutes a
straightforward way to employ ANN regression techniques for the analysis of
count time series. From the (conventional) INGARCH perspective, the ANN
regression function takes the role of an “upgraded” nonlinear response
function which includes the conventional model as a degenerate special case.
With an appropriate choice of the involved activation functions, the gradient
of the neural INGARCH model can be calculated efficiently which allows for
fast likelihood maximization. Information criteria can be used to determine
the optimal order $(p,q)$ and the optimal complexity of the network $H$.
Regarding inference, concepts from non-linear regression such as marginal
effects at the mean and numerical Hessian matrices are employed. For the
present heavily overdispersed counts, a generalized Poisson distribution with
additional variance parameter is useful.
A final remark concerns the implications of the results, in particular in
relation to those obtained by Dungey et al. (2020). One of their main models
is a threshold INARMA model (TINARMA) which leads to the identification of two
regimes which they label “systemic” and “non-systemic”. Among other aspects,
this implies that the number of (countries experiencing) banking crises is
mainly explained endogenously. On the contrary, the (non-linear) time trend
suggested in this paper offers an exogenous explanation and enables the model
to represent the zero counts during the BW period very accurately.

##  5 Modeling bounded counts

As a second example, bounded counts (with upper limit $n$) are considered.
Then, the binomial distribution is employed as the conditional distribution
instead of the Poisson distribution. Because of the parametrization of the
binomial distribution, the INGARCH regression equation now refers to the
(conditional) success probability instead of the (conditional) mean. The
binomial INGARCH model is defined by

$Y_{t}\sim Bin(n,P_{t})\mbox{ with
}P_{t}=f(Y_{t-1},\dots,Y_{t-p},P_{t-1},\dots,P_{t-q}).$ (9)

The main consequence for the ANN architecture is that the activation function
$g_{1}$ has to yield a probability and therefore, should map to the interval
$[0,1]$. An obvious choice is the logistic function which is generally
employed as the activation function for the hidden layer $g_{0}$ in this
paper, so $g_{0}(x)=g_{1}(x)=\frac{1}{1+\exp(-x)}$ for the neural binomial
INGARCH model. As explained previously, the function $g_{1}$ takes the role of
the response function in conventional INGARCH models, meaning here that the
conventional binomial INGARCH model with logit link (i.e. logistic response)
function (cf. Chen et al., 2020) can be interpreted as the corresponding
degenerate version of the neural binomial INGARCH model. An alternative choice
would be the soft-clipping activation function. This function constitutes an
almost linear response function and the idea is similar to the softplus
response function for unbounded counts. A corresponding soft-clipping binomial
INGARCH model is investigated by Weiß & Jahn (2022).

As the data example, the monetary policy decisions of the Monetary Policy
Council (MPC) of the National Bank of Poland are considered. Precisely, the
data are the number of MPC members voting in favor of an adjustment of the
interest rate at the monthly meetings between 2002 and 2013 (139 months).
There are $n=10$ members so we have a monthly time series of bounded counts.
The data were originally analyzed by Sirchenko (2013, 2020) and then also by
Möller et al. (2018).
The procedure to determine an appropriate binomial neural INGARCH model is the
same as for the previously considered Poisson neural INGARCH model for
unbounded counts. First, the INGARCH model order $(p,q)$ is selected based on
the conventional counterpart which is the binomial INGARCH model with logistic
response function (cf. Chen et al., 2020). Table 3 shows the information loss
for different model orders ($p+q\leq 3$ considered). The BIC selects
$(p,q)=(2,0)$ by some margin and the AIC does not provide a clear decision
between $(p,q)=(2,0)$ and $(p,q)=(3,0)$, so the BIC-optimal order is chosen.
Regarding the network complexity $H$, the rule of thumb yields
$H\leq\frac{0.1T}{J+1}=\frac{13.7}{4}=3.425$ and, therefore, $H=3$ is
selected. The corresponding AIC and BIC values are displayed in Table 3 and
they imply that updating the logistic model to a neural model is sensible in
terms of the information loss. Figure 6 helps explaining possible reasons.
First note that the observed counts are characterized by multiple zero runs of
medium length on the one hand and a relatively large share of high values on
the other hand. The predicted conditional success probabilities (Figure 6 (a))
from both models mostly stay at a medium level which is “the best they can
do”. Similar to the previous example of bounded counts, the neural model
achieves a higher probability of staying in the zero-count state (assuming
$y_{t-1}=y_{t-2}=0$). The calculations yields $p_{0|0,0}^{logi}=0.16$ and
$p_{0|0,0}^{ANN}=0.35$. The marginal effect of $y_{t-1}$ in the neural model
resembles a polynomial but is in general agreement with the positive marginal
effect from the logistic model. From subfigure (c), it could be suspected that
the effect of $y_{t-2}$ is insignificant and that allowing for a nonlinear
effect of $y_{t-1}$ eliminates the need to consider $y_{t-2}$.

---
(a) Predicted cond. success probabilities
|
(c) Marginal effect of $y_{t-1}$ |  (d) Marginal effect of $y_{t-2}$
Figure 6: MPC votes: Predictions and marginal effects

The analysis of the Pearson residuals (Table 4, top) shows that none of the
models captures the dispersion behavior of the monetary policy votes in a
satisfactory way. Still, the neural model produces the most adequate fit which
is in line with the lower information loss (Table 3). An extended model with a
time trend was also tested but, other than in the previous data example, the
goodness-of-fit could not be improved further due to the nature of the zero
runs. Möller et al. (2018) consider various zero-inflated models for the same
time series. In this paper, the example primarily serves to motivate the
proposed novel neural INGARCH framework, so only the simplest option to
implement zero-inflation will be discussed. The conditional binomial
distribution is then replaced by the following zero-inflated binomial (ZIB)
distribution with parameters $\omega,n$ and $p$:

$P(X=k)=\mathbbm{1}_{[k=0]}\omega+(1-\omega)P_{n,p}^{bino}(X=k).$ (10)

It can be useful to reparametrize with $\pi=(1-\omega)p$ because the expected
value is $E[X]=n\pi$. The variance is
$Var(X)=n\pi(1-\pi)+n(n-1)\frac{\omega}{1-\omega}\pi^{2}$ (cf. Möller et al.,
2018, eq. 2.4), so corresponding INGARCH models feature (conditional) extra-
binomial variation (for $\omega>0$). Estimating the two ZIB-INGARCH(2,0)
models, a massively reduced information loss is obtained (Table 3, middle).
Therefore, it is not surprising that the zero-inflation parameter is quite
large, $\hat{\omega}=0.43$ for both models. However, considering Table 4
(middle), it is found that the better representation of the dispersion
structure comes at the cost of serial correlation within the Pearson
residuals. Such a trade-off is also observed by Möller et al. (2018) where the
two best models in terms of the variance of the Pearson residuals (closest to
1) feature high levels of serial correlation (Möller et al., 2018, Table 6).
Referring to Sirchenko (2020) a possible solution is to look more closely at
the nature of the zeros. The meaning of the zero count is that all MPC members
vote for the interest rate to remain unchanged. He distinguishes three
different situations when this can occur: in periods of policy tightening; in
periods of maintaining between rate reversals; and in periods of policy easing
(Sirchenko, 2020, p. 1). In the neural INGARCH model, these regimes can be
represented by just a single additional variable with value $-1$ (easing), $0$
(maintaining) or $1$ (tightening). This result in a model with different
regimes for each value of the policy variable. In particular, the probability
of staying in the zero-count state now depends on the policy regime. The
estimation yields $p_{0|0,0,-1}\approx p_{0|0,0,1}=0.32$ and
$p_{0|0,0,0}=0.74$, so a distinction between at least two regimes should be
made. Figure 7 displays the corresponding predicted cond. success
probabilities and indeed, the zero runs are now represented much better. This
is confirmed by the reduced information loss (Table 3, bottom). In fact, the
model outperforms the best model discussed in Möller et al. (2018, Table 5) in
terms of AIC and BIC. Considering the Pearson residuals, the extended model
offers a compromise between the variance and the autocorrelation (Table 4,
bottom). The (two-sided) 95% critical value for the ACF is
$\frac{1.96}{\sqrt{137}}=0.167$, so the autocorrelation is not (highly)
significant.

Figure 7: MPC votes: Predicted cond. success probability by the neural ZIB-INGARCH(2,0) model with policy regime variable response | cond. distribution | policy regimes | order | AIC | BIC
---|---|---|---|---|---
logistic | binomial | no | (1,0) | 1029.1 | 1034.9
logistic | binomial | no | (2,0) | 989.0 | 997.8
logistic | binomial | no | (3,0) | 988.3 | 1000.0
logistic | binomial | no | (1,1) | 1009.8 | 1018.6
logistic | binomial | no | (2,1) | 990.9 | 1002.6
logistic | binomial | no | (1,2) | 989.6 | 1001.3
neural | binomial | no | (2,0) | 868.0 | 903.1
logistic | ZIB | no | (2,0) | 610.0 | 621.6
neural | ZIB | no | (2,0) | 581.9 | 619.9
neural | ZIB | yes | (2,0) | 532.0 | 578.7
Table 3: MPC votes: Information loss for different models response | cond. distribution | policy regimes | order |  mean($r_{t}$) |  var($r_{t}$) |  ACF(1)($r_{t}$) |  ACF(2)($r_{t}$)
---|---|---|---|---|---|---|---
logistic | binomial | no | (1,0) | -0.022 | 5.256 | -0.039 | 0.188
logistic | binomial | no | (2,0) | -0.031 | 5.105 | 0.037 | 0.036
logistic | binomial | no | (3,0) | -0.031 | 5.145 | 0.034 | 0.033
logistic | binomial | no | (1,1) | -0.043 | 5.131 | 0.023 | 0.090
logistic | binomial | no | (2,1) | -0.030 | 5.111 | 0.033 | 0.032
logistic | binomial | no | (1,2) | -0.062 | 5.120 | 0.023 | 0.036
neural | binomial | no | (2,0) | 0.006 | 4.853 | 0.028 | 0.002
logistic | ZIB | no | (2,0) | 0.004 | 1.197 | 0.389 | 0.323
neural | ZIB | no | (2,0) | 0.029 | 1.141 | 0.381 | 0.307
neural | ZIB | yes | (2,0) | 0.017 | 1.470 | 0.193 | 0.151
Table 4: MPC votes: Properties of (empirical) Pearson residuals $r_{t}$ for
different models

Concluding the MPC votes data example, it was shown that the neural INGARCH
model, with a suitable choice of the involved activation functions, can also
be employed for time series of bounded counts. Both layers in the neural model
are then activated by the logistic function as the regression equation refers
to the conditional success probability. The higher likelihood of the neural
model compared with the corresponding conventional INGARCH model generally
justifies the larger number of parameters. The zero-inflation can be accounted
for by using a conditional ZIB distribution instead of the binomial
distribution. Finally, as an extension beyond the pure INGARCH model, a policy
regime variable helps explaining the zero runs in times of stable interest
rates.

##  6 Conclusion and outlook

The paper introduced a novel class of INGARCH models where the response
function corresponds to an artificial neural network, more precisely, a single
hidden layer feedforward network. Due to the universal approximation property,
neural INGARCH models are very flexible regarding possible nonlinear
influences of the considered regressors and interaction effects between them.
Conventional INGARCH models can be interpreted as “degenerate” instances of
the corresponding neural INGARCH models. This implies that the activation
function of the output neuron in the SLFN takes the same role as the response
function in a conventional INGARCH model. Conversely, a neural INGARCH model
can be interpreted as an “upgraded” version of a corresponding conventional
model. Two data examples for unbounded and bounded counts demonstrate the
empirical relevance as the neural INGARCH models are preferred over their
conventional counterparts in terms of the information loss. Additional
explanatory variables such as a time variable or a regime variable can be
easily incorporated. Apart from the computationally demanding numerical
likelihood maximization, the main limitation of the neural INGARCH model is
the relatively high number of parameters which makes it difficult to apply it
to short time series with $T<100$.
Regarding future research, the possibilities to extend the use of ANN are
manifold. First, the framework can obviously be generalized to ANN regression
functions with more than one hidden layer (deep networks). The associated
research question could then be whether or in which situation more layers with
less neurons are preferred over one layer with more neurons. Furthermore, the
nonlinear modeling capability of the neural INGARCH model could be utilized
for multivariate analysis. For example, multiple output neurons could be
considered to obtain a type of nonlinear VAR model. In case of spatial data,
spatio-temporal INGARCH model could provide more accurate estimation of how
disturbances spread over space and time compared to linear models.

## References

  * Chen et al. (2020) Chen H, Li Q, Zhu F. 2020. Two classes of dynamic binomial integer-valued ARCH models. Brazilian Journal of Probability and Statistics 34(4): 685–711.
  * Dungey et al. (2020) Dungey M, Martin V, Tang C, Tremayne A. 2020. A threshold mixed count time series model: estimation and application. Studies in Nonlinear Dynamics & Econometrics 24(2): 20180029.
  * Famoye (1993) Famoye F. 1993. Restricted generalized Poisson regression model. Communications in Statistics - Theory and Methods 22: 1335–1354.
  * Ferland et al. (2006) Ferland R, Latour A, Oraichi D. 2006. Integer-valued GARCH processes. Journal of Time Series Analysis 27(6): 923–942.
  * Fokianos et al. (2009) Fokianos K, Rahbek A, Tjøstheim D. 2009. Poisson autoregression. Journal of the American Statistical Association 104(4): 1430–1439.
  * Fokianos and Tjøstheim (2011) Fokianos K, Tjøstheim D. 2011. Log-linear Poisson autoregression. Journal of Multivariate Analysis 102: 563–578.
  * Jahn (2020) Jahn M. 2020. Artificial neural network regression models in a panel setting: Predicting economic growth. Economic Modelling 91: 148–154.
  * Kock and Teräsvirta (2014) Kock AB, Teräsvirta T. 2014. Forecasting performances of three automated modelling techniques during the economic crisis 2007-2009. International Journal of Forecasting 30: 616–631.
  * Möller et al. (2018) Möller T, Weiß CH, Kim HY, Sirchenko A. 2018. Modeling zero inflation in count data time series with bounded support. Methodology and Computing in Applied Probability 20: 589–609.
  * Reinhart and Rogoff (2009) Reinhart C, Rogoff K. 2009. This Time is Different: Eight Centuries of Financial Folly. USA: Princeton University Press.
  * Rivals and Personnaz (2000) Rivals I, Personnaz L. 2000. Construction of confidence intervals for neural networks based on least squares estimation. Neural Networks 13: 463–484.
  * Sirchenko (2013) Sirchenko A. 2013. A model for ordinal responses with an application to policy interest rate. NBP Working paper No 148.
  * Sirchenko (2020) Sirchenko A. 2020. A model for ordinal responses with heterogeneous status quo outcomes. Studies in Nonlinear Dynamics & Econometrics 24(1): 20180059.
  * Tkacz (2001) Tkacz G. 2001. Neural network forecasting of Canadian GDP growth. International Journal of Forecasting 17: 57–69.
  * Weiß (2018) Weiß CH. 2018. An Introduction to Discrete-Valued Time Series. John Wiley & Sons, Inc., Chichester.
  * Weiß et al. (2020) Weiß CH, Scherer L, Aleksandrov B, Feld M. 2020. Checking Model Adequacy for Count Time Series by Using Pearson Residuals. Journal of Time Series Econometrics 12(1): 20180018.
  * Weiß & Jahn (2022) Weiß C.H., Jahn M. 2022. Soft-clipping INGARCH models for time series of bounded counts. Statistical Modelling. doi:10.1177/1471082X221121223.
  * Weiß et al. (2022) Weiß CH, Zhu F, Hoshiyar A. 2022. Softplus INGARCH models. Statistica Sinica 32: 1099–1120.