# Engineering the Neural Automatic Passenger Counter

Nico Jahn1, Michael Siebert1
Interautomation Deutschland GmbH,
Berlin, Germany
{nico.jahn<EMAIL_ADDRESS>
1 All authors contributed equally to this work

###### Abstract

Automatic passenger counting (APC) in public transportation has been
approached with various machine learning and artificial intelligence methods
since its introduction in the 1970s. While equivalence testing is becoming
more popular than difference detection (Student’s t-test), the former is much
more difficult to pass to ensure low user risk. On the other hand, recent
developments in artificial intelligence have led to algorithms that promise
much higher counting quality (lower bias). However, gradient-based methods
(including Deep Learning) have one limitation: they typically run into local
optima. In this work, we explore and exploit various aspects of machine
learning to increase reliability, performance, and counting quality. We
perform a grid search with several fundamental parameters: the selection and
size of the training set, which is similar to cross-validation, and the
initial network weights and randomness during the training process. Using this
experiment, we show how aggregation techniques such as ensemble quantiles can
reduce bias, and we give an idea of the overall spread of the results. We
utilize the test success chance, a simulative metric based on the empirical
distribution. We also employ a post-training Monte Carlo quantization approach
and introduce cumulative summation to turn counting into a stationary method
and allow unbounded counts.

###### Index Terms:

Automatic passenger counting, cumulative summation layer, ensemble method,
error simulation, grid search, Long Short-Term Memory (LSTM), quantization

##  I Introduction

Assessment of passenger counts is of paramount importance for public transport
agencies to plan, manage and evaluate their transit service. Over the past
three decades, automatic passenger counting (APC) systems have played an
increasingly important role in determining the number of passengers in public
transport. They are used in the daily monitoring of operations, in long-term
demand planning, as well as in revenue sharing within transport associations
around the world, which is in the billions annually [1, 2, 3, 4] by both
ticket revenues, and public funds [5] and directly affects services of general
interest. Systematic over- or undercounts, so-called biases can thus lead to
large unfairness and should thus be as small as possible. The VDV 457 is the
prevalent standard that defines how APC systems are officially validated [6,
7].

Figure 1: Left image: Two automatic passenger counting systems mounted above
doors, a permanently installed stereographic camera to the left, and a
temporary installed, battery-powered system with a LiDAR/ToF camera to the
right. Center image: a person is triggering a door opening as seen from the
LiDAR system. Right image: a floor height color scale for the center image.

Modern APC systems are based on depth-sensing cameras mounted above doors in
busses, trains, trams, and other vehicles of public transport (see Figure 1).
The recorded 3D videos start with a door opening, passengers board and alight,
sometimes pushing and carrying objects along like bicycles and baby buggies,
and at the end of the video, the door closes again. This sequence of events is
called a video of a door opening phase or in the following _video_ for short.

[8] introduce and evaluate the Neural Automated Passenger Counter (NAPC), a
deep neural network based on a long short-term memory (LSTM) [9] architecture
[8]. It is capable of learning automatic passenger counting _tabula rasa_ from
videos and total manual counts of e.g. boarding and alighting passengers at
the end of each video. To illustrate this, the information _there have been 2
boarding and 3 alighting passengers_ already is a label to an entire video,
timestamps are not required. This property lowers the effort in acquiring the
labels drastically over approaches like object detection [10, 11] or even
segmentation [12]. Furthermore, the computational effort of the NAPC is
comparably low to the aforementioned approaches. Training can thus be executed
on consumer GPUs and inference even on legacy hardware systems, see Section
III-E.

We tried to improve the original NAPC architecture by changing popular
hyperparameters like the learning rate, batch size, LSTM layer count and
depth. What can be considered as a success is the introduction of _learning
rate decay_ (LRD): before, we have used a _selection set_ to determine a
suitable epoch. With LRD, we can pick the last epoch, making the process
easier to handle. However, besides LRD, our results stayed inconclusive for
months, so there was no improvement. Therefore, we switched our exploration to
more fundamental parameters like various random seeds and training data
selection. It is known from other fields like operation research [13] and
recent findings for neural networks [14] as well that such seemingly minor
choices can have even more impact than algorithms or other design decisions.
Another challenge was the lack of a suitable metric, so over time, we created
a new one with a universal approach: the _test success chance_. Finally, we
did an evaluation on how _stationary_ the NAPC is, determined a limit, and
found yet another very universal approach to remove it.

We organize the paper as follows: in Section II we introduce the methods we
have used. These comprise efficient labeling, LRD, grid search, quantization,
ensemble building, and simulative metrics. Then, we present challenges and our
results in Section III. Finally draw a conclusion in Section IV.

###  I-A Previous Work

Over the recent years, together with our research partners, we have developed
multiple passenger count algorithms, e.g. [15], which quality-wise was
comparable to non-neuronal methods of other manufacturers. [8] introduced the
NAPC [8] to predict the boarding and alighting passengers on the Berlin-APC
[16] dataset. It consists of roughly 13.000 videos of door opening phases and
their total boarding and alighting passenger counts at the video end. A
sequence-to-sequence LSTM model predicts the current absolute count for every
frame (ranging from 56 to 3275 frames per sequence at 10 frames per second),
which means that the prediction on the last frame is trained to match the
label and thus is solely used to calculate the accuracy metric. Since the
labels are counts, they are all integral, and thus the NAPC predictions are
rounded as they typically have decimals. A special bounding box loss function
covers all intermediate predictions. The maximum amount of passengers in a
_class_ (here: boarding or alighting adults) is 67. However, the dataset
contains mostly labels with low counts, and videos with high counts are scarce
(for details, see Table I within "Number of Events" in [8]).

The NAPC architecture consists of an input and output fully connected (FC)
layer and an LSTM core in between (for details, see Figure 2 in [8]).
Therefore, it does not explicitly use spatial information and interprets each
video frame as a flattened 500-dimensional vector. The output is a regression
of the cumulative counted passengers until the corresponding frame. [8] report
a maximum overall accuracy of ~97% dropping to 55.17% with their best model in
_more difficult_ situations like "More than 20 boarding passengers" [8]. The
global relative bias $\Delta^{(\mathrm{global})}$ and an equivalence test for
a 95% confidence interval are used as an additional metric, fitting the
passenger counting task. This is the same methodology as in the recent VDV 457
v2.1 [7], which, however, requires 99% for high precision systems.

###  I-B Related Work

The NAPC as introduced in [8] can be considered as a _stationary method_
operating on _non-stationary data_ since its counts are bounded (see Section
III) but the labels can basically be arbitrary large. For time series, methods
to change non-stationary to stationary data, two popular methods can be
applied: The forward difference operator, which is defined by [17] as

$\Delta_{h}f(t)=\frac{f(t+h)-f(t)}{h}$

and the backward difference operator as

$\nabla_{h}f(t)=\frac{f(t)-f(t-h)}{h}$

analogously for the time scale $h\in\left]0,1\right]$. According to [17] it
was first proposed by [18] in 1989. The cumulative sum over a sequence, the
approach we use, can be considered the inverse operation of the backward
difference operator. The differentiation and integration approach is applied,
e.g. to make stock-market predictions [19]. In economics, autoregressive
integrated moving average (ARIMA) models are used to make time-series forecast
predictions on non-stationary data (for details, see Chapter 4 of [20]). The
integration step is a cumulative sum over all previous predictions (i.e., an
integration over a stationary process). Autoregressive (AR) and moving average
(MA) properties help the model to predict future time-steps based on previous
predictions (including the errors of previous predictions). Models with those
features but no integration are categorized as ARMA. The integration step
removes the trend of the time series.

Another topic is the use of legacy computational platforms, which we use for
inference: since some have limited floating-point capabilities, we have
developed a _universal post-training Monte Carlo quantization method_ , which
can, in particular, be applied to LSTMs. The term _quantization_ refers to the
process of reducing the resolution from floating-point to fixed-point numbers
while retaining an overall similarity of the values. [21] [21] investigated
the effects of multilayer neural network quantization and concluded that 10
bits are necessary for their experiments. However, they used only weight
quantization on linear-only connections. More than two decades later [22]
propose their schematics to quantize recurrent neural networks down to 8-bit
resolution [22]. They leverage quantization-aware training procedures,
allowing for reduced quantization errors post-training. They successfully
report the quality of their method with wider LSTM layers (compared to [8]).
[23] proposed a more recent solution to LSTM quantization with and without
quantization-aware training in li2021quantization [23]. The authors reduce the
resolution for weights and activations to 8-bit fixed-point arithmetics and
maintain a 16-bit fixed-point when necessary (e.g. cell states). Not only is
quantization relevant to legacy platforms, but it can also be used to fully
utilize specialized neural network ASICs (Application-Specific Integrated
Circuit) like the Google Edge TPU [24].

Another aspect we cover is ensembles: To improve equivalence test success
chances by reducing bias, we have combined multiple networks into _NAPC
ensembles_. Ensemble methods are used to improve the quality of a prediction
of a single _expert model_ through the combination of multiple _experts_.
Those members of an ensemble then vote, or their predictions are combined such
that the resulting selected prediction is a collective solution. Multiple
experiments [25, 26] show that ensemble methods can improve the predictions.
Keeping the ensemble size small is another objective. Since each member
processes the input, the computations grow with each additional member. [27]
mentions that using all available instances (neural networks in our case)
could also result in a worse ensemble performance than choosing a subset of
those [27].
Multiple learners can be created by e.g. _Bagging_ , _Boosting_ , and
_Stacking_ , which have different properties and limitations. Bagging
(bootstrap aggregating) averages through all predictions in regression, or
uses plurality voting for classification tasks [28]. All members are trained
independently and can therefore be parallelized. On the other hand, Boosting
optimizes recursively and is not parallel. The Adaptive Boosting (AdaBoost)
algorithm emphasizes the wrongly predicted samples of one model to train
another model with those. The collection of those repeatedly optimized models
is weighted and then combined [29]. [30] introduced _stacking_ (or _stacked
generalization_) in 1992 as a method to combine the output of one or more
_generalizers_ [30]: _Level 0 algorithms_ are optimized on input data and
_level 1 algorithms_ combine level 0 algorithm outputs to create its own
outputs. More levels can be defined analogously.
Another aspect of ensemble building is the pruning method to limit the number
of learners. Those methods can be categorized into _Ranking_ , _Clustering_ ,
_Optimization_ and others [31]. Ranking-based methods use measurable quality
criteria to rank all the available algorithms and take the $k$ best performing
to build the ensemble (with size $k$). Clustering-based methods do not need a
labeled pruning set as the other methods do, as it clusters the models based
on a distance measurement (could be done on artificial data) and prunes the
clusters until $k$ members are left. Optimization-based ensemble pruning
methods like _Centralized Objection Maximization for Ensemble Pruning_ (COMEP)
by [32] starts with a single ensemble member and optimizes an objective
iteratively for each new member [32]. Other methods like genetic algorithms
[33] can be used as optimization-based ensemble pruners [31] as well.

During our work, another issue emerged: Non-determinism. To conduct research
on and potentially exploit the impact of neural network initialization or the
training order for the NAPC as in [13], _determinism_ is required. This has
been an issue with GPU accelerated training, which relies on modern research
in the area. As stated by [34] even with fixed seeds, a robust reproducibility
of experiments used to be not achievable [34]. The non-determinism they
measure (variance after training with the same seeds) is only traceable to the
GPU floating-point operations, as the initialization of the ResNet [35]
weights and the order of samples (mini-batches) is fixed. After training 50
models with the same and different seeds, they measure the proportion of
variance in the metrics introduced by the GPUs. For the standard deviation of
the model’s accuracy, the GPU non-determinism contributed around 74% of the
overall variance on the test set, and for the loss, it is more than 87% of the
standard deviation’s variance on the test set. However, we could utilize more
recent developments that enable deterministic GPU accelerated training [36].

###  I-C Contributions

This work extends the original NAPC with several engineering and research
optimizations:

  * •

efficient labeling

  * •

post-training quantization, including LSTMs

  * •

introducing LRD to the training

  * •

cumulative summation in training and inference

  * •

test success chance as metric obtained by simulation

  * •

a grid search to identify important parameters

  * •

increased accuracy and less bias by ensemble building

##  II Methods

We use the network architecture mentioned in [8] as baseline predictor with 5
LSTM layers with 50 LSTM units each [8]. The following methods presented in
the subsections are partially combined during the experiments. We did not use
the network optimizations from Section II-B within the grid search (Section
II-C3). We leverage mixed-precision throughout the training supplied by the
Tensorflow framework [37, 38].

###  II-A Labeling

To efficiently gather the large amounts of labels required to train NAPCs, we
have developed _VisualCount_ , a specialized tool that utilizes game
controllers. The main advantage is the analog buttons that allow navigating
through a video in both directions at the most appropriate speed: slow in
crowds, fast in idle situations, and backward to quickly review complex
scenes. It also allows to enable and disable a 3D height map overlay. This
allows to distinguish between children and adults by using a certain height
threshold (e.g. 1.20m), see Figure 2. At time of writing, around half a
million manual video counts have been made with VisualCount.

Figure 2: VisualCount: A specialized video software to perform manual counts,
which we have used to obtain our count data. The height information from 3D
videos can be used to distinguish between children and adults, e.g. by using a
critical height of 1.20 meters. To these standards, the tiny person in the
center of the image above would be considered an adult, while the one to the
left is a child. In the case of our specialized software, we have achieved a
70% speedup over the use of a standard video player with a spreadsheet
program. It runs entirely in the browser; thus, it can operate without a
server by using videos from the user’s filesystem, which increases data
privacy. Users utilize game controllers to navigate through the video with
playback or rewind speeds corresponding to the pressure applied to the analog
buttons to obtain both slow-motion for crowds and high speeds (10x and more)
to skip through long idle timespans. Thanks to a passthrough on filesystem
call level, video loading times are typically seconds, even for hundreds of
gigabytes of video data.

###  II-B Network Optimizations

The NAPC performs weaker in more challenging scenarios (see Figure 6 in [8])
and in particular for high boarding and alighting counts. Predicting non-
stationary data (without trend estimation or elimination) is often avoided in
regression (compare ARMA to ARIMA in Section I-B) by converting it to
stationary data. However, this is not possible in the case of NAPC since
labels always refer to a range of video images in which the event (e.g.
boarding passenger) happens. Therefore, instead of modifying the data, we
modify the neural network and add a cumulative sum at the end, the so-called
_cumsum layer_ : Before, for each video image, the LSTM needed to report the
total count until that image. With the cumsum layer, since frame-wise outputs
are summed up, the LSTM only needs to report a frame-to-frame difference.
Therefore, the cumsum also preserves the temporal correlation of single events
over multiple frames, and the neural part of the network does not need to
perform and understand the concept of _summation_ anymore. Instead of changing
the data, we thus have improved the NAPC to predict on non-stationary data,
compare Figure 3.

###  II-C Training Optimizations

####  II-C1 Data Storage

We have stored all our video data in a raw Numpy [39] array with an additional
file that contains the names and offsets of the individual videos and accessed
it via a simple Python class. The advantage of this bare-metal approach is
that no notable data parsing delays occur once the Numpy file is created,
which can be significant for other file formats. Also, we made use of memory
mapping, which reduces the data load time basically to zero so code changes
can be evaluated _interactively_. Since the Numpy files can become hundreds of
gigabytes large, we created them by reading the video files individually and
adding the pixel data by appending to the file with _npy-append-array_
111https://github.com/xor2k/npy-append-array. Together with memory mapping,
this technically very basic approach allows using data sources larger than the
main memory of the training machine.

####  II-C2 LRD

Due to the problem of diverging and collapsing networks during training, we
introduce LRD. It decreases the gradient step size over time, stabilizes the
learning procedure, and allows us to pick the last epoch of the training run.
It eliminates the need to pick an epoch by using a selection set. Therefore,
more videos are available to the training and test set as compared to the
previous solution. It was already stated by [[]p.20]bengio2012practical that
the learning rate is one of the essential hyperparameters.

####  II-C3 Grid Search

We wanted to know to what amount similar effects as in [13] can occur for deep
learning neural networks. In [14], the authors focus on neural network
initialization while we evaluate multiple randomness aspects at the same time
and their impact on the quality of the NAPC. We split the data into mutually
exclusive groups to measure the impact of the training data on the results.
Our grid search covers the product of the following ranges:

  1. 1.

group selection: 4 options

     * •

Uniformly shuffling 8000 videos

     * •

Dividing into 4 mutually exclusive groups (with index $g$) with 2000 videos
each

  2. 2.

data amount: 2 options

     * •

2000 training videos, selected by group index $g$ (smaller training set)

     * •

6000 training videos, everything but group index $g$ (larger training set)

  3. 3.

weights random seed: 10 options

     * •

Affects only the initialization of the weights

  4. 4.

training random seed: 4 options

     * •

Affects order of training videos in the mini-batches (and the concatenation of
the videos)

     * •

Affects dropout regularization [41] (i.e., the random masking of values)

###  II-D Inference Optimizations

####  II-D1 LSTM Quantization

[42] pointed out that the weights of artificial neural networks are trained as
floating-point numbers since they allow for higher precision and more stable
learning [42]. Previous work on post-training quantized convolutional neural
networks revealed the small impact of lower precision weights on the
prediction quality. Major frameworks such as Tensorflow [38], and PyTorch [43]
have build-in tools to reduce the machine precision of the trained weights and
to quantize them to fixed-point numbers. However, LSTMs are not well supported
or in a beta state. As they are the core element of the NAPC, quantization is
needed to perform faster inference. Our custom framework approach is a greedy
Monte Carlo random search of input and weight scales that processes each layer
in sequential order. This approach reduces the computational effort and time
required instead of optimizing all layers at once. The quantizer uses several
hand-picked calibration videos (less than a dozen) as a reference, and any
quantized model is required to predict within an error margin $\varepsilon$
for every timestep. No labels are required for this process. Also, we have
kept the reference video set intentionally small. A benefit is not only a fast
quantization process but also an _increase in variability_ : we only require
the quantized models to be within an error margin for a few videos and thus
only relatively _loosely related_ to the non-quantized model. We treat a
quantized model as if it had been an _entirely unrelated_ to its non-quantized
counterpart, so it undergoes full validation just like any other NAPC. This
approach allows us to increase our training pool size without training more
models.
From a structural point of view, apart from the activation functions, the
network consists of two FC layers and an LSTM core. In the following,
variables with $\widetilde{\cdot}$ are quantized and therefore scaled
versions, and $\times$ denotes the element-wise product. Scalars are denoted
with lower case letters in the following equations, while vectors and matrices
are denoted with upper case letters.
FC layers are stateless, and thus the layer indexing does not consider
timesteps but only placement within the network. Such layer with index $n$
receives the scaled output $\widetilde{O}_{n-1}$ of the previous layer $n-1$
as input, as well as the normalization factor $o_{n-1}$. Such that

$\displaystyle O_{n-1}=\frac{\widetilde{O}_{n-1}}{o_{n-1}}-\varepsilon$

as $\widetilde{O}_{n-1}$ is the scaled and quantized version of $O_{n-1}$
which could have rounding errors $\varepsilon$. When searching for appropriate
input and weight scales ($i_{n}$ and $w_{n}$ respectively), we have to scale
the input $\widetilde{O}_{n-1}$, weights $W_{n}$ and biases $B_{n}$ for layer
$n$ accordingly. Therefore, we calculate the output $\widetilde{O}_{n}$ and
its scale $o_{n}$ as follows:

$\displaystyle\widetilde{I}_{n}={}$
$\displaystyle\frac{\widetilde{O}_{n-1}i_{n}}{o_{n-1}}$
$\displaystyle\widetilde{W}_{n}={}$ $\displaystyle W_{n}w_{n}$
$\displaystyle\widetilde{B}_{n}={}$ $\displaystyle B_{n}w_{n}i_{n}$
$\displaystyle\widetilde{O}_{n}={}$
$\displaystyle\widetilde{I}_{n}\widetilde{W}_{n}+\widetilde{B}_{n}$
$\displaystyle o_{n}={}$ $\displaystyle w_{n}i_{n}\ \ .$

We extend the previous notation to the stateful case of LSTMs with the
timestep $t$ superscript. Again we have the output $\widetilde{O}^{t}_{n-1}$
of the same timestep, but the previous layer. Additionally, for LSTMs there
are the hidden $\widetilde{H}^{t-1}_{n}$ and cell $C^{t-1}_{n}$ state of the
same layer of the previous time step, where the first one is scaled with
$i_{n}$ and the latter one is always unscaled (i.e., floating-point). We
define $M[\\{i,f,o,c\\}]$ as the non-activated floating-point LSTM gate block
matrix, which is calculated with the additional recurrent kernels $U_{n}$, as
well as the weights $\widetilde{W}_{n}$, and biases $\widetilde{B}_{n}$; which
are block matrices themselves with the same indices. Furthermore,
$\operatorname{\sigma}(\cdot)$ is the logistic sigmoid activation function.
Those definitions lead to the following calculations:

$\displaystyle\widetilde{I}^{t}_{n}={}$
$\displaystyle\frac{\widetilde{O}^{t}_{n-1}i_{n}}{o_{n-1}}$
$\displaystyle\widetilde{W}_{n}={}$ $\displaystyle W_{n}w_{n}$
$\displaystyle\widetilde{U}_{n}={}$ $\displaystyle U_{n}w_{n}$
$\displaystyle\widetilde{B}_{n}={}$ $\displaystyle B_{n}w_{n}i_{n}$
$\displaystyle M[k]={}$
$\displaystyle\frac{\widetilde{I}^{t}_{n}\widetilde{W}_{n}[k]+\widetilde{H}^{t-1}_{n}\widetilde{U}_{n}[k]+\widetilde{B}_{n}[k]}{w_{n}i_{n}},$
$\displaystyle\text{for all }k\in\\{i,f,o,c\\}$ $\displaystyle C^{t}_{n}={}$
$\displaystyle\operatorname{\sigma}\left(M[i]\right)\times\tanh\left(M[c]\right)$
$\displaystyle+\operatorname{\sigma}\left(M[f]\right)\times C^{t-1}_{n}$
$\displaystyle\widetilde{H}^{t}_{n}={}$ $\displaystyle
i_{n}(\tanh\left(C^{t}_{n}\right)\times\operatorname{\sigma}\left(M[o]\right))$
$\displaystyle\widetilde{O}^{t}_{n}={}$ $\displaystyle\widetilde{H}^{t}_{n}$
$\displaystyle o_{n}={}$ $\displaystyle i_{n}\ \ .$

As $\widetilde{H}^{t}_{n}$ is also the input for the next layer and timestep,
it is multiplied again with $i_{n}$.

####  II-D2 Ensemble Building

Ensemble methods are commonly used in machine learning to overcome the
weaknesses of a single predictor. We opt for ranking-based pruning [31], where
we sort the independently trained networks according to the probability of
passing the VDV 457 v2.1 and select the top $N$ networks. The final prediction
of the ensemble is determined via the $\tau$-th percentile or _quantile_ for
each class, which is particularly well suited to compensate for a bias and can
be tuned via a calibration set. This process is a stacking ensemble method as
we optimize a meta-model (quantile regression) based on the specific ensemble.
In the following, unless we combine ensembles specifically for different
training (sub-)datasets, the only differences between most ensemble members
are solely their random seeds. Also, note that, from a mathematical point of
view, the seemingly familiar and straightforward median-like functions like
quantile are highly non-linear, non-differentiable, and thus not trainable by
gradient descent. Therefore, they structurally provide added value to typical
neural network training.

###  II-E Simulative Metrics

Deep Learning uses gradient descent and variations of it, which requires a
_differentiable_ loss function. Accuracy, while not being differentiable, is
constantly measured and reported during the training process and is a key
metric. However, for automatic passenger counting, accuracy is not well suited
since it is very much decoupled from the bias: an NAPC instance can have an
extremely high accuracy (e.g. 98%), but the bias might vary from 0% to
multiple percents because the few videos where the model’s prediction does not
match the label (e.g. the remaining 2%) typically contain difficult situations
with crowds and thus the neural network makes a large error. On the other
hand, to pass the VDV 457 v2.1 or the equivalence test, the standard deviation
is important as well. Our initial idea was to use some linear combination of
the absolute bias and standard deviation. However, this approach is quite
arbitrary, and not less complex as doing an entire equivalence test in the
first place. The drawback is that the sample size cannot be controlled, and
the possible outcomes are only whether the test was passed or failed with
nothing in between. Labeling tens of thousands of videos to run multiple
equivalence tests and average over tests would be affordable thanks to our
efficient labeling software mentioned in Section II-A, but it still scales
badly. Therefore, as called _test success simulation_ in [44], we use another
approach: we assume that the error made on our validation set generalizes well
enough so that it can be used as an _empirical_ error distribution to sample
from [45]. We then draw such a sample of the desired sample size (e.g. 3600
videos) and repeat the process (e.g. 10000 times), and determine the _test
success ratio_ or _test success chance_ ; which is a so-called _bootstrapping_
method [46]. Another advantage of this approach is that no mathematical model
for the error distribution with many incomprehensible parameters to tweak is
required, and multiple categories like boarding and alighting passengers can
be meaningfully combined into one continuous metric.

##  III Results

###  III-A Network Upper Bounds

Investigations revealed a limitation of the architecture from [8] (see Figure
3): the original NAPC (without cumsum) cannot count above a threshold of
around $100$ passengers, which can easily be observed by looping sequences
multiple times, e.g. a sequence with one boarding passenger 200 times. The
training of models with cumsum leads to unaffected predictions during
inference _regardless_ of the number of passengers. However, this is not the
only issue: networks without cumsum also suffer from random predictions of the
opposite class for sequences with more than 70 passengers.

Even though LSTMs are capable of learning non-linearities in general and
should therefore be able to predict non-stationary data, our experiments have
shown that there are limits. When applying cumsum to our model, the LSTM is
not required to learn and perform the arithmetic logic of counting (summation)
and thus does not need to compensate for the stochastic drift induced by
rising passenger counts anymore. Cumsum layers however, as can be seen in
Figure 7, come at a cost: the average model quality gets worse. Therefore,
training more neural networks (e.g. with different random seeds) is required
to obtain a NAPC pool with comparable success chance.

Figure 3: The effect of the cumsum layer: all NAPC instances (floating-point
and quantized) from the qualitative performance experiment displayed in Figure
7. Left plot: the expected number of boarding passengers after the last video
iteration is 200 and 0 alighting. However, without cumsum (continuous lines),
the highest counting NAPC instance has only reached a maximum of 91 boardings
(91.43 without rounding) passengers after the last repetition. This can be
considered a best-case upper limit, as the NAPC counts vary heavily between
individual instances. Although there were supposed to be none for alighting
passengers, the NAPC instances have counted up to 23 (22.64 without rounding)
in the left plot after the last repetition (worst-case scenario). The NAPC
instances with cumsum (dashed lines) counted on average 197 ($197.38\pm 3.3$)
boarding and 0 ($0.32\pm 0.73$) alighting passengers in the left plot, which
almost matches the labels (200 and 0). Additionally, models without cumsum can
– and some do – count backward after a certain number of video repetitions,
while the model with cumsum by design is unable to do so: the absolute
activation, which only produces values in $\mathbb{R}^{+}_{0}$, happens before
the cumulative summation. Right plot: the same scenario as with the left plot,
based on a video with zero boarding and one alighting passenger, the expected
label after 200 iterations would be 0 and 200. As one would expect, the
results are very similar: the NAPC instances without cumsum reached up to 15
(15.26) boarding and 90 (90.46) alighting passengers, while the instances with
cumsum as used in the left plot counted on average 0 ($0.34\pm 0.61$) boarding
and 193 ($193.29\pm 5.7$) alighting passengers, which is again a slight
undercount.

###  III-B Fixed Training

Initially, we could not train deterministically, even after setting all seeds.
Determinism within the framework, drivers, and hardware is required to e.g.
research the influence of random seeds and fortunately became available during
our research [36, 47]. As it turns out, the framework we used (TensorFlow) did
not support deterministic training out-of-the-box (without patching) before
v2.3 [47]. From that version on, it could be activated programmatically.
Further, an issue in one of the GPU libraries (cuDNN) affected determinism in
LSTMs, which was fixed (for details, see [48]) around the time we were about
to start the experiment. There are other factors limiting the reproducibility
of deep learning in general [49]:

  1. 1.

CPU architecture

     * •

E.g. whether it is x86 or ARM

  2. 2.

GPU manufacturer and GPU hardware architecture

     * •

E.g. whether a Nvidia Turing (RTX 20XX) or Ampere (RTX 30XX) was used

  3. 3.

Any software and driver version & build

  4. 4.

The operating system version & build

It turned out that, on top of fixed hardware, having fixed software versions
may suffice for determinism, but not for reproducibility: using the same
framework version, but from different sources (PIP, Conda, compiled from
source) resulted in different randomness and thus different, incompatible
training results. Therefore, we eventually used a TensorFlow docker image
which can be fixed by its hash, and used only one machine equipped with four
identical GPUs.

###  III-C Grid Search

Figure 4: Grid search results, equivalence test success chance as a function
of training progress (sample size $n=3600$). The training dataset size has the
greatest impact: more training data improves the success chance drastically.
This means that the capabilities of our small NAPC neural network variant have
not yet been exhausted. The spread among the different models can be very
high. After the training video count, the data group has the second-largest
impact. Although these groups have been chosen as a random division of the
full training dataset, data group 3 worsens success chance when absent and
improves it when present. The 4x ensemble using a $2/3\approx 66.6\%$ quantile
performs better than the best model most of the time. This is most likely due
to its ability to reduce undercounting since the other ensembles with 50%
quantile (3x ensemble) or $6/9\approx 55.5\%$ quantile (10x ensemble) yield
lower counts on average. In most but not all cases, the last training epoch
has the best quality, in particular when training with the larger 6000 video
set. For a sample size perspective, see Figure 5 for comparison. Figure 5:
Grid search results, equivalence test success chance at 100% training progress
(in contrast to the training process in Figure 4). NAPC instances trained with
more data tend to require a smaller sample size and thus less effort and cost
to succeed in the equivalence test, particularly for reliable success (success
chance $\geq$ 95%). Ensembles perform around as well as the best model. Note
that some lines are not convex. This occurs when there is a substantial,
single error spike (see Figure 6). All in all, NAPC models can be validated
safely with samples of around 3600 door opening phases, which is about 60%
less than 6147 stop door events or around 9000 door opening phases as
suggested in the VDV 457 v2.1 standard in the case of the vehicle we have
used. Latter large sample sizes we have proposed became standard in the pre-
NAPC era in 2018 so that the then prevalent legacy APC systems had a fair
chance to succeed. The left half of the figure shows that using more training
data (generalization) per NAPC training is favorable over the training with
less data (specialization) in general and for creating ensembles. We also
detected an artifact in the bottom right $4\times 4$ grid: The reddish
highlighted subplot at (training videos, weights seed, training seed) = (2000,
2, 3) indicates that not a single NAPC instance or ensemble would achieve a
95% success chance. That means, regardless of the data (group) used, with
those fixed seeds for the initialized weights and the randomness during the
training process, we could not succeed in training an acceptable model,
compare Section III-C. Figure 6: In Figure 5 some unexpected non-convex curves
occur when there is a single substantial error in the population. We extracted
this into a synthetic data example. For small sample sizes, chances to hit the
error are relatively low, thus the high success chance. As the sample grows,
the success chance worsens and increases again as the standard error increases
less. A sawtooth-like shape indicates that this property repeats on smaller
scales and can be found in real-world examples, although it is less pronounced
there.

Since variation among trained NAPC models was generally very high and we were
thus unable to reliable judge whether a modification to the architecture or
hyperparameters improved quality or deteriorated it, we investigated a more
fundamental aspect of training: randomness. Unlike [50, 51] we decided not to
fix random seeds but iterate over multiple. Such experiments can pinpoint a
random variable like the initialization of the neural network or the data
training order and its effect on the overall training. From other domains like
operation research, it is known that randomness can have a far more
significant influence than other popular design aspects [13] and has recently
been shown for deep learning as well [52]. Therefore, to achieve a minimal
level of conclusiveness, iterating over multiple seeds is necessary and can be
considered essential for research in general [53, 54].

There are several kinds of randomness during the training process: Firstly,
the selection of the subset of data used for training222[8] have done this
reproducibly, and independent of the framework in [8] already., secondly the
random initialization of the weights, and thirdly, the randomness during the
training, which spans from the data preprocessing and augmentation, to also
regularization methods. For a visual overview, compare Figure 4 and Figure 5.
The most obvious observation is that more training data (6000 vs. 2000 videos)
increases test success chance. The qualitative difference among the models can
be very high, which needs to be accounted for when making decisions about
deployment or further development, even without non-determinism introduced by
GPUs. Building (unranked) ensembles increase the success chances if a quantile
> 50% is chosen as it compensates undercounting, which the NAPC method seems
to be more prone to than overcounting. Even when all 8000 possible training
videos are available to the ensemble via its members, the count quality is
still higher if every ensemble member receives more training videos, so
generalization is favorable over specialization. There seems not to be a big
difference in the prediction quality introduced by the randomness in
initializing weights and training (dropout layer and composition of batches).
However, some configurations lead to a terrible count quality even with all
training videos being available to a bias-corrected ensemble (see reddish
highlighted subplot in Figure 5). This is an interesting and unexpected case:
when combining a certain neural network initialization seed with a certain
training order seed, _no matter the data_ , the training is almost certain to
fail. The mentioned seed combination would be a starting point for further
investigation and could lead to a better architectural understanding of the
neural network.

###  III-D Inference

####  III-D1 Quantization

We could almost entirely quantize our network, except for the non-linear LSTM
activation functions. Related work [55, 56, 57] leveraged restricted
activations, regularization during the training, or quantized training (fine-
tuning) to improve quantizability, while NAPC uses neither of these. Compared
to feed-forward neural networks, RNNs maintain an internal state. In our case,
videos can range from a few dozens to multiple thousands of frames with a
large variety of counts and situations. The varying length temporal dimension
makes it more difficult to analytically quantize networks as it is done for
other architectures [58]. We quantize all weights, biases, inputs, and outputs
to be an integer (i.e. fixed-point). We defined $\varepsilon$ as the absolute
error margin for each frame and class during quantization. It is set to at
most $0.45$ absolute deviation from the floating-point NAPC’s output as the
final predictions are on-average within less than $0.02$ to the closest
integer. The idea is to increase quantization potential as much as possible
since the quantized model needs to pass the same validation as its non-
quantized template model. With this approach, we discourage divergent counts
in most cases, resulting in less degraded prediction quality. If the quantized
version predicts more than $\varepsilon$ absolute deviation during the
process, the scale is discarded, and new scales for this layer are drawn
randomly. One could choose $\varepsilon$ to be even smaller as this would
result in tighter inference quality, but it also increases the risk of
violations during the quantization process and could therefore take more time
or could not even finish for some layers. Input and weight scales are
optimized concurrently as the algorithm works through each layer. We confirm
that the empirical results of this approach are sufficient to leverage
quantization for real-world usage of NAPC in existing edge devices. Both 16
and 32-bit fixed-point quantization are feasible for the greedy approach.
Lower precisions were only possible for particular layers. However, those
increased the quantization noise for subsequent layers, thus providing no
advantage. With a 32-bit fixed-quantization, we can achieve comparable results
to unquantized models, with a maximum absolute deviation of 4 to the ground
truth (floating-point prediction) sampled from 13 randomly chosen quantized
models and checked against all 4966 hold-out sequences (unseen during training
and calibration).

In other experiments and to our great surprise, a model quantized with this
method is not necessary of inferior or of equal quality as its non-quantized
template but can be _superior_ as well even to an extent where some mediocre
model turns into one of the _best models_ after quantization. Therefore,
quantizing all models can be a convenient way to enlarge the search space and
increase quality. Nevertheless, the average quality of our quantized models is
below those without quantization, compare to Figure 7.

####  III-D2 Ensemble building

For the NAPC, we can confirm studies about random seed ensembles that suggest
they can improve quality [59, 60]. As mentioned previously, we take the
$\tau$-th (with $\tau\geq 0.5$ ) quantile333The choice $\tau=0.5$ is also
known as _median_. of the ensemble for each class for the final prediction.
This is due to the property of NAPC instances undercounting on average, which
is even worse when using the cumsum, compare the lower success in Figure 7.
Also, this approach filters outliers and selectively centers the error
distribution of the ensemble more towards 0, which increases the test success
chance, compared Figure 4 and Figure 5.

Figure 7: Test success simulation, comparison with and without cumsum,
quantization, and a single large count video. All NAPCs shown here have been
trained using the whole 8000 video training set: even though certain models
might improve, quantization reduces the average model quality. Also, in
scenarios without large counts, models seem to perform better without cumsum.
However, even a single video with many boarding passengers (here 170) can
drastically reduce test success chances without cumsum, while NAPC instances
with cumsum are mostly unaffected. An explanation for the non-convexity in the
two right figures can be found in Figure 6. Overall, creating many instances
improves the chances of obtaining high-quality models.

###  III-E Performance

Figure 8: NAPC performance was measured in real-time sensor counts on
different platforms using different compilers. GCC (v11) seems to perform
particularly well on legacy platforms, while Clang (v13) performs better
everywhere else. The APC-R profits from quantization the most as it does not
have a dedicated floating-point unit. A sensor is assumed to operate at 10
FPS.

To compare the performance of a quantized and a floating-point NAPC instance,
we developed a custom inference framework written in C/C++ [61, 62]. To
increase the performance, we include the weights in the binary and do not use
dynamic memory via malloc or new. This allows us to infer multiple sensors
concurrently on various platforms (x86, ARMv7, and ARMv5). For the benchmark,
we use a video with a length of approx. 204 seconds, which can be stored
locally even on a legacy machine with only 32 MB of free storage. We measure
the user time to infer all of these sequences to calculate how many NAPC
instances the system could run in real-time. In Figure 8 we show the
performance of different CPUs. Noticeably the already deployed SSU-110 and
APC-R can reach around 20 (SSU-110) or approx. 6 (APC-R) NAPC instances in
real-time while x86 machines like the Intel i7 can do thousands and even a
Raspberry Pi up to around 200 with multicore systems only using a single core.
The APC-R profits most from quantization, as it would only compute approx. 2
times real-time with floating-point models.

##  IV Conclusion

We proposed several improvements to the training and inference of existing
NAPC implementation, which considerably enhance the quality of the predictions
and usability in real-world scenarios.

For a long time, it was unclear whether the new VDV 457 v2.1 could ever be
passed with 99% count accuracy444An accuracy of 99% refers to the equivalence
test for bias, i.e. the absolute bias being $<1\%$ including confidence
intervals.. Our results prove that it can be done. Thanks to quantization,
both in theory and practice, even on legacy hardware already installed in
vehicles. Not only can the test be passed, but also the validation costs of
this APC system are around 50% less out of the box due to lower sample size
requirements5553600 videos $\approx$ 3000 stop door events instead of the 6147
mentioned in the VDV 457 v2.1 which we proposed based on low precision legacy
systems back in 2018 to give APC manufacturers a fair chance to pass the
test..

We have created and applied a variety of methods for quality control and
identified that the number of videos in the training process matters a lot,
and additional measures like _quantile ensembles_ are required to compensate
for the undercounting that seems to be intrinsic to the NAPC. Further research
has to be done to determine the limits of the architecture, e.g. a maximal
count of training videos at which quality does not further improve. With the
test success chance, we have introduced a _simulative metric_ based on an
empirical distribution. Our approach is very generic and can probably be
applied to many design problems outside of automatic passenger counting.

We have also found out that our LSTMs have an upper bound up to which they can
count. Our _cumsum layer_ thus not only helps create better counts but can be
seen as an improvement to LSTMs in general for regression tasks, where
calculating the differences in time series (trend elimination) is not
applicable. To our knowledge, we are the first to apply this approach to LSTM
predictions.

Finally, we hope that our contribution to the NAPC will raise fairness in
global revenue sharing and distribution to a new level and that other areas
will profit from our results as well.

##  V Acknowledgements

This research is financially supported by the European Regional Development
Fund under Grant 10166961.

## References

  * [1] Lars Wagner  “ÖPNV-Bilanz 2018: erneuter Fahrgastrekord im Nahverkehr” [Online; accessed 25-May-2021]  Verband Deutscher Verkehrsunternehmen e. V, 2019  URL: https://www.vdv.de/190213-pm-bilanz-2018.pdfx
  * [2] Lars Wagner  “ÖPNV-Bilanz 2019: Fahrgastzahlen mit leichtem Plus” [Online; accessed 25-May-2021]  Verband Deutscher Verkehrsunternehmen e. V, 2020  URL: https://www.vdv.de/200128-pm-oepnv-bilanz-2019.pdfx
  * [3] Lars Wagner  “Die ÖPNV-Bilanz des Corona-Jahres 2020” [Online; accessed 25-May-2021]  Verband Deutscher Verkehrsunternehmen e. V, 2021  URL: https://www.vdv.de/210204-pm-oepnv-bilanz-des-corona-jahres-2020.pdfx
  * [4] Wikipedia  “Verkehrsverbund Berlin-Brandenburg — Wikipedia, The Free Encyclopedia” [Online; accessed 25-May-2021], http://de.wikipedia.org/w/index.php?title=Verkehrsverbund%20Berlin-Brandenburg&oldid=211810070, 2021
  * [5] Wikipedia  “Farebox recovery ratio — Wikipedia, The Free Encyclopedia” [Online; accessed 25-May-2021], http://en.wikipedia.org/w/index.php?title=Farebox%20recovery%20ratio&oldid=999229063, 2021
  * [6] Michael Siebert and David Ellenberger  “Validation of automatic passenger counting: introducing the t-test-induced equivalence test”  In _Transportation_ , 2019  DOI: 10.1007/s11116-019-09991-9
  * [7] Silvia Köhler et al.  “Recommendations for the Appliction of APCSs within Public Transport and Regional Rail Transport, Version 2.1”, VDV-Recommendations 457  Köln: Verband Deutscher Verkehrsunternehmen (VDV), 2018  URL: https://www.beka-verlag.de/ebook-vdv-schrift-457-automatic-passenger-counting-systems-v2-1.html
  * [8] Robert Seidel et al.  “NAPC: A Neural Algorithm for Automated Passenger Counting in Public Transport on a Privacy-Friendly Dataset”  In _IEEE Open Journal of Intelligent Transportation Systems_ 3, 2022, pp. 33–44  DOI: 10.1109/OJITS.2021.3139393
  * [9] Sepp Hochreiter and Jürgen Schmidhuber  “Long short-term memory”  In _Neural computation_ 9.8 MIT Press, 1997, pp. 1735–1780
  * [10] Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi  “You Only Look Once: Unified, Real-Time Object Detection”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [11] Alexey Bochkovskiy, Chien-Yao Wang and Hong-Yuan Mark Liao  “YOLOv4: Optimal Speed and Accuracy of Object Detection”, 2020  arXiv:2004.10934 [cs.CV]
  * [12] Jonathan Long, Evan Shelhamer and Trevor Darrell  “Fully Convolutional Networks for Semantic Segmentation”, 2015  arXiv:1411.4038 [cs.CV]
  * [13] Michael Siebert and Marc Goerigk  “An experimental comparison of periodic timetabling models”  In _Computers & Operations Research_ 40.10, 2013, pp. 2251–2259  DOI: https://doi.org/10.1016/j.cor.2013.04.002
  * [14] Johannes Mehrer, Courtney J. Spoerer, Nikolaus Kriegeskorte and Tim C. Kietzmann  “Individual differences among deep neural network models”  In _Nature Communications_ 11.1, 2020, pp. 5725  DOI: 10.1038/s41467-020-19632-w
  * [15] Denis Klauser, Günter Bärwolff and Hartmut Schwandt  “A TOF-based automatic passenger counting approach in public transportation systems”  In _AIP Conference Proceedings_ 1648, 2015  DOI: 10.1063/1.4913168
  * [16] Robert Seidel et al.  “Berlin-APC: A Privacy-Friendly Dataset for Automated Passenger Counting in Public Transport”, 2021  DOI: 10.14279/depositonce-12205
  * [17] Iyad Suwan, Shahd Owies and Thabet Abdeljawad  “Monotonicity results for h-discrete fractional operators and application”  In _Advances in Difference Equations_ 2018.1 Springer, 2018, pp. 1–17
  * [18] Kenneth S Miller and Bertram Ross  “Fractional difference calculus, in” Univalent functions, fractional calculus, and the applications (Koriyama, 1988)”, 139-152”  In _Horwood, Chichester_ 548, 1989
  * [19] Steven Craig Hillmer and George C Tiao  “An ARIMA-model-based approach to seasonal adjustment”  In _Journal of the American Statistical Association_ 77.377 Taylor & Francis, 1982, pp. 63–70
  * [20] George EP Box, Gwilym M Jenkins, Gregory C Reinsel and Greta M Ljung  “Time series analysis: forecasting and control”  John Wiley & Sons, 2015
  * [21] Gunhan Dundar and Kenneth Rose  “The effects of quantization on multilayer neural networks”  In _IEEE Transactions on Neural Networks_ 6.6 IEEE, 1995, pp. 1446–1451
  * [22] Raziel Alvarez, Rohit Prabhavalkar and Anton Bakhtin  “On the efficient representation and execution of deep acoustic models”  In _arXiv preprint arXiv:1607.04683_ , 2016
  * [23] Jian Li and Raziel Alvarez  “On the quantization of recurrent neural networks”  In _arXiv preprint arXiv:2101.05453_ , 2021
  * [24] Amir Gholami et al.  “A Survey of Quantization Methods for Efficient Neural Network Inference”, 2021  arXiv:2103.13630 [cs.CV]
  * [25] Alex Krizhevsky, Ilya Sutskever and Geoffrey E Hinton  “Imagenet classification with deep convolutional neural networks”  In _Advances in neural information processing systems_ 25, 2012, pp. 1097–1105
  * [26] Xueheng Qiu et al.  “Ensemble deep learning for regression and time series forecasting”  In _2014 IEEE symposium on computational intelligence in ensemble learning (CIEL)_ , 2014, pp. 1–6  IEEE
  * [27] Zhi-Hua Zhou, Jianxin Wu and Wei Tang  “Ensembling neural networks: many could be better than all”  In _Artificial intelligence_ 137.1-2 Elsevier, 2002, pp. 239–263
  * [28] Leo Breiman  “Bagging predictors”  In _Machine learning_ 24.2 Springer, 1996, pp. 123–140
  * [29] Yoav Freund and Robert E Schapire  “A decision-theoretic generalization of on-line learning and an application to boosting”  In _Journal of computer and system sciences_ 55.1 Elsevier, 1997, pp. 119–139
  * [30] David H Wolpert  “Stacked generalization”  In _Neural networks_ 5.2 Elsevier, 1992, pp. 241–259
  * [31] Grigorios Tsoumakas, Ioannis Partalas and Ioannis Vlahavas  “An ensemble pruning primer”  In _Applications of supervised and unsupervised ensemble methods_ Springer, 2009, pp. 1–13
  * [32] Yijun Bian, Yijun Wang, Yaqiang Yao and Huanhuan Chen  “Ensemble pruning based on objection maximization with a general distributed framework”  In _IEEE transactions on neural networks and learning systems_ 31.9 IEEE, 2019, pp. 3766–3774
  * [33] Zhi-Hua Zhou and Wei Tang  “Selective ensemble of decision trees”  In _International workshop on rough sets, fuzzy sets, data mining, and granular-soft computing_ , 2003, pp. 476–483  Springer
  * [34] Miguel Morin and Matthew Willetts  “Non-determinism in tensorflow resnets”  In _arXiv preprint arXiv:2001.11396_ , 2020
  * [35] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [36] Duncan Riach  “Deep Learning Determinism”  In _YouTube_ YouTube, 2019  URL: https://www.youtube.com/watch?v=TB07_mUMt0U
  * [37] Paulius Micikevicius et al.  “Mixed precision training”  In _arXiv preprint arXiv:1710.03740_ , 2017
  * [38] Martı́n Abadi et al.  “Tensorflow: A system for large-scale machine learning”  In _12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16)_, 2016, pp. 265–283
  * [39] Charles R. Harris et al.  “Array programming with NumPy”  In _Nature_ 585.7825, 2020, pp. 357–362  DOI: 10.1038/s41586-020-2649-2
  * [40] Yoshua Bengio  “Practical recommendations for gradient-based training of deep architectures”  In _Neural networks: Tricks of the trade_ Springer, 2012, pp. 437–478
  * [41] Nitish Srivastava et al.  “Dropout: a simple way to prevent neural networks from overfitting”  In _The journal of machine learning research_ 15.1 JMLR. org, 2014, pp. 1929–1958
  * [42] Feng Zhu et al.  “Towards unified int8 training for convolutional neural network”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1969–1979
  * [43] Adam Paszke et al.  “Pytorch: An imperative style, high-performance deep learning library”  In _arXiv preprint arXiv:1912.01703_ , 2019
  * [44] David Ellenberger and Michael Siebert  “Introducing the Partitioned Equivalence Test: Artificial Intelligence in Automatic Passenger Counting Validation”, 2021  arXiv:2104.09697 [stat.AP]
  * [45] A.. der Vaart  “Asymptotic Statistics”, Cambridge Series in Statistical and Probabilistic Mathematics  Cambridge University Press, 1998  DOI: 10.1017/CBO9780511802256
  * [46] P Diaconis and B Efron  “Computer-intensive methods in statistics”  In _Sci. Am.; (United States)_ , 1983  DOI: 10.1038/scientificamerican0583-116
  * [47] NVIDIA Corporation  “Framework Determinism”  In _GitHub repository_ GitHub, https://github.com/NVIDIA/framework-determinism, 2021
  * [48] NVIDIA  “Release Notes :: NVIDIA Deep Learning cuDNN Documentation”, 2020  URL: https://docs.nvidia.com/deeplearning/cudnn/release-notes/rel_7xx.html#rel_761
  * [49] Saeed S Alahmari, Dmitry B Goldgof, Peter R Mouton and Lawrence O Hall  “Challenges for the Repeatability of Deep Learning Models”  In _IEEE Access_ 8 IEEE, 2020, pp. 211860–211868
  * [50] Jonathan Frankle and Michael Carbin  “The Lottery Ticket Hypothesis: Training Pruned Neural Networks”  In _CoRR_ abs/1803.03635, 2018  arXiv: http://arxiv.org/abs/1803.03635
  * [51] Travis Dick, E. Wong and Christoph Dann  “How many random restarts are enough”, 2014
  * [52] David Picard  “Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision”, 2021  arXiv:2109.08203 [cs.CV]
  * [53] Matthew Hutson  “Artificial intelligence faces reproducibility crisis”  American Association for the Advancement of Science, 2018
  * [54] Anne-Laure Boulesteix, Sabine Hoffmann, Alethea Charlton and Heidi Seibold  “A replication crisis in methodological research?”  In _Significance_ 17.5 Wiley Online Library, 2020, pp. 18–21
  * [55] Jungwook Choi et al.  “Pact: Parameterized clipping activation for quantized neural networks”  In _arXiv preprint arXiv:1805.06085_ , 2018
  * [56] Milad Alizadeh et al.  “Gradient $\ell_{1}$ Regularization for Quantization Robustness”  In _arXiv preprint arXiv:2002.07520_ , 2020
  * [57] Darryl Lin, Sachin Talathi and Sreekanth Annapureddy  “Fixed point quantization of deep convolutional networks”  In _International conference on machine learning_ , 2016, pp. 2849–2858  PMLR
  * [58] Ron Banner, Yury Nahshan, Elad Hoffer and Daniel Soudry  “Post-training 4-bit quantization of convolution networks for rapid-deployment”  In _arXiv preprint arXiv:1810.05723_ , 2018
  * [59] Lars Kai Hansen and Peter Salamon  “Neural network ensembles”  In _IEEE transactions on pattern analysis and machine intelligence_ 12.10 IEEE, 1990, pp. 993–1001
  * [60] Stanislav Fort, Huiyi Hu and Balaji Lakshminarayanan  “Deep Ensembles: A Loss Landscape Perspective”, 2020  arXiv:1912.02757 [stat.ML]
  * [61] Brian W Kernighan and Dennis M Ritchie  “The C programming language”, 2006
  * [62] Bjarne Stroustrup  “The C++ programming language”  Pearson Education Inc, 2000