# Imbalance-aware Presence-only Loss Function for Species Distribution
Modeling

Robin Zbinden, Nina van Tiel
EPFL, Switerzland
<EMAIL_ADDRESS>&Marc Rußwurm
WUR, Netherlands
<EMAIL_ADDRESS>Devis Tuia
EPFL, Switerzland
<EMAIL_ADDRESS>
###### Abstract

In the face of significant biodiversity decline, species distribution models
(SDMs) are essential for understanding the impact of climate change on species
habitats by connecting environmental conditions to species occurrences.
Traditionally limited by a scarcity of species observations, these models have
significantly improved in performance through the integration of larger
datasets provided by citizen science initiatives. However, they still suffer
from the strong class imbalance between species within these datasets, often
resulting in the penalization of rare species–those most critical for
conservation efforts. To tackle this issue, this study assesses the
effectiveness of training deep learning models using a balanced presence-only
loss function on large citizen science-based datasets. We demonstrate that
this imbalance-aware loss function outperforms traditional loss functions
across various datasets and tasks, particularly in accurately modeling rare
species with limited observations.

##  1 Introduction

Species distribution models (SDMs) play a crucial role in ecology, serving as
indispensable tools for understanding and predicting the spatial distribution
of species. By establishing correlations between species occurrence data and
environmental variables (Elith and Leathwick, 2009), these models provide
valuable insights into the ecological niches and habitat preferences of
diverse organisms, thereby informing conservation efforts (Guisan et al.,
2013). The significance of SDMs in identifying and safeguarding endangered
species becomes even more pronounced as habitats of numerous species face
imminent threats from climate change (Thomas et al., 2004, Dyderski et al.,
2018). Additionally, conservation endeavors aimed at preventing biodiversity
loss not only contribute to mitigating climate change (Shin et al., 2022) but
also play an important role in alleviating its broader impacts (Pörtner et
al., 2023).

These conservation efforts primarily focus on rare and endangered species,
which are inherently difficult to observe. Consequently, we have only a few
observations available, posing challenges to the development of reliable SDMs
(Breiner et al., 2015). Recent initiatives in citizen science present
promising avenues to facilitate the collection of large amounts of species
records. However, there is still a high disparity in the number of
observations per species, ranging from a few handfuls of occurrences to tens
of thousands for the most common or iconic species (Botella et al., 2023, Cole
et al., 2023). Such a significant class imbalance reflects the existence of
various biases within the data, which can be geographical and taxonomic, among
others (Feldman et al., 2021). Additionally, the data gathered through citizen
science initiatives is typically presence-only, i.e., it consists of recorded
occurrences but no data regarding the species’ absence (Pearce and Boyce,
2006). Managing such data limitations brings additional complexities in
developing accurate and reliable SDMs for rare species.

Deep learning (DL) has demonstrated promise for SDMs (Deneu et al., 2021, Teng
et al., 2023), by enabling the simultaneous modeling of multiple species and
the identification of shared environmental patterns, a characteristic
particularly advantageous for rare111In this study, for simplicity, we refer
to rare species as those with the lowest numbers of occurrences in the
training set. It is important to note that the rarity of species is a more
complex concept, dependent on factors such as range size, occupancy, and
abundance (Crisfield et al., 2024). species (Zbinden et al., 2023). However,
the persistence of high class imbalance in training datasets (see Figure 1)
often leads to the neglect of rare species during model training, despite
their critical importance. Recently, Zbinden et al. (2024) introduced a
balanced loss function to account for the class imbalance between species,
demonstrating its effectiveness in improving the model’s performance on rare
species, but only on the relatively small datasets of Elith et al. (2020). In
this study, we extend their work to larger datasets acquired through citizen
science initiatives at continental and global scales (Botella et al., 2023,
Cole et al., 2023). Specifically, we train DL models with different presence-
only losses and evaluate these models on different SDM-related tasks. Our
findings highlight that an imbalance-aware loss function is essential to
achieve optimal performance on rare species.

##  2 Methods

Generally, SDMs learn correlations between the environmental features and
observed species occurrence patterns. Depending on the downstream application,
the model can also be made spatially explicit (Domisch et al., 2019) by
incorporating geospatial coordinates as additional input through location
encoders (Rußwurm et al., 2024). Multi-species distribution models aim to
learn the distribution of multiple species simultaneously within a single
model. The task then involves multi-label classification, as a given location
may host an arbitrary number of species. However, a significant portion of
species records exists in the form of presence-only data, often with only one
species presence observation associated with a given location. This creates a
scenario known as single positive multi-label learning (Cole et al., 2021),
which poses significant challenges. To cope with this scenario, a common
strategy involves sampling pseudo-absences (PAs), designating samples as
negative even when certainty about the absence of a species is lacking, and
incorporating them into the loss function during model training (Cole et al.,
2023). Below, we describe such losses.

Figure 1: Distributions of the number of presence records in the GeoLifeCLEF
2023 (left) and iNaturalist (right) training datasets, obtained through
citizen science initiatives. Both distributions exhibit a long-tailed pattern,
which is crucial to address to avoid penalizing rare species during training.

###  2.1 Loss functions

The predominant approach for multi-label classification uses the binary cross-
entropy loss function (Nam et al., 2014, Ung et al., 2023). For a given
location with species observations represented by $y$, it is defined as
follows:

$\displaystyle\mathcal{L}_{\text{BCE}}(\mathbf{y},\mathbf{\hat{y}})=-\frac{1}{S}\sum^{S}_{s=1}\bigl{[}\mathbbm{1}_{[y_{s}=1]}\log(\hat{y}_{s})+\mathbbm{1}_{[y_{s}=0]}\log(1-\hat{y}_{s})\bigr{]}$
(1)

where $y_{s}$ is $1$ if species $s$ has been observed and $0$ otherwise,
$\hat{y}_{s}\in[0,1]$ is the predicted suitability score for species $s$, and
$S$ denotes the number of species. It is essential to note that $y_{s}=0$
doesn’t necessarily imply that the species is absent; it simply indicates that
it hasn’t been observed. This corresponds to a PA, characterized here by the
observation of another species. These specific PAs are referred to as target-
group background points in the context of SDMs (Phillips et al., 2009). Since
target-group background points are only located where other species’
observations are, they may not entirely cover the area of interest. To address
this limitation, Cole et al. (2023) extended the BCE loss to incorporate, for
each sample, a PA located at a random location with a predicted score
$\hat{y}_{s}^{\prime}$, introducing the full assume negative loss function:

$\displaystyle\mathcal{L}_{\text{full}}(\mathbf{y},\mathbf{\hat{y}})=-\frac{1}{S}\sum^{S}_{s=1}\bigl{[}\mathbbm{1}_{[y_{s}=1]}\lambda\log(\hat{y}_{s})+\mathbbm{1}_{[y_{s}=0]}\log(1-\hat{y}_{s})+\log(1-\hat{y}_{s}^{\prime})\bigr{]}.$
(2)

Here, $\lambda=2048$ is included to counterbalance the larger number of PAs
compared to presences. However, this loss doesn’t address the high class
imbalance between species, which is particularly detrimental for rare species.
To tackle this issue, we introduced class-specific weights, or species
weights, in a prior work (Zbinden et al., 2024), resulting in the full
weighted loss function:

$\displaystyle\mathcal{L}_{\text{full-
weighted}}(\mathbf{y},\mathbf{\hat{y}})=-\frac{1}{S}\sum^{S}_{s=1}\bigl{[}\mathbbm{1}_{[y_{s}=1]}\lambda_{1}w_{s}\log(\hat{y}_{s})+\mathbbm{1}_{[y_{s}=0]}\lambda_{2}\frac{1}{\left(1-\frac{1}{w_{s}}\right)}\log(1-\hat{y}_{s})$
$\displaystyle+(1-\lambda_{2})\log(1-\hat{y}_{s}^{\prime})\bigr{]}.$ (3)

Species weights are defined as
$w_{s}=\frac{n}{n_{\text{p($s$)}}}=\frac{1}{\text{freq({s})}}$, where
$n_{\text{p($s$)}}$ denotes the number of presences of species $s$, and $n$
represents the total number of presence locations, i.e., the number of
training samples. Additionally, the weight $\lambda_{2}$ is introduced to
modulate the impact of different types of PAs. In this work, we extensively
test these loss functions across different large-scale citizen science-based
datasets and tasks, with a particular emphasis on rare species. The datasets
are described in the following section, and training details are presented in
the Appendix A.

###  2.2 Datasets

GeoLifeCLEF 2023 (GLC23). Originally designed for a competition (Botella et
al., 2023), this dataset provides an extensive collection of species
observation records from various citizen science sources. GLC23 includes a
public validation set containing presence-absence data for $2174$ plant
species. We use it to assess performances by calculating the mean AUC across
all species. Additionally, we compute a mean AUC for $468$ rare species (with
$50$ observations or fewer). The distribution of the number of presence
records per species is shown in the left panel of Figure 1.

iNaturalist. We leverage the codebase, model architectures, datasets, and
tasks developed in Cole et al. (2023). Specifically, the training dataset
comprises $35.5$ million observations spanning $47\,375$ species from
iNaturalist222https://www.inaturalist.org/. Unlike the GLC23 dataset, all
species in this dataset have a minimum of $50$ observations, with some species
exceeding $100\,000$ occurrences. For computational efficiency, we limit the
number of observations to $1000$ per species (see right panel of Figure 1),
aligning to Cole et al. (2023). We evaluate with the following three tasks:
first, the eBird Status and Trends (S&T) test set (Fink et al., 2020), derived
from expert range maps, which includes $535$ bird species. The number of
presences for these species is less imbalanced, with almost all the species
having a substantial number of observations during training. Specifically, we
categorize the $96$ out of $535$ species with less than $1000$ occurrences as
rare. Second, the International Union for Conservation of Nature (IUCN) test
set, which is more imbalanced and contains $639$ rare species with $100$
occurrences or less. Performance is evaluated using the mean average precision
(mAP) for both the S&T and IUCN test sets. Third, the models are assessed as
geographic priors for fine-grained image classification. This task enhances
species image classification by incorporating location and environmental
metadata into the model. We calculate the top-1 accuracy gain ($\Delta$ Top-1)
by adding SDMs to complement the vision model. The distribution of the number
of presences of the species considered in these three tasks is presented in
the Appendix B.

##  3 Results

| GeoLifeCLEF | S&T | IUCN | Geo Prior
---|---|---|---|---
| (AUC) | (mAP) | (mAP) | ($\Delta$ Top-1)
| all | rare | all | rare | all | rare | all
$\mathcal{L}_{\text{BCE}}$ | 0.453 | 0.398 | 0.810 | 0.746 | 0.702 | 0.659 | +6.7
$\mathcal{L}_{\text{full}}$ | 0.786 | 0.802 | 0.807 | 0.756 | 0.761 | 0.703 | +6.6
$\mathcal{L}_{\text{full-weighted}}$ with $\lambda_{2}=0.5$ | 0.796 | 0.854 | 0.806 | 0.751 | 0.765 | 0.710 | +7.3
$\mathcal{L}_{\text{full-weighted}}$ with $\lambda_{2}=0.8$ | 0.787 | 0.841 | 0.810 | 0.753 | 0.757 | 0.700 | +7.3

Table 1: Performance of the different losses function on the SDMs tasks.
Results in bold correspond to the best in the column, while the second-best is
underlined.

Figure 2: Performance of the loss functions, grouped by the number of
presences records of species in the training set. The
$\mathcal{L}_{\text{full-weighted}}$ loss, defined here with
$\lambda_{2}=0.5$, is beneficial for rare species.

Results are presented in Table 1 and Figure 2. Firstly, we observe consistent
high performance with the full weighted loss, specifically with
$\lambda_{2}=0.5$, outperforming the other losses in three out of four tasks.
Notably, substantial improvements are observed in the GLC23 dataset,
particularly for rare species. This aligns well with the nature of the
dataset, which contains numerous rare species with a limited number of
observations. As depicted in the left panel of Figure 2, the
$\mathcal{L}_{\text{full-weighted}}$ loss demonstrates improved performance
over the $\mathcal{L}_{\text{full}}$ loss as the number of presences in the
training set decreases.

Results are more nuanced for the S&T dataset, where all loss functions yield
similar performance. This may be attributed to the fact that most bird species
have at least $1000$ observations in the training set, as illustrated in
Figures 3 and 4 in the Appendix, which diminishes the effect of the full
weighted loss. In contrast, the IUCN dataset presents more variability,
leading to a slightly higher performance with the $\mathcal{L}_{\text{full-
weighted}}$ loss. Additionally, we note the significantly higher performance
on the Geo Prior task, potentially due to the test set being more balanced
than the training set. Since this task involves multi-class classification, it
favors balanced loss functions. Finally, the $\mathcal{L}_{\text{full-
weighted}}$ loss with $\lambda_{2}=0.5$ seems to perform slightly better than
its counterpart with $\lambda_{2}=0.8$.

##  4 Conclusion

In this study, we emphasized the importance of effectively modeling the
distribution of rare species using deep learning, which requires addressing
the high class imbalance commonly found in datasets derived from citizen
science initiatives. The presented results illustrate the advantages of
employing a balanced loss function for SDMs across three out of four datasets,
demonstrating substantial performance improvements for rare species. Notably,
achieving equal performance on the S&T dataset, despite its lower imbalance,
suggests that imbalance-aware loss functions do not adversely affect less
imbalanced applications. Lastly, we stress the significance of considering
species with very few observations in benchmark datasets when evaluating SDMs,
as is done in the GeoLifeCLEF 2023 dataset. This aspect is particularly
crucial given that SDMs are most valuable when aimed at predicting the
distribution of rare and endangered species.

## References

  * Botella et al. (2023) C. Botella, B. Deneu, D. M. Gonzalez, M. Servajean, T. Larcher, C. Leblanc, J. Estopinan, P. Bonnet, and A. Joly.  Overview of geolifeclef 2023: Species composition prediction with high spatial resolution at continental scale using remote sensing.  _Working Notes of CLEF_ , 2023.
  * Breiner et al. (2015) F. T. Breiner, A. Guisan, A. Bergamini, and M. P. Nobis.  Overcoming limitations of modelling rare species by using ensembles of small models.  _Methods in Ecology and Evolution_ , 6(10):1210–1218, 2015.
  * Cole et al. (2021) E. Cole, O. Mac Aodha, T. Lorieul, P. Perona, D. Morris, and N. Jojic.  Multi-label learning from single positive labels.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 933–942, 2021.
  * Cole et al. (2023) E. Cole, G. Van Horn, C. Lange, A. Shepard, P. Leary, P. Perona, S. Loarie, and O. Mac Aodha.  Spatial implicit neural representations for global-scale species mapping.  _arXiv preprint arXiv:2306.02564_ , 2023.
  * Crisfield et al. (2024) V. E. Crisfield, F. Guillaume Blanchet, C. Raudsepp-Hearne, and D. Gravel.  How and why species are rare: towards an understanding of the ecological causes of rarity.  _Ecography_ , page e07037, 2024.
  * Deneu et al. (2021) B. Deneu, M. Servajean, P. Bonnet, C. Botella, F. Munoz, and A. Joly.  Convolutional neural networks improve species distribution modelling by capturing the spatial structure of the environment.  _PLoS computational biology_ , 17(4):e1008856, 2021.
  * Domisch et al. (2019) S. Domisch, M. Friedrichs, T. Hein, F. Borgwardt, A. Wetzig, S. C. Jähnig, and S. D. Langhans.  Spatially explicit species distribution models: A missed opportunity in conservation planning?  _Diversity and Distributions_ , 25(5):758–769, 2019.
  * Dyderski et al. (2018) M. K. Dyderski, S. Paź, L. E. Frelich, and A. M. Jagodziński.  How much does climate change threaten european forest tree species distributions?  _Global change biology_ , 24(3):1150–1163, 2018.
  * Elith and Leathwick (2009) J. Elith and J. R. Leathwick.  Species distribution models: ecological explanation and prediction across space and time.  _Annual Review of Ecology, Evolution and Systematics_ , 40(1):677–697, 2009.
  * Elith et al. (2020) J. Elith, C. Graham, R. Valavi, M. Abegg, C. Bruce, S. Ferrier, A. Ford, A. Guisan, R. J. Hijmans, F. Huettmann, et al.  Presence-only and presence-absence data for comparing species distribution modeling methods.  _Biodiversity informatics_ , 15(2):69–80, 2020.
  * Feldman et al. (2021) M. J. Feldman, L. Imbeau, P. Marchand, M. J. Mazerolle, M. Darveau, and N. J. Fenton.  Trends and gaps in the use of citizen science derived data as input for species distribution models: A quantitative review.  _PLoS One_ , 16(3):e0234587, 2021.
  * Fink et al. (2020) D. Fink, T. Auer, A. Johnston, M. Strimas-Mackey, O. Robinson, S. Ligocki, B. Petersen, C. Wood, I. Davies, B. Sullivan, et al.  ebird status and trends, data version: 2018; released: 2020.  _Cornell Lab of Ornithology, Ithaca, New York_ , 10, 2020.
  * Gorishniy et al. (2021) Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko.  Revisiting deep learning models for tabular data.  _Advances in Neural Information Processing Systems_ , 34:18932–18943, 2021.
  * Guisan et al. (2013) A. Guisan, R. Tingley, J. B. Baumgartner, I. Naujokaitis-Lewis, P. R. Sutcliffe, A. I. Tulloch, T. J. Regan, L. Brotons, E. McDonald-Madden, C. Mantyka-Pringle, et al.  Predicting species distributions for conservation decisions.  _Ecology letters_ , 16(12):1424–1435, 2013.
  * Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. pmlr, 2015.
  * Nam et al. (2014) J. Nam, J. Kim, E. Loza Mencía, I. Gurevych, and J. Fürnkranz.  Large-scale multi-label text classification—revisiting neural networks.  In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part II 14_ , pages 437–452. Springer, 2014.
  * Pearce and Boyce (2006) J. L. Pearce and M. S. Boyce.  Modelling distribution and abundance with presence-only data.  _Journal of applied ecology_ , 43(3):405–412, 2006.
  * Phillips et al. (2009) S. J. Phillips, M. Dudík, J. Elith, C. H. Graham, A. Lehmann, J. Leathwick, and S. Ferrier.  Sample selection bias and presence-only distribution models: implications for background and pseudo-absence data.  _Ecological applications_ , 19(1):181–197, 2009.
  * Pörtner et al. (2023) H.-O. Pörtner, R. Scholes, A. Arneth, D. Barnes, M. T. Burrows, S. Diamond, C. M. Duarte, W. Kiessling, P. Leadley, S. Managi, et al.  Overcoming the coupled climate and biodiversity crises and their societal impacts.  _Science_ , 380(6642):eabl4881, 2023.
  * Rußwurm et al. (2024) M. Rußwurm, K. Klemmer, E. Rolf, R. Zbinden, and D. Tuia.  Geographic location encoding with spherical harmonics and sinusoidal representation networks.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2024.
  * Shin et al. (2022) Y.-J. Shin, G. F. Midgley, E. R. Archer, A. Arneth, D. K. Barnes, L. Chan, S. Hashimoto, O. Hoegh-Guldberg, G. Insarov, P. Leadley, et al.  Actions to halt biodiversity loss generally benefit the climate.  _Global change biology_ , 28(9):2846–2874, 2022.
  * Teng et al. (2023) M. Teng, A. Elmustafa, B. Akera, H. Larochelle, and D. Rolnick.  Bird distribution modelling using remote sensing and citizen science data, 2023.
  * Thomas et al. (2004) C. D. Thomas, A. Cameron, R. E. Green, M. Bakkenes, L. J. Beaumont, Y. C. Collingham, B. F. Erasmus, M. F. De Siqueira, A. Grainger, L. Hannah, et al.  Extinction risk from climate change.  _Nature_ , 427(6970):145–148, 2004.
  * Ung et al. (2023) H. Q. Ung, R. Kojima, and S. Wada.  Leverage samples with single positive labels to train cnn-based models for multi-label plant species prediction.  _Working Notes of CLEF_ , 2023.
  * Zbinden et al. (2023) R. Zbinden, N. van Tiel, B. Kellenberger, L. Hughes, and D. Tuia.  Exploring the potential of neural networks for species distribution modeling.  _ICLR climate change AI workshop_ , 2023.
  * Zbinden et al. (2024) R. Zbinden, N. van Tiel, B. Kellenberger, L. Hughes, and D. Tuia.  On the selection and effectiveness of pseudo-absences for species distribution modeling with deep learning.  _arXiv preprint arXiv:2401.02989_ , 2024.

##  Appendix A Training details

GeoLifeCLEF 2023 (GLC23). Our models are trained on the $2\,856\,818$ presence
records corresponding to the $2174$ species in the test set. We provide the
models with $19$ historical bioclimatic variables, $9$ pedological variables,
and $17$ land cover classes as tabular data. We employ a multi-layer
perceptron model (MLP) with $5$ fully connected hidden layers, each containing
$1000$ neurons and connected through residual connections (Gorishniy et al.,
2021). Batch normalization (Ioffe and Szegedy, 2015) is applied to each hidden
layer, and the training process spans $150$ epochs with an SGD optimizer and a
learning rate set to $0.001$. Finally, the $\mathcal{L}_{\text{full-
weighted}}$ loss function uses $\lambda_{1}=1$.

iNaturalist. We adopt the identical configuration and MLP architecture as
described in Cole et al. (2023). We focus on their approach that incorporates
both coordinates and environmental data as inputs. For the
$\mathcal{L}_{\text{full-weighted}}$ loss function, we set $\lambda_{1}=0.1$
since the inverse of the frequency of species presences becomes very large
with the high number of species considered during training.

##  Appendix B Distributions of species presences

Figure 3: Distribution of the number of training presences of the species
considered in the different tasks. The GLC23 training set contains the same
species used in testing.

##  Appendix C Extra S&t results

Figure 4: Performance of the loss functions on the S&T dataset, grouped by the
number of presences records of species in the training set.


# Real Time Monocular Vehicle Velocity Estimation using Synthetic Data

Robert McCraith1, Lukas Neumann1, Andrea Vedaldi1 Authors are members of the
Visual Geometry Group in the University of Oxford, UK. emails: robert, lukas,
vedaldi @robots.ox.ac.uk

###### Abstract

Vision is one of the primary sensing modalities in autonomous driving. In this
paper we look at the problem of estimating the velocity of road vehicles from
a camera mounted on a moving car. Contrary to prior methods that train end-to-
end deep networks that estimate the vehicles’ velocity from the video pixels,
we propose a two-step approach where first an off-the-shelf tracker is used to
extract vehicle bounding boxes and then a small neural network is used to
regress the vehicle velocity from the tracked bounding boxes. Surprisingly, we
find that this still achieves state-of-the-art estimation performance with the
significant benefit of separating perception from dynamics estimation via a
clean, interpretable and verifiable interface which allows us distill the
statistics which are crucial for velocity estimation. We show that the latter
can be used to easily generate synthetic training data in the space of
bounding boxes and use this to improve the performance of our method further.

##  I Introduction

Autonomous driving systems rely on a wide array of sensors including LiDARs,
radars and cameras. LiDAR sensors are especially good at estimating the
position and velocities of vehicles and obstacles ($0.25$m/s at the distance
of $15$ meters [16], $0.71$m/s at a wider range of distances [1]). However,
LiDAR sensors are also very expensive, not reliable in adverse weather
conditions [19] and easily confused by exhaust fumes [9]. Depending on a
single source of information in a context such as autonomous driving is also
inherently fragile. Hence, it is natural to develop alternative sensors either
as redundancies or to improve the accuracy and robustness of sensors like
LiDARs. Cameras are a natural choice as they are very cost effective and in
principle sufficient for navigation given that humans drive cars primarily
using their sense of vision. Furthermore, there is a substantial amount of
computer vision research that is directly applicable to autonomous driving.

In this paper, we aim to predict the velocity of other cars based only on a
video from a standard monocular camera. Because the vehicle where the camera
is mounted on (the _ego-vehicle_) is typically moving as well, the velocity
estimate of other vehicles is relative to the velocity of the ego-vehicle.

Our main contribution is to show that, for this problem, one can separate
perception from velocity estimation by first mapping the visual data to a mid-
level representation — the space of vehicle bounding boxes — with no loss in
performance compared to much more complex estimation approaches. The velocity
estimation problem is then modelled purely on bounding boxes, which change
their position and size over time, tracking the motion of vehicles on the road
in a simplified view of the data. Using such an approach, our simple model
outperforms the winning model [15] of CVPR 2017 Vehicle Velocity Estimation
Challenge [1], which is a much more complex model that combines tracking with
two different deep networks for monocular depth and optical flow estimation.

A major advantage of using such a simple intermediate representation is that
it becomes much easier to _simulate_ training data for the velocity estimator
model. Still, we show that doing so in an effective manner requires to capture
accurately the statistics of real bounding boxes. Our second contribution is
thus to show how such a synthetic dataset is created, by extracting the
necessary data priors from a small set of real data and using it to generate a
much larger training set in the mid-level feature space. We then show that
training only using this synthetic dataset results in excellent performance on
real test data. In the process, we also distill the data statistics that are
crucial for velocity estimation from visual data, clarifying in the process
what is important and what information is not for this task.

The ability to easily generate synthetic data is not only beneficial to
improve velocity estimation accuracy at virtually no cost for existing
scenarios, but can be also used to train the model for different driving
scenarios, such as different countries, and to model different driving styles
and various emergency situations (e.g. unexpected breaking) without the need
for laborious real data collection or for expensive 3D photo-realistic
animation in order to cater for such situations.

Our approach can also be seen as encapsulating perception, which is often
implemented by opaque and difficult-to-diagnose components such as deep
convolutional neural networks, in a module which has a simple, interpretable,
and testable interface. Decomposing systems in modules that are individually
verifiable and that can be connected in a predictable manner is essential for
autonomous systems to meet reliability standards such as ISO 26262 [12, 21].
Hence, while end-to-end trainable systems may be conceptually preferable,
decompositions such as the one we propose may be essential in practice —
fortunately, as we show, this may come with no loss of performance.

The rest of the paper is structured as follows. In section II, prior work is
discussed. In section III, we introduce the method, in section IV we discuss
the data generation. The evaluation is presented in section V and the paper is
concluded in section VI.

##  II Related Work

|
---|---
Figure 1: Sample sequences from the TuSimple dataset at the last frame with
their respective bounding box and velocity annotation in $Z$ and $X$
direction.

#### Depth and ego-motion estimation

In recent years numerous approaches to monocular depth and ego-motion
estimation have been explored thanks to the expressive power of convolutional
neural networks. Supervised approaches [4, 25, 28] depend on pixel-wise depth
annotations to be available for each pixel of the training set, which results
in costly data collection. Unsupervised methods [30, 17] on the other hand
depend heavily on camera intrinsics and rely on camera motion between
successive frames to produce coarse relative depths, which are difficult to
use for predicting exact distances.

#### Velocity estimation

Classical vision techniques for motion detection from a moving camera such as
Yamaguchi _et al._[29] first match points in successive frames and then filter
them based on their location and compatibility with the epipolar geometry. A
similar approach is used in Fanani  _et al._[5], where candidate objects are
first filtered by a CNN which detects vehicles and then tests based on the
epipolar geometry are used to determine whether these vehicles are moving or
not.

The above approaches aim to only differentiate between static and moving
vehicles with no indication of their velocity. Kampelmühler _et al._[15], the
winner of CVPR 2017 Vehicle Velocity Estimation Challenge [1], extends these
approaches to predict the relative velocity of the vehicles in view of the
ego-vehicle. This is achieved by passing the sequence of images through a
depth [7] and flow [11] estimation network, as well as applying a classical
tracking system [13]. The features extracted from these three sub-components
are then concatenated and fed to a small neural network which predicts the
velocity (see fig. 2 top).

#### Tracking

Object tracking is a classical computer vision problem [20]. The multiple
instance learning tracker [2] expresses the problem as a classification task,
where detections in previous frames are used as training data for future
frames. MedianFlow [13] uses the forward-backward error to validate which
points are robust predictors of object movement. The method is further
improved in the TLD tracker [14] by disabling online learning when the object
is occluded and by allowing the algorithm to re-detect the object once it
appears again. More recently, with the emergence of deep learning, specialized
networks have been trained to explicitly track location of objects in video
sequences [10]. Similarly, Siamese CNNs [3, 26] have been exploited to build a
powerful embedding to discriminate whether an image patch contains the same
object or not, therefore tracking objects by appearance similarity. For a
systematic evaluation of tracking algorithms, we refer the reader to the VOT
challenge results [18].

#### Synthetic training data

Synthetic training data have been successfully applied in various domains of
computer vision, ranging from scene text detection [8] to optical flow
estimation [11]. In the autonomous driving domain, the most widely used
synthetic dataset is Virtual KITTI [6], which contains 50 photo-realistic
videos generated by the Unity game engine. Thanks to the synthetic source of
the data, the dataset comes with pixel-level annotations for segmentation,
optical flow and depth, which would be virtually impossible and very expensive
to achieve in real data. More recently, the SYNTHIA [22] and Synscapes [27]
artificial driving datasets have also been introduced.

The main difference to our work is that the above datasets are based on photo-
realistic representation of the world, whereas our mid-level representation
consists of mere bounding boxes, which are significantly easier to generate.

##  III Method

|
---|---
Figure 2: The state-of-the-art architecture of [15] (top) consists of three
different sub-networks and separate models for each of the three distance
intervals. Our architecture (bottom) trained on synthetic data is
significantly more straightforward, uses a single model and achieves
comparable accuracy.

Our goal is to estimate the velocity of other vehicles imaged from a camera
rigidly mounted on the ego-vehicle and looking forward. We can model this
situation as follows. The input to the model is a sequence
$\mathbf{I}=(I_{1},\dots,I_{T})$ of $T$ video frames extracted from the
camera. We also assume to have a bounding box $b_{T}$ tightly enclosing the
vehicle of interest at the end of the sequence $T$. The output of the model
$\Phi$ is a 2D vector $V=\Phi(\mathbf{I},b_{T})\in\mathbb{R}^{2}$ representing
the velocity of the target vehicle at time $T$ projected on the ground plane
relative to the ego-vehicle.111Estimating the vertical velocity $Y$ is
essentially irrelevant for this application.

###  III-A Geometry and elementary velocity estimation

Next, we describe in some detail the geometry of the problem and provide a
naïve solution based solely on projecting bounding boxes into 3D world
coordinate system. The physical constraint of the setup results in several
simplifications compared to the general imaging scenario. Let
$P=(X,Y,Z)\in\mathbb{R}^{3}$ be a 3D point expressed in the ego-vehicle
reference frame. We can assume that the camera is at a fixed height $H$ from
the ground looking straight ahead. Hence, point $P$ projects to the image
point $p=(u,v),$ $u=f\,\frac{X}{Z},$ $v=f\,\frac{Y+H}{Z}$ where $f$ is the
focal length of the camera.222The image coordinates $(u,v)$ are standardized,
with $(0,0)$ corresponding to the view direction, and are in practice related
to pixels coordinates via a non-linear transformation that accounts for the
camera intrinsic parameters, including effects such as radial distortion. We
assume that the intrinsic parameters are known and that their effects has
already been removed from the data.

Now assume that the bounding box $b=(x,y,w,h)\in\mathbb{R}^{4}$ is given by
the image coordinates $(u,v)=(x+w/2,y+h)$ of the mid point of the bottom edge
of the box and by the box width and height $(w,h)$. To a first approximation,
we can assume that $(u,v)$ is the image of a certain virtual 3D point
$P=(X,0,Z)$ rigidly attached to the vehicle at ground level. Hence, since we
know the height $H$ of the camera, we can readily infer the depth or distance
$Z$ of the vehicle, and hence the 3D point, as

$Z=fH\,\frac{1}{v},\quad X=H\,\frac{u}{v}.$ (1)

If we can track the bounding box $b_{t}$ over time $t\in[1,T]$, then we can
use eq. 1 to estimate the coordinates $P_{t}=(X_{t},0,Z_{t})$ of the 3D point
relative to the ego-vehicle and obtain the velocity as the derivative
$V=(\dot{X}_{T},\dot{Z}_{T})$. However, owing to the varying quality of road
surfaces and inclines or declines along a road this technique provides very
poor estimate of vehicle velocity, especially for vehicles which are further
away (see table II). Indeed, analyzing eq. 1 and assuming a camera with
standard image resolution, we come to a conclusion that for vehicles in
distance $d>20$m the approach requires sub-pixel accuracy of the bounding-box
estimate for a reasonable estimate of 3D position and hence velocity, which
simply is not realistic.

Figure 3: Vehicle statistics. Top left: coordinate of the left edge of the
bounding box image location vs the depth $Z$ of the enclosed vehicle. This is
constrained by green lines that represent the pixel coordinates of 3D points
$9$m left and right of the camera and placed at corresponding depths. Other
plots in order: coordinate $y$ (constraints at $[0,2.5\mathrm{m}]$), box
height $h$ ($[1.3\mathrm{m},2.5\mathrm{m}]$) and box width $w$
($[1.5\mathrm{m},3\mathrm{m}]$).

###  III-B Deep learning for velocity estimation

Sophisticated models which combine several streams of information from image
pixels (depth, optical flow) perform significantly better, so one may be
tempted to ascribe the poor performance of the geometric approach to the fact
that too much information is discarded by looking at bounding boxes only. We
show that, somewhat surprising, _this is not the case_. Instead, we show below
that bounding boxes _are_ sufficient provided that the modelling of dynamics
is less naïve.

#### State-of-the art baseline

Our reference model is the one of Kampelmühler  _et al._ [15]. They propose a
complex network that combines three data streams, all derived from the video
sequence $\mathbf{I}$ (see fig. 2 top). One stream applies a pre-trained
monocular depth-estimation network, called MonoDepth [7], to estimate a 3D
depth map from the video. Another stream applies FlowNet2 [11], a state-of-
the-art optical flow estimation network, to estimate instead optical flow. The
last stream uses an off-the-shelf tracker [13] to track the bounding box
$b_{T}$ backward through time, and thus obtain a simplified representation
$(b_{1},\dots,b_{T})$ of the vehicle trajectory in the image. The output of
the different streams are concatenated and passed through a multi-layer
perceptron (MLP) to produce the final velocity estimate
$V=\Phi(\mathbf{I},b_{T})$. In addition to pre-training MonoDepth, FlowNet2,
and the tracker on external data-sources, the MLP, fusing the information, is
trained on an ad-hoc benchmark dataset that contains vehicle bounding boxes
for one frame with their ground-truth velocities and positions measured via a
LiDAR (section IV).

Furthermore, three individual models of different-sized MLPs are used to
determine velocity within the different vehicle distance ranges used in the
evaluation. These range from 3 layers of 40 hidden neurons to 4 layers of 70
hidden neurons with increasing model size for vehicles further away [15]. The
final velocity prediction is given by averaging output of 5 model instances,
therefore in total 15 distance-specific models are used at testing time.

#### Our Model

Similar to [15] we use a fully connected neural network with 4 hidden layers
of 70 neurons each with CReLU[23] activation and Dropout[24] between layers
during training. This allows us to reduce the runtime of our method
significantly as we do not require estimation of optical flow and depth which
allows our runtime to reduce from $423ms$ to $10ms$ and only one model is
trained reducing the inference time further.

#### Estimating velocity from bounding boxes

Next, we describe our architecture to estimate vehicle velocity (see fig. 2
bottom). We consider an input video sequence $\mathbf{I}=(I_{1},\dots,I_{T})$
of $T$ video frames capturing a vehicle at time $t\in[1,T]$. As shown above,
in the reference model, the velocity estimate $V$ is then obtained as

$V=\Phi(\mathbf{I},b_{T})$ (2)

where $\Phi$ is neural network or a combination of several neural networks as
in [15], taking video sequence $\mathbf{I}$ as an input and a bounding box
$b_{T}$ in the last frame.

In our paper, we decompose $\Phi$ into two mappings $\Psi$ and $\Xi$

$V=\Psi(\Xi(\mathbf{I},b_{T}))$ (3)

by introducing an intermediate representation $\mathbf{b}=(b_{1},\dots,b_{T})$
representing the vehicle image location at time $t\in[1,T]$
$\mathbf{b}=\Xi(\mathbf{I},b_{T}),V=\Psi(\mathbf{b}),$ where $\Xi$ is a off-
the-shelf tracker component [13] and $\Psi$ is the vehicle velocity estimator
we train.

Using the above decomposition, we only require $(\mathbf{b},V)$ as supervision
pairs, and not the $(\mathbf{I},V)$ pairs as in the original formulation [15],
which are extremely expensive to obtain as it implies capturing and annotating
many driving video sequences.

As before, bounding boxes $\mathbf{b}$ are represented as quadruples
$(x,y,w,h)$. The trajectories $\mathbf{b}$ are passed to a filter
$g(\mathbf{b})$ that applies temporal Gaussian smoothing to each coordinate
independently. Finally, the output of the filter is flattened to a
$4T$-dimensional vector and fed to a multi-layer perceptron
$\bar{\Psi}:\mathbb{R}^{4T}\rightarrow\mathbb{R}$. Hence, the overall model at
inference time can be written as
$V=\Psi(\mathbf{b})=\bar{\Psi}(\operatorname{vec}(g(\Xi(\mathbf{I},b_{T})))).$

At training time, assuming the tracker $\Xi$ is fixed, we however only need to
train $\Psi$, using the pairs $(\mathbf{b},V)$ by minimizing the loss
$\|V-\Psi(\mathbf{b})\|^{2}$. Next we show how the pairs can be obtained.

##  IV Learning from real or synthetic data

We begin by discussing a real benchmark dataset sufficient to train our model
and then we discuss how we can generate analogous synthetic data effectively
in the space of object bounding boxes.

###  IV-A Real data: TuSimple

While many driving datasets exist in the realm of autonomous driving many
focus on single frame tasks such as object detection in 2D/3D, depth
estimation, semantic segmentation and localisation. The only dataset with
nessecary annotations for our task with comparable work come from TuSimple[1].
The TuSimple dataset was introduced for the CVPR2017 Autonomous Driving
velocity estimation challenge [1]. It contains $1074$ ($269$) driving
sequences with $1442$ ($375$) annotated vehicles in the training (respectively
testing) set, split into three subsets based on the distance between the ego-
vehicle and the observed car (see table I). The dataset was recorded on a
motorway by an standard camera (image resolution $1280\times 720$) mounted on
the roof of the ego-vehicle. Each driving sequence contains 40 frames,
captured at 20fps as seen in fig. 1. LiDAR and radar were simultaneously used
to capture the position and velocity of nearby vehicles and recorded data were
then used to give the ground truth velocity and 3D position of each vehicle in
the last frame of the sequence. Additionally, a manually created bounding box
(in the image space) is available in the last frame for each vehicle. Using
the notation from section III-B, the dataset therefore provides $1442$
training and $375$ testing triplets $(\mathbf{I},b_{T},V)$. As shown in fig. 4
the TuSimple dataset spans a wide range of distances from the camera with a
high bias towards the same lane as is typical in driving imagery. Velocity in
both $Z$ and $X$ is roughly uniformly distributed around $0$ which is expected
in motorway situations.

| Near | Medium | Far | Total
---|---|---|---|---
distance [m] | $d<20$ | $20<d<45$ | $d>45$ |
Train | 166 | 943 | 333 | 1442
Test | 29 | 247 | 99 | 375
TABLE I: The number of annotated vehicles in the TuSimple dataset [1] and
their split based on their distance from the ego-vehicle. Figure 4:
Distribution of vehicle instances in the TuSimple dataset.

Figure 5: Here we show and example of our synthetic boxes in these cases
generated starting at the same location as a real object. The fainter boxes
depict the location of the object in subsequent frames for the velocity
written above which may differ from the actual vehicles path in real data

###  IV-B Synthetic data

By distilling the information contained in an image to bounding boxes
$\mathbf{b}$ we now have data which is highly-interpretable and easy to
characterise statistically; the latter can be used to generate _synthetic_
training data, which ideally will have very similar statistical properties to
the real data. We could also choose to simulate vehicles with speeds not
normally seen in the real world to train a network capable of understanding
such abnormal situations.

We use the training subset of the TuSimple dataset (see section IV) to infer
these statistical properties. In fig. 3-left we plot the bounding box
horizontal coordinates $x$ vs the depth of the enclosed vehicle. There is some
obvious structure in the data; in particular, several lanes to the left and to
the right of the ego-vehicle are clearly visible. The other plots show the $y$
coordinate as well as the bounding box width and height. The latter are highly
constrained by the physical sizes of vehicles.

We found learning to be sensitive to the distribution of vehicles locations
and less so of car velocities and sizes. We thus represent the location
distribution empirically (fig. 5) and sample from TuSimple the first bounding
box in each to obtain its 3D ground point $(X,0,Z)$. This is then re-projected
to the image as explained in section III-A to obtain the bounding box location
$(x,y)$. The box height and width are obtained indexing with the depth $Z$
polynomial fits to the TuSimple height/width data (fig. 3). Finally, the track
is simulated by sampling a velocity vector $V$ from a Gaussian fit of the
TuSimple velocity data (fig. 4) and integrating the motion (fig. 6).

Note that only the bounding box positions are empirical, whereas the other
parameters are sampled form very simple distributions. We show that this is
sufficient to train models that achieve excellent performance on real data.
This helps isolating what are the important/sensitive priors (location) and
what are not (size, velocity) for velocity estimation.

Figure 6: Real vs synthetic vehicle sequences. The motion of the bounding box
in the video sequence - vehicle motion in real video as captured by the
tracker (blue), and synthetic motion generated by our method (red). In the
width and height plot we see some tracking inaccuracy when the blue lines
increase going to the right.

##  V Evaluation

We evaluated our approach using two models. For both models, we used MLP with
4 hidden layers, trained for 150 epochs, using dropout of 0.2 after the
Concatenated ReLU[23] activation function, learning rate with learning rate
$6\times 10^{-4}$ adjusted using exponential decay with decay rate set to
$0.99$.

In the first model denoted as MLP-tracker, we initially processed all video
frames by the MedianFlow [13] tracker, using the bounding box $b_{T}$ from the
ground truth for tracker initialization. We then used the resulting sequences
$\mathbf{b}=(b_{1},\dots,b_{T})$ as the training data for the MLP (see section
III-B). At test time, we followed the same procedure, which is pre-processing
the input video sequence by the tracker, and then feeding the intermediate
representation $\mathbf{b}$ into the MLP to obtain the velocity estimate.

We follow the TuSimple competition protocol, and report velocity estimation
accuracy $E_{v}$ calculated as the average over the three distance-based
subsets (table I):

$E_{v}=\frac{E_{v}^{\mbox{\tiny{near}}}+E_{v}^{\mbox{\tiny{medium}}}+E_{v}^{\mbox{\tiny{far}}}}{3},\qquad
E_{v}^{S}=\frac{1}{|S|}\sum_{i\in S}=\|V_{i}-\hat{V}_{i}\|^{2},$ (4)

where $V_{i}$ is the ground truth velocity from the LiDAR sensor and
$\hat{V}_{i}$ is method’s velocity estimate. We also note that according to
the dataset authors, the overall ground-truth accuracy is at around 0.71m/s,
however the accuracy will almost certainly depend on the distance of the
observed car.

Using the above model, we reach the overall velocity estimation error of
$1.29$, which outperforms the state-of-the-art333Kampelmühler  _et al._ also
report the error of $1.86$ and $1.25$ for their method using only tracking
information, however this was not the competition entry and it is not obvious
from the paper how the latter number was reached and what data were used for
training. and significantly more complex method of Kampelmühler  _et al._[15].
This experiment also shows, that the intermediate representation $\mathbf{b}$
contains sufficient amount of information to successfully infer vehicle
velocity.

In the second model MLP-synthetic, we instead used the synthetic training data
as the intermediate representation $\mathbf{b}$. We generated $11536$ samples
using the generation procedure described in section IV-B and used them as the
training samples for the MLP. At testing time, we again used the MedianFlow
tracker to get the intermediate representation $\mathbf{b}$ from real video
sequences and fed them into the MLP to obtain vehicle velocity estimates (see
fig. 2 bottom). Because our synthetic boxes are generated without noise
inherently coming from the tracker road surface and other sources, at test
time we apply a Gaussian filter with $\sigma=5$ in each coordinate of the
bounding box as pre-processing to remove noise.

The resulting model has the velocity estimation error of $1.28$, with the
biggest improvement in the Far range (see table II). Generating more synthetic
data for training did not result in further improvement in accuracy, which we
contribute to the relatively small size of the testing set and the relative
low variance of vehicle speed owing to motorway driving style.

| $E_{v}$ | $E_{v}^{\mbox{\tiny{near}}}$ | $E_{v}^{\mbox{\tiny{medium}}}$ | $E_{v}^{\mbox{\tiny{far}}}$
---|---|---|---|---
Kampelmühler  _et al._[15] (1st) | 1.30 | 0.18 | 0.66 | 3.07
Wrona (2nd) | 1.50 | 0.25 | 0.75 | 3.5
Liu (3rd) | 2.90 | 0.55 | 2.21 | 5.94
geometric reprojection | 8.5 | 0.48 | 1.50 | 23.60
Ours (MLP-tracker) | 1.29 | 0.18 | 0.70 | 2.99
Ours (MLP-synthetic) | 1.28 | 0.17 | 0.72 | 2.96
LiDAR | 0.71
TABLE II: Vehicle velocity estimation error $E_{v}$ on the TuSimple dataset,
including the top 3 best-performing methods of the CVPR 2017 Vehicle Velocity
Estimation challenge [1]. The average accuracy of the LiDAR sensor used for
ground truth acquisition added for reference. Runtime of Kampelmühler  _et
al._[15] is $423ms$ owing to the varying and complex inputs required for a
forward pass, our method can perform a forward pass in $10ms$ on the same
hardware(runtime for other methods are not public).

##  VI Conclusion

In this paper, we showed how to efficiently predict the velocity of other
vehicles based only on a video from a standard monocular camera by introducing
an intermediate representation which decouples perception from velocity
estimation. We also showed how priors in real data can be exploited to
generate synthetic data in the intermediate representation and that such
synthetic training data can be used to build a system capable of processing
real-world data without any loss in accuracy.

The decomposition is advantageous not only in terms of the ability to easily
generate training data with the desired parameters, but also to model
different driving styles or various emergency situations. Last but not least,
it also ensures each component is individually verifiable, which is essential
to meet reliability standards for autonomous driving.

## References

  * [1] CVPR 2017 workshop on autonomous driving.  https://github.com/TuSimple/tusimple-benchmark.  Accessed: 2020-02-15.
  * [2] Boris Babenko, Ming-Hsuan Yang, and Serge Belongie.  Visual tracking with online multiple instance learning.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 983–990. IEEE, 2009.
  * [3] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr.  Fully-convolutional siamese networks for object tracking.  In European conference on computer vision, pages 850–865. Springer, 2016.
  * [4] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In Advances in neural information processing systems, pages 2366–2374, 2014.
  * [5] Nolang Fanani, Matthias Ochs, Alina Sturck, and Rudolf Mester.  CNN-based multi-frame IMO detection from a monocular camera.  In 2018 IEEE Intelligent Vehicles Symposium, IV 2018, Changshu, Suzhou, China, June 26-30, 2018, pages 957–964, 2018.
  * [6] A Gaidon, Q Wang, Y Cabon, and E Vig.  Virtual worlds as proxy for multi-object tracking analysis.  In CVPR, 2016.
  * [7] Clément Godard, Oisin Mac Aodha, and Gabriel J. Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In CVPR, 2017.
  * [8] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2315–2324, 2016.
  * [9] Sinan Hasirlioglu, Andreas Riener, Werner Huber, and Philipp Wintersberger.  Effects of exhaust gases on laser scanner data quality at low ambient temperatures.  In 2017 IEEE Intelligent Vehicles Symposium (IV), pages 1708–1713. IEEE, 2017.
  * [10] David Held, Sebastian Thrun, and Silvio Savarese.  Learning to track at 100 fps with deep regression networks.  In European Conference on Computer Vision, pages 749–765. Springer, 2016.
  * [11] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  CoRR, abs/1612.01925, 2016.
  * [12] Road vehicles – functional safety.  Standard, International Organization for Standardization, Geneva, Switzerland, 2011.
  * [13] Z. Kalal, K. Mikolajczyk, and J. Matas.  Forward-backward error: Automatic detection of tracking failures.  In 2010 20th International Conference on Pattern Recognition, pages 2756–2759, Aug 2010.
  * [14] Zdenek Kalal, Krystian Mikolajczyk, and Jiri Matas.  Tracking-learning-detection.  IEEE transactions on pattern analysis and machine intelligence, 34(7):1409–1422, 2012.
  * [15] Moritz Kampelmühler, Michael G Müller, and Christoph Feichtenhofer.  Camera-based vehicle velocity estimation from monocular video.  arXiv preprint arXiv:1802.07094, 2018.
  * [16] Dominik Kellner, Michael Barjenbruch, Klaus Dietmayer, Jens Klappstein, and Jürgen Dickmann.  Instantaneous lateral velocity estimation of a vehicle using doppler radar.  In Proceedings of the 16th International Conference on Information Fusion, pages 877–884. IEEE, 2013.
  * [17] M. Klodt and A. Vedaldi.  Supervising the new with the old: learning sfm from sfm.  In European Conference on Computer Vision, 2018.
  * [18] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Luka Cehovin Zajc, Tomas Vojir, Gustav Hager, Alan Lukezic, Abdelrahman Eldesokey, et al.  The visual object tracking vot2017 challenge results.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1949–1972, 2017.
  * [19] Matti Kutila, Pasi Pyykönen, Werner Ritter, Oliver Sawade, and Bernd Schäufele.  Automotive lidar sensor development scenarios for harsh weather conditions.  In 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pages 265–270. IEEE, 2016.
  * [20] Bruce D Lucas, Takeo Kanade, et al.  An iterative image registration technique with an application to stereo vision.  1981\.
  * [21] Rob Palin, David Ward, Ibrahim Habli, and Roger Rivett.  Iso 26262 safety cases: Compliance and assurance.  2011\.
  * [22] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [23] Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee.  Understanding and improving convolutional neural networks via concatenated rectified linear units.  CoRR, abs/1603.05201, 2016.
  * [24] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  J. Mach. Learn. Res., 15(1):1929–1958, January 2014.
  * [25] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox.  Demon: Depth and motion network for learning monocular stereo.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5038–5047, 2017.
  * [26] Jack Valmadre, Luca Bertinetto, João Henriques, Andrea Vedaldi, and Philip HS Torr.  End-to-end representation learning for correlation filter based tracking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2805–2813, 2017.
  * [27] Magnus Wrenninge and Jonas Unger.  Synscapes: A photorealistic synthetic dataset for street scene parsing.  arXiv preprint arXiv:1810.08705, 2018.
  * [28] Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe.  Multi-scale continuous CRFs as sequential deep networks for monocular depth estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5354–5362, 2017.
  * [29] Koichiro Yamaguchi, Takeo Kato, and Yoshiki Ninomiya.  Vehicle ego-motion estimation and moving object detection using a monocular camera.  In 18th International Conference on Pattern Recognition (ICPR’06), pages 610–613. IEEE, 2006.
  * [30] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1851–1858, 2017.