Xuanlong<EMAIL_ADDRESS>Gianni
<EMAIL_ADDRESS>Emanuel Aldeaemanuel.aldea@universite-
paris-saclay.fr1 SATIE, Paris-Saclay University
Gif-sur-Yvette, France U2IS, ENSTA Paris
Institut Polytechnique de Paris
Palaiseau, France Side Learning Uncertainty for Regression Problems with SLURP

# SLURP: Side Learning Uncertainty for Regression Problems

###### Abstract

It has become critical for deep learning algorithms to quantify their output
uncertainties to satisfy reliability constraints and provide accurate results.
Uncertainty estimation for regression has received less attention than
classification due to the more straightforward standardized output of the
latter class of tasks and their high importance. However, regression problems
are encountered in a wide range of applications in computer vision. We propose
SLURP, a generic approach for regression uncertainty estimation via a side
learner that exploits the output and the intermediate representations
generated by the main task model. We test SLURP on two critical regression
tasks in computer vision: monocular depth and optical flow estimation. In
addition, we conduct exhaustive benchmarks comprising transfer to different
datasets and the addition of aleatoric noise. The results show that our
proposal is generic and readily applicable to various regression problems and
has a low computational cost with respect to existing solutions.

##  1 Introduction

The increasing use of Deep Neural Networks (DNNs) fuelled significant advances
in the last decade in Computer Vision and fully benefited from the
improvements in data availability and computational power. Currently, the main
factor impeding a wider adoption of DNNs for critical tasks is their lack of
reliability and interpretability. Subsequently, a lot of effort has been
dedicated to the uncertainty estimation of model predictions during inference,
which aims to alleviate the impact of out-of-distribution and noisy inputs.
The most popular approaches are either based on ensembles, or on some
simplificatory assumptions applied to Bayesian Neural Networks [Gal and
Ghahramani(2016), Lakshminarayanan et al.(2016)Lakshminarayanan, Pritzel, and
Blundell, Wen et al.(2020)Wen, Tran, and Ba]. In both cases, the predictive
uncertainty is related to the output diversity, the quality of which is
correlated to a higher computational cost. Moreover, classification models
have received more attention since their output is simpler and allows for an
easier definition of uncertainty estimation metrics [Hendrycks and
Gimpel(2016), Hendrycks and Dietterich(2019)]. In a regression setting, DNNs
are expected uncertainty-wise to provide a probability distribution over the
target domain, but most existing models also lack a default uncertainty
output. Following the works addressing classification models, some strategies
and metrics for uncertainty estimation and calibration have been proposed more
recently for regression [Ilg et al.(2018)Ilg, Cicek, Galesso, Klein, Makansi,
Hutter, and Brox, Gustafsson et al.(2020)Gustafsson, Danelljan, and Schon].
However, the proposed solutions tend to be more specific due to the higher
variability of the output domain associated with diverse input data, _e.g_
time series forecasting vs. pixel-wise regression. As a default solution,
ensemble-based strategies are relatively effective but incur a significant
training and inference computational cost, which is sometimes prohibitive when
considered along with the memory requirements for some tasks.

Alternatively, using a side learner [Jain et al.(2021)Jain, Lahlou, Nekoei,
Butoi, Bertin, Rector-Brooks, Korablyov, and Bengio, Yoo and Kweon(2019), Hu
et al.(2020)Hu, Pezzotti, and Welling, Corbière et al.(2019)Corbière, Thome,
Bar-Hen, Cord, and Pérez] may provide a generic solution for predictive
uncertainty estimation. The side learner could be regarded as a post-
processing applied to the prediction result. Therefore, it is a
straightforward approach to provide a precise uncertainty estimation without
influencing the performance of the main task model or further refining its
hyper-parameters. However, for some pixel-wise regression tasks, this approach
may be challenging because of the importance of the semantic context. Compared
with the uncertainty for time series data that pays more attention to long
short-term information, the uncertainty of image-based tasks should give more
consideration to the semantics of the image. The uncertainty map is based on
the prediction map because of the similar higher level encoded semantics.
However, obviously, the prediction result might miss relevant semantics with
respect to the ground truth, which may have a detrimental effect on the
uncertainty estimation. Our work intends to fill this gap by introducing
SLURP, a general side learning approach for regression problems able to
recover semantic information absent from the main task prediction. Our
contributions are as follows. 1\. We are the first to solve the uncertainty
estimation problem for general pixel-wise regression tasks with an auxiliary
network. 2\. We propose a transposable architecture which may be used along
with the main task model without modifying/re-training/affecting the
performance of the latter, thus greatly improving our proposal’s adoption
potential. 3\. We demonstrate our side learner’s flexibility on two
fundamental vision tasks. The extensive experiments validate our algorithm’s
consistent performance in line with SOTA uncertainty estimation algorithms.
The main advantage of our proposed solution is the efficiency and simplicity
of its implementation and its competitive uncertainty performance.

##  2 Related Works

Figure 1: An example for uncertainty estimation on optical flow task. The
predicted flow is made by FlowNetS [Dosovitskiy et al.(2015)Dosovitskiy,
Fischer, Ilg, Hausser, Hazirbas, Golkov, Van Der Smagt, Cremers, and Brox].
The prediction error is the end-point-error between ground truth flow and
predicted flow. We can see that our uncertainty estimation map can correspond
well to the true error, including the semantic loss area (middle left) and the
edges in the connected domains of the prediction flow map.

###  2.1 Major uncertainty estimation approaches

1\. Distribution-based approaches By assuming a probability distribution over
the model output and by minimizing the associated negative log-likelihood [Nix
and Weigend(1994), Kendall and Gal(2017)], one may obtain the aleatoric
uncertainty of the prediction result. Based on this idea, [Asai et
al.(2019)Asai, Ikami, and Aizawa] separate the formulation and use it as a
two-task learning which could improve the main task and uncertainty
performance on indoor depth estimation. As for the considered distribution,
Gundavarapu  _et al_ [Gundavarapu et al.(2019)Gundavarapu, Srivastava, Mitra,
Sharma, and Jain] choose to use a multi-variate Gaussian to capture the
aleatoric uncertainty in human pose, while in their experiments [Gast and
Roth(2018), Ilg et al.(2018)Ilg, Cicek, Galesso, Klein, Makansi, Hutter, and
Brox] choose the best performing distribution from an exponential distribution
cluster. Apart from supervised methods, [Poggi et al.(2020)Poggi, Aleotti,
Tosi, and Mattoccia] learns uncertainty from monocular depth in a self-
supervised way with a teacher-student framework using the concept of [Kendall
and Gal(2017)], while being too specific for re-implementation on the other
tasks.
2\. Ensemble-based approaches In the statistics community [Efron and
Tibshirani(1994), Steck and Jaakkola(2003)] and later in computer vision
[Kybic and Nieuwenhuis(2011), Ilg et al.(2018)Ilg, Cicek, Galesso, Klein,
Makansi, Hutter, and Brox], the most conventional way to capture model
uncertainty has been to use bootstrapping in which the uncertainty is linked
to the variance of multiple outputs generated by re-sampling. Although it is
accessible and well studied, this approach is not well suited to deep learning
as it tends to degrade the main task performance. Instead of using part of the
data each time, Deep Ensembles [Lakshminarayanan et al.(2016)Lakshminarayanan,
Pritzel, and Blundell] uses the full data to train the model under different
stochasticity sources and combines the aleatoric uncertainty estimation
proposed by [Kendall and Gal(2017)]. While this method is easily adaptable to
different tasks while still achieving an extremely competitive baseline
[Ashukha et al.(2020)Ashukha, Lyzhov, Molchanov, and Vetrov], its memory and
run-time consumption make it inefficient in real applications due to its
dependence on multiple, potentially heavy models. Eddy Ilg _et al_ [Ilg et
al.(2018)Ilg, Cicek, Galesso, Klein, Makansi, Hutter, and Brox] use a
multiple-hypothesis network combining with the concept of [Kendall and
Gal(2017)] to reduce memory consumption of ensembles. However, it is specific
for optical flow, it has to modify the main task model and add an extra
network in the end for merging the hypotheses, and it requires multiple stages
of training.
3\. Bayesian approaches Learning the posterior distribution of the weights of
the main task neural network given the training dataset is a way to obtain the
uncertainty from the network architecture [Blundell et al.(2015)Blundell,
Cornebise, Kavukcuoglu, and Wierstra]. Gast _et al_ [Gast and Roth(2018)] try
to weight the uncertainty from network weight by using assumed density
filtering [Boyen and Koller(2013)], but in this case the network has to be
modified. MC-Dropout [Gal and Ghahramani(2016)] is a widely used method which
mimics a BNN by using injected Dropout layers to sample the network weights.
It needs to do multiple times forward propagation which is costly especially
for pixel-wise regression tasks. At the same time, an additional challenge is
raised by properly balancing the main performance and the quality of the
generated uncertainty due to the number and position of the injected Dropout
layers.

###  2.2 Side learning for uncertainty estimation

With the development of deep learning, DNNs are becoming more sophisticated,
and the training of DNN and the selection of hyperparameters have also become
particularly important [Sun et al.(2019)Sun, Yang, Liu, and Kautz]. As it
avoids changing the main task model structure, a side learner is in this
respect a convenient approach to estimate uncertainty. Lee  _et al_ [Lee et
al.(2020)Lee, Capuano, Harvard, and Chung] use a conditional GAN [Isola et
al.(2017)Isola, Zhu, Zhou, and Efros] as an auxiliary model to project an
image-pair input in optical flow to an uncertainty provided by a MC-Dropout
version of main task to reduce the time cost, but its performance will be
decreased during dataset shift. Yoo  _et al_ [Yoo and Kweon(2019)] design a
side learner to learn the loss supplied by the main task model. However, the
goal of this work is to provide a single predicted loss for the main task,
which is not suitable for pixel-wise uncertainty estimation. Instead of
predicting the loss from the main task model, based on [Lakshminarayanan et
al.(2016)Lakshminarayanan, Pritzel, and Blundell] and [Kendall and Gal(2017)],
Hu  _et al_ [Hu et al.(2020)Hu, Pezzotti, and Welling] trains sequentially a
network which is identical to the main task model to fit the prediction error.
It can improve the stability of uncertainty training but the memory cost of
the uncertainty estimator will increase with the increase of the main task
model scale. DEUP [Jain et al.(2021)Jain, Lahlou, Nekoei, Butoi, Bertin,
Rector-Brooks, Korablyov, and Bengio] not only uses a side learner to predict
the loss but in order to better obtain the epistemic uncertainty, it enriches
the input of the side learner but it needs to train three models including a
main task model with aleatoric uncertainty estimator, a data density estimator
and a loss side learner. This has high requirements for model selection and
training. Corbière  _et al_ [Corbière et al.(2019)Corbière, Thome, Bar-Hen,
Cord, and Pérez] introduces ConfidNet which can conveniently provide the
uncertainty by adding an auxiliary network after the main task encoder. It
shares a more similar approach with our work. It shows a good performance but
it works only for classification task and it has to re-tune the encoder part
for uncertainty prediction so it structurally depends on the main task network
and requires two-stage training.
For pixel-wise regression tasks, the number of data points is huge and data
preparation will be costly. SLURP does not need to modify the main task model,
does not require data preparation. It faces on general pixel-wise regression
tasks, which is different from all the works mentioned above. We use a direct
and explicit design, and only train the side learner once to get the
uncertainty. This can circumvent the difficulty of modifying the main task
model caused by the complexity of the structure, or the lack of the training
codes. Using only the in-distribution data, without touching the main task
model, we can get better quality and robust uncertainty. Fig. 1 gives an
example on uncertainty estimation of our work.

##  3 Our method

###  3.1 Problem description and motivation

Our training procedure is a two-step process that first consists of training a
DNN $f$ to perform its main regression task. Then in a second stage, the
parameters in $f$ are fixed and will not be updated any more. Based on a
primary DNN $f$, we train a second DNN $g$ to predict the uncertainty of the
first DNN.

Task 1: given dataset $D=\\{(\mathbf{x}_{i},y_{i})\\}_{i}$, we write
$P(Y|\mathbf{x})$ the conditional distribution for the ground truth value
given an input value $\mathbf{x}$ (_e.g_ an RGB image). Let us denote $f$, the
main task predictor, which is trained by minimizing the objective function
$l_{f}(f(\mathbf{x}),y)$ over the dataset.

Task 2: Once the DNN $f$ is trained, we propose to add a new task, namely the
prediction error estimation. Our goal is to predict the error done by the DNN,
i.e., to learn to predict $l_{f}(f(\mathbf{x}),y)$. Note that the loss does
not need to be the same as $l_{f}$, since in some cases, $l_{f}$ could follow
a specific design such as in focal loss [Lin et al.(2017)Lin, Goyal, Girshick,
He, and Dollár], in scale-invariant error [Eigen et al.(2014)Eigen, Puhrsch,
and Fergus], etc. Hence we aim to predict the error loss
$l_{u}(f(\mathbf{x}),y)$. As is the case with regression tasks, a sensible
choice for $l_{u}$ is the mean square or absolute error. Our uncertainty DNN
$g$, will have the following training objective:

$\operatorname*{argmin}_{g}L\Big{(}l_{u}(f(\mathbf{x}),y),g(f(\mathbf{x}),\mathbf{x})\Big{)}$
(1)

where $L$ denotes an objective function which needs to be minimized to obtain
a model able to predict the error of $f$.

Note that the error prediction task of task 2, is related to predicting the
total uncertainty of the DNN. The total uncertainty $U(f,\mathbf{x})$ may then
be interpreted as the sum of prediction errors according to [Jain et
al.(2021)Jain, Lahlou, Nekoei, Butoi, Bertin, Rector-Brooks, Korablyov, and
Bengio]:

$U(f,\mathbf{x})=\int{l_{u}(f(\mathbf{x}),y)dP(y|\mathbf{x})}$ (2)

In subsection 3.2, we explain the structure of the side learner $g$, and in
subsection 3.3, we present the design of the loss $L$ to learn to predict the
uncertainty.

Figure 2: General solution for pixel-wise uncertainty estimation. We take
monocular depth estimation as an example. In main procedure, Main task stream
$f$ is first trained and frozen. Then we use the input and the output of $f$
to train $g$. An uncertainty generation block is implemented on feature pairs
from different stages. The $n$ outputs of context blocks from different stages
will be sent to the final fusion block. The conv-layers in detailed operations
are described as (shape, number, dilation ratio).

###  3.2 Side learner’s architecture

The side learner $g$ takes two inputs: $\mathbf{x}$ and $f(\mathbf{x})$. The
combination of image features and prediction result in uncertainty map that
depends on the initial image and on the prediction of $f$. The design of this
architecture is inspired by empirical observations on the prediction error
maps. We notice that, for pixel-wise regression tasks, prediction errors are
organized 1. along edges of connected domains in the prediction map; 2. as
hard predictable areas which are not captured in the prediction map but only
exist in the RGB image, e.g. distant or small objects and occlusions.
Therefore, the RGB image needs to be combined with the prediction map to
recover the semantic information absent from the main task prediction.
Meanwhile, we use convolutional features given by the encoders with a final
fusion block to better capture the edge information [Liu et al.(2017)Liu,
Cheng, Hu, Wang, and Bai]. The concatenated convolutional features are
followed by a context block [Sun et al.(2018)Sun, Yang, Liu, and Kautz]
described as follows.
We structure the side learner $g$ in the following three parts, with Figure 2
showing the general design for our pixel-wise uncertainty estimator.
1\. Feature encoders: They aim to learn and extract the richer convolutional
feature pyramids [Liu et al.(2017)Liu, Cheng, Hu, Wang, and Bai] from raw
input (RGB data) and from the final prediction map preparing for the next
steps. We choose the widely used backbone DenseNet [Huang et al.(2017)Huang,
Liu, Van Der Maaten, and Weinberger]. Note that the architecture is agnostic,
and DenseNet could be replaced by the other backbones like ResNet [He et
al.(2016)He, Zhang, Ren, and Sun], depending on the context. The image encoder
in $g$ can be trained from scratch but can also be replaced by the trained and
frozen image encoder from $f$. It depends on how closely the image input is
related to the prediction error. If we have multiple images as a concatenated
input, we need to encode the image on which the prediction error is based. We
recommend using the same structure for the feature extractors of the image and
the prediction map, so as to avoid information asymmetry. The following
operations will be done from coarse to fine.
2\. Feature fusion: For each pair of RGB and final prediction features, we
concatenate them followed by the convolutional layers in order to reduce the
channel number. Dropout layers are implemented before the features are input
to the convolutional layer, to reduce the bias caused by focusing on specific
patterns. The goal is to find the similarity and the difference between two
sources of features to bring out the lost information during main task
training.
3\. Context block: The architecture of this block follows that of the context
network in PWC-net [Sun et al.(2018)Sun, Yang, Liu, and Kautz]. With various
dilation ratios, the convolutional layers can take the features from different
size of receptive fields. The output intermediate uncertainty from the
previous stage will be up-sampled and concatenated as a guide feature to the
input of current stage.
After obtaining the output of the context block with different resolutions, we
up-sample these outputs toward the same size as the prediction target and then
use a simple convolutional layer to sum them up with different weights as a
final fusion output.

###  3.3 Loss design

1\. Natural loss: A straightforward way is to use mean square error loss (MSE)
for $l_{u}$ in eq. 1 and eq. 2. In order to prevent the distribution of
training target from being too sparse, we take
$l_{u}(f(\mathbf{x}),y)=|f(\mathbf{x})-y|$.
2\. Target scaling: The uncertainty of the regression task is expressed across
a support based on the current value of the variable, with the support being
contained in $\mathbb{R}$. The uncertainty of the classification task is
related to the confidence of the classifier output, which is set to take
values in the range is $[0,1]$. Hence we process $l_{u}(f(\mathbf{x}),y)$ to
bound it between 0 and 1. This can also avoid the influence of the effect from
the outliers which have significant prediction errors thanks to the following
equation :

$\tilde{l_{u}}(f(\mathbf{x}),y)=\tanh(\lambda*l_{u}(f(\mathbf{x}),y))$ (3)

where $\lambda$ is a stretch hyper-parameter that can spread the training
target as much as possible between 0 and 1, and
$\tilde{l_{u}}(f(\mathbf{x}),y)$ is the normalized uncertainty target.
3\. Cross-Entropy Loss: After having normalized the uncertainty $l_{u}$, we
select cross entropy loss to describe the probability distance between
training target $\tilde{l_{u}}$ and our predicted uncertainty. Since
$\tilde{l_{u}}\in{[0,1]}$, we choose a binary cross entropy (BCE) loss as $L$
in eq. 1:

$L(\tilde{l_{u}},g)=-\sum_{i}[\tilde{l_{u}}(f(\mathbf{x}_{i}),y_{i})\log(\sigma(g(f(\mathbf{x}_{i}),\mathbf{x}_{i})))+(1-\tilde{l_{u}}(f(\mathbf{x}_{i}),y_{i}))\log(1-\sigma(g(f(\mathbf{x}_{i}),\mathbf{x}_{i})))]$

(4)

where $\sigma(.)$ is the sigmoid function. Note that here we use BCE loss not
for doing classification, but for regression, and BCE can support a faster
convergence for a Sigmoid output value. By calculating the first derivation of
eq. 4 with respect to $\sigma(g)$, we can find that the optimum will be
$\sigma(g)=\tilde{l_{u}}$.

Figure 3: 1D synthetic regression task comparison example. X-axis: spatial
coordinate of the Gaussian process. Green curve: ground truth; Blue points:
training samples; Red curves: main task prediction; Orange zooms: the
uncertainty coverage (1-sigma for inner interval, 2-sigma for outer interval).

##  4 Experiments

Our main focus is on obtaining high-quality uncertainty maps on pixel-level
regression tasks. However, in order to illustrate more comprehensively our
approach and show its applicability in a different context, we also apply
SLURP on a 1D toy dataset. Overall, our proposed method is illustrated on a
synthetic 1D regression dataset, and on two fundamental computer vision tasks:
optical flow (OF) and monocular depth (MD). For the former, we just visualize
it to give some reference and insights. For the later ones, we evaluate the
quality of predicted uncertainty maps, the basic idea is to see whether the
predicted uncertainty map matches the prediction error. To this end, we re-
implemented the transferable uncertainty estimation approaches MC-Dropout (MC)
[Gal and Ghahramani(2016)], Single predictive uncertainty (Single-PU) [Kendall
and Gal(2017)], Empirical ensembles (EE), ConfidNet (Confid) [Corbière et
al.(2019)Corbière, Thome, Bar-Hen, Cord, and Pérez] and Deep ensembles (DE)
[Lakshminarayanan et al.(2016)Lakshminarayanan, Pritzel, and Blundell] to the
main task as the comparisons. Due to its particularity, we reproduce the
multi-hypothesis prediction network (MHP) [Ilg et al.(2018)Ilg, Cicek,
Galesso, Klein, Makansi, Hutter, and Brox] only for the optical flow task.
Specifically, to transfer Confid solution from the classification task to the
regression task, we duplicate the last few layers from the last de-
convolutional layer (or up-sample operation) and add three extra 3x3
convolutional layers without changing the resolution as ConfidNet for
regression. We keep the training schedule, and the pixel-wise square error
maps will replace the original training targets. We use two evaluation
criteria: AUSE and AUROC to evaluate uncertainty maps generated by different
methods. Additionally, we measure the efficiency of main task - uncertainty
estimator system from two perspectives: Runtime and Number of parameters. The
specific formulas as well as the implementation details for the methods we
compare with are provided in the supp. material.

###  4.1 1D regression task toy example

We compare our proposed solution with MC [Gal and Ghahramani(2016)], EE,
Single-PU [Kendall and Gal(2017)] and DE [Lakshminarayanan et
al.(2016)Lakshminarayanan, Pritzel, and Blundell] on a 1D regression task
dataset. The toy dataset is generated by Gaussian process, our spatial
coordinate range is $x_{i}\in{[-10,10]}$. From $x_{i}\in{[-7,7]}$, we cross-
select 875 data points as training set and 175 data points as validation set.
From $x_{i}\in{[-10,10]}$, we randomly select 400 points as test set. The main
task predictor $f$ with single output has only one hidden layer, the Single-PU
and DE are implemented based on a modified dual-output $f$. The detailed model
and training settings for all the methods are in the supp. material. Moreover,
we use two modes to train our SLURP side learner $g$. One is that we first
train one $f$, then we freeze it and train the $g$ with the prediction result
from $f$ and the latent feature extracted from the single hidden layer in $f$,
i.e., sequential-training, which is the original design of our approach.
Another one is that we train $f$ and $g$ at the same time by using the
negative log-likelihood loss [Kendall and Gal(2017)], i.e., joint-training.
Because of the data dimension, the side learner shown in Fig 2 is modified
(see supp. material). As illustrated in Fig 3, the results of MC [Gal and
Ghahramani(2016)] and EE give a good uncertainty on the unseen area, but
little on the training part. While DE [Lakshminarayanan et
al.(2016)Lakshminarayanan, Pritzel, and Blundell] and Single-PU [Kendall and
Gal(2017)] can give a sufficient uncertainty to all areas. But in comparison,
their main task prediction accuracy has been affected. SLURP can achieve both
reasonable main task accuracy and tight uncertainty coverage especially for
joint-training one.

###  4.2 Evaluation protocols for pixel-wise tasks

MD and OF are two fundamental regression tasks which have significant
implications in a wide range of applications. In terms of inputs, the main
difference is that MD requires a single RGB image, while OF needs an image
pair. For each pixel, the MD output is a depth value
$y_{d}\in\mathcal{R}^{+}$, while the OF output is a 2-channel displacement
vector $y_{o}\in\mathcal{R}^{2}$. We introduce below the evaluation criteria.
1\. Uncertainty ordering: Let us consider that we want to remove the worst
pixels based on an uncertainty estimator; we expect that a good uncertainty
estimator should allow us to remove the less reliable data. This is evaluated
thanks to the sparsification curve (SC), and the area under sparsification
error (AUSE) [Bruhn and Weickert(2006), Kondermann et al.(2008)Kondermann,
Mester, and Garbe, Kybic and Nieuwenhuis(2011), Ilg et al.(2018)Ilg, Cicek,
Galesso, Klein, Makansi, Hutter, and Brox, Poggi et al.(2020)Poggi, Aleotti,
Tosi, and Mattoccia]. To build the SC, given a set of data and their
uncertainty, we iteratively erase $m\%$ (we take $m=5$ in our experiments) of
the data which exhibit the highest uncertainty. Then we calculate the average
prediction error for the remaining data. Hence we have the SC. To evaluate the
oracle SC, we remove the $m\%$ data with the most significant prediction error
and we calculate the average prediction error for the remaining data. We
denote the area between the two SC curves as AUSE. The smaller the AUSE, the
closer the order of the predicted uncertainty and the order of the Oracle. As
a note, AUSE could be changed if we change the error metric. Therefore, we
denote AUSE based on different error metrics as AUSE-xxx.
Implementations: Specifically, for monocular depth, we choose square error and
absolute relative error [Eigen et al.(2014)Eigen, Puhrsch, and Fergus] .The
corresponding AUSEs are AUSE-RMSE and AUSE-Absrel. For optical flow we use
EPE, which is the error map representing the Euclidean distance between the
ground truth motion and the predicted one, we denote its AUSE as AUSE-EPE.
2\. AUROC: Since we have access to a soft evaluation of uncertainty, it is
feasible to threshold the dataset into two sets, namely the reliable set and
the unreliable set. We propose for MD to set the data as reliable if they
check the inlier metrics threshold d1 criterion proposed in [Hu et
al.(2017)Hu, Li, and Song], and for OF data is reliable if its EPE is below
$k=2$. We scale the predicted uncertainty between 0 and 1 with min-max scaling
and evaluate the ROC curve. The larger the AUROC, the more data points are
given correct confidence (uncertainty) and the better the uncertainty
estimator is.
3\. Model efficiency: Due to the different schemes of uncertainty generation
design, we will measure the number of model parameters and time consumption of
the entire system (# Param.), including the main task model and uncertainty
generator. We count the running time (Runtime) while processing a whole
testing dataset in one NVIDIA TITAN RTX GPU and Intel Core i9-10900X CPU then
take the average according to the number of samples.

###  4.3 Monocular depth

In this section, we introduce uncertainty estimations for MD. We choose BTS
[Lee et al.(2019)Lee, Han, Ko, and Suh] as the depth estimator, and the
uncertainty estimators will be implemented based on this architecture. BTS is
one of the state-of-the-art architectures on MD benchmarks [Silberman et
al.(2012)Silberman, Hoiem, Kohli, and Fergus, Geiger et al.(2013)Geiger, Lenz,
Stiller, and Urtasun]. As an encoder-decoder based network it is well suited
for the extraction of latent image features as shown in Fig 2, however note
that our strategy is agnostic to the main task architecture. In accordance
with the default setting, we choose DenseNet161 [Huang et al.(2017)Huang, Liu,
Van Der Maaten, and Weinberger] as encoder for BTS. So for our side learner,
we take the trained and fixed image encoder of BTS as our image encoder, and
implement a new DenseNet161 encoder for estimated depth map.
Datasets and procedures We choose two widely used datasets and their
variations to train and evaluate uncertainty estimators as follows. Training
set: KITTI [Geiger et al.(2013)Geiger, Lenz, Stiller, and Urtasun, Uhrig et
al.(2017)Uhrig, Schneider, Schneider, Franke, Brox, and Geiger] Eigen-split
training set [Eigen et al.(2014)Eigen, Puhrsch, and Fergus]; Fine-tuning set:
Cityscapes training set [Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld,
Enzweiler, Benenson, Franke, Roth, and Schiele]; Test set: KITTI Eigen-split
test set, Cityscapes test set, Foggy Cityscapes-DBF test set [Sakaridis et
al.(2018)Sakaridis, Dai, Hecker, and Van Gool] with three severity levels and
Rainy Cityscapes test set [Hu et al.(2019)Hu, Fu, Zhu, and Heng] with three
parameter sets which indicate three severities. KITTI depth dataset and
Cityscapes dataset are both outdoor datasets. However, since the ground truth
on KITTI is sparse and has only half the content of the image, the model
performance is limited if training on KITTI and testing on Cityscapes.
Therefore, we first train models on KITTI Eigen-split training set then
evaluate the uncertainty on KITTI Eigen-split test set and Cityscapes test
set. After that, we fine-tune the models on Cityscapes training set and
evaluate the uncertainty on all Cityscapes test sets listed above. Results
Table 1 presents the performance of different methods in normal circumstances
and against gradual input perturbations. The top performing method is
highlighted in bold, while the second one is highlighted in blue. SLURP,
ConfidNet and Deep ensembles exhibit competitive performance being both ranked
close in terms of uncertainty ordering on the different metrics, with our
proposal being clearly more robust against strong input perturbations.

Datasets | Criteria | MC | EE | Single PU | DE | Confid | Ours
---|---|---|---|---|---|---|---
KITTI | AUSE-RMSE | 8.14 | 3.17 | 1.89 | 1.68 | 1.76 | 1.68
AUSE-Absrel | 9.48 | 5.02 | 4.59 | 4.32 | 4.24 | 4.36
AUROC | 0.686 | 0.882 | 0.882 | 0.897 | 0.892 | 0.895
CityScapes | AUSE-RMSE | 9.42 | 11.56 | 9.91 | 11.47 | 10.48 | 9.48
AUSE-Absrel | 9.52 | 13.14 | 9.96 | 9.36 | 5.75 | 10.90
AUROC | 0.420 | 0.504 | 0.386 | 0.501 | 0.519 | 0.400
After fine-tuning on CityScapes
CityScapes | AUSE-RMSE | 7.72 | 8.20 | 4.35 | 3.03 | 4.05 | 3.05
AUSE-Absrel | 8.13 | 7.50 | 6.44 | 6.81 | 6.34 | 6.55
AUROC | 0.705 | 0.786 | 0.741 | 0.856 | 0.821 | 0.849
CityScapes Rainy s=1 | AUSE-RMSE | 7.06 | 7.29 | 4.17 | 3.42 | 4.89 | 3.39
AUSE-Absrel | 8.73 | 6.92 | 6.55 | 6.68 | 7.26 | 5.62
AUROC | 0.659 | 0.757 | 0.731 | 0.746 | 0.697 | 0.788
CityScapes Rainy s=2 | AUSE-RMSE | 7.14 | 6.9 | 4.27 | 3.35 | 4.68 | 3.36
AUSE-Absrel | 8.36 | 6.48 | 6.79 | 6.24 | 6.86 | 5.28
AUROC | 0.667 | 0.767 | 0.731 | 0.756 | 0.714 | 0.794
CityScapes Rainy s=3 | AUSE-RMSE | 7.30 | 6.66 | 4.35 | 3.28 | 4.59 | 3.41
AUSE-Absrel | 8.27 | 6.03 | 6.44 | 5.85 | 6.64 | 5.05
AUROC | 0.665 | 0.778 | 0.742 | 0.767 | 0.729 | 0.801
CityScapes Foggy s=1 | AUSE-RMSE | 7.80 | 7.82 | 3.42 | 3.05 | 3.98 | 3.04
AUSE-Absrel | 8.36 | 7.33 | 6.78 | 6.58 | 6.21 | 6.25
AUROC | 0.700 | 0.783 | 0.842 | 0.852 | 0.824 | 0.847
CityScapes Foggy s=2 | AUSE-RMSE | 7.82 | 7.53 | 3.42 | 2.98 | 3.86 | 3.01
AUSE-Absrel | 8.20 | 7.09 | 6.55 | 6.35 | 6.02 | 6.06
AUROC | 0.704 | 0.791 | 0.847 | 0.857 | 0.833 | 0.852
CityScapes Foggy s=3 | AUSE-RMSE | 7.84 | 7.28 | 3.48 | 2.93 | 3.70 | 3.08
AUSE-Absrel | 7.87 | 6.80 | 6.19 | 6.01 | 5.78 | 5.80
AUROC | 0.715 | 0.801 | 0.851 | 0.863 | 0.846 | 0.857

Table 1: MD uncertainty performance. Bold value: result with the best
performance, Blue value: second performance. s (_e.g_ s=1) indicates severity,
higher the s value, higher the severity.

###  4.4 Optical flow

FlowNetS [Dosovitskiy et al.(2015)Dosovitskiy, Fischer, Ilg, Hausser,
Hazirbas, Golkov, Van Der Smagt, Cremers, and Brox] is a popular OF
architecture, as the first learning-based optical flow estimation method. We
apply it as our main task model and implement the uncertainty estimators based
on it, since it can be regarded as a good example for uncertainty estimation
on early structures with special inputs. FlowNetS is also an encoder-decoder
network, however - differently from the more recent architectures with a
separate feature extractor implemented for the first image [Ilg et
al.(2017)Ilg, Mayer, Saikia, Keuper, Dosovitskiy, and Brox, Sun et
al.(2018)Sun, Yang, Liu, and Kautz, Hui et al.(2018)Hui, Tang, and Loy, Yang
and Ramanan(2019), Poggi et al.(2020)Poggi, Aleotti, Tosi, and Mattoccia],
FlowNetS takes directly image pairs as encoder input, so the features from
different encoder stages will be mixture feature maps of two RGB images. Since
the ground truth motion is based on the first image, the true error will be
based only on the first image, we use a new encoder to extract the RGB
features only for the first image. In this case, in our side learner, we use
two trainable encoders respectively for the first image and the predicted
flow. We choose both DenseNet161 and DenseNet121 [Huang et al.(2017)Huang,
Liu, Van Der Maaten, and Weinberger] as the encoders in our experiments and we
denote the latter Ours-light as a lightweight version of the former. Datasets
Training set: synthetic FlyingChairs training set [Dosovitskiy et
al.(2015)Dosovitskiy, Fischer, Ilg, Hausser, Hazirbas, Golkov, Van Der Smagt,
Cremers, and Brox]; Testing set: FlyingChairs test set, synthetic Sintel
training set [Butler et al.(2012)Butler, Wulff, Stanley, and Black] and real-
world KITTI 2015 training set [Geiger et al.(2013)Geiger, Lenz, Stiller, and
Urtasun, Menze et al.(2015)Menze, Heipke, and Geiger, Menze et al.(2018)Menze,
Heipke, and Geiger]. The train-test split file for FlyingChairs dataset is
provided officially. Sintel has more complex moving objects and movements than
FlyingChairs, while KITTI is taken from real-world, and exhibits larger
movement magnitude.

Datasets | Criteria | MC | EE | Single PU | DE | Confid | MHP | Ours-light | Ours
---|---|---|---|---|---|---|---|---|---
FlyingChairs | AUSE-EPE | 2.75 | 1.97 | 1.28 | 1.26 | 1.92 | 1.88 | 1.16 | 1.20
AUROC | 0.896 | 0.900 | 0.977 | 0.959 | 0.945 | 0.936 | 0.977 | 0.974
KITTI | AUSE-EPE | 3.57 | 4.41 | 3.71 | 3.45 | 6.56 | 5.48 | 5.20 | 4.69
AUROC | 0.870 | 0.904 | 0.848 | 0.866 | 0.687 | 0.854 | 0.791 | 0.800
Sintel Clean | AUSE-EPE | 3.33 | 2.89 | 2.74 | 3.02 | 5.28 | 2.61 | 3.02 | 2.91
AUROC | 0.861 | 0.825 | 0.925 | 0.895 | 0.767 | 0.886 | 0.889 | 0.896
Sintel Final | AUSE-EPE | 3.30 | 3.02 | 3.09 | 3.05 | 6.06 | 2.71 | 2.95 | 2.86
AUROC | 0.858 | 0.814 | 0.916 | 0.899 | 0.728 | 0.878 | 0.901 | 0.906

Table 2: OF uncertainty performance. Bold value: result with the best
performance. Blue value: second performance.

Results Table 2 shows the uncertainty estimation performance on the various OF
datasets. Even though all uncertainty estimators are not fine-tuned on
datasets other than the training set, they maintain a good estimation level.
On this very different scenario, MHP did a good job due to its pertinence, but
our side learner performs competitively and robustly across all tests/metrics.

###  4.5 Model efficiency

Datasets and settings We select to use Sintel training set [Butler et
al.(2012)Butler, Wulff, Stanley, and Black] and KITTI [Geiger et
al.(2013)Geiger, Lenz, Stiller, and Urtasun] Eigen-split validation set [Eigen
et al.(2014)Eigen, Puhrsch, and Fergus] as the testing sets for Runtime
evaluation. We use three models to form EE and DE and eight forward
propagations for MC. In MD, due to the instability of training, we use
sequential-training for Single-PU and DE. More details are in the supp.
material.
Results  Table 3 shows the model efficiency in two tasks. In OF task, due to
the convenience of modifying the main task model, we confirm that Single-PU is
more efficient than the other methods, while our lightweight version can
achieve comparable performance. For MD, due to the complexity in
implementation for joint uncertainty generation, the side-learner-based
methods have a fixed computational footprint and take significant advantage in
the cost. In addition, our solution can reuse without re-tuning the main task
encoder, so it is lighter and faster than the other auxiliary networks.

Task | Criteria | MC | EE | Single-PU | DE | Confid | MHP | Ours-light | Ours
---|---|---|---|---|---|---|---|---|---
MD | Runtime (ms) | 386 | 144 | 98 | 286 | 106 | - | - | 88
# Param. (M) | 47.0 | 141.0 | 94.0 | 282.0 | 94.7 | - | - | 87.2
OF | Runtime (ms) | 79 | 65 | 64 | 66 | 65 | 67 | 65 | 76
# Param. (M) | 38.7 | 116.1 | 38.7 | 116.3 | 78.6 | 78.8 | 57.0 | 105.3

Table 3: Average time cost for processing one image and number of parameters
of the model(s) (main task + uncertainty task). Bold value: result with the
best performance. Blue value: second performance.

##  5 Conclusion

In this work we proposed a novel solution for estimating the total uncertainty
of a DNN. SLURP is a side learner that does not affect the main prediction
estimator. Given a trained main task model, we use the original input and main
task prediction results as input, and the prediction error as the target.
Following feature extraction, fusion and content reconstruction, we train for
the total uncertainty of the main task prediction. SLURP can also exploit
latent features of the original input provided by a frozen feature extractor
of the main task model, such as an image encoder. We compared our proposal
with popular uncertainty estimation approaches, and performed detailed
experiments on 2D pixel-wise regression tasks to prove that our method is
feasible and robust. Future works involve further optimizing the SLURP
structure to decrease its footprint and make it accessible for embedded
processing, as well as applying this side learning concept to other types of
input signals.

## References

  * [Asai et al.(2019)Asai, Ikami, and Aizawa] Akari Asai, Daiki Ikami, and Kiyoharu Aizawa.  Multi-task learning based on separable formulation of depth estimation and its uncertainty.  In _CVPR Workshops_ , pages 21–24, 2019.
  * [Ashukha et al.(2020)Ashukha, Lyzhov, Molchanov, and Vetrov] Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov.  Pitfalls of in-domain uncertainty estimation and ensembling in deep learning.  _arXiv preprint arXiv:2002.06470_ , 2020.
  * [Blundell et al.(2015)Blundell, Cornebise, Kavukcuoglu, and Wierstra] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In _International Conference on Machine Learning_ , pages 1613–1622. PMLR, 2015.
  * [Boyen and Koller(2013)] Xavier Boyen and Daphne Koller.  Tractable inference for complex stochastic processes.  _arXiv preprint arXiv:1301.7362_ , 2013.
  * [Bruhn and Weickert(2006)] Andrés Bruhn and Joachim Weickert.  A confidence measure for variational optic flow methods.  In _Geometric Properties for Incomplete Data_ , pages 283–298. Springer, 2006.
  * [Butler et al.(2012)Butler, Wulff, Stanley, and Black] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black.  A naturalistic open source movie for optical flow evaluation.  In A. Fitzgibbon et al. (Eds.), editor, _European Conf. on Computer Vision (ECCV)_ , Part IV, LNCS 7577, pages 611–625. Springer-Verlag, October 2012.
  * [Corbière et al.(2019)Corbière, Thome, Bar-Hen, Cord, and Pérez] Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez.  Addressing failure prediction by learning model confidence.  _arXiv preprint arXiv:1910.04851_ , 2019.
  * [Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and Schiele] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3223, 2016.
  * [Dosovitskiy et al.(2015)Dosovitskiy, Fischer, Ilg, Hausser, Hazirbas, Golkov, Van Der Smagt, Cremers, and Brox] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2758–2766, 2015.
  * [Efron and Tibshirani(1994)] Bradley Efron and Robert J Tibshirani.  _An introduction to the bootstrap_.  CRC press, 1994.
  * [Eigen et al.(2014)Eigen, Puhrsch, and Fergus] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  _arXiv preprint arXiv:1406.2283_ , 2014.
  * [Gal and Ghahramani(2016)] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR, 2016.
  * [Gast and Roth(2018)] Jochen Gast and Stefan Roth.  Lightweight probabilistic deep networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3369–3378, 2018.
  * [Geiger et al.(2012)Geiger, Lenz, and Urtasun] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * [Geiger et al.(2013)Geiger, Lenz, Stiller, and Urtasun] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _The International Journal of Robotics Research_ , 32(11):1231–1237, 2013.
  * [Gundavarapu et al.(2019)Gundavarapu, Srivastava, Mitra, Sharma, and Jain] Nitesh B Gundavarapu, Divyansh Srivastava, Rahul Mitra, Abhishek Sharma, and Arjun Jain.  Structured aleatoric uncertainty in human pose estimation.  In _CVPR Workshops_ , volume 2, page 2, 2019.
  * [Gustafsson et al.(2020)Gustafsson, Danelljan, and Schon] Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schon.  Evaluating scalable bayesian deep learning methods for robust computer vision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 318–319, 2020.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * [Hendrycks and Dietterich(2019)] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  _arXiv preprint arXiv:1903.12261_ , 2019.
  * [Hendrycks and Gimpel(2016)] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_ , 2016.
  * [Hu et al.(2020)Hu, Pezzotti, and Welling] Shi Hu, Nicola Pezzotti, and Max Welling.  A new perspective on uncertainty quantification of deep ensembles.  _arXiv e-prints_ , pages arXiv–2002, 2020.
  * [Hu et al.(2019)Hu, Fu, Zhu, and Heng] Xiaowei Hu, Chi-Wing Fu, Lei Zhu, and Pheng-Ann Heng.  Depth-attentional features for single-image rain removal.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8022–8031, 2019.
  * [Hu et al.(2017)Hu, Li, and Song] Yinlin Hu, Yunsong Li, and Rui Song.  Robust interpolation of correspondences for large displacement optical flow.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 481–489, 2017.
  * [Huang et al.(2017)Huang, Liu, Van Der Maaten, and Weinberger] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * [Hui et al.(2018)Hui, Tang, and Loy] Tak-Wai Hui, Xiaoou Tang, and Chen Change Loy.  Liteflownet: A lightweight convolutional neural network for optical flow estimation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8981–8989, 2018.
  * [Ilg et al.(2017)Ilg, Mayer, Saikia, Keuper, Dosovitskiy, and Brox] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2462–2470, 2017.
  * [Ilg et al.(2018)Ilg, Cicek, Galesso, Klein, Makansi, Hutter, and Brox] Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox.  Uncertainty estimates and multi-hypotheses networks for optical flow.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 652–667, 2018.
  * [Isola et al.(2017)Isola, Zhu, Zhou, and Efros] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1125–1134, 2017.
  * [Jain et al.(2021)Jain, Lahlou, Nekoei, Butoi, Bertin, Rector-Brooks, Korablyov, and Bengio] Moksh Jain, Salem Lahlou, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio.  Deup: Direct epistemic uncertainty prediction.  _arXiv preprint arXiv:2102.08501_ , 2021.
  * [Kendall and Gal(2017)] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * [Kondermann et al.(2008)Kondermann, Mester, and Garbe] Claudia Kondermann, Rudolf Mester, and Christoph Garbe.  A statistical confidence measure for optical flows.  In _European Conference on Computer Vision_ , pages 290–301. Springer, 2008.
  * [Kybic and Nieuwenhuis(2011)] Jan Kybic and Claudia Nieuwenhuis.  Bootstrap optical flow confidence and uncertainty measure.  _Computer Vision and Image Understanding_ , 115(10):1449–1462, 2011.
  * [Lakshminarayanan et al.(2016)Lakshminarayanan, Pritzel, and Blundell] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _arXiv preprint arXiv:1612.01474_ , 2016.
  * [Lee et al.(2019)Lee, Han, Ko, and Suh] Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh.  From big to small: Multi-scale local planar guidance for monocular depth estimation.  _arXiv preprint arXiv:1907.10326_ , 2019.
  * [Lee et al.(2020)Lee, Capuano, Harvard, and Chung] Serin Lee, Vincenzo Capuano, Alexei Harvard, and Soon-Jo Chung.  Fast uncertainty estimation for deep learning based optical flow.  2020\.
  * [Lin et al.(2017)Lin, Goyal, Girshick, He, and Dollár] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988, 2017.
  * [Liu et al.(2017)Liu, Cheng, Hu, Wang, and Bai] Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Kai Wang, and Xiang Bai.  Richer convolutional features for edge detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3000–3009, 2017.
  * [Menze et al.(2015)Menze, Heipke, and Geiger] Moritz Menze, Christian Heipke, and Andreas Geiger.  Joint 3d estimation of vehicles and scene flow.  In _ISPRS Workshop on Image Sequence Analysis (ISA)_ , 2015.
  * [Menze et al.(2018)Menze, Heipke, and Geiger] Moritz Menze, Christian Heipke, and Andreas Geiger.  Object scene flow.  _ISPRS Journal of Photogrammetry and Remote Sensing (JPRS)_ , 2018\.
  * [Nix and Weigend(1994)] D. Nix and A. Weigend.  Estimating the mean and variance of the target probability distribution.  _Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94)_ , 1:55–60 vol.1, 1994.
  * [Poggi et al.(2020)Poggi, Aleotti, Tosi, and Mattoccia] Matteo Poggi, Filippo Aleotti, Fabio Tosi, and Stefano Mattoccia.  On the uncertainty of self-supervised monocular depth estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3227–3237, 2020.
  * [Sakaridis et al.(2018)Sakaridis, Dai, Hecker, and Van Gool] Christos Sakaridis, Dengxin Dai, Simon Hecker, and Luc Van Gool.  Model adaptation with synthetic and real data for semantic dense foggy scene understanding.  In _European Conference on Computer Vision (ECCV)_ , pages 707–724, 2018.
  * [Silberman et al.(2012)Silberman, Hoiem, Kohli, and Fergus] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _European conference on computer vision_ , pages 746–760. Springer, 2012.
  * [Steck and Jaakkola(2003)] Harald Steck and Tommi S Jaakkola.  Bias-corrected bootstrap and model uncertainty.  In _NIPS_ , pages 521–528. Citeseer, 2003.
  * [Sun et al.(2018)Sun, Yang, Liu, and Kautz] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8934–8943, 2018.
  * [Sun et al.(2019)Sun, Yang, Liu, and Kautz] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Models matter, so does training: An empirical study of cnns for optical flow estimation.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(6):1408–1423, 2019.
  * [Uhrig et al.(2017)Uhrig, Schneider, Schneider, Franke, Brox, and Geiger] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger.  Sparsity invariant cnns.  In _International Conference on 3D Vision (3DV)_ , 2017.
  * [Wen et al.(2020)Wen, Tran, and Ba] Yeming Wen, Dustin Tran, and Jimmy Ba.  Batchensemble: an alternative approach to efficient ensemble and lifelong learning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * [Yang and Ramanan(2019)] Gengshan Yang and Deva Ramanan.  Volumetric correspondence networks for optical flow.  _NeurIPS_ , 5:12, 2019.
  * [Yoo and Kweon(2019)] Donggeun Yoo and In So Kweon.  Learning loss for active learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 93–102, 2019.

SLURP: Side Learning Uncertainty for Regression Problems
— Supplementary Material —

Side Learning Uncertainty for Regression Problems with SLURP

The document is structured as follows. We recall first the abbreviations which
are used and which will help the reader follow the next sections (Section 6).
Then, we present and discuss the ablation study (Section 7). Section 8
provides all the information which is needed in order to duplicate and
evaluate the result of the experiments performed in the main paper, more
specifically the monocular depth estimation (Section 8.2), the optical flow
estimation (Section 8.3) and the toy example (Section 8.4). Information about
the accuracy of the main task models is included as well. Finally, Section 9
presents some additional qualitative results which provide further insights
about the behavior of SLURP and of the other considered strategies.

##  6 Notations

In Table 4 we summarize the abbreviations used in the paper.

Abbreviation | Meaning
---|---
AUROC | Area under receiver operating characteristic curve
AUSE | Area under sparsification error
Confid | ConfidNet uncertainty estimation solution
DE | Deep ensembles
EE | Empirical ensemble
MC | MC-Dropout
MD | Monocular depth
MHP | Multi-hypothesis prediction network for optical flow uncertainty estimation
OF | Optical flow
SC | Sparsification curve
Single-PU | Single predictive uncertainty

Table 4: Summary of the abbreviations of the paper.

##  7 Ablation study

1\. Ablation study settings: The ablation study for the monocular depth (MD)
task is implemented on KITTI [Geiger et al.(2013)Geiger, Lenz, Stiller, and
Urtasun, Uhrig et al.(2017)Uhrig, Schneider, Schneider, Franke, Brox, and
Geiger] Eigen-split test set [Eigen et al.(2014)Eigen, Puhrsch, and Fergus],
Cityscapes test set [Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld,
Enzweiler, Benenson, Franke, Roth, and Schiele], Foggy Cityscapes-DBF test set
[Sakaridis et al.(2018)Sakaridis, Dai, Hecker, and Van Gool] and Rainy
Cityscapes test set [Hu et al.(2019)Hu, Fu, Zhu, and Heng]. The ablation study
for the optical flow (OF) task is implemented on FlyingChairs test set
[Dosovitskiy et al.(2015)Dosovitskiy, Fischer, Ilg, Hausser, Hazirbas, Golkov,
Van Der Smagt, Cremers, and Brox], Sintel training set [Butler et
al.(2012)Butler, Wulff, Stanley, and Black] and KITTI 2015 training set
[Geiger et al.(2013)Geiger, Lenz, Stiller, and Urtasun, Menze et
al.(2015)Menze, Heipke, and Geiger, Menze et al.(2018)Menze, Heipke, and
Geiger]. Brief descriptions of these datasets can be found in the main paper.
We use the same uncertainty evaluation metrics (AUSE and AUROC) as section 4.2
in the main paper.

2\. Ablation study goals: We want to highlight the impact of the two
considered inputs on the final performance (namely the image features and the
prediction results features), and the impact of the considered loss (binary
cross entropy loss and mean square error loss).

3\. Results: The models with different inputs and different loss functions are
presented as follows. Table 5 presents the model performance on OF task and
Table 6 illustrates the results on MD task. Note that in the tables BCE and
MSE denote binary cross entropy loss and mean square error loss respectively,
PredOnly and RGBOnly denote the models taking only prediction map as input and
the models taking only RGB image as input respectively. No special note means
that the model will use both RGB and prediction results as input and BCE as
the loss function (the default behavior).

4\. Discussions: Firstly, regarding the performance of the different loss
functions, we found that the results obtained with the BCE loss are almost
systematically better than those provided when using MSE loss. We think this
is because when we have a correctly trained predictor for the main task, most
of the data points have minor errors, while a small number of data points have
high errors. Using the MSE loss will amplify the more significant prediction
errors and reduce the minor errors, making the model unable to fit well. Our
target scaling uses a soft clipping strategy to centralize the distribution of
data for better fitting.

For different inputs, we found that it is essential to use the prediction map
as the input through the evaluation results. The input of the RGB image
sometimes affects the generalization ability of uncertainty estimation if the
main task model can generate already very good prediction results. According
to the visualizations and evaluation results, we can see that the influence of
the input of the prediction map is dominant because the uncertainty map of
dual input and the one with only the prediction map input are similar. On the
other hand, the RGB image can supplement some missing semantics of the
prediction map, such as the Fig 4 FlyingChairs where RGB input can supplement
the lack of chair legs and in the Fig 5 where RGB input supplement the
uncertainty of the sky (although the sky does not have ground truth of depth,
it should have a high degree of uncertainty).

Conditions |  |  |  |  |
---|---|---|---|---|---
Input source | RGB Input | ✓ | ✗ | ✓ | ✓
Prediction map Input | ✗ | ✓ | ✓ | ✓
Loss | MSE | ✗ | ✗ | ✓ | ✗
BCE | ✓ | ✓ | ✗ | ✓
Datasets | Criteria | Ours RGBOnly | Ours PredOnly | Ours MSE | Ours BCE
FlyingChairs | AUSE-EPE | 1.82 | 1.24 | 1.41 | 1.20
AUROC | 0.944 | 0.972 | 0.967 | 0.974
KITTI | AUSE-EPE | 8.40 | 4.87 | 5.40 | 4.69
AUROC | 0.586 | 0.800 | 0.793 | 0.800
Sintel Clean | AUSE-EPE | 7.43 | 2.73 | 3.19 | 2.91
AUROC | 0.639 | 0.898 | 0.883 | 0.896
Sintel Final | AUSE-EPE | 8.24 | 2.71 | 3.11 | 2.86
AUROC | 0.575 | 0.907 | 0.889 | 0.906

Table 5: Ablation study for the OF task. Bold value: result with the best
performance. Blue value: second performance.

Conditions |  |  |  |  |
---|---|---|---|---|---
Input source | RGB Input | ✓ | ✗ | ✓ | ✓
Predinction map Input | ✗ | ✓ | ✓ | ✓
Loss | MSE | ✗ | ✗ | ✓ | ✗
BCE | ✓ | ✓ | ✗ | ✓
Dataset | Criteria | Ours RGBOnly | Ours PredOnly | Ours MSE | Ours BCE
KITTI | AUSE-RMSE | 1.84 | 1.76 | 1.74 | 1.68
AUSE-Absrel | 4.45 | 4.31 | 4.19 | 4.36
AUROC | 0.879 | 0.890 | 0.894 | 0.895
Cityscapes | AUSE-RMSE | 9.95 | 9.40 | 9.82 | 9.48
AUSE-Absrel | 10.68 | 9.23 | 10.29 | 10.90
AUROC | 0.344 | 0.446 | 0.414 | 0.400
After fine-tuning on Cityscapes
Cityscapes | AUSE-RMSE | 3.47 | 3.45 | 4.77 | 3.05
AUSE-Absrel | 6.71 | 6.47 | 6.93 | 6.55
AUROC | 0.837 | 0.844 | 0.766 | 0.849
Cityscapes Rainy s=1 | AUSE-RMSE | 3.97 | 3.43 | 4.80 | 3.39
AUSE-Absrel | 6.95 | 5.52 | 7.30 | 5.62
AUROC | 0.739 | 0.795 | 0.68 | 0.788
Cityscapes Rainy s=2 | AUSE-RMSE | 3.98 | 3.39 | 4.92 | 3.36
AUSE-Absrel | 6.68 | 5.16 | 7.09 | 5.28
AUROC | 0.747 | 0.801 | 0.689 | 0.794
Cityscapes Rainy s=3 | AUSE-RMSE | 4.11 | 3.41 | 5.07 | 3.41
AUSE-Absrel | 6.77 | 4.85 | 7.06 | 5.05
AUROC | 0.748 | 0.811 | 0.694 | 0.801
Cityscapes Foggy s=1 | AUSE-RMSE | 3.55 | 3.42 | 4.92 | 3.04
AUSE-Absrel | 6.40 | 6.15 | 6.92 | 6.25
AUROC | 0.835 | 0.841 | 0.763 | 0.847
Cityscapes Foggy s=2 | AUSE-RMSE | 3.51 | 3.39 | 5.03 | 3.01
AUSE-Absrel | 6.23 | 5.98 | 6.89 | 6.06
AUROC | 0.838 | 0.845 | 0.767 | 0.852
Cityscapes Foggy s=3 | AUSE-RMSE | 3.48 | 3.36 | 5.24 | 3.08
AUSE-Absrel | 5.97 | 5.72 | 6.75 | 5.80
AUROC | 0.845 | 0.852 | 0.773 | 0.857

Table 6: Ablation study for the MD task. Bold value: result with the best
performance. Blue value: second performance. s (_e.g_ s=1) indicates severity,
higher the s value, higher the severity. Figure 4: Uncertainty estimation
examples in ablation study for OF task. The first row of each dataset block
represents the input image pair, ground truth and predicted optical flow and
the prediction error. The prediction map and error map are made by a single
FlowNetS model as an example. The second row of each dataset block represents
the uncertainty results in using SLURP side learner with different inputs and
different loss functions. Black indicates higher uncertainty, white indicates
lower uncertainty. Figure 5: Uncertainty estimation examples in ablation study
for MD task. The first row of each dataset block represents the input image,
ground truth and predicted depth map and the prediction error. Since the
ground truth is sparse, we use interpolation to rebuild the ground truth map
just for visualization. The predicted depth map and error map are made by a
single BTS model as an example. The second row of each dataset block represent
the uncertainty results in using SLURP side learner with different inputs and
different loss functions. For uncertainty maps, black indicates higher
uncertainty, white indicates lower uncertainty. For depth maps, black
represents deeper depth, and white represents shallower depth.

##  8 Experiments

###  8.1 Evaluation protocol details

####  8.1.1 Optical flow

Let us consider an optical flow dataset
$D=\\{(\mathbf{x}_{i},\mathbf{y}_{i})\\}_{i}$, where
$\mathbf{y}_{i}\in{\mathcal{R}^{2}}$, $\mathbf{y}_{i}=(u_{i},v_{i})$ is the
ground truth optical flow for pixel $\mathbf{x}_{i}$. Below,
$\hat{\mathbf{y}_{i}}=(\hat{u}_{i},\hat{v}_{i})$ represents the optical flow
prediction.
1\. End point error (EPE): The average end point error for valid pixels in $D$
is the Euclidean distance between $\mathbf{y}_{i}$ and $\hat{\mathbf{y}_{i}}$:

$EPE=\frac{1}{|D|}\sum_{(u_{i},v_{i})\in{D}}\sqrt{(u_{i}-\hat{u_{i}})^{2}+(v_{i}-\hat{v_{i}})^{2}}$
(5)

EPE metric is used for illustrating the performance of the models we use in OF
tasks shown in Table 9 and it’s also used in AUSE and AUROC evaluation for OF
task.

####  8.1.2 Monocular depth

Let us consider a monocular depth dataset $D=\\{(\mathbf{x}_{i},d_{i})\\}_{i}$
where $d_{i}\in{\mathcal{R}^{+}}$ is the ground truth depth value for pixel
$\mathbf{x}_{i}$. Below, $\hat{{d}_{i}}$ represents the depth prediction. The
metrics we used in the evaluations are as follows:
1\. Root mean square error (RMSE):

$RMSE=\sqrt{\frac{1}{|D|}\sum_{d_{i}\in{|D|}}||\hat{d_{i}}-d_{i}||^{2}}$ (6)

2\. Absolute relative error (Absrel):

$Absrel=\frac{1}{|D|}\sum_{d_{i}\in{D}}|\hat{d_{i}}-d_{i}|/d_{i}$ (7)

3\. Threshold dk: Inlier metrics as proposed in [Eigen et al.(2014)Eigen,
Puhrsch, and Fergus], $k$ in dk indicates the power of the threshold ($thr$),
we take $thr=1.25$. In this case, d1: $thr=1.25$; d2: $thr=1.25^{2}$; d3:
$thr=1.25^{3}$, and dk represents the proportion of pixels that meet the
threshold condition:

$dk=max(\frac{\hat{d_{i}}}{d_{i}},\frac{d_{i}}{\hat{d_{i}}})=\delta<thr^{k}$
(8) $dk=\frac{|A|}{|D|},\text{where }A=\left\\{\mathbf{x}_{i},\text{such that
}\delta_{i}=\max(\frac{\hat{d_{i}}}{d_{i}},\frac{d_{i}}{\hat{d_{i}}})\text{
and }\delta_{i}<thr^{k}\right\\}$ (9)

4\. Squared relative difference (SqRel):

$SqRel=\frac{1}{|D|}\sum_{d_{i}\in{D}}{||\hat{d_{i}}-d_{i}||^{2}/d_{i}}$ (10)

5\. Root mean square log error (RMSElog):

$RMSElog=\sqrt{\frac{1}{|D|}\sum_{d_{i}\in{|D|}}{||\log\hat{d_{i}}-\log
d_{i}||^{2}}}$ (11)

6\. Average log10 error (log10):

$log10=\frac{1}{|D|}\sum_{d_{i}\in{|D|}}{|\log_{10}\hat{d_{i}}-\log_{10}d_{i}|}$
(12)

These six metrics measure the performance of the MD models we use, and are
shown in Table 7. At the same time, the first three metrics are also applied
for AUSE and AUROC evaluations. Additionally, for AUROC, we choose $k=1$ for
threshold dk metric.

####  8.1.3 Sparsification plot settings

For both MD and OF tasks, the area under the sparsification error curve (AUSE)
made by the sparsification curve is computed image-wise and not dataset-wise
in our evaluations because of the high memory consumption which would be
required for sorting vaues across the entire dataset.
Image-wise: We calculate the standardized AUSE for every image in the test set
by ranking its pixels according to the corresponding predicted uncertainty and
true error, then calculate the average AUSE after traversing the entire test
set.
Dataset-wise: By collecting all pixels of all images in the test dataset, we
calculate the AUSE by sorting their predicted uncertainty and true error.

###  8.2 Monocular depth estimation task supplement

####  8.2.1 Model precision

Table 7 shows the main task model performance for different uncertainty
estimation approaches [Lakshminarayanan et al.(2016)Lakshminarayanan, Pritzel,
and Blundell, Kendall and Gal(2017), Gal and Ghahramani(2016)]. We have
noticed that after the model is trained on KITTI, it cannot obtain reasonable
accuracy on Cityscapes. This is because the ground truth of KITTI dataset is
sparse, and only the lower half of the content is present. At the same time,
the scene of Cityscapes is more complicated. Therefore, we fine-tune all
models on Cityscapes to obtain reasonable accuracy.

Models | Datasets | higher is better | lower is better
---|---|---|---
d1 | d2 | d3 | Abs Rel | Sq Rel | RMSE | RMSE log | log10
MC | KITTI | 0.945 | 0.992 | 0.998 | 0.072 | 0.287 | 2.902 | 0.107 | 0.031
Cityscapes | 0.103 | 0.255 | 0.453 | 1.051 | 18.942 | 18.986 | 0.842 | 0.324
EE (DE) | KITTI | 0.957 | 0.993 | 0.999 | 0.059 | 0.233 | 2.688 | 0.093 | 0.026
Cityscapes | 0.214 | 0.430 | 0.560 | 0.837 | 14.459 | 18.441 | 0.845 | 0.298
Ours | KITTI | 0.955 | 0.993 | 0.998 | 0.060 | 0.249 | 2.798 | 0.096 | 0.027
(Single-PU, Original [Lee et al.(2019)Lee, Han, Ko, and Suh]) | Cityscapes | 0.183 | 0.386 | 0.519 | 0.963 | 17.230 | 18.948 | 0.896 | 0.321
After fine-tuning on Cityscapes
MC | Cityscapes | 0.882 | 0.974 | 0.992 | 0.117 | 0.917 | 5.625 | 0.169 | 0.049
EE (DE) | Cityscapes | 0.920 | 0.983 | 0.995 | 0.098 | 0.635 | 4.889 | 0.149 | 0.043
Ours (Single-PU) | Cityscapes | 0.906 | 0.980 | 0.993 | 0.104 | 0.711 | 5.216 | 0.159 | 0.046

Table 7: The performance on KITTI and the performance on Cityscapes dataset
before and after fine-tuning the models. Before fine-tuning: Training set:
KITTI Eigen-split training set, Test set: KITTI Eigen-split test set and
Cityscapes test set; After fine-tuning: Fine-tuning training set: Cityscapes
training set, Test set: Cityscapes test set. The three main task models used
in EE are also used in DE and Single-PU also shares the same main task model
with our approach.

####  8.2.2 Training settings

In the MD task, we choose to use a sequential training strategy for single
predicted uncertainty (Single-PU) [Kendall and Gal(2017)], deep ensembles (DE)
[Lakshminarayanan et al.(2016)Lakshminarayanan, Pritzel, and Blundell] and our
SLURP side learner. In other words, we first complete the training of the main
task models (BTS [Lee et al.(2019)Lee, Han, Ko, and Suh]) and then train
different uncertainty predictors according to the settings. Specifically, for
Single-PU, we use an identical BTS model to estimate the uncertainty with
using the output of the main task its corresponding main task model in the
loss. For DE, it is a mixture of multiple Single-PUs, so we just repeat the
previous procedures. Because of the ensemble property of empirical ensemble
(EE) and DE, EE and DE can share the same main task predictors. In the same
sense, the main task predictor of our side learner is chosen from one of
EE(DE)’s main task predictors, which is the same one as the main task model of
Single-PU. This method can ensure that the prediction accuracy of the main
task will not be affected by the training of the uncertainty predictor. The
ConfidNet [Corbière et al.(2019)Corbière, Thome, Bar-Hen, Cord, and Pérez]
(Confid) implementation for BTS references its implementation on SegNet. The
detailed operations are consistent with the descriptions in the main paper.
We build our side learner according to SLURP solution in the main paper (also
shown in Fig 2 in the main paper) for BTS and here are some supplements. We
directly use the frozen RGB feature maps from the encoder of main task BTS
model. To convert 1-channel predicted depth map to 3-channel input, we expand
it three times. The detailed settings for different uncertainty estimation
models are listed in Table 8, all main task models are trained identically
according to the original BTS [Lee et al.(2019)Lee, Han, Ko, and Suh] model
training settings.

Hyper-parameters | MC | EE | DE (Single-PU, Confid) | Ours
---|---|---|---|---
| learning rate for
---
main task model (Training on KITTI)
1e-4 | 1e-4 | / | /
number of training epoch (Training on KITTI) | 50 | 50 | 50 | 8
learning rate forside learner (Training on KITTI) | / | / | / | 1e-4
| learning rate for
---
main task model (Fine-tuning on Cityscapes)
5e-5 | 5e-5 | / | /
number of training epoch (Fine-tuning on Cityscapes) | 30 | 30 | 30 | 16
| learning rate for
---
side learner (Fine-tuning on Cityscapes)
/ | / | / | 8e-5
| learning rate for identical uncertainty estimator
---
/ | / | 5e-5 | /
| learning rate for side learner
---
/ | / | / | 1e-4
| batch size
---
4
number of training epoch | 50 | 50 | 50 | 8
| weight decay for
---
main task model
1e-2 | 1e-2 | / | /
| weight decay for
---
identical uncertainty estimator
/ | / | 1e-2 | /
| weight decay for
---
side learner feature extractor
/ | / | / | 1e-3
| weight decay for
---
side learner uncertainty generation blocks
/ | / | / | 4e-4
Model structure and other settings
| encoder backbone for main task model,
---
identical uncertainty estimator and
side learner feature extractor
Densenet 161 [Huang et al.(2017)Huang, Liu, Van Der Maaten, and Weinberger]
loss |  same as BTS [Huang et al.(2017)Huang, Liu, Van Der Maaten, and Weinberger] |  same as BTS [Huang et al.(2017)Huang, Liu, Van Der Maaten, and Weinberger] | Laplacian NLL |  | BCE
---
$\lambda=0.0125$
number of latent stages $n$ | / | / | / | 5
number of latent stage output channel $c$ | / | / | / | 1
number of final uncertainty output channel $C_{out}$ | / | / | / | 1
dropout rate $p_{d}$ | 0.4 | / | / | /
ensemble size $M$ | 1 | 3 | 3 (1) | 1
| during inference time
---
number of forward propagation
8 | 3 | 3 (1) | 1

Table 8: MD model settings for MC, EE, DE, Single-PU, Confid and Ours.

###  8.3 Optical flow supplement

####  8.3.1 Model precision

Table 9 shows the main task precision for different uncertainty estimation
strategies. Our SLURP side learner picks one of the models from EE as our main
task predictor. The main task models are trained only on FlyingChairs training
set with official split and the KITTI dataset we choose for main task
precision evaluation and also uncertainty estimation/evaluation is KITTI 2015
with occlusions. In the original FlowNetS paper [Dosovitskiy et
al.(2015)Dosovitskiy, Fischer, Ilg, Hausser, Hazirbas, Golkov, Van Der Smagt,
Cremers, and Brox], the precision evaluation is based on KITTI 2012 [Geiger et
al.(2012)Geiger, Lenz, and Urtasun].

Datasets | EE | MC | Single-PU | DE | Ours |  Orginal [Dosovitskiy et al.(2015)Dosovitskiy, Fischer, Ilg, Hausser, Hazirbas, Golkov, Van Der Smagt, Cremers, and Brox]
---|---|---|---|---|---|---
FlyingChairs test | 1.79 | 3.71 | 2.04 | 1.93 | 1.96 | 2.71
KITTI 2015 occ | 18.36 | 16.53 | 21.21 | 20.78 | 19.39 | /
KITTI 2012 noc | 6.77 | 19.02 | 8.34 | 8.40 | 7.65 | 8.26
Sintel clean train | 5.10 | 6.31 | 5.12 | 5.00 | 5.20 | 4.50
Sintel final train | 6.50 | 6.97 | 6.53 | 6.41 | 6.62 | 5.45

Table 9: The main task accuracy for the uncertainty estimators in OF task. The
values present the end-point error (EPE). Training set: FlyingChairs training
set [Dosovitskiy et al.(2015)Dosovitskiy, Fischer, Ilg, Hausser, Hazirbas,
Golkov, Van Der Smagt, Cremers, and Brox], Test set: FlyingChairs test set,
KITTI 2012 noc [Geiger et al.(2012)Geiger, Lenz, and Urtasun] which was used
in the original FlowNetS paper, KITTI 2015 occ [Geiger et al.(2013)Geiger,
Lenz, Stiller, and Urtasun, Menze et al.(2015)Menze, Heipke, and Geiger, Menze
et al.(2018)Menze, Heipke, and Geiger] and Sintel full training set [Butler et
al.(2012)Butler, Wulff, Stanley, and Black]

.

####  8.3.2 Training settings

In the optical flow task, for EE, we directly train multiple main task
prediction models FlowNetS [Dosovitskiy et al.(2015)Dosovitskiy, Fischer, Ilg,
Hausser, Hazirbas, Golkov, Van Der Smagt, Cremers, and Brox], and our side
learner selects one of the models as our main task predictor. For Single-PU,
because FlowNetS is relatively simple, we directly modify the original model
to output two values for each pixel, one representing the predicted value of
the main task and the other the uncertainty value. For DE, we train multiple
Single-PU models. For multi-hypothesis prediction network (MHP) [Ilg et
al.(2018)Ilg, Cicek, Galesso, Klein, Makansi, Hutter, and Brox], we modified
FlowNetS so that it can output eight (number of hypothesis) pairs of main task
- uncertainty results. Futhermore, we use another FlowNetS as the MergeNet. It
should be noted that, the authors did not mention the structural information
about MergeNet in the paper. We choose FlowNetS based on the use of model
stacking in the article. We train MHP followed by the two-stage training
schedule provided in the supplementary of this paper.
For our SLURP side learner, since the encoder in FlowNetS is designed for
capturing the object movement for two images and the total uncertainty will
reflect only the semantics from the first image, we use two DenseNet161
backbones [Huang et al.(2017)Huang, Liu, Van Der Maaten, and Weinberger] as
RGB and prediction map encoders respectively. We also used two DenseNet121
backbones for the lighter version. In order to transfer the 2-channel flow
prediction to a 3-channel input, we just add one convolution layer to expend
the channel number before the RGB feature extractor. All uncertainty model
training settings are shown in Table 10.

Hyper-parameters | MC | EE | DE (Single-PU, Confid) | Ours
---|---|---|---|---
learning rate for (modified) main task model | 1e-4 | 1e-4 | 1e-4 | /
learning rate for side learner | / | / | / | 1e-4
batch size | 8
number of training epoch | 216 | 216 | 216 | 30
weight decay for (modified) main task model | 4e-4 | 4e-4 | 4e-4 | /
weight decay for side learner feature extractors | / | / | / | 1e-4
weight decay for side learner uncertainty generation blocks | / | / | / | 4e-4
Model structure and other settings
loss |  | same as
---
FlowNetS [Dosovitskiy et al.(2015)Dosovitskiy, Fischer, Ilg, Hausser,
Hazirbas, Golkov, Van Der Smagt, Cremers, and Brox]
| same as
---
FlowNetS [Dosovitskiy et al.(2015)Dosovitskiy, Fischer, Ilg, Hausser,
Hazirbas, Golkov, Van Der Smagt, Cremers, and Brox]
Laplacian NLL |  | BCE
---
$\lambda=0.05$
number of latent stages $n$ | / | / | / | 5
number of latent stage output channel $c$ | / | / | / | 2
number of final uncertainty output channel $C_{out}$ | / | / | / | 1
dropout rate $p_{d}$ | 0.4 | / | / | /
ensemble size $M$ | 1 | 3 | 3 (1) | 1
during inference time number of forward propagation | 8 | 3 | 3 (1) | 1

Table 10: OF model settings for MC, EE, DE, Single-PU, Confid, MHP and Ours.
MHP training setting is followed by the same schedule provided by is original
paper.

###  8.4 Synthetic 1D regression task supplement

Because of the simplicity of the data, the main task predictor we use is a
neural network composed of one hidden layer and 3000 neurons. In SLURP joint-
training, we train our main task model and side learner at the same time
without freezing any layers, and in SLURP sequential-training, we train our
side learner while freezing the main task and using the latent values of it.
For the side learner, following general SLURP solution, we use the same hidden
layer as the prediction result feature extractor and three hidden layers with
128, 64, 16 neurons respectively as the context block in the uncertainty
generation block, since we have only one stage, there is no fusion block in
the end. The training details for all uncertainty estimation approaches are
listed in Table 11.
We can give an insight that SLURP strategy can also work on 1D-regression
tasks. In addition, the structure of the SLURP side learner is variable, and
other uncertainty estimation methods are limited to the structure of the main
task model, we are able to get better uncertainty results.

Hyper-parameters | MC | EE | DE (Single-PU) |  | SLURP
---
joint-training
| SLURP
---
sequential-training
number of main task latent features | 3000
learning rate for main task model | 1e-1 | 1e-1 | 1e-2 | 1e-1 | /
| learning rate for
---
side learner feature extractor
/ | / | / | 1e-1 | 1e-4
| learning rate for
---
side learner uncertainty generation blocks
/ | / | / | 1e-4 | 1e-4
batch size | 50
number of training epoch | 50
| weight decay for
---
main task model
1e-2 | 1e-2 | 1e-2 | 1e-2 | /
| weight decay for
---
side learner feature extractor
/ | / | / | 1e-2 | 1e-3
| weight decay for
---
side learner uncertainty generation blocks
/ | / | / | 1e-3 | 1e-2
Model structure and other settings
loss | MSE | MSE | Gaussian NLL | Gaussian NLL | MSE
dropout rate | 0.4 | / | / | / | /
ensemble size M | 1 | 3 | 3 (1) | 1 | 1

Table 11: 1D regression task model settings.

##  9 More visualization results

Figure 6: Uncertainty estimation results for MD task. The ground truth maps
are rebuilt by interpolation just for visualization. The depth prediction map
and the error map are generated by a single BTS model as an example. MC-
Dropout uncertainty maps are obtained by eight forward propagation, Deep
ensembles and Empirical ensembles uncertainty maps are obtained from three
models ensembles. For uncertainty maps, black indicates higher uncertainty,
white indicates lower uncertainty. For depth maps, black represents deeper
depth, and white represents shallower depth. Figure 7: Uncertainty estimation
results for OF task. The optical flow prediction map and the error map are
generated by a single FlowNetS model as an example. MC-Dropout uncertainty
maps are obtained by eight forward propagation, Deep ensembles and Empirical
ensembles uncertainty maps are obtained from three models ensembles. For
uncertainty maps, black indicates higher uncertainty, white indicates lower
uncertainty.