11institutetext: Continental AG, Germany
11email: {moussa.kassem.sbeyti, michelle.karg,
christian.2.wirth<EMAIL_ADDRESS>22institutetext:
DAI-Labor, Technische Universität Berlin, Germany
22email<EMAIL_ADDRESS>
# Overcoming the Limitations of Localization Uncertainty: Efficient & Exact
Non-Linear Post-Processing and Calibration

Moussa Kassem Sbeyti(✉) 1122 Michelle Karg 11 Christian Wirth 11
Azarm Nowzad 11 Sahin Albayrak 22

###### Abstract

Robustly and accurately localizing objects in real-world environments can be
challenging due to noisy data, hardware limitations, and the inherent
randomness of physical systems. To account for these factors, existing works
estimate the aleatoric uncertainty of object detectors by modeling their
localization output as a Gaussian distribution
$\mathcal{N}(\mu,\,\sigma^{2})\,$, and training with loss attenuation. We
identify three aspects that are unaddressed in the state of the art, but
warrant further exploration: (1) the efficient and mathematically sound
propagation of $\mathcal{N}(\mu,\,\sigma^{2})\,$ through non-linear post-
processing, (2) the calibration of the predicted uncertainty, and (3) its
interpretation. We overcome these limitations by: (1) implementing loss
attenuation in EfficientDet, and proposing two deterministic methods for the
exact and fast propagation of the output distribution, (2) demonstrating on
the KITTI and BDD100K datasets that the predicted uncertainty is
miscalibrated, and adapting two calibration methods to the localization task,
and (3) investigating the correlation between aleatoric uncertainty and task-
relevant error sources. Our contributions are: (1) up to five times faster
propagation while increasing localization performance by up to 1%, (2) up to
fifteen times smaller expected calibration error, and (3) the predicted
uncertainty is found to correlate with occlusion, object distance, detection
accuracy, and image quality.

###### Keywords:

Aleatoric Localization Uncertainty Object Detection Loss Attenuation
Uncertainty Calibration.

##  1 Introduction

Object detectors in safety-critical systems face multiple challenges,
including limited sensor resolution, difficult weather conditions, and
ambiguous situations [20, 8, 16]. These challenges decrease performance
regardless of the training frequency, as they induce an inevitable uncertainty
called aleatoric uncertainty [17]. Therefore, existing works explicitly
integrated aleatoric uncertainty into object detectors via loss attenuation
[17] for varying applications, such as enhancing safety, robustness, and
performance [5, 20, 24]. This paper prioritizes localization due to the
absence of confidence information from the localization head in object
detectors, when compared to the scores provided by the classification head.

EfficientDet [31], a one-stage anchor-based detector, demonstrates state-of-
the-art performance in terms of both accuracy and speed on various benchmark
datasets, making it an ideal use-case for this paper. An anchor-based detector
predicts anchor-relative offsets, which are subjected to non-linear
transformations during post-processing to compute the final object
coordinates. These offsets are modeled as distributions to account for
uncertainty, which raises the crucial question: How is the output
distribution, including the uncertainty, propagated through non-linear
functions? Le et al. [24] is the only work known to us, that considers the
propagation of the anchor-relative offsets through non-linearities, and
addresses it via sampling from the estimated distribution. However, sampling
has the downside of either a high computation time for a large sample size or
a reduced accuracy for a small sample size. We therefore develop two novel,
fast and exact approaches. The first method is based on normalizing flows,
with the main advantage of a universal applicability to many non-linear,
arbitrarily complex functions and output distributions. The second method is
tailored towards a normal output distribution $\mathcal{N}(\mu,,\sigma^{2}),$
transformed by an exponential function. It utilizes the properties of the log-
normal distribution, and its main advantage is an efficient usage of
computational resources.

Once the uncertainty is propagated, the focus shifts to assessing its quality:
Is the predicted localization uncertainty well-calibrated? Other research on
localization uncertainty estimation in object detection typically overlooks
its calibration [16, 24, 15, 5, 20]. Hence, we introduce different approaches
to calibrate it, inspired by calibration for general classification and
regression tasks. We select two established methods: calibrating via an
auxiliary model, e.g. isotonic regression [3, 33, 21], and factor scaling [14,
22]. We extend the first method to coordinate- and class-specific calibration.
For the second calibration method, we establish and evaluate various loss
functions during the optimization phase of the scaling factor, which directly
adjusts the predicted uncertainty by considering its proximity to the
residuals. Both methods are further improved by incorporating the object size,
where each object’s uncertainty is normalized by its width and height,
resulting in a balanced calibration of objects of all sizes and aspect ratios.
Furthermore, we provide a data selection process for calibration, which
allocates all predictions to their ground-truth based on proximity, in
contrast to, e.g. thresholding detections based on the classification score.

After the localization uncertainty is estimated, propagated and calibrated,
its interpretability is required to define potential applications (see Fig.
1): What correlations exist between the data and the uncertainty? Related
works discover that aleatoric uncertainty correlates with occlusion [8, 10, 5,
20] and object distance due to sparsity in point clouds [8], but not with
detection accuracy [8, 10]. We investigate the latter and discover the
contrary. We verify and show to which extent aleatoric uncertainty correlates
with occlusion and detection performance, and extend the analysis to the
object area in an image, i.e. object distance, and the quality of the image
cropped around each detection.

In summary, the contributions of our work are:

  * •

Development of two novel, exact and fast methods for uncertainty propagation
through non-linear functions, enabling accurate uncertainty estimation without
additional drawbacks.

  * •

Development and extension of two calibration methods and a data selection
approach for accurate calibration in the context of object localization.

  * •

A comprehensive experimental overview of the quality and correlation between
aleatoric uncertainty and traceable metrics, which further advances the
understanding of aleatoric uncertainty.

Figure 1: Aleatoric localization uncertainty $\sigma$ per object normalized by
its width and height (right), and an example crop with $\mu\pm 10\sigma$ per
coordinate (left). Uncertainty correlates with occlusion, distance and
detection performance. Calibration reduces the uncertainty. Per-coordinate
calibration strengthens the correlation with localization accuracy. Per-class
calibration shifts the uncertainty towards classes with lower detection
accuracy. Relative calibration accounts for the impact of the different object
areas and aspect ratios.

##  2 Background and Related Work

This section presents a concise overview of existing works on aleatoric
uncertainty estimation, decoding in object detectors, and calibration for
regression.

Loss Attenuation. A widely adopted approach for estimating aleatoric
uncertainty is the sampling-free loss attenuation, which assumes that
observation noise is dependent on the input [17]. By extending the network
output to include both the mean $\mu$ and variance $\sigma^{2}$, i.e. modeling
it as a Gaussian distribution, and training the network on the negative log-
likelihood (NLL), the uncertainty can be learned as a function of the data.
Choi et al. [5], Kraus and Dietmayer [20] and Feng, Rosenbaum and Dietmayer
[8, 10] show that loss attenuation enhances the performance of 2D and 3D
object detectors. They find that the estimated uncertainty correlates with
occlusion [20, 8, 10] and object distance based on LiDAR data [8, 10], but it
does not correlate with detection accuracy, measured via the intersection over
union (IoU) [8, 10]. The focus of these works is primarily performance
enhancement of object detectors, as they place less emphasis on the
reliability and interpretability of the uncertainty estimates.

Anchor-Relative Localization. Choi et al. [5] and Kraus and Dietmayer [20]
implement loss attenuation in YOLOv3 [30]. Anchor-based object detectors such
as YOLOv3 [30], single-shot detector (SSD) [26], and EfficientDet [31] divide
their final feature maps into a grid. Whereby each grid cell contains a pre-
defined set of static bounding boxes known as anchors. During training, the
detector learns the offsets for the center, width and height between the pre-
defined anchors and the ground truth. In the post-processing, the predicted
offsets are decoded based on their corresponding anchors, usually via non-
linear functions, such as exponential and sigmoid [31, 26, 30, 29]. This
transforms them into bounding box coordinates, which are then scaled to the
original image size. As introduced in Sec. 1, Le et al. [24] is the only work
that considers the non-linearity in the decoding process. They implement loss
attenuation in SSD [26]. To decode the anchor-relative coordinates along their
corresponding variances, they draw samples from the predicted multivariate
normal distribution $\mathcal{N}(\mu,\,\sigma^{2})\,$, decode the samples,
then calculate the mean and variance of the decoded values. Other works do not
explicitly address the non-linearity in the decoding process, i.a. decode the
predicted variance by reversing the encoding equation of the mean [5, 20, 16,
15]. Therefore, there is currently no deterministic and exact method available
for decoding the values of both $\mu$ and $\sigma^{2}$.

Regression Uncertainty Calibration. Calibration is crucial after estimating
and propagating the uncertainty. Approximate Bayesian approaches such as loss
attenuation produce miscalibrated uncertainties [7]. Laves et al. [22] and
Feng et al. [9] argue that minimizing the NLL should result in the estimation
of $\sigma^{2}$ matching the squared error. However, they and Phan et al. [28]
find that the prediction of $\sigma^{2}$ is in reality biased, since it is
predicted relative to the estimated mean. Kuleshov, Fenner and Ermon [21]
propose a calibration method, which is guaranteed to calibrate the regression
uncertainty given sufficient data. Calibration via an (1) auxiliary model
implies training a model, e.g. isotonic regression, on top of a network so
that its predicted distribution is calibrated. Its main disadvantage is that
it is not suitable for fitting heavy-tailed distributions, and is prone to
over-fitting [4]. Laves et al. [23, 22] propose (2) factor scaling, another
approach which consists of scaling the predicted uncertainty using a single
scalar value $s$. The latter is optimized using gradient descent with respect
to the NLL on the validation dataset. Method (2) is more suitable for embedded
applications and requires less data than (1), but it has less calibration
potential since one value is equally applied to all the uncertainties. Phan et
al. [28] adapt method (1) for the localization of single objects, and show
that it results in more reliable uncertainty estimates. Part of their future
work and Kraus et al.’s [20] is to extend it to multiple-object detection; as
addressed in this work.

##  3 Method

This section presents our approach to loss attenuation in EfficientDet [31]
and outlines its decoding process. Furthermore, it introduces our uncertainty
propagation methods, and explains our extensions for uncertainty calibration
in localization tasks. The proposed methods are model agnostic, i.e. they are
identically applicable to any other object detector.

###  3.1 Uncertainty Estimation

The loss attenuation introduced by Kendall and Gal [17] is defined as follows:

$\mathcal{L}_{\mathrm{NN}}=\frac{1}{2N}\sum_{i=1}^{N}\frac{\|\mathbf{y}_{i}^{*}-\mathbf{f}(\mathbf{x}_{i})\|^{2}}{\sigma(\mathbf{x}_{i})^{2}}+\log\sigma(\mathbf{x}_{i})^{2}$
(1)

with $N$ samples, ground truth $\mathbf{y}^{*}$, variance
$\sigma(\mathbf{x})^{2}$ and output $\mathbf{f}(\mathbf{x})$ for input
$\mathbf{x}$.

The output of the localization head in anchor-based object detectors consists
of four variables: the anchor-relative object center coordinates
($\mathbf{\hat{x}},\mathbf{\hat{y}}$), width $\mathbf{\hat{w}}$, and height
$\mathbf{\hat{h}}$. For the estimation of the aleatoric uncertainty, the four
variables are modeled via a multivariate Gaussian distribution
$\mathcal{N}(\mu,\,\sigma^{2})$ with a diagonal covariance approximation.
Hence, we extend Eq. 1 for object detection:

$\mathcal{L}_{\mathrm{NN}}=\frac{1}{8N_{pos}}\sum_{i=1}^{N}\sum_{j=1}^{4}(\frac{\|y^{*}_{ij}-\hat{\mu_{j}}(\mathbf{x}_{i})\|^{2}}{\hat{\sigma_{j}}(\mathbf{x}_{i})^{2}}\\\
+\log\hat{\sigma_{j}}(\mathbf{x}_{i})^{2})\odot m_{i}$ (2)

with $N_{pos}$ as the number of anchors with assigned ground truth in each
batch of input images, and the mask $\mathbf{m}$ consisting of foreground
ground truth boxes $\mathbf{m}=[\mathbf{y}^{*}\neq 0]$. These features are
specific for the EfficientDet baseline loss.

###  3.2 Uncertainty Propagation

The default decoding process of the localization output in EfficientDet is
similar to other anchor-based object detectors such as SSD [26] and YOLO [29].
The final coordinates ($\mathbf{y},\mathbf{x},\mathbf{h}$ and $\mathbf{w}$)
are computed via two post-processing steps. The first step consists of
transforming the anchor-relative center coordinates
$\mathbf{\hat{x}},\mathbf{\hat{y}}$, width $\mathbf{\hat{w}}$ and height
$\mathbf{\hat{h}}$ based on the center coordinates
$\mathbf{x_{a}},\mathbf{y_{a}}$, width $\mathbf{w_{a}}$, and height
$\mathbf{h_{a}}$ of the corresponding anchor:

$\displaystyle\mathbf{y}$
$\displaystyle=\mathbf{\hat{y}}\odot\mathbf{h_{a}}+\mathbf{y_{a}}\quad$
$\displaystyle\mathbf{h}$
$\displaystyle=\exp(\mathbf{\hat{h}})\odot\mathbf{h_{a}}$ (3)
$\displaystyle\mathbf{x}$
$\displaystyle=\mathbf{\hat{x}}\odot\mathbf{w_{a}}+\mathbf{x_{a}}\quad$
$\displaystyle\mathbf{w}$
$\displaystyle=\exp(\mathbf{\hat{w}})\odot\mathbf{w_{a}}$

Eq. 3 is calculated for each prediction in the five feature maps, resulting in
$A_{cell}\cdot(\frac{I_{H}\cdot I_{W}}{128^{2}}+\frac{I_{H}\cdot
I_{W}}{64^{2}}+\frac{I_{H}\cdot I_{W}}{32^{2}}+\frac{I_{H}\cdot
I_{W}}{16^{2}}+\frac{I_{H}\cdot I_{W}}{8^{2}})$ iterations, with $A_{cell}$ as
the number of anchors per grid cell, $I_{H}$ as the height of the input image
and $I_{W}$ as its width. The decoding process yields coordinates that are
relative to the scaled input image rather than the corresponding anchors. As a
result, the second step consists of linearly rescaling the decoded coordinates
to the original image size.

Sampling is the only approach in existing works that enables the
transformation of a distribution via a non-linear function such as the
exponential in Eq. 3. It however either increases computation time or reduces
accuracy. We therefor present two novel, exact and fast methods for decoding,
via (1) normalizing flows and via (2) properties of the log-normal
distribution.

(1) Decoding via Normalizing Flows. As explained by Kobyzev, Prince and
Brubaker [18], a normalizing flow is a transformation of a probability
distribution via a sequence of invertible and differentiable mappings. The
density of a sample in the transformed distribution can be evaluated by
computing the original density of the inverse-transformed sample, multiplied
by the absolute values of the determinants of the Jacobians for each
transformation:

$p_{\mathbf{Y}}(\mathbf{y})=p_{\mathbf{Z}}(\mathbf{f}(\mathbf{y}))|\operatorname{det}\mathbf{Df}(\mathbf{y})|=p_{\mathbf{Z}}(\mathbf{f}(\mathbf{y}))|\operatorname{det}\mathbf{Dg}(\mathbf{f}(\mathbf{y}))|^{-1}$
(4)

where $\mathbf{Z}\in\mathbb{R^{D}}$ is a random variable with a known and
tractable probability density function
$p_{\mathbf{Z}}:\mathbb{R^{D}}\rightarrow\mathbb{R}$, $\mathbf{g}$ is an
invertible function, $\mathbf{f}$ is the inverse of $\mathbf{g}$,
$\mathbf{Y}=\mathbf{g}(\mathbf{Z})$ is a random variable,
$\mathbf{Df}(\mathbf{y})=\frac{\partial\mathbf{f}}{\partial\mathbf{y}}$ is the
Jacobian of $\mathbf{f}$ and
$\mathbf{Dg}(\mathbf{z})=\frac{\partial\mathbf{g}}{\partial\mathbf{z}}$ of
$\mathbf{g}$. The determinant of the Jacobian of $\mathbf{f}$ captures the
scaling and stretching of the space during the transformation, which ensures
that the transformed distribution has the same area as the original
distribution and is a valid probability density function that integrates to
one. In other words, the original density $p_{\mathbf{Z}}$ is pushed forward
by the function $\mathbf{g}$, while the inverse function $\mathbf{f}$ pushes
the data distribution in the opposite normalizing direction, hence the name
normalizing flow. Eq. 3 can be reformulated into four chains of
transformations on normal distributions. Let
$\mathbf{g}_{1}(\mathbf{y}),\mathbf{g}_{2}(\mathbf{y})$ be invertible
functions; the transformation of the distributions corresponding to the width
$\mathbf{\hat{w}}$ and height $\mathbf{\hat{h}}$ is written as:

$\displaystyle\mathbf{g}_{1}(\mathbf{y})=\exp(\mathbf{y})$
$\displaystyle\quad\mathbf{g}_{2}(\mathbf{y})=\mathbf{c}\odot\mathbf{y}$ (5)
$\displaystyle\mathbf{h}=\mathbf{g}_{2}\circ\mathbf{g}_{1}(\mathbf{\hat{h}})\text{
with }\mathbf{c}=\mathbf{h_{a}}$
$\displaystyle\quad\mathbf{w}=\mathbf{g}_{2}\circ\mathbf{g}_{1}(\mathbf{\hat{w}})\text{
with }\mathbf{c}=\mathbf{w_{a}}$

Each of the transformations in Eq. 5 is implemented with the help of
bijectors, which represent differentiable and injective functions. The final
coordinates and variances in the scaled image are then directly calculated
from the transformed distribution. This method can also be applied for
uncertainty propagation in other anchor-based object detectors such as YOLOv3
[30], by including a sigmoid function in the chain of transformations in Eq.
5.

Figure 2: Illustration of our uncertainty propagation methods in the decoding
process of EfficientDet[31]. The output of the localization head is adjusted
to predict an output distribution for each coordinate (in red). These
distributions are relative to the pre-defined anchor coordinates (in orange).
During post-processing, the distributions undergo non-linear transformations
to obtain coordinates that are relative to the image. Both our methods provide
a fast and exact propagation, with (1) allowing universality and (2) being
computationally efficient.

(2) Decoding via Properties of the Log-Normal Distribution. The calculation of
the Jacobi matrix and inverse functions is computationally expensive. We
therefore introduce a different method that directly calculates the
transformed mean and variance for the specific case of a normal distribution
and exponential or sigmoid transformation. If $\mathbf{Z}$ follows a normal
distribution with mean $\mu$ and variance $\sigma^{2}$, then
$\mathbf{Y}=\exp(\mathbf{Z})$ follows a log-normal distribution. The density
function, mean and standard deviation of a log-normal distribution are
calculated as follows [2]:

$\displaystyle f(y;\mu,\sigma^{2})=$
$\displaystyle\frac{1}{y\sigma\sqrt{2\pi}}\exp(\frac{-[\log(y)-\mu]^{2}}{2\sigma^{2}})$
(6) $\displaystyle Mean(\mathbf{Y})=$
$\displaystyle\exp(\mu)\sqrt{\exp(\sigma^{2})}=\exp(\mu+\frac{\sigma^{2}}{2})$
$\displaystyle SD(\mathbf{Y})=$
$\displaystyle\exp(\mu)\sqrt{\exp(\sigma^{2})(\exp(\sigma^{2})-1)}$

Combining Eq. 6 with Eq. 3 results in the transformed mean and variance for
the width and height, as shown in Fig. 2. Due to the preservation of linearity
for Gaussian distributions, Eq. 3 remains unchanged for the mean of the center
coordinates. For the variance, the equations undergo modification in
compliance with the applicable transformation rules.

Log-Normal during Training. Fig. 2 and $Mean(\mathbf{Y})$ in Eq. 6 show that a
factor $\frac{\sigma^{2}}{2}$ is added to the mean of the width and height
during the decoding. This always results in an enlargement of the bounding
boxes ($\sigma^{2}>0$, $\exp{(\sigma^{2})}>1)$. However, the model fits the
offsets during training based solely on the mean, with no regard to the
uncertainty (see Eqs. 1 and 2). We propose incorporating the same factor
during training, thereby accounting for the exponential transformation in the
decoding equations of $\mu_{\mathbf{h}}$ and $\mu_{\mathbf{w}}$. This results
in
$\|y^{*}_{ij}-[\hat{\mu_{j}}(\mathbf{x}_{i})+\frac{\hat{\sigma_{j}}(\mathbf{x}_{i})^{2}}{2}]\|^{2}$
for $j=3,4$ in Eq. 2.

###  3.3 Uncertainty Calibration

The main idea behind post-hoc calibration on the validation set is to map the
uncertainty to the residuals via a model $\mathbf{r}$ or a scaling factor $s$.

Extensions to Caliration by a Model. Since the model predicts a multivariate
Gaussian distribution with a diagonal covariance matrix (see Sec. 3.1), all
four coordinates are predicted independently. Furthermore, the performance of
the object detector varies from one class to the other due to heavy class
imbalance, potentially leading to bias towards one class during calibration
while neglecting the other. Therefore, we extend calibration via an auxiliary
model [3] from calibrating all four uncertainties simultaneously with one
isotonic regression model $\mathbf{r}$ to the calibration of the uncertainty
for each coordinate $c$ with a separate model $\mathbf{r}_{c}$ for
$c\in[1,4]$, each ground truth class $k$ with $\mathbf{r}_{k}$ for
$k\in[1,n_{classes}]$, and each coordinate $i$ plus each ground truth class
$k$ with $\mathbf{r}_{c,k}$. For an input $\mathbf{x}$, a ground truth
$\mathbf{y}$ and predicted output $\mathbf{p}=\mathbf{r}(\mathbf{x})$, an
isotonic regression model minimizes $\sum_{i=0}^{N}w_{i}(y_{i}-p_{i})^{2}$ on
$N$ predictions [3], with $\mathbf{w}\geq 0$ as the observation weight and
$p_{i}\leq p_{j}$ for all $i,j\in\mathbb{E}$, where
$\mathbb{E}=\\{(i,j):x_{i}\leq x_{j}\\}$.

Extensions to Calibration by a Factor.  Laves et al. [22] optimize the factor
$s$ by minimizing the NLL with gradient descent. However, the log-likelihood
objective is highly sensitive towards outliers and mislabeled variables, which
is particularly relevant for real-world datasets [19, 11]. Since their method
only adjusts the predicted uncertainty $\sigma$ in
$\mathcal{N}(\mu,(s\cdot\sigma)^{2})$, we propose to directly optimize the
scaling factor $s$ based on a distance metric between the predicted
uncertainty and the true intervals, similar to the isotonic regression
optimization goal. Therefore, two different loss functions are introduced, the
root-mean-square uncertainty error (RMSUE) and the mean absolute uncertainty
error (MAUE):

$\text{RMSUE}(s)=\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left(\Delta_{i}-s\cdot\sigma_{i}\right)^{2}}\quad\text{MAUE}(s)=\frac{1}{N}\sum_{i=1}^{N}\left|\Delta_{i}-s\cdot\sigma_{i}\right|$
(7)

with $N$ detections, $\mathbf{\sigma}$ as the predicted uncertainty, and
$\mathbf{\Delta}=|\mathbf{y}^{*}-\mu|$ as the residual.

Relative Uncertainty. Existing methods are not attuned for localization tasks
as they do not account for varying aspect ratios and sizes of bounding boxes.
We introduce relative calibration, which consists of calibrating $\sigma$ and
$\Delta$ after normalization with the width and height of their corresponding
object. This prevents the uncertainty of large objects from negatively
influencing the calibration of the uncertainty of smaller objects.
Contextualizing the uncertainty with respect to its object also helps mitigate
the effect of missing depth information in 2D images, which is crucial for the
comprehension of a detector’s confidence in real-world detections.

Proximity-based Data Sorting. Post-hoc calibration is performed on the
validation set. The output of non-maximum suppression (NMS) in object
detectors typically involves selecting top $n$ detections based on their
classification score using a manually specified threshold, resulting in the
exclusion of certain detections. Such exclusions, in turn, could correspond to
actual ground truths and therefore can impede the calibration of the
localization uncertainty. EfficientDet employs soft-NMS, which entails the
adjustment and subsequent sorting of its output based on the classification
score. Nevertheless, a higher score does not necessarily imply a more accurate
detection. We propose resorting the NMS output based on the nearest-neighbor
to the ground truth via a distance metric, such as mean squared error (MSE),
hence retaining and correctly allocating all samples in the validation set.

##  4 Experiments

The datasets used in this work are common in autonomous driving research:
KITTI [12] (all 7 classes, $20\%$ split for validation), and BDD100K [32] (all
10 classes, 12.5% official split). The baseline is EfficientDet-D0 [31] pre-
trained on COCO [25] and fine-tuned on the two datasets respectively for 500
epochs with 8 batches and an input image resolution of 1024x512 pixels. The
default hyperparameters for EfficientDet-D0 are maintained. To prevent the
classification results from affecting the localization output, we use ground
truth classes for the per-class calibration and reorder the detections based
on MSE (the distance measure used during training, see Eq. 2) for both
calibration and evaluation.

###  4.1 Decoding Methods

To showcase the effectiveness of the presented methods, eight metrics are
selected. For localization: Average Precision (AP), root-mean-square error
(RMSE), mean intersection over union (mIoU) and average time: model exporting
time (ET) in seconds (s) and inference time (IT) in milliseconds (ms) per
image. For uncertainty: RMSUE, expected calibration error (ECE) [21], negative
log-likelihood (NLL) and sharpness (Sharp). Sharpness is the average of the
variance, i.e. it relates to the concentration of the predictive distribution
[13]. Each model is trained three times. The results of sampling and IT are
averaged over three trials on the validation set. ET is calculated as the
average of three exporting iterations. Time measurements are performed on one
GPU (RTX 3090). We compare our normalizing flows (N-FLOW) and log-normal
(L-NORM) approaches to the baseline without uncertainty, and to the sampling
method (SAMP) with 30, 100 and 1000 samples, inspired by Le et al. [24]. We
also add false decoding (FALSEDEC), where both $\mu$ and $\sigma$ are decoded
via Eq. 3, as an ablation study to analyze the effect of correct propagation
and including the uncertainty in the decoding process of the mean (see Eq. 6).
The N-FLOW method is implemented using the library TensorFlow Probability [6].

Table 1: KITTI (top) and BDD100K (bottom): Comparison between EfficientDet-D0
baseline and model with uncertainty. Our propagation methods are faster and
more accurate than sampling. Uncertainty estimation increases localization
performance and reduces computation time.

Method | AP$\uparrow$ | RMSE$\downarrow$ | mIoU$\uparrow$ | NLL$\downarrow$ | ET$\downarrow$ | IT$\downarrow$
---|---|---|---|---|---|---
|  |  |  |  | (s) | (ms)
Baseline | 72.8 $\pm$ 0.1 | 5.07 $\pm$ 0.1 | 90.1 $\pm$ 0.1 | - | 115.6 $\pm$ 3 | 34.8 $\pm$ 4
FalseDec | 73.1 $\pm$ 0.5 | 5.27 $\pm$ 0.1 | 90.3 $\pm$ 0.1 | 4.27 $\pm$ 0.1 | 116.0 $\pm$ 3 | 31.1 $\pm$ 3
L-norm |  73.3 $\pm$ 0.5 | 5.17 $\pm$ 0.2 | 90.3 $\pm$ 0.0 | 3.22 $\pm$ 0.0 | 115.6 $\pm$ 2 | 31.0 $\pm$ 3
N-flow |  73.3 $\pm$ 0.5 | 5.17 $\pm$ 0.2 | 90.3 $\pm$ 0.0 | 3.22 $\pm$ 0.0 | 116.6 $\pm$ 1 | 31.6 $\pm$ 3
Samp30 | 68.6 $\pm$ 0.4 | 5.43 $\pm$ 0.1 | 88.7 $\pm$ 0.1 | 3.19 $\pm$ 0.0 | 118.8 $\pm$ 2 | 34.5 $\pm$ 3
Samp100 | 71.8 $\pm$ 0.5 | 5.23 $\pm$ 0.1 | 90.1 $\pm$ 0.0 | 3.20 $\pm$ 0.0 | 117.4 $\pm$ 4 | 47.0 $\pm$ 3
Samp1000 | 73.1 $\pm$ 0.5 | 5.18 $\pm$ 0.2 | 90.4 $\pm$ 0.0 | 3.21 $\pm$ 0.0 | 117.9 $\pm$ 4 | 187.4 $\pm$ 4
Baseline | 24.7 $\pm$ 0.1 | 8.96 $\pm$ 0.2 | 66.6 $\pm$ 1.6 | - | 115.7 $\pm$ 3 | 33.0 $\pm$ 4
FalseDec | 23.9 $\pm$ 0.2 | 8.81 $\pm$ 0.2 | 67.3 $\pm$ 0.0 | 4.40 $\pm$ 0.1 | 115.9 $\pm$ 2 | 30.4 $\pm$ 4
L-norm | 24.4 $\pm$ 0.1 | 8.53 $\pm$ 0.2 | 67.7 $\pm$ 0.0 | 3.69 $\pm$ 0.0 | 115.3 $\pm$ 1 | 30.6 $\pm$ 4
N-flow | 24.4 $\pm$ 0.1 | 8.53 $\pm$ 0.2 | 67.7 $\pm$ 0.0 | 3.69 $\pm$ 0.0 | 116.4 $\pm$ 1 | 31.0 $\pm$ 3
Samp30 | 21.0 $\pm$ 0.1 | 9.02 $\pm$ 0.2 | 64.7 $\pm$ 0.0 | 3.70 $\pm$ 0.0 | 118.0 $\pm$ 3 | 33.6 $\pm$ 4
Samp100 | 23.2 $\pm$ 0.1 | 8.68 $\pm$ 0.2 | 66.7 $\pm$ 0.0 | 3.69 $\pm$ 0.0 | 117.0 $\pm$ 3 | 45.4 $\pm$ 4
Samp1000 | 24.2 $\pm$ 0.1 | 8.55 $\pm$ 0.2 | 67.6 $\pm$ 0.1 | 3.69 $\pm$ 0.0 | 118.4 $\pm$ 3 | 187.3 $\pm$ 4

Baseline vs Uncertainty. Predicting the localization aleatoric uncertainty
increases the original 3,876,321 parameters by only 2,327 (0.06%). It reduces
the required inference time per image, due to the Tensor Cores in the GPU
utilizing the extension of the model output to eight values (mean and
variance) [1]. The exporting time varies by decoding function. Direct
calculation functions (Baseline, FALSEDEC, L-NORM) are faster than
distribution-based (N-FLOW, SAMPL) functions, due to lower complexity of
operations in the graph. Estimating the uncertainty improves the baseline AP
and mIoU by $0.5\%$ on KITTI. On BDD100K, it reduces the AP by $0.3\%$, but
improves both the mIoU and RMSE, as seen in Tab. 1. Therefore, on both
datasets, the localization performance increases. The COCO-style AP is
affected by the classification performance, since it is calculated per class
and detections are sorted based on their classification score to determine the
cumulative true and false positives. This is amplified in the case of BDD100K,
due to the larger number of images and their lower fidelity, and by extension,
the overall decrease in performance and higher misclassification rate (see
Fig. 4) in comparison to KITTI.

Our Methods vs Sampling. The only difference between the N-FLOW and the L-NORM
approaches is the processing time, due to different mathematical complexity
(see Sec. 3.2). The main advantage of the N-FLOW approach is the flexibility
in changing the distribution or the transformations without manually
recalculating the posterior distribution. The latter is especially beneficial,
when the transformations render the posterior distribution intractable. Tab. 1
shows that incorrectly propagating the mean and variance (FALSEDEC) reduces
performance and the precision of the uncertainty. Compared to our methods,
sampling shows on both datasets either a strong reduction in performance (up
to 3% AP and mIoU) or a longer inference time per image (up to 5 times
slower). However, sampling with 30 samples does offer slightly sharper
uncertainties on KITTI, which results in a lower NLL. The opposite is true for
BDD100K. This can be retraced to the overestimation of the uncertainty by the
model. Therefore, any reduction in the uncertainty leads to an enhancement of
its precision. Sampling with a mere 30 samples can result in substantial
deviation in both directions, hence the fluctuation between the datasets.
Based on the results in Tab. 1, we select the L-NORM decoding method for the
calibration evaluation.

###  4.2 Calibration Evaluation

Calibration improves reliability and interpretability of predicted
uncertainties by reducing misalignment between the error distribution and the
standard Gaussian distribution. This is highly relevant for safety-critical
applications, where uncertainty should reflect the true outcome likelihood.

Uncertainty Behavior. We notice that EfficientDet predicts a lower $\sigma$ on
the validation set, despite the higher NLL and RMSE compared to the training
set, in accordance with Laves et al. [22]. We also found that $\sigma^{2}$ is
predicted higher than the MSE, hence being miscalibrated. Reasons therefor can
be found in the optimization of multiple losses and in uneven data
distribution. For both datasets, the model overestimates the uncertainty, with
the interval $\mu\pm\sigma$ containing $99\%$ of the true values instead of
the expected $68.27\%$.

Calibration Methods. For factor scaling (FS), gradient descent is applied for
100 optimization epochs with a learning rate of 0.1 on the validation dataset.
Optimizing the factor $s$ based on MAUE and RMSUE (see Sec. 3.3 and Eq. 7)
results in a lower ECE and sharper uncertainties, but a higher NLL (see Tab.
2). We discover a trade-off between the ECE and NLL, since optimizing $s$
based on the NLL instead results in a higher ECE. For the auxiliary isotonic
regression (IR) model, we compare its extensions to per-coordinate (PCo) and
per-class (CL) calibration. An illustrative example is featured in Fig. 1.
Tab. 2 shows that per-coordinate calibration outperforms the calibration on
all coordinates as expected, since all four normal distributions are assumed
to be independent. Per-class calibration further reduces the ECE, RMSUE and
NLL, since both datasets contain heavily unbalanced classes with different
aspect ratios and localization accuracy. IR outperforms FS for both datasets,
because the size of the calibration dataset is large enough for the auxiliary
model to train on, as also observed by Feng et al. [9]. Relative calibration
results in further improvement for IR in both NLL and ECE. Our hypothesis in
Sec. 3.3 is that relative calibration mitigates bias towards larger objects.
We empirically demonstrate that it effectively achieves this objective by
conducting a comparative analysis on small, medium and large objects based on
their area as defined by the COCO API [25]. Our findings reveal that relative
calibration causes a more substantial reduction in ECE on small objects with a
6-fold further decrease when compared to absolute calibration, whereas it is
2-fold on medium objects and 3-fold on large objects. Accordingly, relative
isotonic regression per-coordinate and per-class (Rel. IR PCo CL) is selected
for further investigations.

Table 2: KITTI (left) and BDD100K (right): Comparison between different
calibration methods. Our factor scaling (FS) losses outperform NLL. Isotonic
regression (IR) per-coordinate (PCo) and per-class (CL) outperforms classic
one-model IR and all FS approaches. Relativity increases calibration
performance.

Method | RMSUE$\downarrow$ | ECE$\downarrow$ | NLL$\downarrow$ | Sharp$\downarrow$
---|---|---|---|---
Uncalibrated | 13.0 $\pm$ 0.0 | 0.384 $\pm$ 0.000 | 3.22 $\pm$ 0.0 | 14.9 $\pm$ 0.0
FS MAUE | 4.6 $\pm$ 0.2 | 0.047 $\pm$ 0.001 | 3.14 $\pm$ 0.4 | 2.5 $\pm$ 0.0
FS RMSUE | 4.6 $\pm$ 0.2 | 0.088 $\pm$ 0.003 | 2.79 $\pm$ 0.2 | 3.0 $\pm$ 0.0
FS NLL | 5.0 $\pm$ 0.3 | 0.194 $\pm$ 0.021 | 2.51 $\pm$ 0.1 | 4.7 $\pm$ 0.5
Rel. FS RMSUE | 7.2 $\pm$ 0.1 | 0.306 $\pm$ 0.002 | 2.74 $\pm$ 0.0 | 8.3 $\pm$ 0.1
Abs. IR | 4.5 $\pm$ 0.2 | 0.032 $\pm$ 0.001 | 3.15 $\pm$ 0.3 | 2.5 $\pm$ 0.0
Abs. IR CL | 4.4 $\pm$ 0.2 | 0.029 $\pm$ 0.001 | 2.86 $\pm$ 0.2 | 2.7 $\pm$ 0.0
Abs. IR PCo | 4.5 $\pm$ 0.2 | 0.032 $\pm$ 0.001 | 3.03 $\pm$ 0.2 | 2.6 $\pm$ 0.0
Abs. IR PCo CL | 4.3 $\pm$ 0.2 | 0.028 $\pm$ 0.000 | 2.70 $\pm$ 0.1 | 2.9 $\pm$ 0.0
Rel. IR | 4.5 $\pm$ 0.2 | 0.027 $\pm$ 0.001 | 3.06 $\pm$ 0.3 | 2.5 $\pm$ 0.0
Rel. IR CL | 4.4 $\pm$ 0.2 | 0.026 $\pm$ 0.001 | 2.78 $\pm$ 0.2 | 3.1 $\pm$ 0.4
Rel. IR PCo | 4.5 $\pm$ 0.2 | 0.027 $\pm$ 0.001 | 3.03 $\pm$ 0.2 | 2.5 $\pm$ 0.1
Rel. IR PCo CL | 4.4 $\pm$ 0.3 | 0.025 $\pm$ 0.000 | 2.69 $\pm$ 0.2 | 3.2 $\pm$ 0.5
RMSUE$\downarrow$ | ECE$\downarrow$ | NLL$\downarrow$ | Sharp$\downarrow$
---|---|---|---
15.1 $\pm$ 0.1 | 0.323 $\pm$ 0.000 | 3.69 $\pm$ 0.0 | 17.22 $\pm$ 0.0
7.5 $\pm$ 0.3 | 0.026 $\pm$ 0.001 | 4.72 $\pm$ 0.2 | 4.28 $\pm$ 0.0
7.6 $\pm$ 0.3 | 0.074 $\pm$ 0.000 | 6.43 $\pm$ 0.3 | 3.21 $\pm$ 0.0
9.4 $\pm$ 0.4 | 0.217 $\pm$ 0.008 | 3.46 $\pm$ 0.0 | 9.72 $\pm$ 0.4
8.5 $\pm$ 0.3 | 0.175 $\pm$ 0.003 | 3.50 $\pm$ 0.1 | 8.06 $\pm$ 0.1
7.5 $\pm$ 0.3 | 0.027 $\pm$ 0.001 | 4.60 $\pm$ 0.1 | 4.09 $\pm$ 0.0
7.4 $\pm$ 0.3 | 0.026 $\pm$ 0.001 | 4.39 $\pm$ 0.1 | 4.23 $\pm$ 0.0
7.5 $\pm$ 0.3 | 0.027 $\pm$ 0.001 | 4.57 $\pm$ 0.2 | 4.11 $\pm$ 0.0
7.4 $\pm$ 0.3 | 0.025 $\pm$ 0.001 | 4.36 $\pm$ 0.1 | 4.33 $\pm$ 0.0
7.4 $\pm$ 0.3 | 0.018 $\pm$ 0.001 | 4.52 $\pm$ 0.1 | 4.07 $\pm$ 0.0
7.3 $\pm$ 0.3 | 0.017 $\pm$ 0.000 | 4.29 $\pm$ 0.1 | 4.24 $\pm$ 0.0
7.4 $\pm$ 0.3 | 0.018 $\pm$ 0.000 | 4.49 $\pm$ 0.1 | 4.08 $\pm$ 0.0
7.3 $\pm$ 0.3 | 0.017 $\pm$ 0.000 | 4.23 $\pm$ 0.1 | 4.27 $\pm$ 0.0

Figure 3: KITTI (top) and BDD100K (bottom): Correlation between aleatoric
uncertainty ($\mu_{\sigma_{obj}}\pm\sigma_{\sigma_{obj}}$) and performance
metrics, object area, BRISQSUE score and occlusion level. Uncalibrated and
calibrated via Rel. IR PCo CL, binned based on quantiles and normalized with
the highest $\mu_{\sigma_{obj}}$ post-binning.

###  4.3 Uncertainty Correlation

We investigate the correlation between the localization aleatoric uncertainty
and performance, object area, i.e. distance in the real world, occlusion level
and the Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [27]. In
the following, $\sigma_{obj}=\frac{1}{4}\sum^{4}_{i=1}\sigma_{i}$ is the
average of all four uncertainties per object.

Uncertainty vs Real-World Metrics. We assume that the distance of an object in
the real world is connected to its area in an image in pixels2 (px2). For both
datasets, Fig. 3 demonstrates that the smaller the object in the image, or the
farther away it is, the higher its aleatoric uncertainty. As mentioned in Sec.
2, aleatoric uncertainty correlates with occlusion. Fig. 3 visualizes the
results based on the annotations for occlusion in both datasets. KITTI has
three occlusion levels: 0 is fully visible, 1 is occluded less than $50\%$ and
2 is occluded more than $50\%$. BDD100K has only two: 0 is visible and 1 is
occluded. The correlation is present in both datasets, but less for BDD100K.
We trace this back to the model predicting double the uncertainty on average
for the traffic light and sign classes, as compared to other classes. While
56568 instances of these classes were labeled as visible, only 5040 were
labeled as occluded (8%). This, combined with the high uncertainty, negatively
impacts the correlation. However, when excluding these two classes, the
average uncertainty of visible objects is 34% lower than occluded objects pre-
calibration and 40% lower post-calibration.

Uncertainty vs Image Quality. The assumption that aleatoric uncertainty
correlates with inherent noise in the data is investigated based on the
BRISQUE score. For every detection, the score is calculated on the
standardized crop around its bounding box in the corresponding image.
Standardizing crops involves mean subtraction and division by the standard
deviation of pixel values. As Fig. 3 shows, the BRISQUE score positively
correlates with the uncertainty, indicating a higher uncertainty for lower
image quality.

Uncertainty vs Detection Performance. The comparison with IoU and RMSE for
both datasets in Fig. 3 shows a correlation with localization accuracy.
Calibration via Rel. IR PCo CL strengthens the correlation with all metrics,
as presented in Fig. 3. The calibrated aleatoric uncertainty can be used for
thresholding between misdetections (IoU $<=$ threshold) and correct detections
(IoU $>$ threshold) for both datasets (see Fig. 4), since the uncertainty of
misdetections is on average higher than the uncertainty of correct detections.
This extends to classification, where the uncertainty of false positives of
each class is on average also higher than the uncertainty of true positives.
Therefore, the localization aleatoric uncertainty allows for the detection of
the model prediction errors.

Figure 4: KITTI (left) and BDD100K (right): Calibrated aleatoric uncertainty
($\mu_{\sigma_{obj}}\pm\sigma_{\sigma_{obj}}$) for misdetections (MD, IoU $<=$
threshold) and correct detections (CD, IoU $>$ threshold) at each IoU
threshold, and true positives (TP) and false positives (FP) for the
classification of each class. The uncertainty of MDs and FPs is on average
higher than CDs and TPs.

##  5 Conclusion

We provide an object detection pipeline with reliable and interpretable
localization uncertainty, by covering the estimation, propagation,
calibration, and explanation of aleatoric uncertainty. Our methods enhance the
safety and reliability of object detectors without introducing drawbacks. We
propose two approaches to propagation, which allow an exact and fast
propagation of distributions, along the corresponding uncertainty, through
non-linear functions such as exponential, sigmoid and softmax. We demonstrate
the efficacy of our techniques through their implementation in the post-
processing of EfficientDet as a use-case. Our propagation methods improve the
localization performance of the baseline detector on both datasets KITTI and
BDD100K, and decrease the inference time. They generalize to any model with a
tractable output distribution requiring its transformation via invertible and
differentiable functions. They particularly alleviate the disadvantages of
sampling, namely either low accuracy and reproducibility or high computation
time. Furthermore, we extend regression calibration to localization, by
considering the relativity of the uncertainty to its bounding box, as well as
per-class and per-coordinate calibration with different optimization
functions. We also investigate the data selection process for calibration and
propose an approach for the allocation of predictions to their corresponding
ground truth, which alleviates the disadvantages of manual thresholding. We
find a correlation between aleatoric uncertainty and detection accuracy, image
quality, object occlusion, and object distance in the real world. We hope the
methods and results presented in this paper will encourage wider adoption of
uncertainty estimation in different industrial and safety-critical
applications, e.g. for safer decision making via more reliable detection
monitoring, and more efficient use of labeled data in active learning.

## References

  * [1] Appleyard, J., Yokim, S.: Programming tensor cores in cuda 9. https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/, Last accessed on 2023-1-13 (2017)
  * [2] Balakrishnan, N., Chen, W.W.S.: Lognormal distributions and properties, pp. 5–6. Springer US (1999)
  * [3] Barlow, R.E., Brunk, H.D.: The isotonic regression problem and its dual. Journal of the American Statistical Association 67(337), 140–147 (1972)
  * [4] Chatterjee, S., Sen, S.: Regret minimization in isotonic, heavy-tailed contextual bandits via adaptive confidence bands. arXiv preprint arXiv:2110.10245 (2021)
  * [5] Choi, J., Chun, D., Kim, H., Lee, H.J.: Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 502–511. IEEE (2019)
  * [6] Dillon, J.V., Langmore, I., Tran, D., Brevdo, E., Vasudevan, S., Moore, D., Patton, B., Alemi, A., Hoffman, M., Saurous, R.A.: Tensorflow distributions. arXiv preprint arXiv:1711.10604 (2017)
  * [7] Feng, D., Harakeh, A., Waslander, S.L., Dietmayer, K.: A review and comparative study on probabilistic object detection in autonomous driving. IEEE Transactions on Intelligent Transportation Systems 23(8), 9961–9980 (2022)
  * [8] Feng, D., Rosenbaum, L., Dietmayer, K.: Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection. arXiv preprint arXiv:1804.05132 (2018)
  * [9] Feng, D., Rosenbaum, L., Glaeser, C., Timm, F., Dietmayer, K.: Can we trust you? on calibration of a probabilistic object detector for autonomous driving. arXiv preprint arXiv:1909.12358 (2019)
  * [10] Feng, D., Rosenbaum, L., Timm, F., Dietmayer, K.: Leveraging heteroscedastic aleatoric uncertainties for robust real-time lidar 3d object detection. In: 2019 IEEE Intelligent Vehicles Symposium (IV). pp. 1280–1287. IEEE (2019)
  * [11] Futami, F., Sato, I., Sugiyama, M.: Variational inference based on robust divergences. In: Proceedings of the 21st International Conference on Artificial Intelligence and Statistics. pp. 813–822. PMLR (2018)
  * [12] Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The kitti vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 3354–3361. IEEE (2012)
  * [13] Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102(477), 359–378 (2007)
  * [14] Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning. pp. 1321–1330. PMLR (2017)
  * [15] Harakeh, A., Smart, M., Waslander, S.L.: Bayesod: A bayesian approach for uncertainty estimation in deep object detectors. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). pp. 87–93. IEEE (2020)
  * [16] He, Y., Zhu, C., Wang, J., Savvides, M., Zhang, X.: Bounding box regression with uncertainty for accurate object detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2883–2892. IEEE (2019)
  * [17] Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems 30 (2017)
  * [18] Kobyzev, I., Prince, S.J., Brubaker, M.A.: Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(11), 3964–3979 (2021)
  * [19] Krähenbühl, P., Koltun, V.: Parameter learning and convergent inference for dense random fields. In: Proceedings of the 30th International Conference on Machine Learning. pp. 513–521. PMLR (2013)
  * [20] Kraus, F., Dietmayer, K.: Uncertainty estimation in one-stage object detection. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC). pp. 53–60. IEEE (2019)
  * [21] Kuleshov, V., Fenner, N., Ermon, S.: Accurate uncertainties for deep learning using calibrated regression. In: Proceedings of the 35th International Conference on Machine Learning. pp. 2796–2804. PMLR (2018)
  * [22] Laves, M.H., Ihler, S., Fast, J.F., Kahrs, L.A., Ortmaier, T.: Recalibration of aleatoric and epistemic regression uncertainty in medical imaging. arXiv preprint arXiv:2104.12376 (2021)
  * [23] Laves, M.H., Ihler, S., Kortmann, K.P., Ortmaier, T.: Well-calibrated model uncertainty with temperature scaling for dropout variational inference. arXiv preprint arXiv:1909.13550 (2019)
  * [24] Le, M.T., Diehl, F., Brunner, T., Knoll, A.: Uncertainty estimation for deep neural object detectors in safety-critical applications. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC). pp. 3873–3878. IEEE (2018)
  * [25] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision – ECCV 2014. pp. 740–755. Springer International Publishing (2014)
  * [26] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. arXiv preprint arXiv:1512.02325 (2016)
  * [27] Mittal, A., Moorthy, A.K., Bovik, A.C.: Blind/referenceless image spatial quality evaluator. In: 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR). pp. 723–727. IEEE (2011)
  * [28] Phan, B., Salay, R., Czarnecki, K., Abdelzad, V., Denouden, T., Vernekar, S.: Calibrating uncertainties in object localization task. arXiv preprint arXiv:1811.11210 (2018)
  * [29] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 779–788. IEEE (2016)
  * [30] Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018)
  * [31] Tan, M., Pang, R., Le, Q.V.: Efficientdet: Scalable and efficient object detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10778–10787. IEEE (2020)
  * [32] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., Darrell, T.: Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2633–2642. IEEE (2020)
  * [33] Zadrozny, B., Elkan, C.: Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the eighth ACM SIGKDD international conference on knowledge discovery and data mining. pp. 694–699. ACM (2002)


# Simulations for novel problems in recommendation: analyzing misinformation
and data characteristics

Alejandro Bellogín<EMAIL_ADDRESS>0000-0001-6368-2510 Universidad
Autónoma de MadridSpain and  Yashar Deldjoo<EMAIL_ADDRESS>0000-0002-6767-358X Politecnico di BariItaly

(2021)

###### Abstract.

In this position paper, we discuss recent applications of simulation
approaches for recommender systems tasks. In particular, we describe how they
were used to analyze the problem of misinformation spreading and understand
which data characteristics affect the performance of recommendation algorithms
more significantly. We also present potential lines of future work where
simulation methods could advance the work in the recommendation community.

evaluation, data characteristics, misinformation, preference sampling

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††conference: SimuRec ’21: The SimuRec workshop held in
conjunction with the 15th ACM Conference on Recommender Systems (RecSys),
2021, in Amsterdam, Netherlands.; Sep 27– Oct 01, 2021; Amsterdam, Netherlands
(online)††booktitle: SimuRec ’21: The SimuRec workshop held in conjunction
with the 15th ACM Conference on Recommender Systems (RecSys), Sep 27– Oct 01,
2021, Amsterdam, Netherlands (online)††ccs: Information systems Recommender
systems††ccs: Information systems Collaborative filtering

##  1\. Importance of sampling in recommendation

Recent years have witnessed an explosion where simulations have been used in
several aspects of Recommender Systems (RS), either inspired from Machine
Learning (ML) or Information Retrieval (IR) problems – where simulations have
been used for a long time, as in click models (Chuklin et al., 2015) or
learning to rank (Ai et al., 2021) approaches –, or to attack inherent and
concrete issues that are prevalent in RS, such as data scarcity and sparsity.

In general, sampling in recommendation has been used to open up (or simulate)
evaluation scenarios. For example, in (Kluver and Konstan, 2014) the authors
used it to downsample the observed interactions and generate different
scenarios of cold-start. We also found works where sampling is employed to
analyze and understand the evaluation procedure, as in (Valcarce et al., 2018)
where it is used to perform a robustness analysis, in (Anelli et al., 2019)
where the authors extend the previous work to study hyperparameter
optimization, or to directly debias the evaluation, as described in (Carraro
and Bridge, 2020).

However, there is also a growing body of literature that has been using
sampling for learning preferences, as in the well-known Bayesian Personalized
Ranking (BPR) algorithm (Rendle et al., 2009), which uses sampling to select
pairs of items which are then provided to the algorithm. However, it is not
obvious how this sampling should be done so that it always work, and several
approaches have been investigated which differ in effectiveness and biases –
such as popularity – learned by the method (Jadidinejad et al., 2019; Chen et
al., 2021). Recent works that also rely on samplings or simulations are those
where reinforcement learning is applied to recommendation (Glowacka, 2019) or
to its evaluation (Huang et al., 2020). Similarly, whenever the IR or ML
approaches mentioned before have even been applied to recommendation, as in
(Hofmann et al., 2014), the sampling process is key for a successful
translation of the approach from one area to the other.

In the rest of this paper, we present two use cases where the authors have
recently applied data sampling on recommendation tasks with different goals in
each case. We later discuss the main lessons learned in this process and the
advantages of using simulations or sampling strategies. We end the paper with
several open questions or ideas to explore in the future.

##  2\. Successful applications of sampling

In this section, we describe two applications of data sampling on recent works
performed by the authors. Section 2.1 shows how sampling allowed to analyze
the effect of misinformation in recommender systems, whereas Section 2.2
describes a sampling strategy that is used to infer the impact of data
characteristics in recommendation performance.

###  2.1. Analyzing misinformation

In (Fernández et al., 2021), we analyzed the effect that some recommendation
algorithms cause on the amplification of misinformation. For this, we first
created a dataset by merging information from fact-checkers and Twitter. Then,
to simulate a wide range of situations – not only the one captured at the
moment of collecting the data – we used Algorithm 1 to generate user profiles
with a pre-defined set of constraints, in particular, the proportion of users
who shared some misinformative items. In this way, we would simulate a
population where 20% (or 50% or 80%) of users share this type of item. This
also gives us more control on the amount of information received by the
algorithms in terms of sparsity or other important dataset statistics.

Function _generate_  _user u, ratio r_

neg $\leftarrow$ { i $\in$ u: i  is misinformative  } ;

// Negative claims

neu $\leftarrow$ u $\setminus$ neg ;

// Neutral claims

desNeg $\leftarrow$ r $\cdot$ $|$u$|$ ;

// Desired negative ratio

desNeu $\leftarrow$ (1 - r) $\cdot$ $|$u$|$ ;

// Desired neutral ratio

while  _(desNeg $>$ $|$neg$|$) OR (desNeu $>$ $|$neu$|$)_ do

_ _ if  _desNeg $>$ $|$neg$|$ ;_

__ // Downsampling negative

 __ then

desNeg $\leftarrow$ desNeg - 1;

end if

_ _if  _desNeu $>$ $|$neu$|$ ;_

__ // Downsampling neutral

 __ then

desNeu $\leftarrow$ desNeu - 1;

end if

newTotal $\leftarrow$ desNeg + desNeu;

desNeg $\leftarrow$ r $\cdot$ newTotal;

desNeu $\leftarrow$ (1 - r) $\cdot$ newTotal;

end while

userProfile(u) $\leftarrow$ sample(neg, desNeg) $\cup$ sample(neu, desNeu);

end

Algorithm 1 Ratio-based user profile generator

It is interesting to note that simulation has been very recently proposed in
(Lucherini et al., 2021) also to study the societal impact of recommender
systems. By a general event-driven simulation model, the authors analyze some
case studies and discuss the implications for reproducibility such a framework
may have.

###  2.2. Understanding data characteristics

In (Deldjoo et al., 2021b), and as an extension of (Deldjoo et al., 2020), we
aim to better understand the impact of an array of characteristics in a
dataset with respect to accuracy and fairness (in (Deldjoo et al., 2021b)) or
robustness (in (Deldjoo et al., 2020)). With this goal in mind, we develop an
explanatory framework using regression models in its core, a methodology
originally proposed in (Adomavicius and Zhang, 2012). In this framework, we
test whether a set of data characteristics (independent variables of the
regression model) are related to a given performance metric (the dependent
variable). To obtain enough points so that the framework could produce
significant results, we need to simulate $N$ different datasets ($N=600$ in
both our papers). These simulated datasets are generated from sampling the
original dataset as described in Algorithm 2, which produces smaller datasets
with slightly different characteristics in each case, although satisfying some
constraints that allow the dataset to be useful in the analysis – for example,
by restricting the maximum number of items (via $\tau_{i}$) on these smaller
datasets. It is possible to perform these experiments by considering different
types of item content (Deldjoo et al., 2021c; Anelli et al., 2020; Deldjoo et
al., 2018) along different evaluation objective, e.g., accuracy, robustness
(Deldjoo et al., 2019; Anelli et al., 2020; Deldjoo et al., 2020), privacy
(Anelli et al., 2021; Slokom et al., 2020), or fairness (Deldjoo et al.,
2021b; Lesota et al., 2021).

Function _data-sampling_  _URM user-item rating matrix_

$n_{u}\leftarrow$ number of users of the URM;

$n_{i}\leftarrow$ number of users of the URM;

$n_{r}\leftarrow$ number of ratings of the URM;

$\tau_{u}\leftarrow$ constraint on average number of ratings for users;

$\tau_{i}\leftarrow$ constraint on maximum number of items;

$n\leftarrow 1$ ;

while  _ $n\leq N$ _ do

Random shuffle the rows of the URM;

$n_{u}\leftarrow rnd([100,n_{u}])$;

$n_{i}\leftarrow rnd([100,n_{i}])$;

$urm_{n}\leftarrow$ Selection of $n_{u}$, $n_{i}$ from shuffled URM;

if  _ $\frac{n_{r}}{n_{u}}<\tau_{u}$ or $n_{i}>\tau_{i}$ _ then

$n\leftarrow n+1$;

end if

end while

Output: $N$ sub-datasets ($urm_{n}$);

end

Algorithm 2 Sample generation procedure

##  3\. Discussion

From the applications presented in Section 2 we have learned that simulation,
and in particular, data sampling, is complex, even though it might be
beneficial – sometimes the only tool to produce the data points needed for an
analysis. Its difficulty relies on sampling the data in a significant but fair
and realistic way, without introducing new biases, as acknowledged in recent
works such as (Jadidinejad et al., 2019). For example, in the case described
in Section2.2 we experimented with imposing additional constraints to obtain
simulated datasets with a particular density, number of ratings per user, or
number of items.

Based on our experience, we foresee a continuous use of simulation techniques,
such as data sampling or synthetic data, to experiment with novel evaluation
conditions, such as those described before. In particular, consider the
limitations of public datasets, with a fixed number of attributes for users
and items and a (sometimes) a small number of interactions, these strategies
will allow evaluating new conditions with little effort – or at least, with
less effort than that of creating datasets including all the required
information.

##  4\. Open questions

Throughout this position paper, we have presented several situations where
simulations or sampling strategies have been successfully applied to
recommendation. However, we believe there is room for improvement, and several
aspects remain unexplored, both in our works and in the community, under a
more general perspective. First, regarding how to extend our sampling
approaches, it would be more realistic if the temporal dimension is
incorporated in the process, so that the interactions follow a compatible
order with the one in the original dataset. When done properly, this would
allow generating and studying feedback loops, like those created by
reinforcement learning algorithms, but at a higher, more global level. As
observed recently, and in agreement with our misinformation analysis
(Fernández et al., 2021), some algorithms are more prone to reproduce biases
at each feedback loop (Mansoury et al., 2020).

Regarding our second application (see Section 2.2), adding more variability
and flexibility in the types of datasets being generated would allow us to
address more complex questions. In particular, we envision a definition of
user types (or personas) which are then simulated or sampled at different
rates, either randomly or controlled via some parameter. Similarly, generating
content features realistically would help go beyond collaborative filtering
algorithms and test content-based methods at varying levels of information,
quality, and sparsity (Deldjoo et al., 2021c). For this, recent advances from
the Natural Language Processing community could be convenient, where Neural
Networks may generate realistic pieces of text in several domains (Devlin et
al., 2019; Brown et al., 2020). Besides content attributes, including
sensitive attributes in the set of controled (or simulated) information to be
generated will allow to explore fairness (Deldjoo et al., 2021a) analyses in a
more comprehensive way than what is being done right now. Finally, an
interesting perspective that could be promising is to shift the focus of the
application, as mentioned earlier, from the data to the algorithms so that the
sampling strategy instead of sampling data should sample algorithms. In this
way, the input data would be fixed, and the data points for the regression
analysis would be obtained from a wide selection of algorithms, probably with
different hyperparameters, to increase their variability. Such an approach
would have connections with automatic hyperparameter tuning techniques like
Bayes Optimization (Bergstra et al., 2011), but it would be applied to solve a
different problem and in a more extensive search space, as the type of
algorithm will also be part of the sampled variables (Bengio et al., 2021).

From a more general perspective, we believe several open questions need to be
considered when doing simulations or, in particular, some data sampling. An
important aspect that should be considered is that of reproducibility (Dacrema
et al., 2021). While usually sampling – or simulations in general – are random
in nature, this hinders reproducing other people’s works. However, by sharing
the code – where customizable seeds are included wherever is needed – and/or
the generated simulated/sampled datasets or the scripts used, the potential to
reproduce these types of works should increase (Bellogín and Said, 2021). By
addressing the reproducibility problem, a related issue we have also striven
to present properly in the past could be solved. Here we refer to present all
the technical decisions involved in a sampling strategy carefully and as
detailed as possible. Sometimes pseudocodes or algorithms do not have the
granularity level needed to specify implementation details that may greatly
impact how the sampling or the simulation is generated.

At the same time, although there are some works already addressing this issue
(see (Huang et al., 2020)), we believe it is very important to understand the
potential biases that a particular simulation may be introducing in the
generated data. This may be intentional but, usually, there should be some
mechanism to avoid them. Related to this problem, there should exist a
definition of when a simulation can be considered faithful to the original (or
intended) data, which we have referred throughout this work as realistic.
Without such definition, we might end up with data samples that are too
different (or different in specific, key aspects) from the actual data, hence
not satisfying their underlying assumptions or constraints. This also applies
when generating synthetic data, a task not so popular in recommendation
because of its difficulty, but where some efforts have been reported in the
last years (Slokom, 2018).

## References

  * (1)
  * Adomavicius and Zhang (2012) Gediminas Adomavicius and Jingjing Zhang. 2012.  Impact of data characteristics on recommender systems performance.  _ACM Trans. Manag. Inf. Syst._ 3, 1 (2012), 3:1–3:17.  https://doi.org/10.1145/2151163.2151166
  * Ai et al. (2021) Qingyao Ai, Tao Yang, Huazheng Wang, and Jiaxin Mao. 2021\.  Unbiased Learning to Rank: Online or Offline?  _ACM Trans. Inf. Syst._ 39, 2 (2021), 21:1–21:29.  https://doi.org/10.1145/3439861
  * Anelli et al. (2021) Vito Walter Anelli, Luca Belli, Yashar Deldjoo, Tommaso Di Noia, Antonio Ferrara, Fedelucio Narducci, and Claudio Pomo. 2021.  Pursuing Privacy in Recommender Systems: the View of Users and Researchers from Regulations to Applications. In _Fifteenth ACM Conference on Recommender Systems_. 838–841.
  * Anelli et al. (2020) Vito Walter Anelli, Yashar Deldjoo, Tommaso Di Noia, Eugenio Di Sciascio, and Felice Antonio Merra. 2020\.  SAShA: Semantic-Aware Shilling Attacks on Recommender Systems Exploiting Knowledge Graphs. In _The Semantic Web - 17th International Conference, ESWC 2020, Heraklion, Crete, Greece, May 31-June 4, 2020, Proceedings_ _(Lecture Notes in Computer Science, Vol. 12123)_. Springer, 307–323.  https://doi.org/10.1007/978-3-030-49461-2_18
  * Anelli et al. (2019) Vito Walter Anelli, Tommaso Di Noia, Eugenio Di Sciascio, Claudio Pomo, and Azzurra Ragone. 2019\.  On the discriminative power of hyper-parameters in cross-validation and how to choose them. In _Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019, Copenhagen, Denmark, September 16-20, 2019_ , Toine Bogers, Alan Said, Peter Brusilovsky, and Domonkos Tikk (Eds.). ACM, 447–451.  https://doi.org/10.1145/3298689.3347010
  * Bellogín and Said (2021) Alejandro Bellogín and Alan Said. 2021.  Improving Accountability in Recommender Systems Research Through Reproducibility.  _CoRR_ abs/2102.00482 (2021).  arXiv:2102.00482 https://arxiv.org/abs/2102.00482
  * Bengio et al. (2021) Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. 2021\.  Machine learning for combinatorial optimization: A methodological tour d’horizon.  _Eur. J. Oper. Res._ 290, 2 (2021), 405–421.  https://doi.org/10.1016/j.ejor.2020.07.063
  * Bergstra et al. (2011) James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011.  Algorithms for Hyper-Parameter Optimization. In _Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain_ , John Shawe-Taylor, Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger (Eds.). 2546–2554.  https://proceedings.neurips.cc/paper/2011/hash/86e8f7ab32cfd12577bc2619bc635690-Abstract.html
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).  https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  * Carraro and Bridge (2020) Diego Carraro and Derek Bridge. 2020.  Debiased offline evaluation of recommender systems: a weighted-sampling approach. In _SAC ’20: The 35th ACM/SIGAPP Symposium on Applied Computing, online event, [Brno, Czech Republic], March 30 - April 3, 2020_ , Chih-Cheng Hung, Tomás Cerný, Dongwan Shin, and Alessio Bechini (Eds.). ACM, 1435–1442.  https://doi.org/10.1145/3341105.3375759
  * Chen et al. (2021) Jiawei Chen, Chengquan Jiang, Can Wang, Sheng Zhou, Yan Feng, Chun Chen, Martin Ester, and Xiangnan He. 2021\.  CoSam: An Efficient Collaborative Adaptive Sampler for Recommendation.  _ACM Trans. Inf. Syst._ 39, 3, Article 34 (May 2021), 24 pages.  https://doi.org/10.1145/3450289
  * Chuklin et al. (2015) Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2015\.  _Click Models for Web Search_.  Morgan & Claypool Publishers.  https://doi.org/10.2200/S00654ED1V01Y201507ICR043
  * Dacrema et al. (2021) Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. 2021.  A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research.  _ACM Trans. Inf. Syst._ 39, 2 (2021), 20:1–20:49.  https://doi.org/10.1145/3434185
  * Deldjoo et al. (2021a) Yashar Deldjoo, Vito Walter Anelli, Hamed Zamani, Alejandro Bellogin, and Tommaso Di Noia. 2021a.  A flexible framework for evaluating user and item fairness in recommender systems.  _User Modeling and User-Adapted Interaction_ (2021), 1–47.
  * Deldjoo et al. (2021b) Yashar Deldjoo, Alejandro Bellogín, and Tommaso Di Noia. 2021b.  Explaining Recommender Systems Fairness and Accuracy through the Lens of Data Characteristics.  _Inf. Process. Manag._ 58, 5 (2021).  https://doi.org/10.1016/j.ipm.2021.102662
  * Deldjoo et al. (2019) Yashar Deldjoo, Tommaso Di Noia, and Felice Antonio Merra. 2019\.  Assessing the Impact of a User-Item Collaborative Attack on Class of Users. In _ImpactRS@RecSys’19 Workshop on the Impact of Recommender Systems_.
  * Deldjoo et al. (2020) Yashar Deldjoo, Tommaso Di Noia, Eugenio Di Sciascio, and Felice Antonio Merra. 2020.  How Dataset Characteristics Affect the Robustness of Collaborative Recommendation Models. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_. ACM, 951–960.  https://doi.org/10.1145/3397271.3401046
  * Deldjoo et al. (2018) Yashar Deldjoo, Markus Schedl, Paolo Cremonesi, and Gabriella Pasi. 2018.  Content-Based Multimedia Recommendation Systems: Definition and Application Domains. In _Proceedings of the 9th Italian Information Retrieval Workshop_.
  * Deldjoo et al. (2021c) Yashar Deldjoo, Markus Schedl, and Peter Knees. 2021c.  Content-driven Music Recommendation: Evolution, State of the Art, and Challenges.  _arXiv preprint arXiv:2107.11803_ (2021).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171–4186.  https://doi.org/10.18653/v1/n19-1423
  * Fernández et al. (2021) Miriam Fernández, Alejandro Bellogín, and Iván Cantador. 2021.  Analysing the Effect of Recommendation Algorithms on the Amplification of Misinformation.  _CoRR_ abs/2103.14748 (2021).  arXiv:2103.14748 https://arxiv.org/abs/2103.14748
  * Glowacka (2019) Dorota Glowacka. 2019\.  Bandit Algorithms in Information Retrieval.  _Found. Trends Inf. Retr._ 13, 4 (2019), 299–424.  https://doi.org/10.1561/1500000067
  * Hofmann et al. (2014) Katja Hofmann, Anne Schuth, Alejandro Bellogín, and Maarten de Rijke. 2014.  Effects of Position Bias on Click-Based Recommender Evaluation. In _Advances in Information Retrieval - 36th European Conference on IR Research, ECIR 2014, Amsterdam, The Netherlands, April 13-16, 2014. Proceedings_ _(Lecture Notes in Computer Science, Vol. 8416)_ , Maarten de Rijke, Tom Kenter, Arjen P. de Vries, ChengXiang Zhai, Franciska de Jong, Kira Radinsky, and Katja Hofmann (Eds.). Springer, 624–630.  https://doi.org/10.1007/978-3-319-06028-6_67
  * Huang et al. (2020) Jin Huang, Harrie Oosterhuis, Maarten de Rijke, and Herke van Hoof. 2020.  Keeping Dataset Biases out of the Simulation: A Debiased Simulator for Reinforcement Learning based Recommender Systems. In _RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22-26, 2020_ , Rodrygo L. T. Santos, Leandro Balby Marinho, Elizabeth M. Daly, Li Chen, Kim Falk, Noam Koenigstein, and Edleno Silva de Moura (Eds.). ACM, 190–199.  https://doi.org/10.1145/3383313.3412252
  * Jadidinejad et al. (2019) Amir Jadidinejad, Craig Macdonald, and Iadh Ounis. 2019\.  How Sensitive is Recommendation Systems’ Offline Evaluation to Popularity?. In _Proceedings of the Workshop on Offline Evaluation for Recommender Systems (REVEAL ’19), co-located with the 13th ACM Conference on Recommender Systems (RecSys 2019)_.
  * Kluver and Konstan (2014) Daniel Kluver and Joseph A. Konstan. 2014.  Evaluating recommender behavior for new users. In _Eighth ACM Conference on Recommender Systems, RecSys ’14, Foster City, Silicon Valley, CA, USA - October 06 - 10, 2014_ , Alfred Kobsa, Michelle X. Zhou, Martin Ester, and Yehuda Koren (Eds.). ACM, 121–128.  https://doi.org/10.1145/2645710.2645742
  * Lesota et al. (2021) Oleg Lesota, Alessandro Melchiorre, Navid Rekabsaz, Stefan Brandl, Dominik Kowald, Elisabeth Lex, and Markus Schedl. 2021.  Analyzing Item Popularity Bias of Music Recommender Systems: Are Different Genders Equally Affected?. In _Fifteenth ACM Conference on Recommender Systems_. 601–606.
  * Lucherini et al. (2021) Eli Lucherini, Matthew Sun, Amy Winecoff, and Arvind Narayanan. 2021.  T-RECS: A Simulation Tool to Study the Societal Impact of Recommender Systems.  _CoRR_ abs/2107.08959 (2021).  arXiv:2107.08959 https://arxiv.org/abs/2107.08959
  * Mansoury et al. (2020) Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2020\.  Feedback Loop and Bias Amplification in Recommender Systems. In _CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020_ , Mathieu d’Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudré-Mauroux (Eds.). ACM, 2145–2148.  https://doi.org/10.1145/3340531.3412152
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian Personalized Ranking from Implicit Feedback. In _UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009_ , Jeff A. Bilmes and Andrew Y. Ng (Eds.). AUAI Press, 452–461.  https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1&smnu=2&article_id=1630&proceeding_id=25
  * Slokom (2018) Manel Slokom. 2018\.  Comparing recommender systems using synthetic data. In _Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018_ , Sole Pera, Michael D. Ekstrand, Xavier Amatriain, and John O’Donovan (Eds.). ACM, 548–552.  https://doi.org/10.1145/3240323.3240325
  * Slokom et al. (2020) Manel Slokom, Martha Larson, and Alan Hanjalic. 2020\.  Partially Synthetic Data for Recommender Systems: Prediction Performance and Preference Hiding.  _arXiv preprint arXiv:2008.03797_ (2020).
  * Valcarce et al. (2018) Daniel Valcarce, Alejandro Bellogín, Javier Parapar, and Pablo Castells. 2018.  On the robustness and discriminative power of information retrieval metrics for top-N recommendation. In _Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018_ , Sole Pera, Michael D. Ekstrand, Xavier Amatriain, and John O’Donovan (Eds.). ACM, 260–268.  https://doi.org/10.1145/3240323.3240347