# DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera
3D Object Detection and Tracking

Qing Lian1,2 Tai Wang1,3 Dahua Lin1,3 Jiangmiao Pang1✉
1Shanghai AI Laboratory 2The Hong Kong University of Science and Technology
3The Chinese University of Hong Kong
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent multi-camera 3D object detectors usually leverage temporal information
to construct multi-view stereo that alleviates the ill-posed depth estimation.
However, they typically assume all the objects are static and directly
aggregate features across frames. This work begins with a theoretical and
empirical analysis to reveal that ignoring the motion of moving objects can
result in serious localization bias. Therefore, we propose to model Dynamic
Objects in RecurrenT (DORT) to tackle this problem. In contrast to previous
global Bird-Eye-View (BEV) methods, DORT extracts object-wise local volumes
for motion estimation that also alleviates the heavy computational burden. By
iteratively refining the estimated object motion and location, the preceding
features can be precisely aggregated to the current frame to mitigate the
aforementioned adverse effects. The simple framework has two significant
appealing properties. It is flexible and practical that can be plugged into
most camera-based 3D object detectors. As there are predictions of object
motion in the loop, it can easily track objects across frames according to
their nearest center distances. Without bells and whistles, DORT outperforms
all the previous methods on the nuScenes detection and tracking benchmarks
with 62.5% NDS and 57.6% AMOTA, respectively. The source code is available at
https://github.com/SmartBot-PJLab/DORT.

††✉ Corresponding author.

##  1 Introduction

Figure 1: Visualization of object localization from temporal correspondence.
Previous work ignores the motion of moving objects, which leads to imprecise
localization. Our work progressively refines the object’s location and motion
so that the preceding features can be precisely aggregated.

Multi-camera 3D object detection is critical to robotic systems such as
autonomous vehicles. As object depth estimation from a single image is
naturally ill-posed, recent works use large-scale depth pre-trained models
[31] and leverage geometric relationships [16, 57, 46, 44] to alleviate the
problem. Because stereo correspondence exists in consecutive frames, some
works resort to temporal information for accurate depth predictions. For
example, BEVDet4D [11] and BEVFormer [20] warp preceding features to the
current frame to enrich the single-frame BEV representations. DfM [44]
constructs temporal cost volumes that explicitly establish the stereo
correspondence. However, these cross-frame feature aggregations do not
consider the motion of moving objects and assume all the objects are static,
which results in serious 3D localization bias.

In this paper, we first provide a theoretical and empirical analysis to reveal
the negative effects of inaccurate object motion to object depth (Fig 1). In
particular, if the object is moving toward or backward to the ego-car, the
incorrect correspondence would derive a farther or closer depth. In the
driving scenarios, it is critical that a misleading farther depth is
estimated, which might reduce the reaction time of the decision system,
leading to catastrophic collision accidents. This motivates us to devise an
explicit mechanism to involve object motion estimation in the temporal-based
3D object detection pipeline.

Modeling dynamic objects in this context has several challenges: (1) We need a
flexible object-wise representation for potential object-wise operations based
on motion modeling. (2) Jointly estimating object location and motion is an
inherent chicken and egg problem [7]: The temporal correspondence can derive
accurate object location only when accurate object motion is given and vice
versa. (3) Simultaneously predicting object location and motion from only two
frames is also an ill-posed problem theoretically, and thus involving right-
body assumption and more frames in the framework to pose reasonable
constraints is desired.

To address these problems, we model Dynamic Objects in RecurrenT (DORT) that
simultaneously estimates object motion and location, and then progressively
refines them for accurate 3D object detection. It benefits from a local 3D
volume representation that not only extracts object-wise 3D features but also
alleviates the heavy computational costs of global BEV in previous methods
[44, 20, 12]. Based on the object-wise 3D volume, temporal cost volumes are
constructed by warping the volumes from the preceding frame to the current
frame according to the object motion. Then the obtained cost volumes act as
the features for updating the candidate location and motion. We model this
estimation and update pipeline as a recurrent process to alleviate the
aforementioned chicken and egg problem. In addition, our framework can take
into more than two frames and pose constraints to the object motion across
different frames. It inherently provides a feasible solution to avoid the ill-
posed dilemma of estimating object location and motion from only a single pair
of correspondence observations. As there is object motion prediction in the
loop, the framework is naturally capable of joint object detection and
tracking by utilizing object motion to align the detection results into the
same timestamp. It also can be plugged into most camera-based 3D object
detectors for flexible and practical use.

We validate the effectiveness of our framework on the nuScenes detection and
tracking benchmarks. Benefiting from the dynamic objects modeling, DORT
outperforms all the previous methods with a large margin, leading to 62.5%
nuScenes detection metric (NDS) and 57.6% and average multi-object tracking
accuracy (AMOTA), respectively.

##  2 Related work

##### Monocular 3D Object Detection

Monocular-based 3D object detection was first approached from single-frame
based methods and evolved into multi-frame based to alleviate the ill-posed
depth estimation problem.

_(a) Methods with A Single Frame_ The single-frame based methods [59, 27, 57,
60, 2] first extend 2D object detectors and insert several parallel regression
heads (i.e. 3D size, depth, and orientation) to predict 3D bounding boxes. To
alleviate the ill-posed depth recovery problem, several methods improve the
model from the perspectives of loss module [27], network architecture [3, 45],
regression objective [57, 46], etc. Besides directly regressing object depth,
later approaches [16, 25, 21] further design 2D-3D geometry constraints to
better extract visual cues for object depth estimation. To align the detection
features with output space, another line of methods [34] design several
transformation modules to lift 2D inputs into 3D space. Pseudo-lidar based
methods [47, 55, 48, 48] first predict the per-pixel depth and convert the raw
pixel into point cloud for 3D detection. BEV-based methods [34, 33, 12, 19]
propose orthographic feature transformation (OFT) to transform the image
features into a 3D voxel and then adopt a LiDAR-based head to localize
objects. Later work improves the OFT from the perspectives of explicit depth
distribution modeling [33, 12, 19], incorporating deformable attention module
[20] or designing 3D-based position encoding for attention [49, 22].

_(b) Methods with Multiple Frames_ Although many techniques are designed in
single-frame based methods, they still suffer from ill-posed depth recovery
problems, leading to unsatisfactory performance for deployment. To augment the
single-view observation, recent work [15, 3, 44, 11, 20, 17] leverages
previous frames as additional observations for geometry modeling and features
augmentation. Kinematic3D [3] leverages 3D Kalman Filter to associate objects
across frames and refines the estimated 3D box. Later studies [44, 11, 20]
construct cross-frame cost-volumes as another visual cue for 3D detection. The
cost volumes is based on the multi-view stereo, which assumes objects are
static across frames. However, this assumption does not align with the driving
scenario, where the objects can move. More critically, our analysis
demonstrates that the inaccurate object motion would introduce misleading
visual cues for object localization.

##### Monocular 3D Object Tracking

3D object tracking associates the objects across frames and generates a set of
trajectories for motion prediction and planning. Traditional methods adopt a
tracking-by-detection paradigm that first detects objects in each frame and
then associates them by the appearance features [15] or objects’ displacement
with Kalman filter [1, 54, 5, 35, 51, 6]. Besides the above paradigm, several
methods [10, 13] design a two-stage paradigm that first associates objects
based on appearance features and then utilizes the temporal motion to improve
the object detection performance. In this work, we utilize temporal cost
volumes to bridge the spatial location and temporal motion and derive a
recurrent paradigm that iteratively updates them to obtain tightly coupled
results for joint 3D detection and tracking.

##### Multi-View 3D Perception

Leveraging multi-view images to recover 3D information is a fundamental topic,
such as structure from motion [43], multi-view stereo [52], simultaneous
localization and mapping [38], etc. One line of methods develop neural-
network-based cost volumes [52, 53, 36, 40, 41, 37] to construct cross-frame
visual cues for 3D perception. Another line of methods [42, 39, 14] constructs
geometry constraints and leverage optimization techniques to obtain a tight-
coupled 3D structure. However, most of the work assumes the scene and objects
are static, making them fail to handle the moving objects in driving
scenarios.

##  3 Object Motion in Temporal Modeling

In this section, we first provide a theoretical and empirical analysis to
demystify the adverse effects of neglecting object motion in temporal modeling
[44, 11], and then briefly discuss the challenges of modeling 3D motion in the
monocular setting.

###  3.1 Localization Bias from the Static Assumption

In previous temporal-based methods [44, 11, 20], the object motion is ignored
by assuming objects are static across frames and the temporal features are
directly aggregated after converting the past frames to the current frame. We
first show that the static assumption would derive a biased depth in temporal
modeling. Without loss of generality, we consider the two-view case here, and
the derived analysis can be naturally extended to more than two views. We
denote the camera intrinsic as $K$ and the ego-motion from frame $t_{0}$ to
frame $t_{1}$ as $T_{t_{0}\rightarrow t_{1}}^{ego}$:

$\displaystyle K=\left[\begin{array}[]{cccc}f&0&c_{u}\\\ 0&f&c_{v}\\\ 0&0&1\\\
\end{array}\right],T_{t_{0}\rightarrow
t_{1}}^{ego}=\left[\begin{array}[]{cccc}1&0&0&x^{ego}\\\ 0&1&0&0\\\
0&0&1&z^{ego}\\\ \end{array}\right].$ (7)

Here, $f$ is the camera’s focal length, and $(c_{u},c_{v})$ is the camera
center coordinates in the image. For simplicity, we assume the ego-motion only
contains the translation $(x^{ego},0,z^{ego})$ on the horizontal plane. The
analysis also can be easily extended to a more complicated case that the
motion contains rotation.

Given the multiple-view images, temporal-based methods can utilize photometric
or featuremetric similarity to find the correspondence of pixel
$p_{t_{0}}=(u_{t_{0}},v_{t_{0}})$ in the past frame $t_{0}$ and the pixel
$p_{t_{1}}=(u_{t_{1}},v_{t_{1}})$ in the current frame $t_{1}$. The depth
$z_{t_{1}}$ can be recovered by solving the correspondence:

$\displaystyle T_{t_{0}\rightarrow
t_{1}}^{ego}\cdot\pi(p_{t_{0}},K)=\pi(p_{t_{1}},K),$ (8) $\displaystyle
z_{t_{1}}=\frac{z^{ego}(u_{t_{0}}-c_{u})-fx^{ego}}{u_{t_{0}}-u_{t_{1}}},$ (9)

where $\pi$ denotes the projection from 2D image coordinate to 3D camera
coordinate. The above derivation assumes that the object is static across two
frames. However, in the driving scenario, objects can move with a
corresponding object motion $T^{obj}_{t_{0}\rightarrow t_{1}}$. Suppose the
object motion only contains the translation on the horizontal plane in a short
time interval, it can be represented as:

$\displaystyle T^{obj}_{i\rightarrow
j}=\left[\begin{array}[]{cccc}1&0&0&x^{obj}\\\ 0&1&0&0\\\ 0&0&1&z^{obj}\\\
\end{array}\right].$ (13)

Then the corresponding equation to recover depth can be revised as follows:

$\displaystyle T^{obj}_{t_{0}\rightarrow t_{1}}T_{t_{0}\rightarrow
t_{1}}^{ego}\cdot\pi(p_{t_{0}},K)=\pi(p_{t_{1}},K),$ (14)
$\displaystyle\hat{z}_{t_{1}}=\frac{(z^{ego}+z^{obj})(u_{t_{0}}-c_{u})-f(x^{ego}+x^{obj})}{u_{t_{0}}-u_{t_{1}}}.$
(15)

Based on Eq (8) and (14), we can obtain the depth gap:

$\displaystyle\Delta
z=\frac{z^{obj}(u_{t_{0}}-c_{u})-fx^{ego}}{u_{t_{0}}-u_{t_{1}}}.$ (16)

Figure 2: Empirical analysis of the depth bias on the nuScenes dataset if
objects are assumed static. Figure 3: Pipeline overview. Given a video
sequence, we first extract the features from 2D images and generate the
candidate boxes and their motion by a single-frame detector. Then the boxes
and motion are progressively refined from the concurrently updated
corresponding 3D volume features. A fusion process in the recurrent module
finally combines the estimation from each pair of frames. Based on the tightly
coupled modeling of object location and motion, the framework can achieve
joint 3D detection and tracking during inference.

From Eq (16), we can observe that the depth bias is linearly correlated with
the object motion. In Fig 2, we also display the empirical statistics of
object motion and the corresponding depth bias from the nuScenes dataset. We
can observe that the empirical depth error is also linearly correlated to the
object velocity and increases as the time interval enlarges. Besides, the
right part in Fig. 2 also shows that almost 51% of objects are moving across
frames, demonstrating the necessity of modeling object motion in the temporal-
based framework.

###  3.2 Ill-Posed Problem in Motion Modeling

Except for demonstrating the necessity of modeling object motion in temporal-
based frameworks, we also want to mention that simultaneously estimating
object location and motion is a non-trivial problem, especially in the two-
frame case. As shown in Fig 1, the correspondence of two points from the
temporal frames can come from infinite combinations of object location and
motion (We provide more examples and the visualization in the supplementary).
This illustrates that joint estimation of location and motion from only one
correspondence is an ill-posed problem. To alleviate this issue, we first
simplify the object motion as a right-body movement so that multiple
correspondences from the points in the object can be used to solve a shared
motion. Furthermore, we also leverage more than two frames to constrain the
flexibility of object motion. More details can be referred to Sec 4.3.3.

##  4 Methodology

This section describes the details of DORT. DORT is a general joint detection
and motion prediction module that can estimate coupled object location and
motion results across frames. Based on the tightly coupled location and motion
results, DORT is also capable of simultaneously 3D object detection and
tracking. Basically, it can be based on most temporal 3D detectors [11, 32,
20]. In this work, we select the popular temporal detector BEVDepth [19] as
the base detector and extend it to handle both static and moving objects in
temporal modeling. We first present an overview of temporal-based frameworks
in Sec 4.1 and then introduce our modifications: the local volume for object-
wise representation in Sec 4.2, the key recurrent dynamic objects modeling in
Sec. 4.3, and the object association for monocular 4D object detection in Sec
4.4.

###  4.1 Overview of Temporal-Based Frameworks

Previous temporal-based camera-only 3D detection methods contain three stages:
(1) The 2D features extraction stage extracts the features from the input
images. (2) The view transformation and stereo matching process that first
lift the 2D features to a global 3D volume and then warp the features in each
frame to an aligned canonical space for stereo matching. Depending on the
model design, the view transformation and stereo matching order may reverse.
(3) The detection stage fuses the 3D features from different observations
(monocular and stereo) and estimates 3D bounding boxes based on a BEV-based
detection head.

In this work, we follow previous methods [44, 12, 19] and adopt the widely-
used 2D backbone (e.g. ResNet [9]) to do the features extraction. For the view
transformation stage, we design an object-wise local volume that leverages the
candidate 3D boxes to obtain the potential foreground regions and only models
them with local object-wise 3D volumes. For the stereo matching and detection
stages, we propose a recurrent dynamic objects modeling module. Specifically,
we adopt a recurrent paradigm that progressively refines the detection and
motion results by concurrently updating the corresponding 3D volume features
thereon.

Figure 4: The process of extracting local volumes in the current and past
frames according to predicted bounding boxes and object motion.

###  4.2 Object-wise Local Volume

In previous work, the 2D-3D transformation considers each candidate 3D grid
point and constructs a global volume for 3D detection. However, there are
several limitations: (1) the global 3D volume contains lots of background
regions, which is not vital for 3D detection but largely increases the
computation burden. (2) Modeling a global volume needs to pre-define a
detection range during training, making the detectors fail to detect objects
with arbitrary depths. (3) It is inconvenient to maintain and manipulate a
global 3D volume with various object-wise operations.

Hence, we replace the global 3D volume with an object-wise local volume.
Specifically, we leverage the candidate bounding boxes to determine the 3D
region of interest (RoI) and set the local volume center as the bounding box
center. To keep the object ratio and achieve cross-view warping, we assign
each 3D RoI volume $V\in\mathcal{R}^{W\times H\times L\times C}$ with the same
3D dimension $(W,H,L)$ and channel size $C$. Different from 2D detection, the
objects’ dimension in 3D space has less variance and empirically relies less
on the RoI-Align [8] operation. We display the construction of object-wise
local volumes in Fig 4. For the 2D to 3D transformation, we first follow
LiftSplat [33] and lift the images to a 2.5D frustum by weighting with depth
probability. Then we utilize the grid sample operation to warp the features
from the 2.5D frustum to each 3D local volume. Benefiting from the accurate 2D
detection performance of perspective-view detection, the local volume features
sampled from the 2.5D frustum would have a large overlap with the foreground
objects. Hence, if the proposal 3D location is inaccurate, the later
refinement module still has the ability to use the features to do the
refinement.

###  4.3 Recurrent Dynamic Objects Modeling

The pipeline of the recurrent framework is illustrated in Fig. 3. Given the
candidate 3D bounding boxes and motion as input, each iteration first
constructs the temporal cost volumes thereon, and aggregates these cues to
refine the proposal boxes and motion. In particular, we adopt a perspective-
view based 3D detector (i.e. PGD) to generate the initialized candidate 3D
boxes and motion and only predict their residuals for refinement in the
subsequent recurrent updates. Next, we elaborate on these steps in detail.

Table 1: Experimental results of monocular 3D object detection and tracking on
the nuScenes test set. The input resolution is $1600\times 900$ with using
ConvNeXt-Base [24] as the backbone.

Table 2: 3D detection results on the nuScenes test set.

Method | mAP$\uparrow$ | mATE$\downarrow$ | mASE$\downarrow$ | mAOE$\downarrow$ | mAVE$\downarrow$ | mAAE$\downarrow$ | NDS$\uparrow$
---|---|---|---|---|---|---|---
Ego3RT [26] | 42.5 | 0.55 | 0.26 | 0.43 | 1.01 | 0.14 | 47.3
UVTR [18] | 47.2 | 0.57 | 0.25 | 0.39 | 0.51 | 0.12 | 55.1
BEVFormer [20] | 48.1 | 0.58 | 0.25 | 0.37 | 0.37 | 0.12 | 56.9
PETRv2 [23] | 51.2 | 0.55 | 0.25 | 0.36 | 0.40 | 0.13 | 58.6
BEVDepth [19] | 52.0 | 0.45 | 0.24 | 0.35 | 0.35 | 0.13 | 60.9
BEVStereo [17] | 52.5 | 0.43 | 0.24 | 0.36 | 0.35 | 0.14 | 61.0
SOLOFusion [32] | 54.0 | 0.45 | 0.26 | 0.37 | 0.27 | 0.14 | 61.9
DORT (Ours) | 54.5 | 0.44 | 0.26 | 0.37 | 0.25 | 0.14 | 62.5

Table 3: 3D tracking results on the nuScenes test set.

Method | AMOTA$\uparrow$ | AMOTP$\downarrow$ | MOTAR $\uparrow$
---|---|---|---
QD-3DT [10] | 21.7 | 1.550 | 56.3
Time3D [13] | 21.4 | 1.360 | -
PolarDETR [5] | 27.3 | 1.185 | 60.7
MUTR3D [56] | 27.0 | 1.494 | 64.3
SRCN3D [35] | 39.8 | 1.317 | 70.2
QTrack [51] | 48.0 | 1.100 | 74.7
UVTR [18] | 51.9 | 1.125 | 76.4
DORT(Ours) | 57.6 | 0.951 | 77.1

####  4.3.1 Cross-Frame Cost Volumes Construction

Given the initial predictions of 3D boxes and their motion, we first obtain
object-wise volume features following Sec. 4.2. Then we can construct the
temporal cost volumes by warping features from past frames to the current
frame coordinates based on _ego-motion_. In contrast to previous work [44, 11]
assuming objects are static, we further involve the _object motion_ into the
warping procedure. Specifically, for each point $p\in\mathcal{R}^{3}$ in the
object-wise local volume $V$, we query the corresponding features in previous
frame $t-\Delta t$ with the consideration of ego-motion $T^{ego}$ and the
object motion $T^{obj}$ and construct the cost as $\left[V(p),V_{t-\Delta
t}(T^{obj}T^{ego}p)\right]$. Note that we simplify the point motion as the
object motion with a rigid-body assumption, which can approximate most of the
cases in driving scenarios, especially for vehicles [15, 41].

####  4.3.2 3D Boxes and Motion Residual Estimation

Given the object-wise temporal features built from input 3D boxes and motion,
we leverage a refinement network to estimate the residual between the input 3D
boxes and motion with the ground truth. The refinement network contains
several 2D/3D residual-based convolutional layers to extract the 3D volumes
and 2D BEV features. The detailed architecture is presented in Supplementary.
Formally, the refinement is formulated as the regression of 3D attribute
residuals $\mathcal{B}$, including the object’s 3D center $x,y,z$, 3D size
$w,h,l$, rotation $\theta$ and velocity $v_{x},v_{y}$. Since we use the object
velocity in the current frame to represent the object motion and assume
constant velocity across frame, the supervision for some frames may contain
noise(e.g. inaccurate labels, violation of rigid-body assumption, etc). Hence,
we model the residual as a Laplacian distribution and design the loss function
as:

$\displaystyle\mathcal{L}_{refine}=\sum_{b\in\mathcal{B}}(\frac{\sqrt{2}}{\sigma_{b}}\|\Delta\hat{b}-\Delta
b\|+\log{\sigma_{b}}).$ (17)

Here, $\Delta b$, $\Delta\hat{b}$, and $\sigma_{b}$ are all the network
outputs, and represent the ground truth residual, the estimated residual, and
the estimated standard deviation of residual for each 3D attribute,
respectively.

####  4.3.3 Multiple Estimation Fusion

Given $n$ frame as inputs, we can obtain $n$ 3D volumes ($1$ for the local
volumes from the referenced single view and $n-1$ for the paired cross-view
cost volumes) and obtain $n$ estimated residuals from the above residual
estimation module. Then we weigh the importance of each residual by the
estimated deviation and fuse them to obtain an ensemble result. The fusion
formulation can be represented as:

$\displaystyle\hat{b}_{fused}=\sum_{i=1}^{n}\frac{e^{\sigma_{b_{i}}}b_{i}}{\sum_{i=1}^{n}e^{\sigma_{b_{i}}}},$
(18)

where $i$ denotes the volume index.

For simplicity, here we only estimate the velocity measurement for the
referenced frame, _i.e._ , the fluctuation of object velocity across different
frames would not be considered explicitly. The mechanism for multi-frame
fusion is expected to handle this problem adaptively. At the same time, this
constraint also provides additional cues when simultaneously estimating object
location and motion from more than two frames.

####  4.3.4 Recurrent Location and Motion Update

After each iteration, we can obtain the refined bounding boxes with their
motion and thus can derive the updated bounding boxes in different frames.
With these updated locations and RoIs, we can further update the volume
features and proceed to the next-round refinement. Note that any complex or
learnable motion modeling can be integrated into this procedure. Here, to be
consistent with the multiple estimation fusion designs, we still keep the
constant velocity prediction for simplicity to derive the bounding boxes of
previous frames.

In the training stage, we follow the recurrent methods [40, 41] in other tasks
and set the loss weight for each iteration as the same. The overall loss is
represented as:

$\displaystyle\mathcal{L}=\mathcal{L}_{pv}+\sum_{i=1}^{k}\mathcal{L}^{i}_{refine},$
(19)

where $\mathcal{L}_{pv}$ is the loss in the perspective-view detector [46],
$\mathcal{L}^{i}_{refine}$ is the refinement loss in each iteration and $k=3$
is the number of the iterations.

In the inference stage, we first take the perspective-view detector (i.e. PGD
[46]) to generate the initial 3D bounding boxes and their motion and then
progressively refine them with three iterations. During each iteration, we
first construct the volume features as discussed in Sec 4.3.1 and feed them
into the recurrent refinement module to estimate the 3D boxes and motion
residuals for each paired frame input. Then we utilize the multiple estimation
fusion module in Sec 4.3.3 to fuse the estimated results and obtain the
refined 3D boxes and motion as the next stage input.

###  4.4 Monocular 4D Object Detection

So far, we have introduced our recurrent framework for 3D detection from
monocular videos. Based on progressive refinement, our model can estimate
tightly coupled object location and motion results and thus can easily
associate the object detection results across frames, leading to joint 3D
detection and tracking. Specifically, we follow [1, 58, 54, 30] and associate
the detection results by warping current detection to the past frames with
object motion. Based on the ego-motion, we first convert the predicted object
location to the past frame coordinate and then warp with the estimated object
velocity. Then we follow the popular distance-based tracker in 2D task [1, 54]
and associate the objects by the closest distance matching. We also utilize
the Kalman filter to maintain the trackers’ location and velocity. We provide
more details of the tracking pipeline in the supplemental materials.

Table 4: Experimental results on the nuScenes validation set. The input
resolution is $704\times 256$ using ResNet-50 as the backbone. * denotes the
re-implementation based on the provided code.

Methods | # frame | mAP$\uparrow$ | mATE$\downarrow$ | mASE$\downarrow$ | mAOE$\downarrow$ | mAVE$\downarrow$ | mAAE$\downarrow$ | NDS$\uparrow$
---|---|---|---|---|---|---|---|---
PGD* [46] | 1 | 28.8 | 0.75 | 0.27 | 0.52 | 1.13 | 0.18 | 37.0
BEVDet [12] | 29.8 | 0.73 | 0.28 | 0.59 | 0.86 | 0.24 | 37.9
PETR [22] | 31.3 | 0.77 | 0.28 | 0.56 | 0.92 | 0.23 | 38.1
DETR3D [49] | 34.9 | 0.72 | 0.27 | 0.38 | 0.84 | 0.20 | 43.4
BEVDet4D [11] | 2 | 32.2 | 0.70 | 0.28 | 0.50 | 0.35 | 0.21 | 45.7
BEVDepth [19] | 35.1 | 0.64 | 0.27 | 0.48 | 0.43 | 0.20 | 47.5
DORT (Ours) | 37.9 | 0.62 | 0.27 | 0.35 | 0.31 | 0.20 | 51.4
BEVDepth* [11] | 8 | 39.8 | 0.57 | 0.27 | 0.49 | 0.27 | 0.18 | 52.3
DORT (Ours) | 41.8 | 0.57 | 0.26 | 0.43 | 0.25 | 0.19 | 53.4
SOLOFusion [32] | 16 | 42.7 | 0.57 | 0.27 | 0.41 | 0.25 | 0.18 | 53.4
DORT (Ours) | 43.6 | 0.56 | 0.26 | 0.41 | 0.24 | 0.18 | 54.0

Table 5: 3D object tracking results on the nuScenes validation set. We adopt
ResNet-50 as the backbone and set the input resolution as $704\times 256$.

Method | AMOTA$\uparrow$ | AMOTP$\downarrow$ | Recall$\uparrow$
---|---|---|---
QD-Track3D [10] | 24.2 | 1.518 | 39.9
Time3D [13] | 21.4 | 1.360 | N/A
TripletTrack [28] | 28.5 | 1.485 | N/A
MUTR3D [56] | 29.4 | 1.498 | 42.7
QTrack [51] | 34.7 | 1.347 | 46.2
DORT | 42.4 | 1.264 | 49.2

##  5 Experiments

We validate the effectiveness of our method on the large-scale nuScenes
detection and tracking benchmark [4].

###  5.1 Experimental Setup

In this section, we describe the used dataset, the evaluation metrics, and the
implementation details.

Dataset NuScenes [4] is a large-scale autonomous driving dataset, which
contains 1,000 video sequences. The official protocol splits the video
sequences into 700 for training, 150 for validation, and 150 for testing. Each
sequence is annotated with 3D bounding boxes, object velocity, and the
tracking id to connect the objects across the frame. The camera calibration
information, timestamps, and ego-motion in both training and test sets are
provided.

Detection Metrics We adopt the official evaluation protocol provided by
nuScenes benchmark [4]. The official protocol evaluates 3D detection
performance by the metrics of average translation error (ATE), average scale
error (ASE), average orientation error (AOE), average velocity error (AVE),
and average attribute error (AAE). Besides, it also measures the mean average
precision (mAP) with considering different recall thresholds. Instead of using
3D Intersection over Union (IoU) as the criterion, nuScenes defines the match
by 2D center distance $d$ on the ground plane with thresholds
$\\{0.5,1,2,4\\}m$. The above metrics are finally combined into a nuScenes
Detection Score (NDS).

Tracking Metrics Regarding the tracking metrics, the nuScenes benchmark mainly
measures the average multi-object tracking accuracy (AMOTA), average multi-
object tracking precision (AMOTP), and tracking recall. In particular, AMOTA
and AMOTP are the averages of multi-object tracking accuracy (MOTA) and multi-
object tracking precision (MOTP) under different recall thresholds.

Network Details As discussed in Sec 4.3, the recurrent module requires a
proposal detector to generate candidate foreground regions as the $1^{st}$
stage input. We adopt the popular monocular 3D detector PGD [46] due to its
high 2D object detection recall. Following [12, 11, 19], we adopt the
ResNet-50 [9] with FPN as the 2D feature extractor and mainly conduct
experiments on this setting. The 2D feature extractors in PGD and the
recurrent module are shared to save computation time. The grid size of the 3D
volume is set as $0.8m$ with the range of $[-5m,5m]$ in the X and Z (depth)
axis and $[-4m,2m]$ in the Y (height) axis. During 2D to 3D features
transformation, we follow [19] and adopt the depth distribution guided 2D to
3D features lifting. Regarding the test set submission, we follow [19, 17] and
adopt the ConvNeXt-Base [24] as the image backbone. The image backbone is
initialized with ImageNet pre-trained weights, and no other external data is
used. We provide more details about the network architecture of the recurrent
module and the training configuration in the supplementary materials.

Training Configurations The model is optimized by AdamW optimizer with weight
decay $10^{-2}$. We first follow [46] to train the proposal detector and
refine the recurrent module with 24 epochs, where the initial learning rate is
set as $2\times 10^{-4}$ and decreases to $2\times 10^{-5}$ and $2\times
10^{-6}$ at the $18^{th}$ and $22^{th}$ epochs. Following [12, 19], we use the
class balance sampling strategy (CBGS) to alleviate the class imbalance
problem. We adopt the commonly used 2D data augmentation that randomly flips
the image, resizes the image with the range of $[0.36,0.55]$, and crops the
image to the resolution of $704\times 256$. Regarding the input video
sequences, we follow [11, 19] and sample the preceding keyframes to obtain the
past video sequences. Regarding the test set submission, we enlarge the input
resolution to $1600\times 640$ and reduce the volume size to $0.4m$.

###  5.2 Main Results

In Table 4.3, we first provide the comparison of our framework with existing
state-of-the-art methods on the nuScenes test benchmarks. We draw the
following observations: (i) Benefiting from dynamic objects modeling, our
method displays a significant improvement in both object detection (mAP) and
motion estimation (mAVE), and 0.5% and relatively 7.4% better than the
previous best method. These localization and motion estimation improvements
also contribute to state-of-the-art results in terms of the nuScenes detection
metric (NDS). (ii) With strong localization and motion estimation results, our
tracking module can better associate the detected objects in different
timestamps, resulting in superior performance over all the other trackers with
different metrics. Specifically, we improve the second best tracker [18],
another distance-based tracker with 10.9% and 7.5% relative improvements on
the AMOTA and AMOTP metrics. Compared with the joint detection and tracking
methods QDTrack3D [10] and Time3D [13], the performance gain of our method
demonstrates the effectiveness of our dynamic objects modeling framework in
jointly modeling object motion and location. (iii) In Table 4 and 5, we also
report our method on the nuScenes validation set with different settings. For
the detection performance, we can draw the same observation as in the test set
that our method can outperform previous temporal-based methods [11, 19, 32] in
terms of mAP and mAVE. Note that our method is also compatible with the
components designed in the current BEV-based frameworks, such as the training
techniques in BEVDepth [19] and the depth estimation module in SOLOFusion
[32]. Furthermore, the local 3D volume is more friendly to practical
applications, which can handle objects with arbitrary depth in the image.

(a)

(b)

Figure 5: Detection results of each iteration in the recurrent pipeline. 1
past frame is used in temporal modeling. Table 6: Ablation study of the local
volumes on the nuScenes validation set. * denotes the single frame version
without using EMA for fair comparisons. The first term and the second term in
FLOPS denote the operation in perspective view and bird-eye-view,
respectively.

Method | mAP | NDS | FLOPS
---|---|---|---
PGD [46] | 28.8 | 37.0 | 238.2 + 0
BEVDepth [19] | 33.3 | 40.6 | 120.4 + 74.4
PGD + Local Volume | 33.1 | 40.5 | 238.2 + 25.2

###  5.3 Ablation Study

Effectiveness of Object-wise Local Volume We first show the effectiveness of
the object-wise local volume in the single-frame setting. Since there is no
motion modeling and temporal volume construction, the experiments are
conducted with one iteration in the recurrent step. As shown in Table 6, the
object-wise local volume can improve the proposal detector PGD with a large
margin and achieve comparable performance with BEVDepth, which exploits a
global volume. In addition, the computation cost for the local volume is
relatively small compared to the global 3D volume, enabling us to achieve
multiple iterations and use more preceding frames to boost the performance.
Note that the proposal detector also can be replaced by a more efficient one.

Ablation Study of Object Motion We further validate the influence of different
dynamic object modeling strategies on the detection performance The first
experiment compares the assumed static case with that of using the ground
truth object motion. As shown in Table 7, the model with ground truth object
motion outperforms the assumed static with 4.3% mAP, demonstrating the
necessity of object motion for obtaining accurate temporal correspondence
features. When we replace the ground truth object motion with an estimated
one, it still can bring 2.9% mAP improvements, illustrating the usefulness of
our dynamic objects modeling module.

Table 7: Experimental results of different motion modeling strategies on the
nuScenes validation set. The GT motion is obtained from the ground truth
velocity. 1 past frame is used in temporal modeling.

Setting | mAP | NDS | mAVE
---|---|---|---
Assumed static | 35.0 | 47.1 | 0.37
GT motion | 39.3 | - | -
Pred motion | 37.9 | 52.1 | 0.31

Experiments with Different Iterations In Figure 5, we provide the comparison
of modeling object motion and assumed static with different recurrent
iterations. Benefiting from the BEV features modeling, the two configurations
display almost 2% mAP improvements in the first iterations. In the later
iterations, the improvement in assumed static stops, mainly due to the lack of
accurate temporal features. While with more and more accurate temporal
features, the model with modeling object motion can progressively improve the
detection and motion estimation results.

##  6 Conclusion

This work proposes a novel framework to better leverage temporal information
for camera-only 3D detection by modeling dynamic objects. We first designed an
object-wise local volume to save the computation time and maintain a object-
wise representation for motion and detection modeling. Then we propose a
recurrent module to tackle the challenging motion and location modeling
problem. Specifically, we progressively update the motion and location results
from the concurrently updated corresponding 3D volume features thereon. As the
object motion and location results are tightly coupled in the recurrent stage,
we also demonstrate the framework can naturally achieve 3D object tracking.
Based on dynamic objects motion, our method achieves state-of-the-art
performance on both the nuScenes detection and tracking benchmarks.

## References

  * [1] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  In ICIP, 2016.
  * [2] Garrick Brazil and Xiaoming Liu.  M3d-rpn: Monocular 3d region proposal network for object detection.  In ICCV, 2019.
  * [3] Garrick Brazil, Gerard Pons-Moll, Xiaoming Liu, and Bernt Schiele.  Kinematic 3d object detection in monocular video.  In ECCV, 2020.
  * [4] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In CVPR, 2020.
  * [5] Shaoyu Chen, , Xinggang Wang, Tianheng Cheng, Qian Zhang, Chang Huang, and Wenyu Liu.  Polar parametrization for vision-based surround-view 3d detection.  arXiv:2206.10965, 2022.
  * [6] Tobias Fischer, Yung-Hsu Yang, Suryansh Kumar, Min Sun, and Fisher Yu.  CC-3DT: Panoramic 3d object tracking via cross-camera fusion.  In CORL, 2022.
  * [7] Richard Hartley and Andrew Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, New York, NY, USA, 2 edition, 2003.
  * [8] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [10] Hou-Ning Hu, Yung-Hsu Yang, Tobias Fischer, Trevor Darrell, Fisher Yu, and Min Sun.  Monocular quasi-dense 3d object tracking.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [11] Junjie Huang and Guan Huang.  Bevdet4d: Exploit temporal cues in multi-camera 3d object detection.  arXiv preprint arXiv:2203.17054, 2022.
  * [12] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  arXiv preprint arXiv:2112.11790, 2021.
  * [13] Peixuan Li and Jieyu Jin.  Time3d: End-to-end joint monocular 3d object detection and tracking for autonomous driving.  In CVPR, June 2022.
  * [14] Peiliang Li, Siqi Liu, and Shaojie Shen.  Multi-sensor 3d object box refinement for autonomous driving.  arXiv preprint arXiv:1909.04942, 2019.
  * [15] Peiliang Li, Jieqi Shi, and Shaojie Shen.  Joint spatial-temporal optimization for stereo 3d object tracking.  In CVPR, 2020.
  * [16] Peixuan Li, Huaici Zhao, Pengfei Liu, and Feidao Cao.  Rtm3d: Real-time monocular 3d detection from object keypoints for autonomous driving.  In ECCV, 2020.
  * [17] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li.  Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo.  arXiv preprint arXiv:2209.10248, 2022.
  * [18] Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia.  Unifying voxel-based representation with transformer for 3d object detection.  In NeurIPS, 2022.
  * [19] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li.  Bevdepth: Acquisition of reliable depth for multi-view 3d object detection.  arXiv preprint arXiv:2206.10092, 2022.
  * [20] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In ECCV, 2022.
  * [21] Qing Lian, Peiliang Li, and Xiaozhi Chen.  Monojsg: Joint semantic and geometric cost volume for monocular 3d object detection.  In CVPR, 2022.
  * [22] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Petr: Position embedding transformation for multi-view 3d object detection.  In ECCV, 2022.
  * [23] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Petrv2: A unified framework for 3d perception from multi-camera images.  arXiv:2206.-1256, 2022.
  * [24] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  CVPR, 2022.
  * [25] Zongdai Liu, Dingfu Zhou, Feixiang Lu, Jin Fang, and Liangjun Zhang.  Autoshape: Real-time shape-aware monocular 3d object detection.  In ICCV, 2021.
  * [26] Jiachen Lu, Zheyuan Zhou, Xiatian Zhu, Hang Xu, and Li Zhang.  Learning ego 3d representation as ray tracing.  In ECCV, 2022.
  * [27] Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang.  Delving into localization errors for monocular 3d object detection.  In CVPR, 2021.
  * [28] Nicola Marinello, Marc Proesmans, and Luc Van Gool.  Triplettrack: 3d object tracking using triplet embeddings and lstm.  In CVPRW, 2022.
  * [29] Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu.  Quasi-dense similarity learning for multiple object tracking.  In CVPR, 2021.
  * [30] Ziqi Pang, Zhichao Li, and Naiyan Wang.  Simpletrack: Understanding and rethinking 3d multi-object tracking.  In CVPR, 2021.
  * [31] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon.  Is pseudo-lidar needed for monocular 3d object detection?  In ICCV, 2021.
  * [32] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris M. Kitani, Masayoshi Tomizuka, and Wei Zhan.  Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection.  In The Eleventh International Conference on Learning Representations, 2023.
  * [33] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In ECCV, 2020.
  * [34] Thomas Roddick, Alex Kendall, and Roberto Cipolla.  Orthographic feature transform for monocular 3d object detection.  arXiv preprint arXiv:1811.08188, 2018.
  * [35] Yining Shi, Jingyan Shen, Yifan Sun, Yunlong Wang, Jiaxin Li, Shiqi Sun, Kun Jiang, and Diange Yang.  Srcn3d: Sparse r-cnn 3d surround-view camera object detection and tracking for autonomous driving.  arXiv:2206.14451, 2022.
  * [36] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume.  In CVPR, 2018.
  * [37] Jiaming Sun, Linghao Chen, Yiming Xie, Siyu Zhang, Qinhong Jiang, Xiaowei Zhou, and Hujun Bao.  Disp r-cnn: Stereo 3d object detection via shape prior guided instance disparity estimation.  In CVPR, 2020.
  * [38] Takafumi Taketomi, Hideaki Uchiyama, and Sei Ikeda.  Visual slam algorithms: a survey from 2010 to 2016.  IPSJ Transactions on Computer Vision and Applications, 9, 12 2017\.
  * [39] Chengzhou Tang and Ping Tan.  BA-net: Dense bundle adjustment networks.  In ICLR, 2019.
  * [40] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In ECCV, 2020.
  * [41] Zachary Teed and Jia Deng.  Raft-3d: Scene flow using rigid-motion embeddings.  In CVPR, 2021.
  * [42] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In CVPR, 2019.
  * [43] Kaixuan Wang and Shaojie Shen.  MVDepthNet: real-time multiview depth estimation neural network.  In 3DV, 2018.
  * [44] Tai Wang, Jiangmiao Pang, and Dahua Lin.  Monocular 3d object detection with depth from motion.  In ECCV, 2022.
  * [45] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin.  FCOS3D: Fully convolutional one-stage monocular 3d object detection.  In ICCVW, 2021.
  * [46] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin.  Probabilistic and geometric depth: Detecting objects in perspective.  In CoRL, 2021.
  * [47] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q. Weinberger.  Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.  In CVPR, 2019.
  * [48] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger.  Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.  In CVPR, 2019.
  * [49] Yue Wang, Vitor Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, , and Justin M. Solomon.  Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.  In CoRL, 2021.
  * [50] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  Sensors, 18(10):3337, 2018.
  * [51] Jinrong Yang, En Yu, Zeming Li, Xiaoping Li, and Wenbing Tao.  Quality matters: Embracing quality clues for robust 3d multi-object tracking.  arXiv:2208.10976, 2022.
  * [52] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In ECCV, 2018.
  * [53] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan.  Recurrent mvsnet for high-resolution multi-view stereo depth inference.  In CVPR, 2019.
  * [54] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Center-based 3d object detection and tracking.  CVPR, 2021.
  * [55] Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger.  Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving.  arXiv preprint arXiv:1906.06310, 2019.
  * [56] Tianyuan Zhang, Xuanyao Chen, Yue Wang, Yilun Wang, and Hang Zhao.  Mutr3d: A multi-camera tracking framework via 3d-to-2d queries.  arXiv preprint arXiv:2205.00613, 2022.
  * [57] Yunpeng Zhang, Jiwen Lu, and Jie Zhou.  Objects are different: Flexible monocular 3d object detection.  In CVPR, 2021.
  * [58] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Tracking objects as points.  In ECCV, 2020.
  * [59] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  In arXiv preprint arXiv:1904.07850, 2019.
  * [60] Yunsong Zhou, Yuan He, Hongzi Zhu, Cheng Wang, Hongyang Li, and Qinhong Jiang.  Monocular 3d object detection: An extrinsic parameter free approach.  In CVPR, 2021.

Appendix

##  A Implementation Details

In the main paper, we have introduced our overall multi-camera 3D object
detection and tracking framework and the details of the proposed components.
In this supplemental section, we present the details of the other basic
modules.

###  A.1 Network Architecture

Our framework is built based on BEVDet and BEVDepth, and we follow them to
design the basic modules.

2D Feature Extraction. Given $N$ multi-view images $I\in\mathcal{R}^{N\times
W\times H\times 3}$in each frame, we use a shared 2D backbone to extract the
corresponding features. We adopt the standard ResNet-50 [9] as the backbone
and initialize it with ImageNet pre-trained weights. Then we adopt a modified
Feature Pyramid Network (FPN) [50] to extract the multiple-level features and
the output 2D features are downsampled with the ratio of $\frac{1}{16}$ with
channel size $256$:
$F_{pv}\in\mathcal{R}^{\frac{W}{16}\times\frac{H}{16}\times 256}$.

View Transformation. Our work is the same as BEVDet and BEVDepth that contains
a 2D to 3D view transformation module. Specifically, we first leverage a depth
prediction head to predict the depth probability for each pixel. Then we lift
the 2D features to a 2.5D frustum space via out-product it with the depth
probability. The depth probability range is set as $[0m,60m]$ with grid size
$0.5m$. With the 2.5D frustum features, the 3D features for each local volume
are obtained via utilizing the camera intrinsic to project the 3D grid back to
the frustum and bi-linear sample the corresponding features. As mentioned in
the main paper, we aggregate the 3D volume features along the height dimension
and obtain the corresponding object-wise BEV features
$F^{obj}_{bev}\in\mathcal{R}^{N\times W^{obj}\times H^{obj}\times 256}$, where
$W^{obj}$ and $H^{obj}$ are the object features dimension and set as 28 in the
main setting.

RefineNet. Given the object-wise features extracted based on the proposal 3D
box and motion, RefineNet takes several convolutional neural networks to
extract the object-wise features and estimate the bounding box and motion
residual. Specifically, we first adopt an average pooling layer to aggregate
the 3D features along the height dimension and obtain the BEV features. Then
we filter each object-wise BEV features with 6 basic 2D residual blocks, where
each residual block consists of two 2D convolution layers and a skip
connection module as in ResNet. The channel size of the residual blocks in the
first three layers is 256 and decreases to 64 in the last three layers. Then
we aggregate the features along the spatial dimension via average pooling and
take 4 layers MLP network to estimate the bounding box and motion residuals.

###  A.2 The Tracking Module

In this section, we provide the details of the tracking module that omit in
the paper. Since DORT can estimate tightly coupled object location and motion,
object tracking can be easily achieved via nearest center distances
association [1, 54, 30]. Hence, our tracking module is mainly adapted from the
previous distance-based object tracker [1, 54, 30]. Specifically, the tracking
module contains four parts: Pre-processing, Association, Status Update and
Life-cycle Management.

Pre-processing. Given the detection results, the pre-processing stages mainly
focus on filtering false negative objects. In our work, we first adopt Non-
maximum Suppression to remove the duplicated bounding boxes with the threshold
of 0.1 in terms of 3D IoU. Then we filter out the bounding boxes that the
confidence threshold is lower than 0.25.

Association. This stage associates the detection results in the current frame
with tracklets in the past frame. Specifically, we first utilize the estimated
object motion (velocity) to warp the detection results back to the past frame
and then utilize the L2 distances of object centers to compute the similarity
between the detected objects and the tracklets. Then we utilize the linear
greedy matching strategy to achieve multi-object matching.

Status Update. This stage updates the status of the tracklets. For the
tracklets that do not match with any bounding boxes, we replace it object
center location with the corresponding detection results. For the unmatched
objects, we utilize the estimated object velocity to update its object center
location.

Life-cycle Management. The life-cycle management module controls the “birth”
and “depth” of the tracklets (i.e. birth, depth). Specifically, for the
unmatched bounding boxes, they will be initialized as new tracklets. For the
unmatched tracklets, we remove them when they are consecutive unmatched more
than 2 times.

##  B Ablation Studies

In this section, we provide the additional ablation studies that omit in the
main paper. We will release the code afterward for providing the details of
the methods and reproducing the experimental results. DORT with Different
Proposal Detector. We first show that DORT is agnostic with different proposal
detectors (e.g. PGD [46], BEVDepth [19]). In Table 8, we display the
experimental results of DORT with using PGD and BEVDepth as the proposal
detectors. We can observe that the DORT is insensitive to the proposal
detector and can consistently improve BEVDepth. We Benefiting from the low
computation overhead of BEVDepth in the perspective part and the designed
local volume, DORT also can achieve a more lightweight pipeline for dynamic
object modeling.

Table 8: Experimental results on the nuScenes validation set. 1 past frame is
adopted in the temporal modeling. ∗ denotes the BEV FLOPS from the proposal
detector.

Method | mAP | NDS | Flops
---|---|---|---
PV | BEV
BEVDepth | 35.1 | 47.5 | 120.4 | 94.5
DORT with PGD | 37.9 | 52.1 | 238.2 | 40.2
DORT with BEVDepth | 38.1 | 52.1 | 120.4 | 74.4∗+40.2

Tracking with Semantic Embedding or Geometry Distance. In this work, DORT
achieves 3D object tracking via the nearest centerness association. To have a
more comprehensive comparison of the tracking pipeline designed, we further
provide the comparison of DORT with using semantic embedding to associate
objects. Specifically, we follow previous methods [10] and adopt the widely-
used quasi-dense similarity learning [29] to learn the tracking embedding. We
extract two kinds of embedding features, one is from the perspective-view (PV)
and another is from the bird-eye-view (BEV). In Table 9, we display the
tracking results on the nuScenes tracking set. We can observe that DORT with
geometry distance association can outperform the embedding-based methods by a
large margin. Furthermore, it is also much simpler and more efficient that
does not need to maintain an extra object embedding. Besides, the PV embedding
is worse than the BEV-based embedding, which may be due to the view change in
different cameras.

Table 9: Experimental results on the nuScenes validation set. 1 past frame is
adopted in the temporal modeling.

Method | AMOTA$\uparrow$ | AMOTP$\downarrow$ | MOTAR$\uparrow$
---|---|---|---
PV-Embedding | 36.8 | 1.412 | 44.2
BEV-Embedding | 40.1 | 1.356 | 46.7
DORT (Geometry Distance) | 42.4 | 1.264 | 49.2

##  C Theoretical Analysis of Ignoring Object Motion

In the main paper, we have shown that when ignoring object motion, the
temporal correspondence would derive a biased depth. In this supplementary, we
provide the full details of how ignoring object motion introduces a biased
depth. We denote the camera intrinsic as $K$ and the ego-motion from frame
$t_{0}$ to frame $t_{1}$ as $T_{t_{0}\rightarrow t_{1}}^{ego}$:

$\displaystyle K=\left[\begin{array}[]{cccc}f&0&c_{u}\\\ 0&f&c_{v}\\\ 0&0&1\\\
\end{array}\right],T_{t_{0}\rightarrow
t_{1}}^{ego}=\left[\begin{array}[]{cccc}1&0&0&x^{ego}\\\ 0&1&0&0\\\
0&0&1&z^{ego}\\\ \end{array}\right].$ (26)

Here, $f$ is the camera’s focal length, and $(c_{u},c_{v})$ is the camera
center coordinates in the image. For simplicity, we assume the ego-motion only
contains the translation $(x^{ego},0,z^{ego})$ on the horizontal plane. The
analysis also can be easily extended to a more complicated case that the
motion contains rotation. Given the multiple-view images, temporal-based
methods can utilize photometric or featuremetric similarity to find the
correspondence of pixel $p_{t_{0}}=(u_{t_{0}},v_{t_{0}})$ in the past frame
$t_{0}$ and the pixel $p_{t_{1}}=(u_{t_{1}},v_{t_{1}})$ in the current frame
$t_{1}$.

When we ignore the object motion, the depth $z_{t_{1}}$ of pixel $p_{t_{1}}$
can be recovered as:

$\displaystyle T_{t_{0}\rightarrow
t_{1}}^{ego}\cdot\pi(p_{t_{0}},K)=\pi(p_{t_{1}},K),$ $\displaystyle
z_{t_{1}}\frac{u_{t_{1}}+c_{u}}{f}-x^{ego}=\frac{u_{t_{0}}+c_{u}}{f}(z_{t_{1}}-z^{ego}),$
$\displaystyle
z_{t_{1}}=\frac{z^{ego}(u_{t_{0}}-c_{u})-fx^{ego}}{u_{t_{0}}-u_{t_{1}}},$ (27)

where $\pi$ denotes the projection from 2D image coordinate to 3D camera
coordinate.

But as we showed in the main paper, the moving objects occupy large ratios in
the driving scenarios. For example, when the object contains the translation
$(x^{obj},0,z^{obj})$ in the horizontal plane, the object’s motion can be
represented as

$\displaystyle T^{obj}_{i\rightarrow
j}=\left[\begin{array}[]{cccc}1&0&0&x^{obj}\\\ 0&1&0&0\\\ 0&0&1&z^{obj}\\\
\end{array}\right].$ (31)

With the object motion, the depth $z_{t_{1}}$ of pixel $p_{t_{1}}$ is
recovered as:

$\displaystyle T^{obj}_{t_{0}\rightarrow t_{1}}T_{t_{0}\rightarrow
t_{1}}^{ego}\cdot\pi(p_{t_{0}},K)=\pi(p_{t_{1}},K),$ $\displaystyle
z_{t_{1}}\frac{u_{t_{1}}+c_{u}}{f}-x^{ego}-x^{obj}=\frac{u_{t_{0}}+c_{u}}{f}(z_{t_{1}}-z^{ego}-z^{obj})$
$\displaystyle\hat{z}_{t_{1}}=\frac{(z^{ego}+z^{obj})(u_{t_{0}}-c_{u})-f(x^{ego}+x^{obj})}{u_{t_{0}}-u_{t_{1}}}.$
(32)

From Eq (C) and Eq (C), we can obtain the depth gap for the temporal
correspondence with and without considering object motion:

$\displaystyle\Delta
z=\frac{z^{obj}(u_{t_{0}}-c_{u})-fx^{ego}}{u_{t_{0}}-u_{t_{1}}}.$ (33)

In Figure 6, we also provide a toy example to illustrate that one temporal
correspondence can come from multiple combinations of object depth and motion
(i.e. inaccurate depth with zero motion and accurate depth and GT motion).
This means that if we inaccurately assume that objects are static across
frames, the temporal correspondence would derive a misleading depth.

Figure 6: Different object motion can make the same temporal correspondence
derive different depth.

###  C.1 Ill-posed Problem of Simultaneously Estimating 3D Location and Motion

Although object motion plays a critical role in temporal correspondence,
however, it is non-trivial to estimate it from the monocular images. As shown
in Figure 6, the one correspondence can come from infinite combinations of
location and motion (the location can be the point in the ray
$\overrightarrow{O_{t_{0}}P_{t_{0}}}$ and
$\overrightarrow{O_{t_{1}}P_{t_{1}}}$, and the motion can be the line that
connects the points.) Hence, it is an ill-posed problem that simultaneously
estimates the 3D location and motion from the monocular images. To alleviate
this issue, we leverage the rigid-body assumption for the objects in the
driving scenarios and elaborate more temporal frames with constant velocity
regularization to further constrain the motion.


# Expose Uncertainty, Instill Distrust, Avoid Explanations: Towards Ethical
Guidelines for AI

Claudio S. Pinhanez
IBM Research - Brazil
Rua Tutoia 1157 - São Paulo - SP - Brazil
<EMAIL_ADDRESS>

###### Abstract

In this position paper, I argue that the best way to help and protect humans
using AI technology is to make them aware of the intrinsic limitations and
problems of AI algorithms. To accomplish this, I suggest three ethical
guidelines to be used in the presentation of results, mandating AI systems to
expose uncertainty, to instill distrust, and, contrary to traditional views,
to avoid explanations. The paper does a preliminary discussion of the
guidelines and provides some arguments for their adoption, aiming to start a
debate in the community about AI ethics in practice.

##  1 Machine Computation of Similarity

The goal of this position paper is to explore whether and how some intrinsic
limitations of modern _Artificial Intelligence (AI)_ technology should lead to
ethical guidelines mandating that the presentation of their results must: (1)
expose the underlying uncertainty of the results; (2) foster distrust and
doubt about their appropriateness and accuracy; (3) avoid the use of
explanations which further increase user confidence on the results. I argue
here for those guidelines as a way to safeguard AI users from mistakenly
making decisions based on unfounded beliefs about the accuracy of machines, on
biased data, and on other known ailments of current AI technologies [8].

It is first very important to acknowledge how welcomed the debate around
ethics in AI is, although, in my view, it is still insufficient in scope. The
opacity of the current, successful generation of AI algorithms, based on
_neural networks (NN)_ technology, seem to have brought to light a fundamental
discussion about the use of _information systems (IS)_ in the society, and
particularly for decision-making processes involving its most vulnerable and
discriminated members.

However, the key issues in discussion, such as bias towards individuals and
groups, transparency, explainability, and correctness, apply to a much larger
group of ISs, including ones based on statistical methods, optimization
algorithms, and data science in general [6, 11, 8]. That debate is better
done, and served, considering all kinds of ISs which are often used to
reproduce, and sometimes hide, questionable or deplorable political and social
agendas [6, 11, 8].

That point taken, a couple of issues of AI technology seem to have brought, in
my opinion, the spotlight of ethics discussion from ISs to almost exclusively
on AI. First, the notorious and inherent opacity of _machine learning (ML)_
methods, particular NNs, whose behavior is not comprehensible by human beings,
tends to instill, understandably, fear. Second, the excessive and unfounded
propaganda that the current AI technology is, indeed, intelligent in the human
sense, trumpeted by the tech industry and its pundits, has made people and
society much more concerned about its spread.

As a researcher in AI for more than 30 years, I can not see how current AI
technology can be considered as intelligent, in any human sense. From a
technical point of view, ML algorithms, and particularly NN-based ones, are
simply mimicking, often inaccurately, the human perception of _similarity_.
What a NN algorithms does is to create a complex mathematical function which
maps an input pattern into known patterns, provided as labelled examples by
its developers. Based on those, the algorithm produces a classification token,
the one most similar according to the examples. As brilliantly put by Alan
Blackwell [2], _“… ‘intelligent’ behaviour of machines is no more than human
behaviour reflected back to us.”_ Being able to computer similarity is
essential for intelligence, but not sufficient.

So how to appropriately portray and ethically use a opaque, inscrutable
technology based on the machine replication, often inaccurately, of the human
admirable ability of identifying patterns and perceiving similarity? I propose
here that to ethically use modern AI algorithms we should follow 3 basic
ethical guidelines: it is necessary to expose uncertainty, that is, make their
users aware of the level of imprecision of their results; instill distrust on
their results and predictions, in a systematic and deliberate way; and avoid
explanations, which tend to foster unfounded beliefs of true machine
intelligence in non-technical users. Albeit explainability is often portrayed
as important way to deal with ethical issues, I contend here that is a
misguided effort if intended to help the common user navigate modern AI
technology.

I am not denying here the extraordinary technical advances in the last 10
years which have substantially increased the ability to machines to more
accurately deal with similarity. The early AI of the 1970-1980s was plagued by
brittleness and its effects on the rule-based and logic programming systems of
that era, impeding its application in real problems. The pendulum has swung,
and current AI technology suffers from a chronic softness which has to be
better communicated to its users.

For the rest of the paper, I will use the term _AI_ to mean ML-based
algorithms and, particularly, NN-based ones. Although this is a somewhat
unfair portrayal of the richness of methods created and used by the AI
community and profession, it matches the current use of the term by most
people, enterprises, societal actors, and media.

##  2 Exposing Uncertainty

In a recent meeting at my laboratory, a question surfaced related to what is
the world’s _Gross Domestic Product (GDP)_ , or _Gross World Product (GWP)_. I
typed the question in _Google_ , using Portuguese language, and got the
answer, translated here to English, _“… the total wealth should climb from US$
226 trillion in 2019…”_. The same question, in English, yielded _“In 2020,
global GDP amounted to about 84.54 trillion US dollars,…”_.

The reason for the wild discrepancy was the incorrect matching of the term
_total wealth_ to _global GDP_ in the Portuguese version. Although similar,
they are quite different concepts in economics, a distinction which the
Portuguese seemed to lack but the English version had. The answers, in both
cases, were not portrayed as part of a list of search results, which users
have learned to take with some care. Those statements were provided by the
Google search engine as _the answers_ to the question, in spite the of
incorrect matching of the terms in Portuguese.

The designers of the Google search interface could have conveyed the
uncertainty of their matching algorithms easily by preceding the answer with a
sentence such as _“A related answer to you question may be: …”_ , able to
expose, in a meaningful way to humans, the intrinsic uncertainty of the
algorithm, and of language itself. However, they did not so, as almost all
designers and developers of interfaces which publish results of AI algorithms,
especially for non-technical users.

What I question here is the ethics of not ascertaining and making clear the
uncertainty of an AI algorithm, which often creates a unsubstantiated claim of
correctness which is beyond the technology. In many cases, there seems to be a
deliberate intent to hide the uncertainty to increase the perception of
quality of the ISs. However, omitting the uncertainty takes a completely new
dimension when the result of the algorithm has more impacts on people’s lives,
such as when used to determine a parole claim, the identity of a criminal, the
provision of a home loan, or to stop an automatic car from running over a
pedestrian. To fix this, I propose the following ethical guideline:

Ethical Guideline 1 - Expose Uncertainty: Interfaces with results of an AI
algorithm should properly disclose to its users realistic estimates of the
uncertainty of the algorithm, in easy-to-understand forms.

To effectively implement this guideline, it is important to address two key
terms: “realistic estimates” and “easy-to-understand”. Estimates of
uncertainty are not the same as confidence levels, likelihoods, or even
probabilities, in the rare cases where AI algorithms actually compute them.
Real uncertainty must consider the errors of the algorithm, the labeling
mistakes of the training datasets, and possible new conditions of the moment
when the results are presented. For example, AI algorithms used to predict
stock prices had its real uncertainty increased when the COVID-19 pandemic
started. Similarly, e-commerce recommendations portrayed a lot of nonsensical
suggestions after the lock-downs started, since it was the first time they
were seeing data of entire populations buying all their needs in the Internet.
In practice, factoring in the issues affecting uncertainty is a hard problem,
sometimes harder than the one the AI algorithm is trying to solve.

On the other hand, handling the creation of easy-to-understand portrayals of
uncertainty has been in the scope of the HCI research community for decades,
and should be a key area in Human-Centered AI. For data displays, there is
very rich research in Visual Analytics around uncertainty visualization, with
a set of well-established and well-researched methods [13, 5, 3, 24, 17].
Similarly, there are many ways to convey uncertainty through language, for
instance, by preceding statements with phrases such as _“I believe…”_ or _“I
guess…”_ ; by using uncertainty adjectives and adverbs such as _“approximate”_
and _“about”_ ; and by using informal language [16]. Further investigation is
needed whether the use by machines of such terms and methods indeed produces
similar effects as when used by people, even considering that some previous
work on human perception of computers points towards a positive answer [23,
21].

##  3 Instilling Distrust

It is remarkable that the most remembered moment of the famous _Jeopardy_
competition of 2011, which inaugurated the current AI era, was the one where
the _IBM Watson_ computer made a silly mistake. In the category _“U.S.
Cities”_ , for the prompt _“Its largest airport is named for a World War II
hero; its second largest, for a World War II battle”_ , Watson answered _“What
is Toronto?”_ , instead of the right answer _“What is Chicago”_. If IBM Watson
had lost the match, that mistake would probably have been the definite example
of how machines can not be as intelligent as people. The Watson victory framed
that error in a more benign view, which nevertheless highlights how different,
in fact, the reasoning processes between AI technologies and people are.

Silly errors have been reported in every milestone of AI, as in the more
recent case of _OpenAi GPT-3_. Asked, _“How many eyes does a blade of grass
have?”_ , GPT-3 answered _“A blade of grass has one eye.”_ At the core of the
issue is the reliance on similarity by the technology behind the text
generator. Being able to produce text recognized as natural language is a far
cry from being able to think or even to use language properly. In fact, recent
work has shown that SOTA transformer-based models such as BERT have issues to
deal with even basic syntax usage, including word order [26].

Albeit Watson and GPT-3 are both examples of the amazing advances in AI in the
last years, they are also a testament of their limitations, showcased in their
famous mistakes. I argue here that, in fact, the open nature of their errors
is in fact positive, because it instilled in the public the distrust that both
systems fairly deserve, as systems which we could hardly classify as
intelligent. Unfortunately, the users of the majority of AI systems do not see
their systems failing, and often trust them beyond what they can in fact
deliver. More worryingly, developers and entrepreneurs often work hard to hide
mistakes and overhype the AI systems they build. Letting people make decisions
with wrong or inadequate knowledge about the correctness of systems they use
is wrong and dangerous. This suggests a second guideline:

Ethical Guideline 2 - Instill Distrust: Interfaces displaying results of an AI
algorithm should include mechanisms to foster the user to question, doubt, and
distrust the results and the overall capabilities of the algorithm.

As the examples above suggest, one of the simplest ways to create distrust in
machines seems to be by exposing their errors. In fact, a study on
_algorithmic aversion_ [9] shows that people, after seeing a machine making
errors, prefer a human counterpart even when given evidence that the machine’s
performance is better. Showing and acknowledging mistakes is a practical way
to implement this guideline, and the research in the area has explored many
ways to do so [13, 15, 7, 20].

There is, however, much more research needed on how to inspire distrust and
doubt in machine results, especially when they are part of a larger context.
The research on countering beliefs in _deepfakes_ has showcased how difficult,
in practice, is to raise doubt in people when the information provider does
not want to [4, 19, 22, 14, 25]. Having an ethical guideline mandating the
fostering of distrust seems to be needed, as a way to discourage willful
omission.

##  4 Explaining AI Algorithms Considered Harmful

An enormous body of work has been created in the last years to develop methods
to explain the workings of AI algorithms [1, 12, 27, 10]. However, at best,
explainability will help users understand how an AI algorithm is computing its
similarities. But, even for human beings, are explanations about why things
are similar useful? If people are asked very basic similarity questions, such
as why two objects seem to have the same color, they often can not explain or,
worse, provide technical explanations (such as color spectrum) which has
little to do with their actual perception of similarity.

Moreover, AI algorithms are often trained with datasets built by crowdsourced
workers [8], therefore containing the workers own perceptual mistakes and bias
of how sentences or images are similar, or how good an answer to a question
is. Those workers, most often than not, do not rely on systematic reasons to
make their choice: for speed, they often answer based on perception.
Similarly, when professional data is used, such is in medical diagnosis, the
reasoning behind those choices is often a combination of ill-defined
professional rules and the perceived similarity to cases from the past.

The key argument is that the replication of human perception of similarity,
the greatest achievement of current AI technology, mimics a process which
human beings can hardly explain, using data which is not explainable itself.
This poses the question of how can then a machine explain its results besides
stating that it is replicating a non-explainable process of human beings?

Besides that, the only explanations which can be deduced from most AI
algorithms are of numeric nature, such as an input is close in a very complex
space to a certain group of examples; or that there is a correlation between
inputs, examples, and outputs. After the debacle of using scientific data to
communicate to people basic facts in the COVID-19 pandemic [22], it is hard to
imagine that using probabilities and correlations is likely to work to make
people understand AI systems which use much more complex and sophisticated
mathematical methods. Explaining the workings of an AI system to help users
understand its limitations and inaccuracies seems to be a doomed objective.

Moreover, providing explanations to users, in fact, seem to go against the two
ethical guidelines proposed above. Reasonably phrased explanations are likely
to decrease the perception of uncertainty and increase trust. This is
summarized in the following guideline:

Ethical Guideline 3 - Avoid Explanations: Interfaces with results of an AI
algorithm should not provide explanations which increase the confidence or
trust of users on their results.

It is important to acknowledge that explainability has an important value for
professional use, when creating, developing, and approving systems. Similarly,
better methods to understand the workings of AI algorithms can be very
important for auditing, regulating, and certifying AI systems. But I believe
it is wrong to expect that the kind of explanations that can be derived from
AI systems will help users of AI to understand its limitations and when to
believe or not in AI-produced results.

##  5 Towards a Trustworthy Use of AI

The basic proposal of this paper is that the best way to make AI trustworthy
is to act to decrease the users trust in it. I argue that, since modern AI
algorithms are essentially human-behavior replicating machines [2], trained on
noisy human-similarity data [18, 8], they are doomed to produce uncertain,
incorrect, and inexplicable results. Therefore, the ethical way to present
their results is to highlight those characteristics, what was captured in the
three proposed ethical guidelines: expose uncertainty, instill distrust, and
avoid explanations.

This is certainly a provocative and controversial statement, which also
includes many open research questions, such as the techniques and methods to
implement the guidelines, and how human beings will actually perceive and deal
with issues of uncertainty and trust in information from machines [9]. My hope
with this paper is to present those ideas to provoke and foster a discussion
in the community with the ultimate goal of producing ethical guidelines for
the use of AI.

## References

  * [1] Amina Adadi and Mohammed Berrada.  Peeking inside the black-box: a survey on explainable artificial intelligence (xai).  IEEE access, 6:52138–52160, 2018.
  * [2] Alan F Blackwell.  Objective functions:(in)humanity and inequity in artificial intelligence.  In Geoffrey E. R. Lloyd and Aparecida Vilaça, editors, Science in the Forest, Science in the Past, chapter 9, pages 191–208. Hau Books, Chicago,IL, 2020.
  * [3] Georges-Pierre Bonneau, Hans-Christian Hege, Chris R Johnson, Manuel M Oliveira, Kristin Potter, Penny Rheingans, and Thomas Schultz.  Overview and state-of-the-art of uncertainty visualization.  In Scientific Visualization, pages 3–27. Springer, 2014.
  * [4] Danah Boyd.  Did media literacy backfire?  Journal of Applied Youth Studies, 1(4):83–89, 2017.
  * [5] Ken Brodlie, Rodolfo Allendes Osorio, and Adriano Lopes.  A review of uncertainty in data visualization.  Expanding the frontiers of visual analytics and visualization, pages 81–109, 2012.
  * [6] Simone Browne.  Dark matters.  Duke University Press, 2015.
  * [7] Erin K Chiou and John D Lee.  Trusting automation: Designing for responsivity and resilience.  Human Factors, page 00187208211009995, 2021.
  * [8] Kate Crawford.  The Atlas of AI.  Yale University Press, 2021.
  * [9] Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey.  Algorithm aversion: People erroneously avoid algorithms after seeing them err.  Journal of Experimental Psychology: General, 144(1):114, 2015.
  * [10] Filip Karlo Došilović, Mario Brčić, and Nikica Hlupić.  Explainable artificial intelligence: A survey.  In 2018 41st International convention on information and communication technology, electronics and microelectronics (MIPRO), pages 0210–0215. IEEE, 2018.
  * [11] Virginia Eubanks.  Automating Inequality: How High-tech Tools Profile, Police, and Punish the Poor.  St. Martin’s Press, 2018.
  * [12] Juliana J Ferreira and Mateus S Monteiro.  What are people doing about xai user experience? a survey on ai explainability research and practice.  In International Conference on Human-Computer Interaction, pages 56–73. Springer, 2020.
  * [13] Henning Griethe, Heidrun Schumann, et al.  The visualization of uncertain data: Methods and problems.  In SimVis, pages 143–156, 2006.
  * [14] Michael Hameleers, Thomas E Powell, Toni GLA Van Der Meer, and Lieke Bos.  A picture paints a thousand lies? the effects and mechanisms of multimodal disinformation and rebuttals disseminated via social media.  Political Communication, 37(2):281–301, 2020.
  * [15] Wenkai Han and Hans-Jörg Schulz.  Beyond trust building—calibrating trust in visual analytics.  In 2020 IEEE workshop on trust and expertise in visual analytics (TREX), pages 9–15. IEEE, 2020.
  * [16] Francis Heylighen and Jean-Marc Dewaele.  Formality of language: definition, measurement and behavioral determinants.  Interner Bericht, Center “Leo Apostel”, Vrije Universiteit Brüssel, 4, 1999.
  * [17] Jessica Hullman, Xiaoli Qiao, Michael Correll, Alex Kale, and Matthew Kay.  In pursuit of error: A survey of uncertainty visualization evaluation.  IEEE transactions on visualization and computer graphics, 25(1):903–913, 2018.
  * [18] Tom Kelleher.  Conversational voice, communicated commitment, and public relations outcomes in interactive online communication.  Journal of communication, 59(1):172–188, 2009.
  * [19] David M. J. Lazer, Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zittrain.  The science of fake news.  Science, 359(6380):1094–1096, 2018.
  * [20] Eva Mayr, Nicole Hynek, Saminu Salisu, and Florian Windhager.  Trust in information visualization.  In TrustVis@ EuroVis, pages 25–29, 2019.
  * [21] Clifford Ivar Nass and Scott Brave.  Wired for speech: How voice activates and advances the human-computer relationship.  MIT Press, Cambridge,MA, 2005.
  * [22] Gordon Pennycook, Jonathon McPhetres, Yunhao Zhang, and David Rand.  Fighting covid-19 misinformation on social media: Experimental evidence for a scalable accuracy nudge intervention.  Psychological Science, 0(0), 2020.
  * [23] Byron Reeves and Clifford Nass.  The media equation: How people treat computers, television, and new media like real people.  Cambridge University Press, Cambridge, UK, 1996.
  * [24] Dominik Sacha, Hansi Senaratne, Bum Chul Kwon, Geoffrey Ellis, and Daniel A Keim.  The role of uncertainty, awareness, and trust in visual analytics.  IEEE transactions on visualization and computer graphics, 22(1):240–249, 2015.
  * [25] Norbert Schwarz, Eryn Newman, and William Leach.  Making the truth stick & the myths fade: Lessons from cognitive psychology.  Behavioral Science & Policy, 2(1):85–95, 2016.
  * [26] Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams.  UnNatural Language Inference.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7329–7346, Online, August 2021. Association for Computational Linguistics.
  * [27] Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, and Jun Zhu.  Explainable ai: A brief survey on history, research areas, approaches and challenges.  In CCF international conference on natural language processing and Chinese computing, pages 563–574. Springer, 2019.


[a]M. Bonesini

# One inch LaBr3:Ce detectors, with temperature control and improved time
resolution for low energy X-rays spectroscopy

R. Benocci  R. Bertoni  A. Abba  F. Caponio  A. Menegolli  M.C. Prata  M.
Rossella  R. Rossini

###### Abstract

Large area LaBr3:Ce detectors with a SiPM array readout have been developed
for the FAMU experiment at RAL. The aim was to have a good energy resolution
for low energy X-rays detection ($\sim 100$ keV) and good timing properties of
the signal pulse. Sixteen 1" detectors and twelve 1/2" detectors have been
presently installed in the FAMU experiment and are taking data since March
2023, at RIKEN RAL Port 1.

##  1 Introduction

LaBr3:Ce crystals, with Photomultiplier (PMT) or SiPM readout, have
applications in many fields from medical imaging, to homeland security and
gamma-ray astronomy [1]. A readout based on SiPMs or SiPM arrays, instead of
PMTs, allows the construction of compact detector, that may be used in
external magnetic fields, avoiding the use of complicate shielding [2]. Good
energy resolutions: around $3\%$ at the Cs137 peak (661.7 keV), comparable
with the ones obtained with PMTs [3], have been recently obtained for large
detectors (area > 1 $cm^{2}$) with a SiPM array readout [4], [5], [6]. Our
efforts aimed at the development of detectors with large area (initially 1/2")
with both a good energy resolution, down to 100 keV, and good timing
properties, as shown in reference [7]. The purpose was to contribute to the
X-rays detection system of the FAMU experiment [8] at the RIKEN-RAL muon
facility. The experimental target of FAMU is the precision measurement of the
hyperfine splitting (HFS) of the $\mu$p ground state and thus of the proton
Zemach radius, with impinging muons.

##  2 Detectors’ construction

The developed detectors are based on 1" round LaBr3:Ce crystals (0.5 " thick)
read by Hamamatsu S14161-6050AS-04 square 1" SiPM array and are shown in
figure 1.

Figure 1: Left panel: image of a complete detector. Middle panel: images of
some details of the crystal holder; h) with PCB inside. The two SAMTEC
connectors for SiPM array mounting are shown. k) with mounted SiPM array; m)
with crystal inside. Right panel: exploded view of a 1" detector. From top to
bottom: a) heat dissipator, b) detector base; c) gap filler, d) PCB , e) PCB
holder, f) crystal holder, g) support to hang a detector’s base (b) from the
LaBr3:Ce crown support.

All the needed mechanics items for the detector mounting were realized with a
3D printer. Further details are reported in references [7] and [10].

Figure 2: Linearity (top panel) and FWHM energy resolution (bottom panel) for
a typical 1" detector both with parallel ganging and the new 4-1 circuit from
Nuclear Instruments.

The ouput signals of four nearby 6x6 mm2 SiPM are grouped together and for
them an individual pole-zero compensation and amplification is applied.
Afterwords, the four signals of the sub-arrays (2x2 cells) are summed together
and inverted, to give a positive output. All is realized with a custom circuit
realized with Nuclear Instruments srl, see reference [9] for additional
details. In this way a good energy resolution is obtained and pulse timing
(risetime/falltime) is reduced by a factor 2X. Figure 2 shows the FWHM energy
resolution and the linearity for a typical 1" detector, both with the standard
parallel ganging for the 16 cells of the used SiPM array and with the custom
4-1 circuit from Nuclear Instruments srl. No deterioration of FWHM energy
resolution is evident. The well-known drift of the SiPM gain with temperature
is corrected online via a NIM custom module, based on CAEN A7585D power supply
modules, with temperature feedback. In the range 10-35 ∘C the effect on pulse
height (P.H.) variation at the 137Cs photo-peak is reduced from 40 % to 7 %,
see references [10], [11] for additional details.

##  3 Experimental Results

Laboratory tests were performed at INFN MIB with exempt sources from Spectrum
Techniques, covering the energy range 80-1274.5 keV (109Cd - 22Na). Detectors
under test were put inside a Memmert IPV30 climatic chamber, with a precision
of $\pm 0.1^{\circ}$C on the temperature control. Detectors’ signals were then
fed into a CAEN VME V1730 FADC and then acquired by the FAMU custom DAQ, via a
CAEN V2718 interface. Output ROOT ntuples are then processed for data
analysis.

Figure 3: Distribution of FWHM energy resolution for the used 1" detectors at
the 137Cs peak (top panel), at the 57Co peak (middle panel) and at the 109Cd
peak (bottom panel).

FWHM energy resolution (in $\%$) and 10-90 $\%$ risetime/falltime for the
sixteen 1" detectors used in the FAMU experiment are shown in figure 3 and 4.
The main experimental challenge in the FAMU experiment is the detection of
low-energy X-rays around 130 keV separating a "delayed" signal component
(after 600 ns) from a prompt one.

Figure 4: Distribution of the 10-90 $\%$ risetime-falltime for the used 1"
detectors.

The reported results: especially the $\sim 8\%$ energy resolution at the 57Co
peak and the signal falltime ($\sim 150$ ns) show that the required target may
be reached [7], [9]. Prelimary data taken at RIKEN RAL with 55 MeV muons from
March 2023 confirm this assumption.

###  3.1 Installation at RIKEN RAL

In the 2023 run configuration for the FAMU experiment at RIKEN-RAL, sixteen 1"
and twelve 1/2" detectors, with SiPM readout, were installed at RAL in the
X-ray detection system. They are arranged in three crowns : upstream (ten 1"
detectors), central (six 1" detectors) and downstream (twelve 1/2" detectors),
as shown in figure 5. In addition six old 1" detectors with a PMT readout [12]
are used in the central crown.

Figure 5: Drawing of the FAMU X-rays system, used in the 2023 run, with (a)
the beam hodoscope in front of a Pb collimator, (b) the upstream crown of 1"
LaBr3:Ce detectors; (c) the central crown of detectors: based on six old
detectors read by PMTs [12] and six detectors read by SiPM arrays (visible
near the bottom). All are of 1" size. (d) the downstream crown of 1/2"
Ce:LaBr3 detectors.

With the 4-1 innovative circuit from Nuclear Instruments srl, that divide the
readout of a 1" SiPM array into four, a good compromise in the optimisation of
energy resolution and pulse timing is obtained. The major drawback of this
solution is the increase of heat dissipation, due to the introduction of
several Texas Instruments OPA695 amplifiers: about 1 W. As the working
environment at RIKEN-RAL is kept at constant temperature (20 ∘C) by air-
conditioning, a simple passive heat dissipation is enough for proper
operations.

The main characteristics of the Ce:LaBr3 detectors used in the FAMU
experiment, as obtained from laboratory measurements, are resumed in table 1.
10-90 $\%$ risetime (falltime) are measured with a Cs137 source, using a
Lecroy 1 GHZ scope.

Table 1: FWHM energy resolution and timing characteristics of the Ce:LaBr3 detectors used in the FAMU experiment at RIKEN-RAL  | risetime (ns) | falltime (ns) | resolution % | resolution % | resolution %
---|---|---|---|---|---
|  |  | Cs137 | Co57 | Ce109
|  |  | (662 keV) | (122 keV) | (88 keV)
1/2" detectors | 42.8 $\pm$ 4.7 | 372.4 $\pm$ 17.4 | 3.27 $\pm$ 0.11 | 8.44 $\pm$ 0.63 | 10.63 $\pm$ 1.16
1" detectors | 29.3 $\pm$ 1.5 | 147.1 $\pm$ 12.8 | 3.01 $\pm$ 0.16 | 7.93 $\pm$ 0.38 | 9.82 $\pm$ 0.54

The worse timing properties of the 1/2" detectors, as compared to the 1" ones,
are mainly due to the adoption of a standard parallel ganging instead of the
4-1 solution from Nuclear Instruments and probably to a different Ce
concentration. Data were taken since March 2023 with a 55 MeV/c muon beam at
Port 1 at RIKEN RAL and a preliminary analysis is under way. No major issues
have been encountered up to now. Preliminary results on the reconstructed
muonic Ag peak ($\sim 142$ keV) confirm our laboratory ones.

##  4 Conclusions

Good FWHM energy resolution is obtained with 1" LaBr3:Ce crystals read by
Hamamatsu S14161-6050AS-04 SiPM arrays. Resolutions better than $3\%(8\%)$ are
obtained at the Cs137 (Co57) peak in laboratory tests. The use of the
innovative 4-1 circuit from Nuclear Instruments allowed a factor two reduction
of signal risetime (falltime), as respect to the conventional solution with
parallel ganging. Detectors are presently installed in the FAMU apparatus at
RIKEN RAL.

## Acknowledgements

We would like to thank all members of the FAMU collaboration for friendly
discussions, in particular Andrea Vacchi, Ludovico Tortora and E. Mocchiutti.
We acknowledge also the help in detectors’ mounting and the realization of the
test setup of Luca Pastori, Nuclear Instruments srl, Roberto Gaigher, Stefano
Banfi and Giancarlo Ceruti, INFN Milano Bicocca mechanics workshop and of
Maurizio Perego , INFN Milano Bicocca, for SPICE simulation of the used
electronic circuits.

## References

  * [1] W.W. Moses, K.S Shah, _Potential of RbGd 2Br7:Ce, LaBr3:Ce and LuI3:Ce in nuclear medical imaging_, Nucl. Instr. Meth. 537 (2005) 317;
G. Zentai, _X-ray imaging for homeland security_ , IEEE Int. Workshop on
Imaging Systems and Techniques, Chania, Greece, 2008, p 1-6;
A. Gostojic et al., _Characterization of LaBr 3:Ce and CeBr3 calorimeter
modules for 3D imaging in gamma-ray astronomy_, Nucl. Instr. Meth A832 (2016)
24.

  * [2] M. Bonesini et al., _Behaviour in Magnetic Fields of Fast Conventional and Fine-Mesh Photomultipliers_ , Nucl. Instrum. Meth. A 693 (2012), 130.
  * [3] R. Pani et al.,_Gamma-ray spectroscopy with LaBr3:Ce scintillation crystal coupled to an ultra high quantum efficiency PMT_ , IEEE Nucl. Science Symp., Dresden, 2008, pp. 2462-2466, // E.M. Gandolfo et al., _Response of the G-NUMEN LaBr3(Ce) Detectors to High Counting Rates_ , Instruments 2023, 7, 28
  * [4] D. Di Vita et al., _A 144-SiPM 3" LaBr 3 readout module for PMT’s replacement in Gamma spectroscopy_, Nucl. Instr. Meth. A1040 (2022) 167179.
  * [5] O. Poleshchuk et al.,_Performance tests of a LaBr 3:Ce detector coupled to a SiPM array and the GET electronics for $\gamma$-ray spectroscopy in a strong magnetic field_, Nucl. Instr. Meth. A987 (2022) 164863.
  * [6] M. Bonesini et al., _Detection of low-energy X-rays with 1/2 and 1 inch LaBr 3:Ce crystals read by SiPM arrays_, PoS (EPS-HEP 2021) 770.
  * [7] M. Bonesini et al., _Large area LaBr 3:Ce crystals read by SiPM arrays with improved timing and temperature gain drift control_, Nucl. Instr. Meth. A1046 (2023) 167677.
  * [8] A. Vacchi et al., _Measuring the size of the proton_ , SPIE Newsroom (2012), DOI:10.1117/2/1201207.004274;
C. Pizzolotto et al., _The FAMU experiment: muonic hydrogen high precision
spectroscopy studies_ , EPJ A56 (2020) 7,185 ;
M. Bonesini, _The FAMU experiment at RIKEN RAL for a precise measurement of
the proton radius_ , PoS EPS-HEP2019 (2019) 132.

  * [9] M. Bonesini et al., _Improving the time resolution of large area LaBr 3:Ce detectors with SiPM array readout_, Condens. Matter 2023, 8,99; doi.org/10.3390/condmat8040099.
  * [10] M. Bonesini et al., _Ce:LaBr 3 crystals with SiPM array readout and temperature control for the FAMU experiment at RAL_, JINST 15 (2020) 05,C05065.
  * [11] M. Bonesini et al., _Online control of the gain drift with temperature of SiPM arrays used for the readout of LaBr 3:Ce crystals_, JINST 17 (2022), C10004.
  * [12] A. Adamczack at al., _Steps towards the hyperfine splitting measurement of the muonic hydrogen ground state: pulsed muon beam and detection system characterization_ , JINST 11 (2016)05, P05007