11institutetext: TuSimple 22institutetext: Computational Imaging Lab.,
University of Central Florida
22email<EMAIL_ADDRESS>22email:
<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>
# CenterFormer: Center-based Transformer for 3D Object Detection

Zixiang Zhou *1*122 Xiangchen Zhao 11
Yu Wang 11 Panqu Wang 11 Hassan Foroosh 22

###### Abstract

Query-based transformer has shown great potential in constructing long-range
attention in many image-domain tasks, but has rarely been considered in LiDAR-
based 3D object detection due to the overwhelming size of the point cloud
data. In this paper, we propose CenterFormer, a center-based transformer
network for 3D object detection. CenterFormer first uses a center heatmap to
select center candidates on top of a standard voxel-based point cloud encoder.
It then uses the feature of the center candidate as the query embedding in the
transformer. To further aggregate features from multiple frames, we design an
approach to fuse features through cross-attention. Lastly, regression heads
are added to predict the bounding box on the output center feature
representation. Our design reduces the convergence difficulty and
computational complexity of the transformer structure. The results show
significant improvements over the strong baseline of anchor-free object
detection networks. CenterFormer achieves state-of-the-art performance for a
single model on the Waymo Open Dataset, with 73.7% mAPH on the validation set
and 75.6% mAPH on the test set, significantly outperforming all previously
published CNN and transformer-based methods. Our code is publicly available at
https://github.com/TuSimple/centerformer

###### Keywords:

LiDAR point cloud, 3D Object detection, Transformer, Multi-frame fusion

11footnotetext: Work done during an internship at TuSimple

##  1 Introduction

Figure 1: The comparison of CenterFormer with RCNN-style detector. RCNN
aggregates point or grid features in RoI, while CenterFormer can learn object-
level contextual information and long range features through an attention
mechanism.

LiDAR is an important sensing and perception tool in autonomous driving due to
its ability to provide highly accurate 3D point cloud data of the scanned
environment. LiDAR-based 3D object detection aims to detect the bounding boxes
of the objects in the LiDAR point cloud. Compared to image-domain object
detection, the scanned points in LiDAR data may be sparse and irregularly
spaced depending on the distance from the sensor. Most recent methods rely on
discretizing the point clouds into voxels [59, 49] or projected bird’s eye
view (BEV) feature maps [18] to use 2D or 3D convolution networks. Sometimes,
it requires a second stage RCNN [11]-style refinement network to compensate
for the information loss in the voxelization. However, current two-stage
networks [34, 53] lack contextual and global information learning. They only
use the local features of the proposal (RoI) to refine the results. The
features in other boxes or neighboring positions that could also be beneficial
to the refinement are neglected. Moreover, the environment of the autonomous
driving scene is not stationary. The local feature learning has more
limitations when using a sequence of scans.

In the image domain, transformer encoder-decoder structure has become a
competitive method for the detection [4, 64] and segmentation [43, 2] tasks.
The transformer is able to capture long-range contextual information in the
whole feature map and different feature domains. One of the most
representative methods is DETR [4], which uses the parametric query to
directly learn object information from an encoder-decoder transformer. DETR is
trained end-to-end as a set matching problem to avoid any handcrafting
processes like non-maximum suppression (NMS). However, there are two major
problems in the DETR-style encoder-decoder transformer network: First, the
computational complexity grows quadratically as the input size increases. This
limits the transformer to take only low-dimensional features as input which
leads to low performance on small objects. Second, the query embedding is
learned through the network so that the training is hard to converge.

Can we design a transformer encoder-decoder network for the LiDAR point cloud
in order to better perceive the global connection of point cloud data?
Considering the sheer size of LiDAR point cloud data, and the relatively small
sizes of objects to be detected, voxel or BEV feature map representations need
to be large enough to keep the features for such objects to be separable. As a
result, it is impractical to use the transformer encoder structure on the
feature map due to the large input size. In addition, if we use a large
feature map for the transformer decoder, the query embedding is also difficult
to focus on meaningful attention during training. To mitigate these converging
problems, one solution is to provide the transformer with a good initial query
embedding and confine the attention learning region to a smaller range. In the
center-based 3D object detection network [53], the feature at the center of an
object is used to capture all object information, hence the center feature is
a good substitute for the object feature embedding. Multi-scale image pyramid
and deformable convolution [16] are two common methods to increase the
receptive field of the feature learning without significantly increasing the
complexity. Some recent works [64, 15] apply these two methods in the
transformer networks.

Taking the aforementioned aspects into consideration, we propose a center-
based transformer network, named Center Transformer (CenterFormer), for 3D
object detection. Specifically, we first use a standard voxel-based backbone
network to encode the point cloud into a BEV feature representation. Next, we
employ a multi-scale center proposal network to convert the feature into
different scales and predict the initial center locations. The feature at the
proposed center is fed into a transformer decoder as the query embedding. In
each transformer module, we use a deformable cross attention layer to
efficiently aggregate the features from the multi-scale feature map. The
output object representation then regresses to other object properties to
create the final object prediction. As shown in Figure 1, our method can model
object-level connection and long-range feature attention. To further explore
the ability of the transformer, we also propose a multi-frame design to fuse
features from different frames through cross-attention. We test CenterFormer
on the large scale Waymo Open Dataset [38] and the nuScenes dataset [3]. Our
method outperforms the popular center-based 3D object detection networks which
are dominant on public benchmarks by a large margin, achieving state-of-the-
art performance, with 73.7% and 75.6% mAPH on the waymo validation and test
sets, respectively. The contributions of our method can be summarised as
follows:

  * •

We introduce a center-based transformer network for 3D object detection.

  * •

We use the center feature as the initial query embedding to facilitate
learning of the transformer.

  * •

We propose a multi-scale cross-attention layer to efficiently aggregate
neighboring features without significantly increasing the computational
complexity.

  * •

We propose using the cross-attention transformer to fuse object features from
different frames.

  * •

Our method outperforms all previously published methods by a large margin,
setting a new state-of-the-art performance on the Waymo Open Dataset.

##  2 Related Work

###  2.1 LiDAR-based 3D Object Detection

Compared to the well-established point cloud processing networks like PointNet
[31] and PointNet++ [32], most recent LiDAR detection and segmentation methods
[59, 18, 56, 60, 63] voxelize the point cloud in a fixed 3D space into a
BEV/voxel representation and use conventional 2D/3D convolutional networks to
predict the 3D bounding boxes. Other methods [46, 1, 39, 9] detect the objects
on a projected range image. There are also some methods that use hybrid
features along with the voxel network [34, 30, 51], and combine multi-view
features in the voxel feature representation [34]. VoxelNet [59] uses a
PointNet inside each voxel to encode all points into a voxel feature. This
feature encoder later became an essential method in voxel-based point cloud
networks. PointPillar [18] proposes the pillar feature encoder to directly
encode the point cloud into the BEV feature map so that only 2D convolution is
needed in the network.

Similar to image object detection, 3D object detection methods can be divided
into anchor-based [36, 34, 18, 49] and anchor-free [53, 10] methods. Anchor-
based methods detect objects through a classification of all predefined object
anchors, while anchor-free methods generally consider objects as keypoints and
find those keypoints at the local heatmap maxima. Even though anchor-based
methods can achieve a good performance, they rely heavily on hyper-parameter
tuning. On the other hand, as the anchor-free methods become more prevailing
in image-domain tasks, many 3D and LiDAR works have adopted the same design
and show a more efficient and competitive performance. Many works [34, 20, 6,
24] also require an RCNN-style second stage refinement. Feature maps for each
bounding box proposal are aggregated through RoIAlign or RoIPool. CenterPoint
[53] detects objects using center heatmap and regresses other bounding box
information using center feature representation.

Most methods directly concatenate points from different frames based on the
ego-motion estimation to use the multi-frame information. This assumes the
model can align object features from different frames. However, independently
moving objects cause misalignment of features across frames. Recent multi-
frame methods [14, 52] use an LSTM or a GNN module to fuse the previous state
feature with the current feature map. 3D-MAN [50] uses a multi-frame alignment
and aggregation module to learn the temporal attention of predictions from
multiple frames. The feature of each box is generated from the RoI pooling.

###  2.2 Vision Transformer

Originally proposed in the Natural Language Processing (NLP) community,
transformer [40] is becoming a competitive feature learning module in computer
vision. Compared to traditional CNN, the transformer has a bigger receptive
field, and feature aggregation is based on the response learned directly from
pairwise features. A transformer encoder [7, 54, 15] usually serves as a
replacement for the convolution layer in the backbone network. Meanwhile, the
transformer decoder uses high-level query feature embedding as the input and
extracts features from feature encoding through cross-attention, which is more
common in detection and segmentation tasks [4, 64, 58, 43]. DETR [4] uses a
transformer encoder-decoder structure to predict objects from learned query
embedding. Deformable DETR [64] improves the DETR training through a
deformable attention layer. Some recent methods [22, 26, 55] show that DETR is
easier to converge using guidance like anchor boxes.

###  2.3 3D Transformer

An important design in transformer structure is the position embedding due to
permutation invariance of the transformer input. However, 3D point clouds
already have position information in them, which leads to deviance in the
design of 3D transformers. Point transformer [57] proposes a point transformer
layer in the PointNet structure, where the position embedding in the
transformer is the pairwise point distances. 3DETR [27] and [23] use a DETR-
style transformer decoder in the point cloud except that the query embedding
in the decoder is sampled from Farthest Point Sampling (FPS) and learned
through classification. Voxel Transformer [25] introduces a voxel transformer
layer to replace the sparse convolution layer in the voxel-based point cloud
backbone network. SST [8] uses a Single-stride Sparse Transformer as the
backbone network to prevent information loss in downsampling of the previous
3D object detector. CT3D [33] uses a transformer to learn a refinement of the
initial prediction from local points. In contrast to the above methods, our
CenterFormer tailors the DETR to work on LiDAR point clouds with lower memory
usage and faster convergence. Moreover, CenterFormer can learn both object-
level self-attentions and local cross-attentions without requiring a first-
stage bounding box prediction.

##  3 Method

###  3.1 Preliminaries

Center-based 3D Object Detection is motivated by the recent anchor-free image-
domain object detection methods [19, 17]. It detects each object as a center
keypoint by predicting a heatmap on the BEV feature map. Given the output of a
common voxel point cloud feature encoder $M\in R^{h*w*c}$, where $h$ and $w$
are the BEV map size and $c$ is the feature dimension, center-based LiDAR
object detection predicts both a center heatmap $H\in R^{h*w*l}$ and the box
regression $B\in R^{h*w*8}$ through two separated heads. Center heatmap $H$
has $l$ channels, one for each object class. In training, the ground truth is
generated from the Gaussian heatmap of the annotated box center. Box
regression $B$ contains 8 object properties: the grid offset from the
predicted grid center to the real box center, the height of the object, the 3D
size, and the yaw rotation angle. During the evaluation, it takes the class
and regression predictions at the top $N$ highest heatmap scores and uses NMS
to predict the final bounding box.

Transformer Decoder aggregates features from the source representation to each
query based on the query-key pairwise attention. Each transformer module
consists of three layers: A multi-head self-attention layer, a multi-head
cross-attention layer, and a feed-forward layer. In each layer, there is also
a skip connection that connects the input and the output features and layer
normalization. Let $f^{q}$ and $f^{k}$ be the query feature and key feature.
The multi-head attention can be formulated as:

$f_{i}^{out}=\sum_{m=1}^{M}W_{m}[\sum_{j\in\Omega_{j}}\sigma(\frac{Q_{i}K_{j}}{\sqrt{d}})\cdot
V_{j}]$ (1)
$Q_{i}=f^{q}_{i}W_{q}+E^{pos}_{i},K_{j}=f^{k}_{j}W_{k}+E^{pos}_{j},V_{j}=f^{k}_{j}W_{v}$
(2)

where $i$ and $j$ are the indices of query feature and source feature
respectively, $m$ is the head index, $\Omega_{j}$ is the set of attending key
features, $\sigma$ is the softmax function, $d$ is the feature dimension,
$E^{pos}$ is the position embedding and $W$ is the learnable weight. In the
self-attention layer, the query feature and the key feature come from the same
set of query feature embedding, while in the cross-attention layer, the set of
key features is the source feature representation.

Figure 2: The overall architecture of CenterFormer. The network consists of
four parts: a voxel feature encoder that encodes the raw point cloud into a
BEV feature representation, a multi-scale center proposal network (CPN), the
center-based transformer decoder, and a regression head to predict the
bounding box.

###  3.2 Center Transformer

The architecture of our model is illustrated in Figure 2. We use a standard
sparse voxel-based backbone network [53] to process each point cloud into a
BEV feature representation. We then encode the BEV feature into a multi-scale
feature map and predict the center proposals. The proposed centers are then
used as the query feature embedding in a transformer decoder to aggregate
features from other centers and from multi-scale feature maps. Finally, we use
a regression head to predict the bounding box at each enhanced center feature.
In our multi-frame CenterFormer, the last BEV features of frames are fused
together in both the center prediction stage and the cross-attention
transformer.

Figure 3: Illustration of the cross-attention layer. (Left) Multi-scale cross-
attention. (Right) Multi-scale deformable cross-attention.

####  3.2.1 Multi-scale Center Proposal Network

A DETR-style transformer encoder requires the feature map to be compressed
into a small size so that the computation cost is acceptable. This makes the
network lose fine-grained features that are crucial for the detection of small
objects, which typically occupy $<1\%$ of the space in the BEV map. Therefore,
we propose a multi-scale center proposal network (CPN) to replace the
transformer encoder for the BEV feature. In order to prepare a multi-scale
feature map, we use a feature pyramid network to process the BEV feature
representation into three different scales. At the end of each scale, we add a
convolutional block attention module (CBAM) [45] to enhance the feature via
channel-wise and spatial attention.

We use a center head on the highest scale feature map $\mathcal{C}$ to predict
an $l$-channel heatmap of object centers. Each channel contains the heatmap
score of one class. The location of the top $N$ heatmap scores will be taken
out as the center proposals. We used $N=500$ in our experiments empirically.

####  3.2.2 Multi-scale Center Transformer Decoder

We extract the features at the proposed center locations as the query
embedding for the transformer decoder. We use a linear layer to encode the
location of the centers into a position embedding. Traditional DETR decoder
initializes the query with a learnable parameter. Consequently, the attention
weights acquired in the decoder are almost the same among all features. By
using the center feature as the initial query embedding, we can guide the
training to focus on the feature that contains meaningful object information.
We use the same self-attention layer in the vanilla Transformer decoder to
learn contextual attention between objects. The complexity of computing the
cross-attention of a center query to all multi-scale BEV features is
$O(\sum^{S}_{s=1}h_{s}w_{s}N)$. Since the BEV map resolution needs to be
relatively large to maintain the fine-grained features for small objects, it
is impractical to use all BEV features as the attending keypoints.
Alternatively, we confine the attending keypoints to a small $3\times 3$
window near the center location at each scale, as illustrated in Figure 3. The
complexity of this cross-attention is $O(9SN)$, which is more efficient than
the normal implementation. Because of multi-scale features, we are able to
capture a wide range of features around proposed centers. The Multi-scale
cross-attention can be formulated as:

$\text{MSCA}(p)=\sum_{m=1}^{M}W_{m}[\sum_{s=1}^{S}\sum_{j\in\Omega_{j}}\sigma(\frac{Q_{i}K^{s}_{j}}{\sqrt{d}})\cdot
V^{s}_{j}],$ (3)

where $p$ denotes the center proposal, $\Omega_{j}$ here is the window around
the center, and $s$ is the index of the scale. The feed-forward layer is also
kept unchanged.

####  3.2.3 Multi-scale Deformable Cross-attention layer

Inspired by [64], we also used a deformable cross-attention layer to sample
the attending keypoints automatically. Figure 3 shows the structure of the
deformable cross-attention layer. Compared to the normal multi-head cross-
attention layer, deformable cross-attention uses a linear layer to learn 2D
offsets $\Delta p$ of the reference center location $p$ at all heads and
scales. The feature at $p+\Delta p$ will be extracted as the cross-attention
attending feature through bilinear sampling. We use a linear layer to directly
learn the attention scores from the query embedding. Features from multiple
scales are aggregated together to form the cross-attention layer output:

$\text{MSDCA}(p)=\sum_{m=1}^{M}W_{m}[\sum_{s=1}^{S}\sum_{k=1}^{K}\sigma(W_{msk}\mathcal{C}(p))x^{s}(p+\Delta
p_{msk})],$ (4)

where $x^{s}$ is the multi-scale BEV feature, $\mathcal{C}(p)$ is the center
feature, and $\sigma(W_{msk}\mathcal{C}(p))$ is the attention weight. We used
$K=15$ in our experiments.

###  3.3 Multi-frame CenterFormer

Multi-frame is commonly used in 3D detection to improve performance. Current
CNN-based detectors cannot effectively fuse features from a fast-moving
object, while the transformer structure is more suitable for the fusion due to
the attention mechanism. To further explore the potential of CenterFormer, we
propose a multi-frame feature fusing method using the cross-attention
transformer. As shown in Figure 2, we process each frame individually using
the same backbone network. The last BEV feature of the previous frames is
transformed to current coordinates and fused with the current BEV feature in
both the center head and cross-attention layer.

Figure 4: The network structure of spatial-aware fusion. To focus on the
current centers, we use current BEV feature as the reference to learn
attention.

Due to object movements, the center of an object may shift in different
frames. Since we only need to predict the center in the current frame, we use
a spatial-aware fusion in the center head to alleviate the misalignment error.
As shown in Figure 4, the spatial-aware module uses a similar spatial
attention layer as CBAM [45] to calculate pixel-wise attention based on the
current BEV feature. We concatenate the current BEV feature and weighted
previous BEV feature and use an additional convolution layer to fuse them
together. We also add the time embedding to the BEV features based on their
relative time. Finally, we feed the output fused features to the center head
to predict the center candidates.

In the cross-attention layer, we use the location of the center proposal to
find the corresponding features in the aligned previous frames. The extracted
features will be added to the attending keys. Since our normal cross-attention
design uses features in a small window close to the center location, it has
limited learnability if the object was out of the window area due to fast
movement. Meanwhile, our deformable cross-attention is able to model any level
of movement, and is more suitable for the long time-range case. Because our
multi-frame model only needs the final BEV feature of the previous frame, it
is easy to be deployed to the online prediction by saving the BEV feature in a
memory bank.

###  3.4 Loss Functions

Besides the general classification and regression loss functions, we add two
additional loss functions to better account for the center-based object
detection. First, Inspired by the design in CIA-SSD [47], we move the IoU-
aware confidence rectification module from the second stage of other methods
to the regression head. More specifically, we predict an IoU score $iou$ for
each bounding box proposal, which is supervised with the highest IoU between
the prediction and all ground truth annotations in a smooth L1 loss. During
the evaluation, we rectify the confidence score with the predicted IoU score
using $\alpha^{\prime}=\alpha*iou^{\beta}$, where $\alpha$ is the confidence
score and $\beta$ is a hyperparameter controlling the degree of rectification.
Second, similar to [42, 13], we also added a corner heatmap head alongside the
center heatmap head as auxiliary supervision. For each box, we generate the
corner heatmap of four bounding box edge centers and the object center using
the same methods to draw the center heatmap except that the Gaussian radius is
half size. During training, we supervise the corner prediction with an MSE
loss on the region where the ground truth heatmap score is above 0.

The final loss used in our model is the weighted combination of the following
four parts:
$\mathcal{L}=w_{hm}\mathcal{L}_{hm}+w_{reg}\mathcal{L}_{reg}+w_{iou}\mathcal{L}_{iou}+w_{cor}\mathcal{L}_{cor}.$
We use focal loss [21] and L1 loss for the heatmap classification and box
regression. The weights of heatmap classification loss, box regression loss,
IoU rectification loss, and corner classification loss are [1, 2, 1, 1],
respectively, in our experiment.

##  4 Experiments

We present our experimental results on two large-scale LiDAR object detection
benchmarks: Waymo Open Dataset [38] and nuScenes dataset [3]. Due to page
limitation, the experimental results on the nuScenes dataset, more analysis as
well as the details on the choice of network parameters, and additional
visualizations are included in the supplementary material.

###  4.1 Dataset

Waymo Open Dataset (WOD) is a large-scale LiDAR point cloud dataset for the
autonomous driving environment. It contains 798, 202, and 150 sequences for
training, validation, and testing, respectively. Each sequence is 20-second
long, captured by a 10 FPS LiDAR sensor with 64 lines in $360^{\circ}$. WOD
provides bounding box annotations for three classes: vehicles, pedestrians,
and cyclists. The evaluation metrics used in WOD are mean average precision
(mAP) and mAP weighted by heading accuracy (mAPH). Each object is categorized
into two levels of difficulty, where LEVEL_1 (L1) denotes that there are more
than 5 points on the object and LEVEL_2 (L2) denotes that there are $1\sim 5$
points on the object or the object is manually labeled as L2. The evaluation
of the primary metric mAPH L2 includes both L1 and L2 objects. We set the
range of the 3D voxel space as $[-75.2\text{m},75.2\text{m}]$ for the $X$ and
$Y$ axes, and $[-2\text{m},4\text{m}]$ for the $Z$ axis. The size of each
voxel is set to $(0.1\text{m},0.1\text{m},0.15\text{m})$.

###  4.2 Implementation Details

We follow the same VoxelNet backbone network design as [61, 53, 34]. In our
center proposal network, we process the output of the BEV feature into three
scales through one upsample layer and one downsample layer. We set the
transformer layer/head numbers to 3/4 when using the normal cross-attention,
and to 2/6 when using the deformable cross-attention. During evaluation, we
use the NMS IoU threshold of $[0.8,0.55,0.55]$ and $\beta=[1,1,4]$ for
vehicle, pedestrian and cyclist. For our 8 frames model, we use
$\beta=[1,1,1]$ to get a better result for the cyclist. We also increase the
center proposal number $N$ to 1000 in evaluation. We trained our model using
AdamW optimizer with the one-cycle policy. We trained the single-frame and
multi-frame models on 8 Nvidia A100 GPUs with batch sizes 32 and 16. Due to
the memory limitation, 4 frames and 8 frames are first split into two 2 frames
and 4 frames mini-batch. The points from frames in each mini-batch will be
first concatenated together. Hence our multi-frame model only needs to fuse
two BEV features together. We apply the object copy & paste augmentation
during training with the same fade strategy in [41]. More details are included
in the supplementary material.

###  4.3 Object Detection Results

Table 1: The detection results on WOD validation set. $\dagger$: Deformable
CenterFormer. $\ddagger$ Reported by PVRCNN++.

Method | Frame |  | Vehicle L1
---
(mAP/APH)
| Vehicle L2
---
(mAP/APH)
| Pedestrain L1
---
(mAP/APH)
| Pedestrain L2
---
(mAP/APH)
| Cyclist L1
---
(mAP/APH)
| Cyclist L2
---
(mAP/APH)
| Mean L2
---
mAPH
StarNet [28] | 1 | 55.1/54.6 | 48.7/48.3 | 68.3/60.9 | 59.3/52.8 | \- / - | \- / - | -
SECOND‡ [49] | 1 | 72.3/71.7 | 63.9/63.3 | 68.7/58.2 | 60.7/51.3 | 60.6/59.3 | 58.3/57.1 | 57.2
LiDAR R-CNN [20] | 1 | 73.5/73.0 | 64.7/64.2 | 71.2/58.7 | 63.1/51.7 | 68.6/66.9 | 66.1/64.4 | 60.1
Part-A2‡ [37] | 1 | 77.1/76.5 | 68.5/68.0 | 75.2/66.9 | 66.2/58.6 | 68.6/67.4 | 66.1/64.9 | 63.8
3D-MAN [50] | 16 | 74.5/74.0 | 67.6/67.1 | 71.7/67.7 | 62.6/59.0 | \- / - | \- / - | -
PV-RCNN++ [35] | 1 | 79.1/78.6 | 70.3/69.9 | 80.6/74.6 | 71.9/66.3 | 73.5/72.4 | 70.7/69.6 | 68.6
CenterPoint [53] | 1 | \- / - | \- /67.9 | \- / - | \- /65.6 | \- / - | \- /68.6 | 67.4
CenterPoint [53] | 2 | \- / - | \- /69.7 | \- / - | \- /70.3 | \- / - | \- /70.9 | 70.3
CenterFormer | 1 | 75.0/74.4 | 69.9/69.4 | 78.0/72.4 | 73.1/67.7 | 73.8/72.7 | 71.3/70.2 | 69.1
CenterFormer† | 1 | 75.2/74.7 | 70.2/69.7 | 78.6/73.0 | 73.6/68.3 | 72.3/71.3 | 69.8/68.8 | 69.0
CenterFormer | 2 | 77.1/76.6 | 72.2/71.7 | 80.9/77.6 | 76.2/73.0 | 76.0/75.1 | 73.6/72.7 | 72.5
CenterFormer† | 2 | 77.0/76.5 | 72.1/71.6 | 81.4/78.0 | 76.7/73.4 |  76.6/75.7 |  74.2/73.3 | 72.8
CenterFormer† | 4 | 78.1/77.6 | 73.4/72.9 | 81.7/78.6 | 77.2/74.2 | 75.6/74.8 | 73.4/72.6 | 73.2
CenterFormer† | 8 |  78.8/78.3 |  74.3/73.8 |  82.1/79.3 |  77.8/75.0 | 75.2/74.4 | 73.2/72.3 | 73.7
|  |  |  |  |  |  |  |

In Table 1, we show the results on the validation set. Anchor-free center-
based method CenterPoint achieves better performance than the anchor-based
methods. PVRCNN++ is the best anchor-based method so far, yet shows weak
performance on small objects. This demonstrates the limitation of using
manually designed anchors for the detection of objects that have large size
variations. Our single frame model outperforms CenterPoint by 1.7%. Using a
multi-frame model can significantly increase the mAPH performance. Our 2/4/8
frames model reaches the mAPH of 72.8%, 73.2% and 73.7%, respectively,
becoming the new state-of-the-art. The pedestrian class benefits most from the
multi-frame since the pedestrian point cloud suffers most from the occlusion
and noise, as well as its small size. The overall better performance verifies
the effectiveness of our proposed transformer model.

Table 2: The single-model detection result on WOD testing set.

Method |  | Vehicle L1
---
(mAP/APH)
| Vehicle L2
---
(mAP/APH)
| Pedestrain L1
---
(mAP/APH)
| Pedestrain L2
---
(mAP/APH)
| Cyclist L1
---
(mAP/APH)
| Cyclist L2
---
(mAP/APH)
| Mean L2
---
mAPH
StarNet [28] | 61.5/61.0 | 54.9/54.5 | 67.8/59.9 | 61.1/54.0 | \- / - | \- / - | -
PPBA [5] | 64.6/64.1 | 56.2/55.8 | 69.7/61.7 | 63.0/55.8 | \- / - | \- / - | -
M3DETR [12] | 77.7/77.1 | 70.5/70.0 | 68.2/58.5 | 60.6/52.0 | 67.3/65.7 | 65.3/63.8 | 61.9
3D-MAN [50] | 78.7/78.3 | 70.4/70.0 | 70.0/66.0 | 64.0/60.3 | \- / - | \- / - | -
RSN [1] | 80.7/80.3 | 71.9/71.6 | 78.9/75.6 | 70.7/67.8 | \- / - | \- / - | -
PV-RCNN++ [35] | 81.6/81.2 | 73.9/73.5 | 80.4/75.0 | 74.1/69.0 | 71.9/70.8 | 69.3/68.2 | 70.2
CenterPoint [53] | 81.1/80.6 | 73.4/73.0 | 80.5/77.3 | 74.6/71.5 | 74.6/73.7 | 72.2/71.3 | 71.9
SST_3f [8] | 81.0/80.6 | 73.1/72.7 | 83.3/79.7 | 76.9/73.5 | 75.7/74.6 | 73.2/72.2 | 72.8
AFDetV2 [13] | 81.7/81.2 | 74.3/73.9 | 81.3/78.1 | 75.5/72.4 |  76.4/75.4 |  74.1/73.0 | 73.1
CenterFormer |  84.7/84.4 |  78.1/77.7 |  84.6/81.8 |  79.4/76.6 | 75.5/74.5 | 73.3/72.4 | 75.6
|  |  |  |  |  |  |

In Table 2, we show the single model results on the test set. The prediction
result is submitted to the online server for evaluation. Our method
outperforms all the previous methods by a large margin. The result on vehicle
and pedestrian classes have significant improvements ($+3.8\%$ and $+3.1\%$ on
L2 mAPH) as a result of the long-range contextual information learning of the
transformer.

Table 3: The comparison of single-frame CenterFormer (trained on vehicle only)
with other methods on WOD validation set for vehicle class.

| Method | Ref | L2 mAP | L2 mAPH
---|---|---|---|---
CNN | RSN [1] | CVPR 2021 | 66.0 | 65.5
Voxel R-CNN [25] | AAAI 2021 | 66.6 | -
Pyramid R-CNN [24] | ICCV 2021 | 67.2 | 66.7
LiDAR R-CNN [20] | CVPR 2021 | 68.3 | 67.9
BtcDet [48] | AAAI 2022 | 70.1 | 69.6
Transformer | BoxeR-3D [29] | CVPR 2022 | 63.9 | 63.7
Voxel transformer [25] | ICCV 2021 | 65.9 | 65.3
M3DETR [12] | WACV 2022 | 66.6 | 66.0
3D-MAN [50] | CVPR 2021 | 67.6 | 67.1
SST_1f [8] | CVPR 2022 | 68.0 | 67.6
CT3D [33] | ICCV 2021 | 69.0 | -
CenterFormer | ECCV 2022 | 70.7 | 70.2
|  |  |  |

To fairly compare with more recent methods [25, 6, 24, 33] that train the
model and report the result on only the vehicle class, we train the single
frame CenterFormer only on Vehicle class too. We show the results in Table 3.
As shown in the table, even with the simplest design, CenterFormer outperforms
both the recent transformer-based methods and CNN-based baselines.

###  4.4 Ablation Study

Table 4: The ablation of the LEVEL_2 mAPH result improvement of each component
on the validation set using single frame. SA, CA and DCA denote the self-
attention layer, cross-attention layer and deformable cross-attention layer.
IoU and Corner denote IoU rectification and Corner auxiliary supervision. Fade
denotes the fade augmentation strategy.

CPN | SA | CA | Corner | DCA | IoU | Fade | Vehicle | Pedstrain | Cyclist | Mean
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  | 66.4 | 63.4 | 67.8 | 65.9
$\checkmark$ |  |  |  |  |  |  | 68.5 | 62.7 | 67.0 | 66.1
$\checkmark$ | $\checkmark$ |  |  |  |  |  | 68.7 | 64.5 | 67.3 | 66.8
$\checkmark$ |  | $\checkmark$ |  |  |  |  | 68.7 | 64.2 | 66.7 | 66.5
$\checkmark$ | $\checkmark$ | $\checkmark$ |  |  |  |  | 69.3 | 64.8 | 66.8 | 67.0
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  |  | 69.5 | 65.3 | 66.4 | 67.1
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  | 69.5 | 66.9 | 69.7 | 68.7
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$ | 69.4 | 67.7 | 70.2 | 69.1
$\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$ |  |  | 69.3 | 65.1 | 67.5 | 67.3
$\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 69.2 | 66.7 | 69.1 | 68.3
$\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 69.7 | 68.3 | 68.8 | 69.0
|  |  |  |  |  |  |  |  |  |

In Table 4, we investigate the effect of each added component in our method on
a single frame. We use the previous center-based method as the baseline. After
changing the RPN to multi-scale CPN and separating the detection into the
center proposal and box regression, our method reaches a similar performance
despite we flatten the regression head to 1D. The transformer self-attention
layer and cross-attention layer both can improve the results, and when used
together, the result reaches 67.0%. This indicates that the self-attention
layer and the cross-attention layer learn features separately. Corner
auxiliary supervision can additionally improve the result by 0.1%. On the
other hand, deformable cross-attention achieves a better result of 67.3%. When
trained with IoU rectification, the results get a significant boost to 68.7%
and 68.3% for the models using cross-attention and deformable cross-attention.
The fade augmentation strategy can further improve the result by 0.4% and
0.7%. This is because the model can adjust to the real data distribution at
the end of the training.

###  4.5 Analysis

Comparison with deformable DETR Deformable DETR [64] aims to speed up the
learning speed and reduce the computation cost in the DETR structure. Compared
to Deformable DETR, our method has three major differences. First, we
completely remove the transformer encoder to enable a larger encoded feature
map. Second, we use the center feature rather than the learnable parameter as
the query embedding for the transformer decoder. Experiment shows that using
the center feature as the initial query embedding outperforms the parametric
embedding by 1.5% mAPH. This is because the center feature already contains
object-level information, which makes it easier to learn pairwise attentions.
Third, we use a similar training strategy as [53] rather than the end-to-end
set matching training strategy. The set matching training is known for being
hard to converge. Since we already have an initial center proposal, we can
limit the network to learn only when the proposal is close to the ground truth
annotation to speed up the training. Experiment shows that if we use the same
set matching training strategy in DETR, the mAPH result is 46.3%, which is
more than 20% lower than our current training method.

Figure 5: The visualization of the learned self- and cross-attention weight.
The lightness or darkness of the color represents the value of attention
weight. The red box denotes the ground truth box and blue box denotes the
predicted box. In cross-attention, the sampled keypoints are drawn with red,
green and blue for the scale from low to high. Best viewed in color.

Visualization of the learned attention The visualizations of the learned self-
and cross-attention are illustrated in Figure 5. The self-attention learning
is mainly focusing on the feature of the same class or nearby objects that
have the same attributes. For instance, the vehicles on the same line or same
parking area will have higher attention weight. The offsets learned by the
deformable attention layer vary among different scales. The offsets in two
lower scales mostly lead to the keypoints inside or around the object, whereas
the offsets in the higher scale can sample far-range features.

##  5 Conclusion

In this paper, we propose a novel center-based transformer for 3D object
detection. Our method provides a solution to improve the anchor-free 3D object
detection network through object-level attention learning. Compared to the
DETR-style transformer networks, we use the center feature as the initial
query embedding in the transformer decoder to speed up the convergence. We
also avoid high computational complexity by focusing the cross-attention
learning of each query in a small multi-scale window or a deformable region.
Results show that the proposed method outperforms the strong baseline in the
Waymo Open Dataset, and reaches state-of-the-art performance when extended to
multi-frame. We hope our design will inspire more future work in query-based
transformers for LiDAR point cloud analysis.

Acknowledgements We thank Yufei Xie for his help with refactoring the code for
release.

## References

  * [1] Bewley, A., Sun, P., Mensink, T., Anguelov, D., Sminchisescu, C.: Range conditioned dilated convolutions for scale invariant 3d object detection. In: CoRL (2021)
  * [2] Bowen, C., Alexander, G.S., Alexander, K.: Per-pixel classification is not all you need for semantic segmentation. In: NeurIPS (2021)
  * [3] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. In: CVPR (2020)
  * [4] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV (2020)
  * [5] Cheng, S., Leng, Z., Cubuk, E.D., Zoph, B., Bai, C., Ngiam, J., Song, Y., Caine, B., Vasudevan, V., Li, C., et al.: Improving 3d object detection through progressive population based augmentation. In: ECCV (2020)
  * [6] Deng, J., Shi, S., Li, P., Zhou, W., Zhang, Y., Li, H.: Voxel r-cnn: Towards high performance voxel-based 3d object detection. In: AAAI (2021)
  * [7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
  * [8] Fan, L., Pang, Z., Zhang, T., Wang, Y.X., Zhao, H., Wang, F., Wang, N., Zhang, Z.: Embracing single stride 3d object detector with sparse transformer. In: CVPR (2022)
  * [9] Fan, L., Xiong, X., Wang, F., Wang, N., Zhang, Z.: Rangedet: In defense of range view for lidar-based 3d object detection. In: ICCV (2021)
  * [10] Ge, R., Ding, Z., Hu, Y., Wang, Y., Chen, S., Huang, L., Li, Y.: Afdet: Anchor free one stage 3d object detection. In: CVPRW (2020)
  * [11] Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014)
  * [12] Guan, T., Wang, J., Lan, S., Chandra, R., Wu, Z., Davis, L., Manocha, D.: M3detr: Multi-representation, multi-scale, mutual-relation 3d object detection with transformers. In: WACV (2022)
  * [13] Hu, Y., Ding, Z., Ge, R., Shao, W., Huang, L., Li, K., Liu, Q.: Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds. In: AAAI (2022)
  * [14] Huang, R., Zhang, W., Kundu, A., Pantofaru, C., Ross, D.A., Funkhouser, T., Fathi, A.: An lstm approach to temporal 3d object detection in lidar point clouds. In: ECCV (2020)
  * [15] Jianwei, Y., Chunyuan, L., Pengchuan, Z., Xiyang, D., Bin, X., Lu, Y., Jianfeng, G.: Focal self-attention for local-global interactions in vision transformers. In: NeurIPS (2021)
  * [16] Jifeng, D., Haozhi, Q., Yuwen, X., Yi, L., Guodong, Z., Han, H., Yichen, W.: Deformable convolutional networks. In: ICCV (2017)
  * [17] Kaiwen, D., Song, B., Lingxi, X., Honggang, Q., Qingming, H., Qi, T.: Centernet: Keypoint triplets for object detection. In: ICCV (2019)
  * [18] Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: PointPillars: Fast encoders for object detection from point clouds. In: CVPR (2019)
  * [19] Law, H., Deng, J.: Cornernet: Detecting objects as paired keypoints. In: ECCV (2018)
  * [20] Li, Z., Wang, F., Wang, N.: Lidar r-cnn: An efficient and universal 3d object detector. In: CVPR (2021)
  * [21] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: CVPR (2017)
  * [22] Liu, S., Li, F., Zhang, H., Yang, X., Qi, X., Su, H., Zhu, J., Zhang, L.: Dab-detr: Dynamic anchor boxes are better queries for detr. In: ICLR (2021)
  * [23] Liu, Z., Zhang, Z., Cao, Y., Hu, H., Tong, X.: Group-free 3d object detection via transformers. In: ICCV (2021)
  * [24] Mao, J., Niu, M., Bai, H., Liang, X., Xu, H., Xu, C.: Pyramid r-cnn: Towards better performance and adaptability for 3d object detection. In: CVPR (2021)
  * [25] Mao, J., Xue, Y., Niu, M., Bai, H., Feng, J., Liang, X., Xu, H., Xu, C.: Voxel transformer for 3d object detection. In: CVPR (2021)
  * [26] Meng, D., Chen, X., Fan, Z., Zeng, G., Li, H., Yuan, Y., Sun, L., Wang, J.: Conditional detr for fast training convergence. In: ICCV (2021)
  * [27] Misra, I., Girdhar, R., Joulin, A.: An end-to-end transformer model for 3d object detection. In: CVPR (2021)
  * [28] Ngiam, J., Caine, B., Han, W., Yang, B., Chai, Y., Sun, P., Zhou, Y., Yi, X., Alsharif, O., Nguyen, P., et al.: Starnet: Targeted computation for object detection in point clouds. In: arXiv (2019)
  * [29] Nguyen, D.K., Ju, J., Booji, O., Oswald, M.R., Snoek, C.G.: Boxer: Box-attention for 2d and 3d transformers. In: CVPR (2022)
  * [30] Noh, J., Lee, S., Ham, B.: Hvpr: Hybrid voxel-point representation for single-stage 3d object detection. In: CVPR (2021)
  * [31] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: CVPR (2017)
  * [32] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In: NeurIPS (2017)
  * [33] Sheng, H., Cai, S., Liu, Y., Deng, B., Huang, J., Hua, X.S., Zhao, M.J.: Improving 3d object detection with channel-wise transformer. In: ICCV (2021)
  * [34] Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., Li, H.: Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In: CVPR (2020)
  * [35] Shi, S., Jiang, L., Deng, J., Wang, Z., Guo, C., Shi, J., Wang, X., Li, H.: Pv-rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. In: arXiv (2022)
  * [36] Shi, S., Wang, X., Li, H.: Pointrcnn: 3d object progposal generation and detection from point cloud. In: CVPR (2019)
  * [37] Shi, S., Wang, Z., Shi, J., Wang, X., Li, H.: From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. In: TPAMI (2020)
  * [38] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: CVPR (2020)
  * [39] Sun, P., Wang, W., Chai, Y., Elsayed, G., Bewley, A., Zhang, X., Sminchisescu, C., Anguelov, D.: Rsn: Range sparse net for efficient, accurate lidar 3d object detection. In: CVPR (2021)
  * [40] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017)
  * [41] Wang, C., Ma, C., Zhu, M., Yang, X.: Pointaugmenting: Cross-modal augmentation for 3d object detection. In: CVPR (2021)
  * [42] Wang, G., Tian, B., Ai, Y., Xu, T., Chen, L., Cao, D.: Centernet3d: An anchor free object detector for autonomous driving. In: arXiv (2020)
  * [43] Wang, H., Zhu, Y., Adam, H., Yuille, A., Chen, L.C.: Max-deeplab: End-to-end panoptic segmentation with mask transformers. In: CVPR (2021)
  * [44] Wang, Y., Fathi, A., Kundu, A., Ross, D.A., Pantofaru, C., Funkhouser, T.A., Solomon, J.M.: Pillar-based object detection for autonomous driving. In: ECCV (2020)
  * [45] Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: ECCV (2018)
  * [46] Wu, B., Wan, A., Yue, X., Keutzer, K.: Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud. In: ICRA (2018)
  * [47] Wu, Z., Weiliang, T., Sijin, C., Li, J., Chi-Wing, F.: Cia-ssd: Confident iou-aware single-stage object detector from point cloud. In: AAAI (2021)
  * [48] Xu, Q., Zhong, Y., Neumann, U.: Behind the curtain: Learning occluded shapes for 3d object detection. In: AAAI (2022)
  * [49] Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. In: Sensors (2018)
  * [50] Yang, Z., Zhou, Y., Chen, Z., Ngiam, J.: 3d-man: 3d multi-frame attention network for object detection. In: CVPR (2021)
  * [51] Ye, M., Xu, S., Cao, T.: Hvnet: Hybrid voxel network for lidar based 3d object detection. In: CVPR (2020)
  * [52] Yin, J., Shen, J., Guan, C., Zhou, D., Yang, R.: Lidar-based online 3d video object detection with graph-based message passing and spatiotemporal transformer attention. In: CVPR (2020)
  * [53] Yin, T., Zhou, X., Krähenbühl, P.: Center-based 3d object detection and tracking. In: CVPR (2021)
  * [54] Ze, L., Yutong, L., Yue, C., Han, H., Yixuan, W., Zheng, Z., Stephen Ching-Feng, L., Baining, G.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV (2021)
  * [55] Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., Shum, H.Y.: Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In: arXiv (2022)
  * [56] Zhang, Y., Zhou, Z., David, P., Yue, X., Xi, Z., Gong, B., Foroosh, H.: Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In: CVPR (2020)
  * [57] Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V.: Point transformer. In: CVPR (2021)
  * [58] Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H., Zhang, L.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: CVPR (2021)
  * [59] Zhou, Y., Tuzel, O.: Voxelnet: End-to-end learning for point cloud based 3d object detection. In: CVPR (2018)
  * [60] Zhou, Z., Zhang, Y., Foroosh, H.: Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation. In: CVPR (2021)
  * [61] Zhu, B., Jiang, Z., Zhou, X., Li, Z., Yu, G.: Class-balanced grouping and sampling for point cloud 3d object detection. In: arXiv (2019)
  * [62] Zhu, X., Ma, Y., Wang, T., Xu, Y., Shi, J., Lin, D.: Ssn: Shape signature networks for multi-class object detection from point clouds. In: ECCV (2020)
  * [63] Zhu, X., Zhou, H., Wang, T., Hong, F., Ma, Y., Li, W., Li, H., Lin, D.: Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In: CVPR (2021)
  * [64] Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection. In: ICLR (2020)

##  Appendix 0.A Implementation Details

VoxelNet backbone network We adopt the same VoxelNet backbone network design
in CenterPoint [53]. In specific, we first use an average pooling in each
voxel to encode the point cloud into a voxel feature map. Then, a VoxelNet
[59] with sparse convolution is used to extract features in the voxel map.
Except for the down-sample layer, all residual blocks use the submanifold
sparse convolution layer to minimize the computation cost. The VoxelNet
backbone network down-samples the dimensions of the x-axis and y-axis with a
factor of 8 and the z-axis with a factor of 16. Finally, the output voxel
feature map is reshaped to the BEV for the following process.

Multi-scale CPN We design the Multi-scale CPN to achieve two goals: First, we
want to encode the BEV feature into different scale levels for the transformer
decoder. Second, the BEV feature map should be large enough to separate small
objects like the pedestrian. Since in our experiment, the size of each BEV
grid in the VoxelNet output feature is [0.8m $\times$ 0.8m], which is similar
to the average pedestrian object size, we need to up-sample the feature map to
avoid the voxelization error. We also use a down-sample layer to extract
larger-scale features. The overall network structure is shown in Figure 6.

Figure 6: The network structure of multi-scale CPN. Each Conv block contains a
convolution layer with kernel size $3\times 3$, a batchnorm layer and a relu
activation layer. We use convolution layer with stride and transpose
convolution layer as the down-sample and up-sample layers.

Spatial-aware heatmap fusion To focus the heatmap fusion on the current object
center location, we use the current BEV feature as the reference to learn
spatial attention. We concatenate the current feature and the weighted
previous features together and use another $3\times 3$ convolution layer to
compress the BEV feature into the same size as the single frame input in the
heatmap head.

Training details Generally, the transformer decoder requires a matching
process, e.g. Hungarian matching, to find the closest ground truth bounding
box to the prediction in training. The computation cost of this process
becomes unacceptable when we match two 3D bounding boxes with orientation.
Hence, we use the same training strategy in the center-based object detection
network, i.e. only training the network when the proposed center is at the
same position as the ground truth center. To utilize all annotation
information in training, we manually select the center positions of all ground
truth bounding boxes as the initial center proposals in training. And the
position with the highest heatmap response other than those positions are
selected as the remaining proposals. This allows the network to have a
meaningful training objective from the beginning of the training, and thus
converges faster.

##  Appendix 0.B NuScenes Dataset Result

NuScenes Dataset (ND) [3] is a large-scale dataset created by Motional. It
contains 1000 scenes of 20s each, which are split into 700,150,150 sequences
for the training, validation and testing. ND uses a 32 lines LiDAR with the
frequency of 20 FPS. In each keyframe that is sampled every 0.5s, ND provides
bounding box annotations for 10 different classes. The evaluation metrics used
by ND are mean average precision (mAP) and nuScenes detection score (NDS). In
contrast to WOD, the mAP used by NP only matches objects according to the 2D
center distance in BEV rather than IoU. NDS is computed based on a weighted
sum of Average Translation/ Scale/ Orientation/ Velocity and Attribute Errors
(ATE/ ASE/ AOE/ AVE/ AAE) on the set of true positives. Followed by [53], we
set the range of the 3D voxel space as $[-54\text{m},54\text{m}]$ for the $X$
and $Y$ axes, and $[-5\text{m},3\text{m}]$ for the $Z$ axis. The size of each
voxel is set to $(0.075\text{m},0.075\text{m},0.2\text{m})$.

Table 5: The detection result on the ND validation set. $\ddagger$: Base
CenterFormer model without IoU rectification and multi-frame fusion. $*$ : Our
implementation uses the same backbone network and training configuration.

Method | mAP $\uparrow$ | NDS $\uparrow$ | ATE $\downarrow$ | ASE $\downarrow$ | AOE $\downarrow$ | AVE $\downarrow$ | AAE $\downarrow$
---|---|---|---|---|---|---|---
PointPillars [18] | 39.3 | 53.3 | - | - | - | - | -
Pillar-OD [44] | 44.4 | 56.8 | - | - | - | - | -
SSN [62] | 45.3 | 57.0 | - | - | - | - | -
CBGS [61] | 51.4 | 62.6 | - | - | - | - | -
CenterPoint∗ [53] | 55.2 | 64.4 | 29.3 | 25.7 | 29.6 | 27.2 | 19.5
CenterFormer‡ | 55.4 | 65.2 | 27.5 | 25.2 | 27.5 | 24.3 | 20.8
|  |  |  |  |  |  |

We show the comparison of the results on the nuScenes validation set in Table
5. We compare our base CenterFormer model with the CenterPoint baseline using
the same training configuration. Due to the time limitation, we did not
include further experiments on ND using some of our more complex structures,
like deformable cross-attention and multi-frame fusion. Nevertheless, our base
CenterFormer already outperforms CenterPoint as shown in the table. The
improvement comes mainly from the bounding box regression since these two
methods share a similar center-based classification design. This result
validates the superiority of our proposed CenterFormer over the traditional
center-based detector in different point cloud structures.

##  Appendix 0.C Analysis

The effect of our multi-frame design In Table 8, we show the improvement of
our multi-frame CenterFormer compared to the point concatenation method used
by most LiDAR object detectors. The point concatenation method has significant
improvement ($+2.6\%$) on two frames. But it has less effect when using more
frames. In contrast, our multi-frame CenterFormer has constant improvement
when using more frames. In 2, 4 and 8 frames, multi-frame CenterFormer
achieves $1.0\%$, $0.5\%$ and $1.1\%$ higher mAPH than the point concatenation
method. Our deformable CenterFormer achieves better performance than the base
model on multi-frame due to the ability to model cross-attention in a larger
range. We also compared the performance on different speeds in Figure 8. The
speed of a object is categorized into stationary ($<0.2m/s$), slow ($0.2\sim
1m/s$), medium ($1\sim 3m/s$), fast ($3\sim 10m/s$) or very fast ($>10m/s$).
We can see that the main improvement in the point concatenation method comes
from the stationary objects, and the slow-speed objects even have worse
performance. On the contrary, our multi-frame CenterFormer achieves better
performance throughout all categories.

Figure 7: The LEVEL_2 mAPH results comparison of the multi-frame CenterFormer
on WOD validation set. All models are trained without the IoU rectification.
$\star$: CenterFormer using point concatenation. $\dagger$: Deformable
CenterFormer.

Method | Frame | Vehicle | Pedestrian | Cyclist | Mean
---|---|---|---|---|---
CenterFormer | 1 | 69.0 | 66.8 | 68.0 | 67.9
CenterFormer⋆ | 2 | 70.6 | 70.2 | 70.7 | 70.5
CenterFormer⋆ | 4 | 71.7 | 70.8 | 71.6 | 71.4
CenterFormer⋆ | 8 | 72.0 | 71.6 | 71.6 | 71.7
CenterFormer | 2 | 70.9 | 70.4 | 71.8 | 71.0
CenterFormer† | 2 | 70.7 | 71.1 | 72.6 | 71.5
CenterFormer† | 4 | 71.9 | 72.2 | 71.5 | 71.9
CenterFormer† | 8 | 73.4 | 73.4 | 71.7 | 72.8
|  |  |  |  |

Figure 8: The LEVEL_2 mAPH results comparison breakdown by speed.

Table 6: The LEVEL_2 mAPH result comparison of using different layer and head
configurations on WOD validation set. (Left) Base CenterFormer. (Right)
Deformable CenterFormer. All experiments use only 20% of uniformly sampled
training data.

layer | head | Vehicle | Pedestrian | Cyclist | Mean
---|---|---|---|---|---
2 | 4 | 65.5 | 61.7 | 63.9 | 63.7
2 | 6 | 65.2 | 61.0 | 64.5 | 63.7
3 | 2 | 65.2 | 61.0 | 63.3 | 63.2
3 | 4 | 65.4 | 61.6 | 65.1 | 64.0
3 | 6 | 65.0 | 61.6 | 64.5 | 63.7
4 | 2 | 64.2 | 60.7 | 64.1 | 63.0
4 | 4 | 64.9 | 61.4 | 64.1 | 63.5
|  |  |  |  |

layer | head | Vehicle | Pedestrian | Cyclist | Mean
---|---|---|---|---|---
1 | 3 | 65.1 | 60.2 | 64.7 | 63.3
2 | 3 | 64.9 | 60.7 | 64.8 | 63.4
2 | 6 | 65.3 | 60.4 | 66.0 | 63.9
3 | 3 | 65.7 | 60.7 | 65.1 | 63.7
3 | 6 | 64.5 | 60.3 | 64.5 | 63.1
|  |  |  |  |

Transformer layer and head number We show the comparison of results using
different transformer layers and head numbers in Table 6. The results indicate
that more transformer layers and heads do not assure better performance. The
base transformer model with 3 layers and 4 heads and the deformable
transformer model with 2 layers and 6 heads has the best performance.

Table 7: The LEVEL_2 mAPH result comparison of using different window sizes in
our base CenterFormer and different offset numbers in deformable CenterFormer
on WOD validation set. (Left) Base CenterFormer. (Right) Deformable
CenterFormer. All experiments use only 20% of uniformly sampled training data.

Window size | Vehicle | Pedestrian | Cyclist | Mean
---|---|---|---|---
3,3,3 | 65.4 | 61.6 | 65.1 | 64.0
5,3,3 | 65.4 | 61.7 | 65.0 | 64.0
5,5,3 | 65.4 | 61.8 | 64.1 | 63.8
5,5,5 | 64.1 | 60.1 | 63.8 | 62.7
7,3,3 | 64.9 | 61.1 | 64.7 | 63.6
|  |  |  |

Offset number | Vehicle | Pedestrian | Cyclist | Mean
---|---|---|---|---
5 | 64.8 | 59.5 | 64.2 | 62.8
9 | 64.4 | 60.6 | 64.5 | 63.2
15 | 65.3 | 60.4 | 66.0 | 63.9
20 | 65.0 | 60.2 | 64.3 | 63.2
|  |  |  |

Cross-attention field We show the comparison of results using different cross-
attention window sizes and offset numbers in Table 7. Increasing the window
size does not have any performance gain. This is because the sizes of each
grid in three scales are $[0.4m,0.8m,1.6m]$ in our setting. The $3\times 3$
attention window can encompass the region of almost all pedestrian and cyclist
objects. If we increase it to $5\times 5$ or $7\times 7$, although it can
include more features for the vehicle, the added feature for the pedestrian
and cyclist is almost unrelated. On the other hand, the number of offsets used
in the deformable cross-attention also does not increase the performance
monotonically. We find the offset number of 15 has the best performance.

Figure 9: The LEVEL_2 mAPH result comparison of the position encoding methods
on WOD validation set. All experiments use only 20% of uniformly sampled
training data.

Encoder | Vehicle | Pedestrian | Cyclist | Mean
---|---|---|---|---
None | 60.3 | 56.3 | 61.3 | 59.5
Sinusoidal | 62.7 | 58.6 | 63.5 | 61.7
Linear | 65.2 | 60.9 | 66.0 | 64.0
|  |  |  |

Figure 10: The LEVEL_2 mAPH results comparison of CenterFormer and DETR in
different epochs.

Position embedding Position embedding is important in the transformer model to
capture the spatial relationship between input features. Standard position
embedding is either crafted manually using sine and cosine distances or
learned through a linear layer. However, 3D point clouds, as a specific type
of data, contain the position information in the raw feature. They do not
necessarily need the position embedding since the spatial information is
already in the encoded feature. We test the performance of different position
embedding methods using 20% of training data. The LEVEL_2 mAPH result is shown
in Table 10. We can see without the position embedding, the result drops
significantly to 59.5%. This indicates the absolute position information is
still an important feature to guide the attention learning of the transformer
model. The learnable encoding method also outperforms the sinusoidal encoding
method by a large margin.

Converging difficulty compared with DETR In Figure 10, We show the LEVEL_2
mAPH result comparison of CenterFormer and DETR in a 20 epochs training cycle.
We implement the DETR-style set matching training strategy using the same
backbone network in CenterFormer. We can see that not only CenterFormer can
reach a much higher result than DETR, but also converge much faster than DETR.

Figure 11: Visualization of CenterFormer predictions. The red box denotes the
ground truth bounding box. The blue box denotes the predictions with
confidence score $>0.4$. Truncated objects whose center is outside of the 50m
range are not visualized. Best viewed in color.

Comparison with two-stage LiDAR detection Most two-stage LiDAR detection
methods [36, 34, 53, 20] apply the RCNN-style refinement network in the 3D
domain. The second stage aggregates RoI features in each first-stage proposal
to refine both the classification and regression prediction. There are two
drawbacks in this design. First, the second stage only utilizes local RoI
features. It cannot retrieve global information and depends heavily on the
quality of the first feature and proposal. Second, it has a large computation
overhead, especially when used in LiDAR point clouds with a large size of
points or voxel features. The network will predict the same object information
twice, which results in a cumbersome structure and prohibitive run-time. In
contrast, our method works between one-stage and two-stage. We use a center
proposal network to generate initial center queries. The self-attention layer
allows the network to directly learn object-level contextual information. The
cross-attention layer can also capture long-range information in the multi-
scale BEV feature. The classification and regression are done only once in our
method.

##  Appendix 0.D Qualitative Results

Figure 11 shows the qualitative result of our proposed method. Our method can
make accurate predictions with a high confidence score.


17(0.5,0.2) Published in: missing2022 IEEE 25th International Conference on
Intelligent Transportation Systems (ITSC).
Please cite this paper as:
Zhaobin Mo, Yongjie Fu, and Xuan Di. “Quantifying Uncertainty In Traffic State
Estimation Using Generative Adversarial Networks.”
2022 IEEE 25th International Conference on Intelligent Transportation Systems
(ITSC). IEEE, 2022.

# Quantifying Uncertainty In Traffic State Estimation Using Generative
Adversarial Networks

Zhaobin Mo1, Yongjie Fu1, and Xuan Di1,2 Accepted by the 25th IEEE
International Conference on Intelligent Transportation Systems (IEEE ITSC
2022)${}^{\text{\textdagger}}$Zhaobin Mo and Yongjie Fu contributed equally to
this work. ${}^{\text{\textdaggerdbl}}$This work is partially sponsored by NSF
CPS-2038984.1Department of Civil Engineering and Engineering Mechanics,
Columbia University, New York, NY 10027, USA
<EMAIL_ADDRESS>for Smart Cities, Data Science
Institute, Columbia University, New York, NY 10027, USA.

###### Abstract

This paper aims to quantify uncertainty in traffic state estimation (TSE)
using the generative adversarial network based physics-informed deep learning
(PIDL). The uncertainty of the focus arises from fundamental diagrams, in
other words, the mapping from traffic density to velocity. To quantify
uncertainty for the TSE problem is to characterize the robustness of predicted
traffic states. Since its inception, generative adversarial networks (GAN) has
become a popular probabilistic machine learning framework. In this paper, we
will inform the GAN based predictions using stochastic traffic flow models and
develop a GAN based PIDL framework for TSE, named “PhysGAN-TSE”. By conducting
experiments on a real-world dataset, the Next Generation SIMulation (NGSIM)
dataset, this method is shown to be more robust for uncertainty quantification
than the pure GAN model or pure traffic flow models. Two physics models, the
Lighthill-Whitham-Richards (LWR) and the Aw-Rascle-Zhang (ARZ) models, are
compared as the physics components for the PhysGAN, and results show that the
ARZ-based PhysGAN achieves a better performance than the LWR-based one.

##  I INTRODUCTION

Traffic states, represented by traffic velocity or density, can contain
uncertainty from various sources, including model errors, measurement noise,
inherent stochasticity in driving behavior, and random initial or boundary
conditions. Imposed with these stochastic terms, traffic states are
essentially spatiotemporal (ST) random fields. In this paper, we are primarily
focused on uncertainty arising from driving behaviors and their resultant
effect on traffic states. We aim to infer the distribution of traffic state
random fields using sparse data collected from fixed-location sensors, i.e.,
loop detectors.

Traffic state estimation (TSE), a classical traffic problem, has long been
studied aiming to estimate traffic states using various sensor data [1]. The
methods that predict traffic states and characterize the robustness of the
prediction can be categorized into model-based and data-driven methods. The
model-based methods assume that observed data are generated by underlying
models and filtering methods can be applied to these models to propagate
uncertainty. The data-driven methods learn distributions of uncertainty
directly from data without using a prior assumption of underlying physics. The
model based methods suffer from limitations such as the non-Gaussian
likelihoods and high-dimensional posterior distributions [2], while the data-
driven approaches require a huge amount of data to learn the posterior
distributions from high-dimensional random inputs. Thus, an integration of
model based and data-driven methods has become a rapidly growing research
arena, namely, physics-informed deep learning (PIDL) [3, 4, 5, 6].

Recent years have seen a growing trend of learning random fields using machine
learning models, including but not limited to: Gaussian Processes [7],
Bayesian neural networks [8], and most recently, generative models [2]. Among
the generative models, generative adversarial networks (GAN) have demonstrated
its robustness in various applications, from image or video generation [9] to
uncertainty quantification (UQ) [10]. For UQ problems, physics-informed GANs
(PhysGAN) is becoming increasingly popular to quantify uncertainty in
stochastic differential equations [11, 2, 12, 13]. Little research, though,
has been documented when randomness inherently arises from human behaviors or
inherent dynamics. In other words, the existing literature on PhysGAN
integrates a deterministic physics model into GANs because it is assumed that
the equations that generate data are deterministic and the prediction contains
stochasticity due to randomness from initial or boundary conditions. In this
paper, in contrast, we will assume that the models that generate data are
themselves stochastic due to randomness in parameters, and thus, stochastic
physics models need to be integrated to capture the randomness from these
sources.

The contributions of this paper include:
(1) Physics-informed GANs are used to quantify uncertainty in TSE. Because
traffic, constituted by human drivers, is inherently stochastic due to
randomness in driving behaviors, stochastic traffic flow models are
incorporated into GANs to help regularize the training process;
(2) Parameters in physics are estimated jointly with traffic state prediction;
(3) PhysGAN-TSE is applied to the real-world dataset, Next Generation
SIMulation (NGSIM), to demonstrate its predictive accuracy compared to other
baselines.

The rest of the paper is organized as follows: Section II reviews the state-
of-the-art of PIDL on TSE. Section III introduces the preliminaries of GAN
based UQ. Section IV introduces how we integrate PIDL and GAN into the
framework of PhysGAN for TSE problems. Section V demonstrates how PhysGAN is
applied to NGSIM to characterize uncertainty from the real-world data. Section
VI concludes our work and projects future research directions in this
promising arena.

##  II Related Work

For studies that use model-based UQ for TSE, a prior assumption is usually
made about the distribution of randomness in inputs or traffic states. The
existing practice includes two ways: one to add a Brownian motion on the top
of the deterministic traffic flow models, leading to Gaussian stochastic
traffic flow models [14]; and the other to derive intrinsic stochastic traffic
flow models with more complex probabilistic distributions [15]. The former is
more amenable to applying standard filtering methods while the latter is more
challenging to deal with and requires a large population approximation to
extract mean and variance before any filtering can be applied.

If the noise in the output traffic states is non-Gaussian and follows complex
distributions, the filtering methods could fail.  GANs have been used to learn
complex distributions of stochasticity residing in random fields from ST data
[11]. Key issues with GANs include mode collapse and training instability
[16]. One solution is to regularize the training of GANs by incorporating
physics into the structure of generators (and potentially discriminators),
where unlabeled data are generated from generators that are guided by physics-
based models, so that the search space of parameters is narrowed down.

PhysGAN have been applied to quantify real-world data uncertainty in various
domains, which includes flood prediction [17], blood alcohol concentration
prediction [18], and porous media flow modeling [19]. PhysGANs have not been
used to quantify uncertainty in traffic state estimation. Traffic, different
from physical systems, can exhibit highly nonlinear randomness arising from
inherent human driving behavior. Thus, it poses new challenges when PhysGANs
are applied.

##  III Preliminaries

Let us start with a partial differential equation (PDE). Define location
$x\in[0,L]$ and time $t\in[0,T]$ and $L,T\in\mathbb{R}^{+}$. Then the ST
domain of interest is a continuous set of points:
$D=\\{(x,t)|x\in[0,L],t\in[0,T]\\}$. A PDE defined over the ST domain is
written as:

$\begin{split}{\begin{array}[]{*{20}l}\
\partial_{t}\mathbf{u}(x,t)+\mathcal{N}_{x}[\mathbf{u}(x,t);\mathbf{\lambda}]=0,(x,t)\in
D,\\\ \ \mathcal{B}_{x}[\mathbf{u}(x,t)]=0,\end{array}}\end{split}$ (1)

where, $\mathcal{N}_{x}(\cdot)$ is the nonlinear differential operator,
$\mathcal{B}_{x}(\cdot)$ is the boundary condition operator, $\mathbf{u}(x,t)$
is the exact solution of the PDE, and $\lambda$ is the physics parameter
vector.

Now we will approximate the PDE solution, $\mathbf{u}(x,t)$, by a (deep)
neural network (DNN) parametrized by $\theta$, i.e.
$\hat{\mathbf{u}}(x,t)=f_{\theta}(x,t)$, which is called “physics uninformed
neural network (PUNN).” If this PUNN is exactly equivalent to the PDE
solution, then we have

$\partial_{t}\hat{\mathbf{u}}(x,t)+\mathcal{N}_{x}[\hat{\mathbf{u}}(x,t);\lambda]=0,(x,t)\in
D.$ (2)

Otherwise we define a residual function:

$r_{\theta,\lambda}(x,t)=\partial_{t}\hat{\mathbf{u}}(x,t)+\mathcal{N}_{x}[\hat{\mathbf{u}}(x,t);{\lambda}].$
(3)

If PUNN is well trained, the residual needs to be as close to zero as
possible. Since NNs are normally trained with discrete data points, below we
will define the potentially accessible training data in the context of TSE.

Assume the dataset consists of (1) (labeled) observation data
$O=\\{(x_{o}^{(i)},t_{o}^{(i)};\mathbf{u}_{o}^{(i)})|i=1,...,N_{o}\\}$, and
(2) (unlabeled) collocation points
$C=\\{(x_{c}^{(j)},t_{c}^{(j)})|j=1,...,N_{c}\\}$. $i$ and $j$ are the indexes
of observation points and collocation points, respectively. The numbers of
observed data and collocation states are denoted as $N_{o},N_{c}$,
respectively. Observation data $O$ are limited to the time and locations where
traffic sensors are placed. In contrast, collocation points $C$ have neither
measurement requirements nor location limitations, and thus are controllable.
The target values associated with collocation points are used to regularize
the loss function. Accordingly, we define a loss function as the weighted
average of data discrepancy and physics discrepancy. Denote the weight of the
data discrepancy in the loss function as $\alpha$. The loss function is:

$\begin{split}{\begin{array}[]{*{20}l}\mathcal{L}({\theta,\lambda})&=\alpha\cdot\frac{1}{N_{o}}\sum_{i=1}^{N_{o}}\underbrace{\left|\left|\hat{\mathbf{u}}_{o}(x_{o}^{(i)},t_{o}^{(i)})-\mathbf{u}_{o}^{(i)}\right|\right|^{2}}_{\text{data
discrepancy}}\\\
&+(1-\alpha)\frac{1}{N_{c}}\sum_{j=1}^{N_{c}}\underbrace{\left|\left|r_{\theta,\lambda}(x_{c}^{(j)},t_{c}^{(j)})\right|\right|^{2}}_{\text{physics
discrepancy}}.\end{array}}\end{split}$ (4)

We will introduce stochasticity into the PDE solution. Below we will discuss
how the probabilistic physics-informed neural networks [2] can be used for UQ.
When considering the stochasticity of a PDE, we assume the uncertainty mainly
comes from two sources, the intrinsic uncertainty of the physics parameter
$\lambda$ and the measurement uncertainty of the variable $\mathbf{u}$. In
connection with this, we treat the physics parameter as a random variable,
i.e. $\lambda\sim p_{\lambda}(\lambda)$, and assume the variable $\mathbf{u}$
follows a conditional distribution of the spatial-temporal coordinates, i.e.
$\mathbf{u}\sim p_{\mathbf{u}}(\mathbf{u}|x,t)$.

After formulating the uncertainty of the physics $\lambda$ and the variable
$\mathbf{u}$, a UQ problem further assumes that there exists a mapping between
the variable $\mathbf{u}$ and a latent random variabe $\mathbf{z}\sim
p_{\mathbf{z}}(\mathbf{z})$, i.e. $G:(x,t,\mathbf{z})\rightarrow\mathbf{u}$,
such that the PDE holds in a stochastic manner:

$\displaystyle\mathbf{u}=G(x,t,\mathbf{z})\sim
p_{\mathbf{u}}(\mathbf{u}|x,t),\mathbf{z}\sim
p_{\mathbf{z}}(\mathbf{z});\lambda\sim p_{\lambda}(\lambda),$ (5)
$\displaystyle
s.t.\mathbb{E}_{p_{\mathbf{z}}(\mathbf{z})p_{\lambda}(\lambda)}\left[\partial_{t}\mathbf{u}(x,t)+\mathcal{N}_{x}\left[\mathbf{u}(x,t);\lambda\right]\right]=0,$

where $\mathbf{z}$ encodes possible randomness sources that lead to
stochasticity in the observable output $\mathbf{u}$.

With Eq. 5 generating noised output $\mathbf{u}$ that are constrained by a
PDE, the UQ problem based on adversarial inference is to match the conditional
distribution of the generated data, denoted as
$\hat{\mathbf{u}}=G_{\theta}(x,t,\mathbf{z})\sim
p_{\theta}(\hat{\mathbf{u}}|x,t)$ parametrized by $\theta$, and the
conditional distribution of the observed output, which is
$p_{\mathbf{u}}(\mathbf{u}|x,t)$. One widely used metric to match two
distributions is the reverse Kullback-Leibler (KL) divergence.

Now we will formulate the UQ problem in the context of conditional GANs. The
generator is a neural network surrogate model, denoted as $G_{\theta}$, that
approximates the ground-truth mapping $G$ mentioned before. The objective of
the generator $G_{\theta}$ is to fool an adversarially trained discriminator
$D_{\phi}$. Then the adversarial UQ problem becomes a min-max game:

$\displaystyle\min_{G_{\theta}}\max_{D_{\phi}}\
\mathcal{L}_{\theta,\phi}=\mathbb{E}_{q(x,t)p_{\mathbf{z}}(\mathbf{z})}\left[\log
D_{\phi}(x,t,G_{\theta}(x,t,z))\right]$ (6)
$\displaystyle+\mathbb{E}_{q(x,t,\mathbf{u})}\left[\log(1-D_{\phi}(x,t,\mathbf{u}))\right],$

where $\theta$ and $\phi$ are the parameters of the generator and the
discriminator, respectively.
$q(x,t,\mathbf{u})=p_{\mathbf{u}}(\mathbf{u}|x,t)q(x,t)$ is the joint
distribution of the spatial-temporal coordinates and the output $\mathbf{u}$,
and $q(x,t)$ is the marginal distribution of the spatial-temporal coordinates.
$p_{\mathbf{z}}(\mathbf{z})$ is a standard normally distributed random
variable.

With the physics introduced into the generator, the loss functions of the
generator $G_{\theta}$ and the discriminator $D_{\phi}$ can be decoupled and
defined below, respectively:

$\displaystyle\mathcal{L}_{G}(\theta,\lambda)$ (7)
$\displaystyle=\alpha\mathbb{E}_{q(x_{o},t_{o})p_{\mathbf{z}}(\mathbf{z})}\left[D_{\phi}(x_{o},t_{o},\hat{\mathbf{u}}_{o})\right]+(1-\alpha)\mathcal{L}_{phy}(\theta,\lambda)\quad\quad\quad$
$\displaystyle\simeq\frac{\alpha}{N_{o}}\sum_{i=1}^{N_{o}}D_{\phi}(x_{o}^{(i)},t_{o}^{(i)},\hat{\mathbf{u}}_{o}^{(i)})+(1-\alpha){\cal
L}_{phy}(\theta,\lambda),$ $\displaystyle\mathcal{L}_{D}(\phi)$ (8)
$\displaystyle=-\mathbb{E}_{q(x_{o},t_{o})p_{\mathbf{z}}(\mathbf{z})}\left[\log
D_{\phi}(x_{o},t_{o},\hat{\mathbf{u}}_{o})\right]$
$\displaystyle-\mathbb{E}_{q(x_{o},t_{o},\mathbf{u}_{o})}\left[\log(1-D(x_{o},t_{o},\mathbf{u}_{o}))\right]$
$\displaystyle\simeq-\frac{1}{N_{o}}\sum_{i=1}^{N_{o}}\log
D_{\phi}(x_{o}^{(i)},t_{o}^{(i)},\hat{\mathbf{u}}_{o}^{(i)})+\log(1-D_{\phi}(x_{o}^{(i)},t_{o}^{(i)},\mathbf{u}_{o}^{(i)})),$
$\displaystyle\mathcal{L}_{phy}(\theta,\lambda)=\mathbb{E}_{q(x_{c},t_{c})}\left[\mathbb{E}_{p_{\mathbf{z}}(\mathbf{z})p_{\lambda}(\lambda)}\left|\left|r_{\theta,\lambda}\right|\right|^{2}\right],\quad\quad\quad\quad\quad$
(9)

where
$\hat{\mathbf{u}}_{o}^{(i)}=G_{\theta}(x_{o}^{(i)},t_{o}^{(i)},\mathbf{z}^{(i)})$
and
$\hat{\mathbf{u}}_{c}^{(j)}=G_{\theta}(x_{c}^{(j)},t_{c}^{(j)},\mathbf{z}^{(j)})$
are predictions of the generator on observation and collocation points,
respectively. Note that ${\cal L}_{D}$ only depends on the observation points,
${\cal L}_{phy}$ only depends on the collocation points, and ${\cal L}_{G}$
depends on both the observation and the collocation points.

##  IV GAN based PIDL framework for TSE (PhysGAN-TSE)

In this section, we will introduce how PhysGANs are introduced to solve the UQ
problem with latent variables.

###  IV-A Problem Statement

Let $\mathcal{N}^{(1)}[\cdot]$ and $\mathcal{N}^{(2)}[\cdot]$ be two general
nonlinear differential operators. The problem of interest is to quantify
uncertainty in the traffic density $\rho(x,t)$ and velocity $u(x,t)$ fields at
each point $(x,t)$ in the ST domain $D$, such that the following PDEs of a
traffic flow model can be satisfied:

$\begin{split}{\begin{array}[]{*{20}l}\
\partial_{t}\rho(x,t)+\mathcal{N}^{(1)}_{x}[\rho(x,t),u(x,t);\lambda,\xi^{(1)}(x,t)]=\omega^{(1)}(x,t),\\\
\
\partial_{t}u(x,t)+\mathcal{N}^{(2)}_{x}[\rho(x,t),u(x,t);\lambda,\xi^{(2)}(x,t)]=\omega^{(2)}(x,t),\end{array}}\end{split}$
(10)

where $\left\\{\rho(x,t),u(x,t)\right\\}$ is the exact solution of PDEs.
$\xi^{(i)}(x,t)$ and $\omega^{(i)}(x,t),i\in[1,2],$ are potential intrinsic
and observation uncertainties, respectively, represented by random fields.

We aim to develope PUNNs to approximate $\rho(x,t)$ and $u(x,t)$ with time $t$
and location $x$ as inputs, respectively. We denote the approximation from
PUNN as $\hat{\rho}(x,t)$ and $\hat{u}(x,t)$. To use physics to guide the
training of PUNN, we customize the residuals on the collocation points as:

$\displaystyle
r^{(1)}_{\theta,\lambda}(x_{c},t_{c})=\partial_{t}\hat{\rho}_{c}(x_{c},t_{c})+\mathcal{N}^{(1)}_{x}[\hat{\rho}_{c}(x_{c},t_{c}),\hat{u}_{c}(x_{c},t_{c});\lambda],$
(11a) $\displaystyle
r^{(2)}_{\theta,\lambda}(x_{c},t_{c})=\partial_{t}\hat{u}_{c}(x_{c},t_{c})+\mathcal{N}^{(2)}_{x}[\hat{\rho}_{c}(x_{c},t_{c}),\hat{u}_{c}(x_{c},t_{c});\lambda],$
(11b)

which are defined according to the traffic flow model in Eq. 10.

###  IV-B Architecture Design

We adopt two types of traffic model as the physics component, the Lighthill-
Whitham-Richards (LWR) model [20] and the Aw–Rascle–Zhang (ARZ) model [21], to
construct the LWR-based PhysGAN (LWR-PhysGAN) and the ARZ-based PhysGAN (ARZ-
PhysGAN).

LWR-PhysGAN. The LWR model is depicted as:

$\displaystyle\partial_{t}\rho+\partial_{x}(\rho u)=0,$ (12a) $\displaystyle
u=u_{max}(1-\rho/\rho_{max}).$ (12b)

Eq. 12a is the conservation law; Eq. 12b defines the relation between the
traffic density $\rho$ and the traffic velocity $u$, where $\rho_{max}$ and
$u_{max}$ are the maximum traffic density and the maximum traffic velocity,
respectively. Note that the time $t$ and locations $x$ of $\rho$ and $u$ are
omitted for simplicity. Applying Eq. 12a on the collocation points, we can
rewrite Eq. 11a as below below:

$\displaystyle
r_{\theta,\lambda}^{(1)}=\partial_{t}\hat{\rho}_{c}+\partial_{x}(\hat{\rho}_{c}\hat{u}_{c}).$
(13)

With this residual, we can re-write Eq. LABEL:eqn:physics in the form of the
LWR model as:

$\displaystyle\mathcal{L}_{phy}^{LWR}(\theta,\lambda)$
$\displaystyle=\mathbb{E}_{q(x_{c},t_{c})}\left[\mathbb{E}_{p_{\mathbf{z}}(\mathbf{z})p_{\lambda}(\lambda)}\left|\left|r_{\theta,\lambda}^{(1)}\right|\right|^{2}\right].$
(14)

The architecture of the LWR-PhysGAN is illustrated in Fig. 1. It consists of
three sub-models, namely the generator $G_{\theta}$, the discriminator
$D_{\phi}$, and the physics $f_{\lambda}$, which are emphasized by shaded
boxes. The blue and red colors illustrate the processes of calculating the
data discrepancy and physics discrepancy, respectively. The dashed lines
indicate the process of updating parameters.

Figure 1: Architecture of the LWR-PhysGAN.

As for calculating the data discrepancy, both the observation points
$(x_{o},t_{o})$ and the collocation points $(x_{c},t_{c})$ are fed into the
generator $G_{\theta}$ along with the random number $\mathbf{z}$. The outputs
of the generator $G_{\theta}$ are the predicted traffic density
$\hat{\rho}_{o}$ and $\hat{\rho}_{c}$, which correspond to the observation and
collocation points, respectively. Then the physics $f_{\lambda}$ takes as
input the predicted traffic density $\hat{\rho}_{o}$ and outputs the predicted
traffic velocity $\hat{u}_{o}$ according to the LWR equation depicted in Eq.
12b. The prediction tuple $(x_{o},t_{o},\hat{\rho}_{o},\hat{u}_{o})$ and the
ground-truth tuple $(x_{o},t_{o},\rho_{o},u_{o})$ are then fed into the
discriminator $D_{\phi}$ to calculate discriminator loss function
$\mathcal{L}_{D}(\phi)$.

As for calculating the physics discrepancy, $\hat{\rho}_{c}$ is used to
calculate the physics loss function $\mathcal{L}^{LWR}_{phy}(\theta,\lambda)$
using Eq. 14. The physics loss is then added to the generator loss function as
a regularization term.

As for updating the parameters, the discriminator loss function
$\mathcal{L}_{D}(\phi)$ is used to update the discriminator parameters
${\phi}$. The generator loss function $\mathcal{L}_{G}(\theta,\lambda)$, which
is regularized by the physics loss, is used to update both the generator
parameters $\theta$ and the physics parameter $\lambda$.

ARZ-PhysGAN. The ARZ model is depicted as:

$\displaystyle\partial_{t}\rho+\partial_{x}(\rho u)=0,$ (15a)
$\displaystyle\partial_{t}(u+h(\rho))+u\cdot\partial_{x}(u+h(\rho))=(U_{eq}(\rho)-u)/\tau,$
(15b) $\displaystyle h(\rho)=U_{eq}(0)-U_{eq}(\rho),$ (15c) $\displaystyle
U_{eq}(\rho)=u_{max}(1-\rho/\rho_{max}).$ (15d)

Eq. 15a is the conservation law, which is the same as Eq. 12a; $\rho_{max}$
and $u_{max}$ are the maximum traffic density and the maximum traffic
velocity, respectively; $\tau$ is the relaxation time. Applying Eq. 15b on the
collocation points, we can rewrite Eq. 11b as below:

$r_{\theta,\lambda}^{(2)}=\partial_{t}(\hat{u}_{c}+h(\hat{\rho}_{c}))+\hat{u}_{c}\cdot\partial_{x}(\hat{u}_{c}+h(\hat{\rho}_{c}))-(U_{eq}(\hat{\rho}_{c})-\hat{u}_{c})/\tau.$
(16)

Thus, we can re-write Eq. LABEL:eqn:physics in the form of the ARZ model as:

$\displaystyle\mathcal{L}_{phy}^{ARZ}(\theta,\lambda)$
$\displaystyle=\mathbb{E}_{q(x_{c},t_{c})}\left[\mathbb{E}_{p_{\mathbf{z}}(\mathbf{z})p_{\lambda}(\lambda)}\left|\left|r_{\theta,\lambda}^{(1)}\right|\right|^{2}\right]$
(17)
$\displaystyle+\mathbb{E}_{q(x_{c},t_{c})}\left[\mathbb{E}_{p_{\mathbf{z}}(\mathbf{z})p_{\lambda}(\lambda)}\left|\left|r_{\theta,\lambda}^{(2)}\right|\right|^{2}\right].$

The structure of the ARZ-PhysGAN is illustrated in Fig. 2, which is similar to
the LWR-PhysGAN except for two modifications due to different physics
components used:

  1. 1.

The generator $G_{\theta}$ of ARZ-PhysGAN have two outputs, the predicted
traffic density $\hat{\rho}$ and the predicted traffic velocity $\hat{u}$.

  2. 2.

To compute the physics loss as shown in Eq. 17 , both $\hat{\rho}_{c}$ and
$\hat{u}_{c}$ are fed into the physics.

Figure 2: Architecture of the ARZ-PhysGAN.

###  IV-C Training Algorithm

We use the Adam [22], one widely used gradient descent algorithm, to update
our models. The training algorithm is shown in Algorithm 1.

Algorithm 1 PhysGAN-TSE Training Algorithm.

Initialization:
Initialized physics parameters $\lambda^{0}$.
Initialized networks parameters $\theta^{0}$, $\phi^{0}$.
Training iterations $Iter$.
Batch size $m$.
Learning rate $lr$.
Weights of loss functions $\alpha$.
Require: Adam optimizer.
Input: The observation data
$\\{(x_{o}^{(i)},t_{o}^{(i)},\mathbf{u}_{o}^{(i)})\\}_{i=1}^{N_{o}}$ and
collocation points $\\{x_{c}^{(j)},t_{c}^{(j)}\\}_{j=1}^{N_{c}}$.

1: for $k\in\\{0,...,Iter\\}$ do

2: Sample batches
$\\{(x_{o}^{(i)},t_{o}^{(i)},\mathbf{u}_{o}^{(i)})\\}_{i=1}^{m}$ and
$\\{x_{c}^{(j)},t_{c}^{(j)}\\}_{j=1}^{m}$ from the observation and collocation
points, respectively

3: Calculate ${\cal L}_{D}$ by Eq. 8

4:
$\phi^{k+1}\leftarrow\phi^{k}-lr\cdot\text{Adam}(\phi^{k},\nabla_{\phi}{\cal
L}_{D})$

5: Calculate ${\cal L}_{phy}$ by Eq. LABEL:eqn:physics

6: Calculate ${\cal L}_{G}$ by Eq. 7

7:
$\theta^{k+1}\leftarrow\theta^{k}-lr\cdot\text{Adam}(\theta^{k},\nabla_{\theta}{\cal
L}_{G})$

8:
$\lambda^{k+1}\leftarrow\lambda^{k}-lr\cdot\text{Adam}(\lambda^{k},\nabla_{\lambda}{\cal
L}_{G})$

9: end for

##  V PhysGAN-TSE on Real World Data

To validate the performance and effectiveness of our proposed model and
algorithm, in this section, we will apply LWR-PhysGAN and ARZ-PhysGAN to the
real-world dataset.

###  V-A NGSIM Dataset

The Next Generation SIMulation (NGSIM) dataset [23] is an open dataset that is
widely used to evaluate various transportation models, which contains high-
resolution vehicular information including the position, velocity, occupied
lane, and spacing at every 0.1 seconds. We focus on trajectories of
automobiles from all five lanes in the mainline.

###  V-B Baseline and Metrics

Two baselines are used for comparison:

  1. 1.

Pure GAN. Pure GAN shares the same architecture with the PhysGAN except for
the physics component. We adopt this baseline to verify the effectiveness of
adding physics component.

  2. 2.

Extended Kalman Filter (EKF). EKF applies a nonlinear version of the Kalman
filter and is widely used in nonlinear systems like the TSE. We use the ARZ-
based EKF, of which details can be found in [6].

We adopt the relative error (RE) to measure the difference between the mean of
the prediction and that of the ground-truth. Also, we use the KL divergence to
quantify the distributional difference between the prediction and the ground-
truth.

###  V-C Experiment Setup

Experiments are conducted on an AWS cloud workstation with 8 Intel Xeon
E5-2686 v4 processors and an NVIDIA V100 Tensor Core GPU with 16 GB memory in
Ubuntu 18.04.3. The initial value of $\rho_{max}$ and $u_{max}$ are 1.0
$m^{-1}$ and 50.0 $m\cdot s^{-1}$, respectively. The learning rate for the
Adam optimizer is $0.0005$, and other configurations are kept as default.

###  V-D Results and Discussion

Effect of training sizes. Fig. 3 shows the REs of our proposed PhysGANs and
the baselines, the left column for the RE of the traffic density and the right
column for the RE of the traffic velocity. The x-axis is the number of loop
detectors, and the y-axis is the RE. Different scatter types and colors are
used to distinguish with different models, namely blue lines with squares for
the ARZ-PhysGAN; red lines with crosses for the LWR-PhysGAN; yellow lines with
rotated triangles for the EKF; purple lines with triangles for the pure GAN.
From this figure, we can see that ARZ-PhysGAN outperforms others across all
numbers of loop detectors, followed by LWR-PhysGAN. The superior performance
of ARZ-PhysGAN compared to LWR-PhysGAN can be explained by the better
compatibility of ARZ with the NGSIM data. In addition, both ARZ- and LWR-
PhysGAN outperform the pure GAN by a significant margin, which demonstrates
the effectiveness of exploiting physics information. Fig. 4 shows the KL
divergences under different loop detector numbers. The x-axis is the number of
loop detectors, and the y-axis is the value of KL divergence for the
prediction of the traffic density (left) and the traffic velocity (right). A
lower KL divergence indicates a better performance. Although the LWR-PhysGAN,
ARZ-PhysGAN and pure GAN achieve similar KL divergences of the traffic
velocity, the ARZ-PhysGAN outperforms other models by a significant margin in
terms of the KL divergence of the traffic density.

Figure 3: The REs of our proposed PhysGANs and the baselines. Figure 4: The KL
divergence of our proposed PhysGANs and the baselines.

Estimation of physics parameters. Fig. 5 shows the mean of the estimated
physics parameters, the left for $\rho_{max}$ and the right for $u_{max}$,
where the x-axis is the number of loop detectors. The line specification is
the same as Fig. 3. From the figure we can see that both $\rho_{max}$ and
$u_{max}$ converge as the number of loop detectors increase. The converged
values for $(\rho_{max}$,$u_{max})$ of LWR and ARZ model are $(0.3820,22.31)$
and $(0.3766,22.86)$, respectively, when the number of loop detectors is 18.
The units of $\rho_{max}$ and $u_{max}$ are $m^{-1}$ and $m\cdot s^{-1}$,
respectively. The converged values of $(\rho_{max}$,$u_{max})$ are reasonable
for the highway scenario.

Figure 5: The mean value of the jointly estimated physics parameters for the
NGSIM dataset.

Visualization of prediction distribution. Fig. 6 presents the comparison
between the ground-truth traffic density distribution and that predicted by
the ARZ-PhysGAN, each subfigure for a randomly sampled spatial-temporal
coordinates. The x-axis is the traffic density and the y-axis is the
corresponding distribution, where the blue is for the ground-truth and the
transparent is for the prediction. Most parts of the predicted and ground-
truth distributions overlap with each other, which demonstrates that our
proposed model can estimate the real-world traffic states uncertainties well.

Figure 6: Visualization of predictions of PhysGAN for NGSIM dataset.

##  VI Conclusions

This paper proposes PhysGANs to quantify the uncertainty in the TSE problem.
The effectiveness of the proposed model is verified by conducting experiments
on the real-world NGSIM dataset, where the ARZ-PhysGAN has the best
performance, followed by the LWR-PhysGAN. In summary, we show that the GAN
model can achieve an enhanced prediction accuracy in terms of the RE and KL
divergence if physics information is used to guide the training. We also show
that the ARZ-PhysGAN outperforms the LWR-PhysGAN because the ARZ model can
better capture the real-world traffic dynamics.

This work can be further improved in two directions. First, although adding
physics can help stabilize the training process, GAN is still hard to train
due to the min-max game. One possible solution is to use the normalizing flow
[24] as the generator, which can explicitly compute the data likelihood and
thus the discriminator is not needed. Second, the proposed model needs to be
re-trained if applied to other roads or to the same road but within a new time
slot, which limits the real-world application of the proposed model. Thus, the
model generalizability for unseen scenarios is worth investigating.

## References

  * [1] T. Seo, A. M. Bayen, T. Kusakabe, and Y. Asakura, “Traffic state estimation on highway: A comprehensive survey,” _Annual Reviews in Control_ , vol. 43, pp. 128–151, 2017.
  * [2] Y. Yang and P. Perdikaris, “Adversarial uncertainty quantification in physics-informed neural networks,” _Journal of Computational Physics_ , vol. 394, pp. 136–152, 2019.
  * [3] M. Raissi, “Deep hidden physics models: Deep learning of nonlinear partial differential equations,” _Journal of Machine Learning Research_ , vol. 19, no. 1, pp. 932–955, Jan. 2018.
  * [4] R. Shi, Z. Mo, and X. Di, “Physics-informed deep learning for traffic state estimation: A hybrid paradigm informed by second-order traffic models,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 1, pp. 540–547, May 2021.
  * [5] Z. Mo, R. Shi, and X. Di, “A physics-informed deep learning paradigm for car-following models,” _Transportation Research Part C: Emerging Technologies_ , vol. 130, p. 103240, 2021.
  * [6] R. Shi, Z. Mo, K. Huang, X. Di, and Q. Du, “A physics-informed deep learning paradigm for traffic state and fundamental diagram estimation,” _IEEE Transactions on Intelligent Transportation Systems_ , 2021.
  * [7] Y. Yuan, Z. Zhang, X. T. Yang, and S. Zhe, “Macroscopic traffic flow modeling with physics regularized gaussian process: A new insight into machine learning applications in transportation,” _Transportation Research Part B: Methodological_ , vol. 146, pp. 88–110, 2021.
  * [8] P. L. McDermott and C. K. Wikle, “Bayesian recurrent neural network models for forecasting and quantifying uncertainty in spatial-temporal data,” _Entropy_ , vol. 21, no. 2, p. 184, 2019.
  * [9] M. Chu, Y. Xie, J. Mayer, L. Leal-Taixé, and N. Thuerey, “Learning temporal coherence via self-supervision for gan-based video generation,” _ACM Transactions on Graphics (TOG)_ , vol. 39, no. 4, pp. 75–1, 2020.
  * [10] D. V. Patel and A. A. Oberai, “Gan-based priors for quantifying uncertainty in supervised learning,” _SIAM/ASA Journal on Uncertainty Quantification_ , vol. 9, no. 3, pp. 1314–1343, 2021.
  * [11] D. Zhang, L. Lu, L. Guo, and G. E. Karniadakis, “Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems,” _Journal of Computational Physics_ , vol. 397, p. 108850, 2019.
  * [12] L. Yang, D. Zhang, and G. E. Karniadakis, “Physics-informed generative adversarial networks for stochastic differential equations,” _SIAM Journal on Scientific Computing_ , vol. 42, no. 1, pp. A292–A317, 2020.
  * [13] A. Daw, M. Maruf, and A. Karpatne, “Pid-gan: A gan framework based on a physics-informed discriminator for uncertainty quantification with physics,” in _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 2021, pp. 237–247.
  * [14] Y. Wang, M. Papageorgiou, A. Messmer, P. Coppola, A. Tzimitsi, and A. Nuzzolo, “An adaptive freeway traffic state estimator,” _Automatica_ , vol. 45, no. 1, pp. 10–24, 2009.
  * [15] S. E. Jabari and H. X. Liu, “A stochastic model of traffic flow: Theoretical foundations,” _Transportation Research Part B: Methodological_ , vol. 46, no. 1, pp. 156–174, 2012.
  * [16] M. Wiatrak, S. V. Albrecht, and A. Nystrom, “Stabilizing generative adversarial networks: A survey,” _arXiv preprint arXiv:1910.00927_ , 2019\.
  * [17] K. Qian, A. Mohamed, and C. Claudel, “Physics informed data driven model for flood prediction: application of deep learning in prediction of urban flood development,” _arXiv preprint arXiv:1908.10312_ , 2019.
  * [18] C. Oszkinat, S. E. Luczak, and I. Rosen, “Uncertainty quantification in estimating blood alcohol concentration from transdermal alcohol level with physics-informed neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [19] B. Siddani, S. Balachandar, W. C. Moore, Y. Yang, and R. Fang, “Machine learning for physics-informed generation of dispersed multiphase flow using generative adversarial networks,” _Theoretical and Computational Fluid Dynamics_ , vol. 35, no. 6, pp. 807–830, 2021.
  * [20] M. J. Lighthill and G. B. Whitham, “On kinematic waves II. A theory of traffic flow on long crowded roads,” _Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences_ , vol. 229, no. 1178, pp. 317–345, 1955.
  * [21] A. Aw and M. Rascle, “Resurrection of “second order” models of traffic flow,” _SIAM Journal on Applied Mathematics_ , vol. 60, no. 3, pp. 916–938, 2000.
  * [22] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [23] V. Punzo, M. T. Borzacchiello, and B. Ciuffo, “On the assessment of vehicle trajectory data accuracy and application to the next generation simulation (ngsim) program data,” _Transportation Research Part C: Emerging Technologies_ , vol. 19, no. 6, pp. 1243–1262, 2011.
  * [24] L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real nvp,” _arXiv preprint arXiv:1605.08803_ , 2016.