# InterTrack: Interaction Transformer for 3D Multi-Object Tracking

John Willes* Cody Reading* Steven L. Waslander
University of Toronto Robotics Institute
{john.willes<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

3D multi-object tracking (MOT) is a key problem for autonomous vehicles,
required to perform well-informed motion planning in dynamic environments.
Particularly for densely occupied scenes, associating existing tracks to new
detections remains challenging as existing systems tend to omit critical
contextual information. Our proposed solution, InterTrack, introduces the
Interaction Transformer for 3D MOT to generate discriminative object
representations for data association. We extract state and shape features for
each track and detection, and efficiently aggregate global information via
attention. We then perform a learned regression on each track/detection
feature pair to estimate affinities, and use a robust two-stage data
association and track management approach to produce the final tracks. We
validate our approach on the nuScenes 3D MOT benchmark, where we observe
significant improvements, particularly on classes with small physical sizes
and clustered objects. As of submission, InterTrack ranks $1^{st}$ in overall
AMOTA among methods using CenterPoint [45] detections.

00footnotetext: indicates equal contribution

##  1 Introduction

Figure 1: 3D tracking visualization. (a) When object interactions are not
considered, the highlighted track produces multiple similar affinity scores
leading to an incorrect match at $t=1$. The track is killed at $t=2$,
resulting in an identity switch (shown as new color). (b) Interaction modeling
encourages a single highest affinity for the correct detection resulting in an
uninterrupted track.

3D multi-object tracking (MOT) is a vital task for many fields such as
autonomous vehicles and robotics, enabling systems to understand their
environment in order to react accordingly. A common approach is the tracking-
by-detection paradigm [41, 45, 16, 8], in which trackers consume independently
generated 3D detections as input. Existing tracks are matched with new
detections at each frame, by estimating a track/detection affinity matrix and
matching high affinity pairs. Incorrect associations can lead to identity
switching and false track initialization, confusing decision making in
subsequent frames.

Affinities are estimated by extracting track and detection features, followed
by a pairwise comparison to estimate affinity scores. Feature extraction can
simply extract the object state [41, 16] or use a neural network to extract
features from sensor data [43, 7]. For effective association, features should
be discriminative such that feature comparison results in accurate affinity
estimation [42].

3D MOT methods often perform feature extraction independently for each track
and detection. Independent methods, however, tend to suffer from high feature
similarity, particularly for densely clustered objects (see Figure 1). Similar
features are non-discriminative, and lead to ambiguous data association as the
correct match cannot be distinguished from the incorrect match hypotheses.

Feature discrimination can be improved through learned feature interaction
modeling [42]. Interaction modeling allows individual features to affect one
another, adding a capacity for the network to encourage discrimination between
matching and non-matching feature pairs. Previous methods [42, 43] have
adopted a graph neural network (GNN) to model feature interactions, but limit
the connections to maintain computational efficiency. We argue all
interactions are important, as for example, short range interactions can be
helpful for differentiating objects in dense clusters, while long range
interactions can assist with fast-moving objects with large motion changes.

Additionally, most prior 3D MOT works [45, 7] lack a filtering of duplicate
tracks, leading to additional false positives and reduced performance.
Duplicate tracks can be an issue for trackers that ingest detections [45] with
high recall and many duplicate false positives.

To resolve the identified issues, we propose InterTrack, a 3D MOT method that
introduces the Interaction Transformer to 3D MOT, in order to generate
discriminative affinity estimates in an end-to-end manner, and introduces a
track rejection module. We summarize our approach with three contributions.
(1) Interaction Transformer. We introduce the Interaction Transformer for 3D
MOT, a learned feature interaction mechanism leveraging the Transformer model.
The Interaction Transformer models spatio-temporal interactions between all
object pairs, leveraging attention layers [40] to maintain high computational
efficiency. Through complete interaction modeling, we encourage feature
discrimination between all object combinations, leading to increased overall
feature discrimination and tracking performance shown in Section 4.3.
(2) Affinity Estimation Pipeline. We design a novel method to estimate
track/detection affinities in an end-to-end manner. Using detections and LiDAR
point clouds, our method extracts full state and shape information for each
track and detection, and aggregates contextual information via the Interaction
Transformer. Each track/detection feature pair is used to regress affinity
scores. Our learned feature extraction offers increased representational power
over simply extracting the object state and allows for interaction modeling.
(3) Track Rejection. We introduce a duplicated track rejection strategy, by
removing any tracks that overlap greater than a specified 3D intersection-
over-union (IoU) threshold. We validate the track rejection module leads to
performance improvements as demonstrated in Section 4.3.
InterTrack is shown to rank 1 on the nuScenes 3D MOT test benchmark [4] among
methods using public detections, with margins of 1.00%, 2.20%, 4.70%, and
2.60% AMOTA on the Overall, Bicycle, Motorcycle, and Pedestrian categories,
respectively.

##  2 Related Work

Figure 2: InterTrack Architecture. For each frame, track and detection
features $\mathbf{T}_{t-1}^{\mathrm{feat}},~{}\mathbf{D}_{t}^{\mathrm{feat}}$
are extracted from object and frame level information, followed by a
Transformer module to produce interaction-aware track/detection features
$\tilde{\mathbf{T}}_{t-1}^{\mathrm{feat}},~{}\tilde{\mathbf{D}}_{t}^{\mathrm{feat}}$.
The track/detection features are then used to estimate an affinity matrix
$\mathbf{A}_{t}$, which is consumed by a 3D tracking module to produce the
final tracks $\mathbf{T}_{t}$.

Transformer. The Transformer [40] architecture has become the standard
sequence modeling technique for natural language processing (NLP) tasks [12,
21, 3]. The key element is the attention mechanism used to model dependencies
between all sequence elements, leading to improved computational efficiency
and performance. Transformers have seen use in computer vision tasks [13, 6,
37], in which image features are extracted to be consumed by a Transformer as
input. Transformers have also seen use in 3D prediction [19] wherein the
Transformer operates on object-level features extracted from autonomous
vehicle sensor input. In this work, we use the Transformer to operate on
object-level features, and leverage the attention mechanism to model
interactions between all tracks and detections.
Image Feature Matching. Image feature matching consists of matching points
between image pairs, often used in applications such as simultaneous
localization and mapping (SLAM) and visual odometry. Feature descriptors are
extracted from local regions surrounding each point, in order to match points
with the highest feature similarity. Feature extraction can be performed by
hand-crafting features [23, 5, 32] or with a learned convolutional neural
network (CNN) [9, 29]. SuperGlue [33] and LoFTR [37] introduce a GNN and
Transformer module respectively, as a means to incorporate both local and
global image information during feature extraction. We follow the Transformer
design of LoFTR [37] to incorporate global object information in our feature
extraction.
Multi-Object Tracking. MOT is a key focus in both 2D [11] and 3D [41]
settings. Tracking methods either follow a tracking-by-detection approach [41,
2, 10], wherein object detections are independently estimated and consumed by
a tracking module for track estimation, or perform joint detection and
tracking in a single stage [26, 28, 24, 38].

In 2D MOT, recent methods [38, 28, 46] leverage the Transformer for joint
detection and tracking, extending DETR [6] to associate detections implicitly.
A drawback is the lack of explicit affinity estimates, resulting in a less
interpretable solution often required by safety-critical systems.

Most 3D MOT methods [41, 47, 8, 7] implement the tracking-by-detection
paradigm, which is dependent on high-quality, high-frequency detections [45,
49]. These trackers solve a data association problem [48, 34], in which
detections are assigned to existing tracks or used to birth new tracks.
Similar to feature matching methods, tracking-by-detection methods extract
object features and form correspondences with the highest feature similarity
[31, 15]. Affinity matrices are estimated that represent the feature
similarity between tracks and detections, and are fed to either a Greedy or
Hungarian [18] matching algorithm to generate matches. Based on how data
association is performed, trackers can be divided into two categories:
heuristic and learned association methods. Heuristic-based methods compute
affinity scores using a heuristic similarity or distance metric. The metric is
based on the object state, with some metrics including 3D IoU, Euclidean
distance, or Mahalanobis distance [27]. One of the first methods for 3D
tracking is AB3DMOT [41], which uses 3D IoU as a similarity metric and a
Kalman filter for track prediction and update. Introduced by CenterPoint [45],
methods [1, 22] can compute affinity using Euclidean distance, and replaces
the Kalman filter prediction with motion prediction based on learned velocity
estimates. EagerMOT [16] performs separate data association stages for LiDAR
and camera streams, and modifies the Euclidean distance to include orientation
difference. Without contextual information, heuristic-based affinity
estimation leads to many similar affinity scores and ambiguous data
association, notably for clustered objects.

Learning-based methods use a neural network to extract object features from
detection and sensor data. Object affinities are computed from the extracted
object features using a distance metric [47] or a learned scoring function
[42, 7]. GNN3DMOT [42] and PTP [43] aggregate object state and appearance
information from LiDAR and image streams to be consumed by a GNN to model
object interactions. The object features are represented as GNN nodes that are
passed to an edge regression module to estimate affinities. Edges are sparsely
constructed due to computational requirements of the GNN, where edges are only
formed between tracks and detections if the objects are within specified 2D
and 3D distance thresholds. Chiu et al. [7] similarly extract object features
from LiDAR and image data, but simply apply a pairwise concatenation and
regression to estimate affinities, which are combined with the Mahalanobis
distance [27] to generate affinity scores. AlphaTrack [47] improves the
CenterPoint [45] detector by fusing LiDAR point clouds with image features,
and extracts object appearance information for data association. SimTrack [26]
performs data association implicitly in a learned joint detection and tracking
framework, based on reading off identities from the prior frame’s centerness
map. Current learned association methods either extract features independently
or with sparse interaction modeling, limiting feature discrimination and
region of influence that is considered during matching.

InterTrack directly address the limitation of interaction modeling for data
association, by introducing the Interaction Transformer to 3D MOT. Doing so
allows for complete interaction modeling, leading to improved object feature
discrimination. InterTrack only leverages the Transformer for data association
to provide an interpretable solution for safety-critical applications in
autonomous vehicles.

##  3 Methodology

InterTrack learns to estimate affinity matrices for data association, followed
by a 3D tracking module to estimate tracks. An overview of InterTrack is shown
in Figure 2.

###  3.1 Affinity Learning

Our network learns to estimate discriminative affinity matrices for effective
data association. Taking object detections and LiDAR point clouds as input, we
extract object-level features for both existing tracks and new detections.
Interaction information is embedded in the track/detection features with a
Transformer model, followed by an affinity estimation module to regress
affinity scores.
Object Feature Extraction. We extract features for both tracks and detections,
using a shared network to extract both state and shape features for each
object. The inputs to the feature extractor are either track or detection
states, $\mathbf{T}_{t-1}\in\mathbb{R}^{M\times 11}$ or
$\mathbf{D}_{t}\in\mathbb{R}^{N\times 11}$, and frame corresponding LiDAR
point clouds, $\mathbf{P}_{t-1}\in\mathbb{R}^{P\times 5}$ or
$\mathbf{P}_{t}\in\mathbb{R}^{P\times 5}$. We extract state directly as
$(x,y,z,l,w,h,\theta,\dot{x},\dot{y},c,s)$ including 3D location $(x,y,z)$,
dimensions $(l,w,h)$, heading angle $\theta$, velocity $(\dot{x},\dot{y})$,
class $c$, and confidence $s$, while shape information is extracted from the
LiDAR point cloud, $\mathbf{P}$. We adopt the point cloud preprocessing and 3D
feature extractor used in SECOND [44] (Voxel Backbone in Figure 2) to generate
bird’s-eye-view (BEV) features, $\mathbf{B}$, where multiple LiDAR sweeps are
combined and each point is represented as $(x,y,z,r,\Delta t)$ including 3D
location $(x,y,z)$, reflectance $r$, and relative timestep $\Delta t$. 3D
object states are projected into the BEV grid, $\mathbf{B}$, and used with ROI
Align to extract shape features for each object. Let $\mathbf{G}(\cdot)$
represent a feed-forward network (FFN) consisting of 2 Linear-BatchNorm-Relu
blocks, used to modify the number of feature channels. We apply a state FFN,
$\mathbf{G}_{\mathrm{st}}$, and a shape FFN, $\mathbf{G}_{\mathrm{sh}}$ (State
and Shape Layers in Figure 2) to modify the number of channels to $C/2$, and
then concatenate to form track or detection features,
$\mathbf{T}_{t-1}^{\mathrm{feat}}\in\mathbb{R}^{M\times C}$ or
$\mathbf{D}_{t}^{\mathrm{feat}}\in\mathbb{R}^{N\times C}$.

Figure 3: Attention block.
$\mathbf{A}^{\mathrm{feat}},\mathbf{B}^{\mathrm{feat}}$ is the input feature
pair and attention is based on the original Multi-Head Attention [40].

Interaction Transformer. The track and detection features,
$\mathbf{T}_{t-1}^{\mathrm{feat}}$ and $\mathbf{D}_{t}^{\mathrm{feat}}$, are
used as input to the Interaction Transformer (see Figure 2), producing
interaction-aware track and detection features,
$\mathbf{\tilde{T}}_{t-1}^{\mathrm{feat}}\in\mathbb{R}^{M\times C}$ and
$\mathbf{\tilde{D}}_{t}^{\mathrm{feat}}\in\mathbb{R}^{N\times C}$. We adopt
the transformer module from LoFTR [37] and interleave the self and cross
attention blocks $N_{c}$ times. In this work, we retain the value of
$N_{c}=4$.

The design of the attention block is shown in Figure 3. For self-attention
blocks, the input features, $\mathbf{A}^{\mathrm{feat}}$ and
$\mathbf{B}^{\mathrm{feat}}$, are the same (either
$\mathbf{T}_{t-1}^{\mathrm{feat}}$ or $\mathbf{D}_{t}^{\mathrm{feat}}$). For
cross attention blocks, the input features are different (either
$\mathbf{T}_{t-1}^{\mathrm{feat}}$ and $\mathbf{D}_{t}^{\mathrm{feat}}$ or
$\mathbf{D}_{t}^{\mathrm{feat}}$ and $\mathbf{T}_{t-1}^{\mathrm{feat}}$,
depending on the direction of cross-attention.) Self-attention blocks perform
feature attenuation within a frame, therefore embedding spatial relationships
through intra-frame interactions. Conversely, cross-attention blocks perform
feature attenuation across frames, embedding temporal relationships through
inter-frame interactions. Leveraging both self and cross attention provides a
complete interaction modeling solution.
Affinity Head. The interaction-aware track and detection features,
$\mathbf{\tilde{T}}_{t-1}^{\mathrm{feat}}$ and
$\mathbf{\tilde{D}}_{t}^{\mathrm{feat}}$, are used to generate an affinity
matrix, $\mathbf{A}_{t}\in\mathbb{R}^{M\times N}$. Every feature pair is
concatenated to form affinity features, $\mathbf{F}_{t}\in\mathbb{R}^{M\times
N\times 2C}$. The number of channels is reduced to 1 with a FFN,
$\mathbf{G}_{\mathrm{aff}}$ (Channel Reduce in Figure 2), followed by a
sigmoid layer to convert the predicted affinities into probabilities. The
channel axis is flattened forming the affinity matrix, $\mathbf{A}_{t}$, where
each element, $\mathbf{A}_{t}(m,n)$, represents the match confidence between a
given track, $\mathbf{T}_{t-1}^{m}$, and detection, $\mathbf{D}_{t}^{n}$.

###  3.2 3D Tracking

The 3D tracking module is only used during inference, taking affinity
estimates, $\mathbf{A}_{t}$, and 2D and 3D detections as input to produce
tracks, $\mathbf{T}_{t}$. We adapt the EagerMOT [16] framework due to the
excellent performance of the two-stage data association. The framework
performs track motion prediction, associates 3D and 2D detections with a
fusion module, and separately associates 3D and 2D detections with tracks in
two stages. Our adaptation modifies the first stage data-association, the
track prediction and update, and introduces a track overlap rejection module.
Fusion. We first match 3D and 2D detections directly following the fusion
stage from the EagerMOT methodology [16]. 3D detections are projected into the
image, and greedily associated with 2D detections based on 2D IoU. Any matches
below a 2D IoU threshold, $\tau_{fuse}$, are discarded.
First Stage Data Association. We use our learned affinity, $\mathbf{A}_{t}$,
for the first stage data association. In this stage all tracks,
$\mathbf{T}_{t-1}$, are matched with all 3D object detections,
$\mathbf{D}_{t}$, of the current frame. The Hungarian algorithm [18] is used
find the optimal assignment matrix, $\mathcal{A}_{t}^{*}$, that minimizes the
assignment affinity cost:

$\displaystyle\mathcal{A}_{t}^{*}=$
$\displaystyle\underset{\mathcal{A}_{t}}{\operatorname*{arg\,min}}\sum_{m,n}\mathbf{A}_{t}(m,n)\mathcal{A}_{t}(m,n)$
(1)

where each element $\mathcal{A}_{t}(m,n)\in\\{0,1\\}$ and rows/columns are
one-hot encoded. We maintain the match filtering strategy in EagerMOT [16], by
removing assignments $(\mathbf{T}_{t-1}^{m},\mathbf{D}_{t}^{n})$ with scaled
Euclidean distance [16] greater than a threshold, $\tau_{3D}$. Experimentally,
we find that the Hungarian algorithm outperforms greedy assignment with our
learned affinity (see Table 4).
Second Stage Data Association. The second stage data association directly
follows the EagerMOT methodology [16]. We use 2D detections without a matched
3D detection (from the fusion stage) and unmatched tracks (from the first
stage) as input. 3D tracks are projected into the image plane, and are
greedily associated with 2D detections based on 2D IoU. Matches below a 2D IoU
threshold, $\tau_{2D}$, are removed.
Track Prediction/Update. For track prediction, we discard the 3D Kalman filter
and use the motion model used in CenterPoint [45] as track prediction, which
uses the latest velocity estimate from the detection as follows:

$\mathbf{p}_{t}=\mathbf{p}_{t-1}+d_{t}\mathbf{v}_{t-1},$ (2)

where $\mathbf{p}_{t}=(x_{t},y_{t})$ is the track centroid,
$\mathbf{v}_{t-1}=(\dot{x}_{t-1},\dot{y}_{t-1})$ is the predicted velocity of
a detection and $d_{t}$ is the timestep duration at time $t$. Discarding the
Kalman prediction equations removes the covariance predictions required for
the Kalman update equations. Therefore, we simply assign the detection state
as the latest measurement.
Track Overlap Rejection. Due to the permissive structure of the two stage data
association framework of EagerMOT, we note that duplicate tracks occur
regularly in the output stream. Duplicate tracks often occur when ingesting
detections [45] with many duplicates with high recall, often with low
confidence scores. Detections with low confidence could be filtered to remove
most duplicates, but would result in reduced track coverage due to a loss of
true positive detections. Rather, we detect duplicates by computing the 3D IoU
between all tracks at each frame, and flag any pairs of the same class with a
3D IoU greater than a rejection threshold, $\tau_{\mathrm{rej}}$. For each
pair, we reject the track with the lower age in order to reduce instances of
identity switching.

###  3.3 Training Loss

We treat the affinity matrix estimation as a binary classification problem
where each element represents the probability of a match between any given
track, $\mathbf{T}_{t-1}^{m}$, and detection, $\mathbf{D}_{t}^{n}$. We use the
binary focal loss [20] as supervision in order to deal with the
positive/negative imbalance:

$\displaystyle L$ $\displaystyle=\frac{1}{M\cdot
N}\sum_{m=1}^{M}\sum_{n=1}^{N}\mathrm{FL}(\mathbf{A}_{t}(m,n),\mathbf{\hat{A}}_{t}(m,n))$
(3)

where $\mathbf{\hat{A}}_{t}$ is the affinity matrix label. To generate the
affinity label, each track, $\mathbf{T}_{t-1}^{m}$, and detection,
$\mathbf{D}_{t}^{n}$, is matched to ground truth boxes via 3D IoU to retrieve
an identity label. Only matches greater than a threshold,
$\tau_{\mathrm{gt}}$, are considered valid. We set
$\mathbf{\hat{A}}_{t}(m,n)=1$ for pairs
$(\mathbf{T}_{t-1}^{m},\mathbf{D}_{t}^{n})$ with the same identity and
$\mathbf{\hat{A}}_{t}(m,n)=0$ for all remaining pairs.

###  3.4 Data Augmentation

As the 3D object detector and InterTrack are trained on the same training
splits, the quality of detections encountered at training time will be higher
than at test time. Therefore, we apply two augmentations during training to
reduce the gap between training and test detections.
Positional Perturbation. To simulate positional error, we add a perturbation
$(\delta_{x},\delta_{y})$ on the 2D position $(x,y)$ of detections by randomly
sampling from a normal distribution:

$\displaystyle\begin{aligned} x&=x+\delta_{x}\\\ y&=y+\delta_{y}\\\
\end{aligned}\qquad\begin{aligned}
\delta_{x}\sim\mathcal{N}(0,\sigma_{x}^{2})\\\
\delta_{y}\sim\mathcal{N}(0,\sigma_{y}^{2})\\\ \end{aligned}$ (4)

Detection Dropout. We employ detection dropout to simulate missing detections.
At each frame, a fraction $d$ of all detections are removed, where the
fraction is randomly sampled from a uniform distribution
$d\sim\mathcal{U}(d_{\mathrm{min}},d_{\mathrm{max}})$.

##  4 Experimental Results

Method | AMOTA$\uparrow$ | AMOTP$\downarrow$ | AMOTA$\uparrow$
---|---|---|---
Bicycle | Bus | Car | Motor | Ped | Trailer | Truck
AB3DMOT [41] | 15.1 | 1.501 | 0.0 | 40.8 | 27.8 | 8.1 | 14.1 | 13.6 | 1.3
CenterPoint [45] | 63.8 | 0.555 | 32.1 | 71.1 | 82.9 | 59.1 | 76.7 | 65.1 | 59.9
ProbTrack [7] | 65.5 | 0.617 | 46.9 | 71.3 | 83.0 | 63.1 | 74.1 | 65.7 | 54.6
EagerMOT [16] | 67.7 | 0.550 | 58.3 | 74.1 | 81.0 | 62.5 | 74.4 | 63.6 | 59.7
InterTrack (Ours) | 68.7 | 0.560 | 60.5 | 72.7 | 80.4 | 67.2 | 77.0 | 62.9 | 60.0
Improvement | +1.00 | +0.01 | +2.20 | -1.40 | -0.60 | +4.70 | +2.60 | -0.70 | +0.30
Table 1: 3D MOT results on the nuScenes test set [4], reporting methods that
use CenterPoint [45] detections. We indicate the best result in bold and
improvements relative to the next best method EagerMOT [16]. Full results for
InterTrack can be accessed here.

To demonstrate the effectiveness of InterTrack we present results on both the
nuScenes 3D MOT benchmark [4] and the KITTI 3D MOT benchmark [14].

The nuScenes dataset [4] consists of dense urban environments, and is divided
into 750 training scenes, 150 validation scenes, and 100 testing scenes. We
compare InterTrack with existing methods on the test set, and use the
validation set for ablations.

The KITTI dataset [14] consists of mid-size city, rural and highway
environments, and is divided into 21 training scenes and 29 testing scenes. We
follow AB3DMOT [41] and divide the training scenes into a train set (10
scenes) and a val set (11 scenes), using the val set for comparison.
Input Parameters. The voxel grid is defined by a range and voxel size in 3D
space. On nuScenes, we use $[-51.2,51.2]\times[-51.2,51.2]\times[-5,3]$
($\mathrm{m}$) for the range and $[0.1,0.1,0.2]$ ($\mathrm{m}$) for the voxel
size. On KITTI, we use $[0,70.4]\times[-40,40]\times[-3,1]$ ($\mathrm{m}$) for
the range and $[0.05,0.05,0.1]$ ($\mathrm{m}$) for the voxel size. We use
leverage the same 3D and 2D detections as EagerMOT [16].
Training and Inference Details. Our learned affinity estimation is implemented
in PyTorch [30]. We use OpenPCDet [39] for the Voxel Backbone described in
Section 3.1. The network is trained on a NVIDIA Tesla V100 (32GB) GPU. The
Adam optimizer [17] is used with an initial learning rate of 0.03 and is
modified using the one-cycle learning rate policy [36]. We train the model for
20 epochs on the nuScenes dataset [4] and 100 epochs on the KITTI dataset [14]
using a batch size of 4. We use previous detections $\mathbf{D}_{t-1}$ to
represent tracks $\mathbf{T}_{t-1}$ during training. We use the 3D Kalman
filter for track prediction on the KITTI [14] dataset due to the lack of
velocity estimates in KITTI detections. The values $\alpha=0.25$ and
$\gamma=2.0$ are used for the focal loss parameters in Equation 3. We set
$\tau_{\mathrm{rej}}=0.6$ and $\tau_{\mathrm{gt}}=0.55$ for the IoU thresholds
in the track overlap rejection and affinity label generation steps,
respectively. We set our detection perturbation standard deviations as
$(\sigma_{x},\sigma_{y})=(0.01,0.01)$, and set our detection drop fraction
range as $(d_{\mathrm{min}},d_{\mathrm{max}})=(0.0,0.2)$. We inherit all
tracking related parameters from EagerMOT [16].

###  4.1 nuScenes Dataset Results

We use the official nuScenes 3D MOT evaluation [4] for comparison, reporting
the AMOTA and AMOTP metrics. Table 1 shows the results of InterTrack on the
nuScenes test set [4], compared to 3D MOT methods using CenterPoint [45]
detections for fair comparison. We observe that InterTrack outperforms
previous methods by a large margin on overall AMOTA (+1.00%) with a similar
AMOTP (+0.01m).

InterTrack greatly outperforms previous methods on classes with smaller
physical sizes, with considerable margins of +2.20%, +4.70%, and +2.60% on the
Bicycle, Motorcycle, and Pedestrian classes. We attribute the performance
increase to the dense clustering of smaller objects that exists in the
nuScenes dataset. Heuristic metrics often have trouble distinguishing between
clustered objects (see Figure 1), while our Interaction Transformer produces
features with more discrimination leading to correct association. We reason
heuristic metrics are sufficient on larger objects due to increased
separation, as InterTrack lags behind prior methods on the Bus, Car, and
Trailer classes with margins of -1.40%, -0.60%, and -0.70%, and outperforms
the Truck class (+0.30%). We note InterTrack achieves 5 first place rankings.

###  4.2 KITTI Dataset Results

We follow AB3DMOT [41] for 3D MOT evaluation on the KITTI dataset [14]. Table
2 shows the results of InterTrack on the KITTI val set [14] for the Car class
compared to 3D MOT methods. InterTrack achieves 1st place on the KITTI
benchmark [41] with a 0.3% margin on sAMOTA.

Method |  sAMOTA$\uparrow$ |  AMOTA$\uparrow$ |  AMOTP $\uparrow$
---|---|---|---
AB3DMOT† [41] | 91.8 | 44.3 | 77.4
GNN3DMOT† [41] | 93.7 | 45.3 | 78.1
EagerMOT [16] | 94.9 | 48.8 | 80.4
InterTrack (Ours) | 95.2 | 48.8 | 80.3
Improvement | +0.30 | +0.00 | -0.10
Table 2: 3D MOT results on the KITTI val set [14] for the Car class with the
AB3DMOT [41] evaluation. † indicates methods using 3D detections from
PointRCNN [35].

###  4.3 Ablation Studies

Exp | Model |  AMOTA$\uparrow$ |  AMOTP$\downarrow$ |  AssA $\uparrow$
---|---|---|---|---
1 | FFN | 63.4 | 0.749 | 48.2
2 | GNN | 66.3 | 0.688 | 51.1
3 | Trans | 72.1 | 0.566 | 56.2
Table 3: Object Feature Learning Ablation on the nuScenes [4] validation set.
We compare tracking performance when using a feed-forward network, a graph
neural network, and a transformer for feature learning.

We provide ablation studies on our network to validate our design choices. The
results are shown in Tables 3 and 4.

Figure 4: Discriminative Comparison for Feature Learning Models. We plot
cosine similarity and affinity estimate $\mathbf{A}_{t}(m,n)$ distributions
for both same and different object pairs for the FFN, GNN, and Transformer
feature learning models. The distribution means (shown as dotted lines) and
Jensen–Shannon divergence (JSD) between the distributions are shown.

Object Feature Learning. Table 3 shows the tracking performance when different
feature learning models are used for affinity estimation. Specifically, we
swap out the Interaction Transformer (Section 3.1) with a feed-forward network
(FFN) and a graph neural network (GNN). We evaluate the models using AMOTA,
AMOTP, AssA on the nuScenes [4] validation set.

Experiment 3 in Table 3 uses a FFN to learn object features independently,
applying 8 Linear-BatchNorm-ReLU blocks. Experiment 3 uses a GNN to add sparse
interaction modeling. We follow the model architecture of GNN3DMOT [42], and
only construct edges between objects that are (1) in different frames and (2)
within 5 meters in 3D space. Adding the GNN leads to an improvement (+2.90%,
-0.061m, +2.90%) when interactions are considered. We implement our own GNN
for fair comparison instead of using the official GNN3DMOT [42] results, which
report poor performance as a result of using lower quality MEGVII [49]
detections. Experiment 3 uses the Transformer model (described in Section 3.1)
for feature learning, with both self-attention and cross-attention layers.
Leveraging the Transformer model leads to large performance improvements
(+5.80%, -0.122m, +5.10%) due to the ability to model all object interactions,
validating the use of the Transformer for 3D MOT.

We also directly compare the discrimination ability of the feature learning
models in Figure 4. Specifically, we compute both feature similarity and
affinities for all object pairs, and plot separate distributions for same and
different object pairs. Same objects are object pairs with the same identity
and different objects are intra-class object pairs with different identities.
We assign ground truth object identities to objects using the 3D IoU matching
strategy described in Section 3.3. Feature similarity is evaluated using the
cosine similarity between track and detection feature vectors,
$\mathbf{\tilde{T}}_{t-1}^{\mathrm{feat}},\mathbf{\tilde{D}}_{t}^{\mathrm{feat}}$,
produced by the Interaction Transformer (see Section 3.1). Affinity is
directly taken from the affinity matrices, $\mathbf{A}_{t}$, produced by the
Affinity Head (see Section 3.1).

We measure discrimination as the amount of separation between same and
different object distributions, as increased separation reduces ambiguity when
matching object pairs. As a measure of separation, we compare the distribution
means and compute the Jensen–Shannon divergence (JSD). The Jensen–Shannon
divergence (JSD) is used over the standard Kullback–Leibler (KL) divergence
due to the symmetrical and always finite output of the metric.

As in Table 3, we observe improvements with increased interaction modeling in
Figure 4. Sparse interaction modeling with the GNN shows an improvement over
independent feature extraction with the FFN, with increased mean difference
(+0.24) and JSD difference (+0.22) for feature similarities and an increased
mean difference (+0.06) and JSD difference (+0.01) for affinities. Adding
complete interaction modeling with the Transformer leads to an increased mean
difference (+0.21) and JSD difference (+0.21) for feature similarities and an
increased mean difference (+0.01) and JSD difference (+0.03) for affinities,
validating the use of the Transformer for discriminative feature and affinity
estimation. Based on Table 3 and Figure 4, we note that discrimination
increases lead to increases in AMOTA, indicating the importance of object
discrimination for tracking performance.

Further, we observe different characteristics between the cosine similarity
and affinity distributions in Figure 4, which we attribute to the difficulty
and frequency imbalance between positive (same object) and negative (different
object) matches in the nuScenes dataset [4]. For the GNN and Transformer, we
note a dense clustering of same object feature similarities near 1 (maximum
similarity) and a larger spread of different object similarities. We reason a
high difficulty of positive matches would require a high feature similarity to
identify correctly, while a negative match could be identified with a range of
lower similarities. Conversely, we observe a dense clustering of different
object affinities near 0 (minimum affinity), attributed to the large amount of
easy, negative matches to identify in the dataset. We note a larger spread of
same object affinities which we attribute to the challenging nature of
positive match identification.

Exp | MA | Rej | $\mathbf{A}_{t}$ | Pred |  AMOTA$\uparrow$ |  AMOTP$\downarrow$
---|---|---|---|---|---|---
1a | G |  | Heuri | KF | 71.2 | 0.569
2a | ✓ | Heuri | KF | 71.3 | 0.568
3a | ✓ | Learn | KF | 71.4 | 0.586
4a | ✓ | Learn | Vel | 71.6 | 0.585
1b | H |  | Heuri | KF | 63.8 | 0.635
2b | ✓ | Heuri | KF | 65.8 | 0.628
3b | ✓ | Learn | KF | 71.8 | 0.573
4b | ✓ | Learn | Vel | 72.1 | 0.566
Table 4: 3D Tracking Ablation on the nuScenes [4] validation set. MA indicates
the matching algorithm in the 1 stage data association, where G is Greedy and
H is Hungarian. Rej indicates track overlap rejection. $\mathbf{A}_{t}$
indicates the 1 stage affinity estimation method, where Heuri indicates the
EagerMOT [16] heuristic and Learn indicates our learned approach. Pred
indicates the prediction model, where KF indicates the Kalman filter and Vel
indicates the CenterPoint [45] velocity-based motion model.

Track Rejection. Experiment 4 in Table 4 shows the tracking performance of the
EagerMOT [16] baseline without any additions, while Experiment 4 shows the
performance when using the Hungarian algorithm [18] in the first stage data
association. We add a track rejection based on overlap (see Section 3.2) in
Experiments 4 and 4, leading to improvements on the Greedy (+0.10%, -0.001m)
and Hungarian (+2.00%, -0.007m) trackers on AMOTA and AMOTP. We attribute the
performance increase to the removal of duplicated trajectories estimated by
our tracker.
Affinity Matrix Estimation. Experiments 4 and 4 in Table 4 replace the
heuristic affinity matrix estimation method used in EagerMOT [16] with the
learned affinity $\mathbf{A}_{t}$ estimation method outlined in Section 3.1.
We note improvements (+0.10%, -0.002m and +6.00%, -0.055m), which we attribute
the increased discrimination of our feature interaction mechanism.
Track Prediction/Update. Experiments 4 and 4 in Table 4 replace the Kalman
filter with the estimated velocity prediction and assignment update strategy
outlined in Section 3.2). We observe a performance improvement (+0.20%,
-0.001m and +0.30%, -0.007m) over the Kalman filtering approach, which we
credit to the low accuracy of velocity propagation when filtering, especially
when operating at the low 2 Hz update rate of the nuScenes dataset. When
ingesting detections without velocity estimates or when operating at a higher
update rate, InterTrack may benefit from a Kalman filtering approach.
Data Association Method. Table 4 shows the performance for all experiments
when both the Greedy and Hungarian algorithms are used in the first stage data
association. We observe that the Greedy algorithm (Experiments 4 and 4) offers
higher performance (+7.40%, -0.066m and +5.50%, -0.060m) over the Hungarian
algorithm (Experiments 4 and 4) when a heuristic affinity metric is used. We
attribute this performance difference to the globally optimal solution that
the Hungarian algorithm provides [18] which considers all affinity scores for
all matches. We reason that a heuristic affinity metric produces more
incorrect scores. For a majority of matches, the greedy approach only
considers a subset of affinities in which incorrect affinity scores are
ignored.

We observe better performance (+0.40%, -0.013m and +0.50%, -0.019m) for the
Hungarian algorithm (Experiments 4 and 4) compared to the Greedy algorithm
(Experiments 4 and 4) when the affinity is learned. Similarly, we attribute
this performance difference to the globally optimal solution from the
Hungarian algorithm that takes advantage of our improved affinity estimates.
The combination of learned affinity estimation and Hungarian algorithm leads
to the best results, validating its use for our method.

Additionally, we observe larger performance gains (+2.00%, -0.007m and +6.00%,
-0.055m) for the Hungarian methods from Experiments 4 to 4 compared to gains
(+0.10%, -0.001m and +0.10%, -0.002m) for the Greedy methods from Experiments
4 to 4. We credit the performance gain difference to the high baseline
performance (71.2%, 0.569m) when using the Greedy algorithm (Experiment 4), in
which further additions have less effect.

##  5 Conclusion

We have presented InterTrack, a novel 3D multi-object tracking method that
extracts discriminative track and detection features for data association via
end-to-end learning. The Transformer is used as a feature interaction
mechanism, shown to be effective for increasing 3D MOT performance due to its
integration of complete interaction modeling. We outline updates to the
EagerMOT [16] tracking pipeline leading to improved performance when paired
with the Interaction Transformer. Our contributions lead to a 1 place ranking
on the nuScenes 3D MOT benchmark [4] among methods using CenterPoint [45]
detections.

## References

  * [1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai.  TransFusion: Robust Lidar-Camera Fusion for 3d Object Detection with Transformers.  CVPR, 2022.
  * [2] Alex Bewley, ZongYuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  ICIP, 2016.
  * [3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  NeurIPS, 2020.
  * [4] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  CVPR, 2020.
  * [5] Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua.  Brief: Binary robust independent elementary features.  ECCV, 2010.
  * [6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  ECCV, 2020.
  * [7] Hsu-kuang Chiu, Jie Li, Rares Ambrus, and Jeannette Bohg.  Probabilistic 3d multi-modal, multi-object tracking for autonomous driving.  ICRA, 2021.
  * [8] Hsu-kuang Chiu, Antonio Prioletti, Jie Li, and Jeannette Bohg.  Probabilistic 3d multi-object tracking for autonomous driving.  arXiv preprint, 2020.
  * [9] Christopher B Choy, JunYoung Gwak, Silvio Savarese, and Manmohan Chandraker.  Universal correspondence network.  NeurIPS, 2016.
  * [10] Peng Chu and Haibin Ling.  Famnet: Joint learning of feature, affinity and multi-dimensional assignment for online multiple object tracking.  ICCV, 2019.
  * [11] Gioele Ciaparrone, Francisco Luque Sánchez, Siham Tabik, Luigi Troiano, Roberto Tagliaferri, and Francisco Herrera.  Deep learning in video multi-object tracking: A survey.  Neurocomputing, 381:61–88, 2020.
  * [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  NAACL, 2019.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [14] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  CVPR, 2012.
  * [15] Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang, Ji Lin, Min Sun, Philipp Krähenbühl, Trevor Darrell, and Fisher Yu.  Joint monocular 3d vehicle detection and tracking.  ICCV, 2019.
  * [16] Aleksandr Kim, Aljosa Osep, and Laura Leal-Taixé.  Eagermot: 3d multi-object tracking via sensor fusion.  ICRA, 2021.
  * [17] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  ICLR, 2015.
  * [18] H. W. Kuhn.  The hungarian method for the assignment problem.  Naval Research Logistics Quarterly, 1955.
  * [19] Lingyun Luke Li, Bin Yang, Ming Liang, Wenyuan Zeng, Mengye Ren, Sean Segal, and Raquel Urtasun.  End-to-end contextual perception and prediction with interaction transformer.  IROS, 2020.
  * [20] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  PAMI, 2018.
  * [21] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  arXiv, 2019.
  * [22] Zhijian Liu, Haotian Tang, Alexander Amini, Xingyu Yang, Huizi Mao, Daniela Rus, and Song Han.  Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation.  arXiv, 2022.
  * [23] David G Lowe.  Distinctive image features from scale-invariant keypoints.  IJCV, 60(2):91–110, 2004.
  * [24] Zhichao Lu, Vivek Rathod, Ronny Votel, and Jonathan Huang.  Retinatrack: Online single stage joint detection and tracking.  CVPR, 2020.
  * [25] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe.  Hota: A higher order metric for evaluating multi-object tracking.  IJCV, 2021.
  * [26] Chenxu Luo, Xiaodong Yang, and Alan L. Yuille.  Exploring simple 3d multi-object tracking for autonomous driving.  ICCV, 2021.
  * [27] Prasanta Chandra Mahalanobis.  On the generalized distance in statistics.  Proceedings of the National Institute of Sciences (Calcutta), 1936\.
  * [28] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixé, and Christoph Feichtenhofer.  Trackformer: Multi-object tracking with transformers.  arXiv preprint, 2021.
  * [29] Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han.  Large-scale image retrieval with attentive deep local features.  ICCV, 2017.
  * [30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS. Curran Associates, Inc., 2019.
  * [31] Hamed Pirsiavash, Deva Ramanan, and Charless C. Fowlkes.  Globally-optimal greedy algorithms for tracking a variable number of objects.  CVPR, 2011.
  * [32] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  Orb: An efficient alternative to sift or surf.  ICCV, 2011.
  * [33] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  CVPR, 2020.
  * [34] Samuel Scheidegger, Joachim Benjaminsson, Emil Rosenberg, Amrit Krishnan, and Karl Granström.  Mono-camera 3d multi-object tracking using deep learning detections and PMBM filtering.  IV, 2018.
  * [35] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  PointRCNN: 3D object proposal generation and detection from point cloud.  CVPR, 2019.
  * [36] Leslie N. Smith.  A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay.  arXiv preprint, 2018.
  * [37] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  CVPR, 2021.
  * [38] Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo.  Transtrack: Multiple-object tracking with transformer.  arXiv preprint, 2020.
  * [39] OpenPCDet Development Team.  Openpcdet: An open-source toolbox for 3d object detection from point clouds.  https://github.com/open-mmlab/OpenPCDet, 2020.
  * [40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 2017.
  * [41] Xinshuo Weng and Kris Kitani.  A baseline for 3d multi-object tracking.  IROS, 2020.
  * [42] Xinshuo Weng, Yongxin Wang, Yunze Man, and Kris Kitani.  GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with 2D-3D Multi-Feature Learning.  CVPR, 2020.
  * [43] Xinshuo Weng, Ye Yuan, and Kris Kitani.  PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling.  ICRA RA-L, 2021.
  * [44] Yan Yan, Yuxing Mao, , and Bo Li.  SECOND: Sparsely embedded convolutional detection.  Sensors, 2018.
  * [45] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Center-based 3d object detection and tracking.  CVPR, 2021.
  * [46] Fangao Zeng, Bin Dong, Tiancai Wang, Xiangyu Zhang, and Yichen Wei.  Motr: End-to-end multiple-object tracking with transformer.  arXiv preprint, 2021.
  * [47] Yihan Zeng, Chao Ma, Ming Zhu, Zhiming Fan, and Xiaokang Yang.  Cross-modal 3d object detection and tracking for auto-driving.  IROS, 2021.
  * [48] Li Zhang, Yuan Li, and Ramakant Nevatia.  Global data association for multi-object tracking using network flows.  CVPR, 2008.
  * [49] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu.  Class-balanced grouping and sampling for point cloud 3d object detection.  arXiv preprint, 2019.

##  Appendix Appendix A Additional Results

We note that InterTrack reports more ID switches than comparable methods.
Table 5 reports the ID switch results on the nuScenes validation set for the
EagerMOT baseline and InterTrack. At first glance, the results would suggest
that InterTrack poorly performs data association, however we argue that the
nuScenes ID switch metric is unreliable for comparison between methods. The ID
switch metric is computed at a single confidence threshold, which is selected
as the threshold that results in the highest MOTA [4]. The confidence
threshold operating point can differ between methods, resulting in an unfair
comparison. For example, if the confidence threshold for one method is higher,
fewer track predictions will be considered for evaluation, which will result
in fewer ID switches from simply evaluating at a different confidence
threshold. When analysing the association performance between trackers we
recommend using the more recent HOTA [25] metric. HOTA is advantageous since
it is evaluated over a range of confidence thresholds and can decompose the
evaluation of the tracking task into Detection Accuracy (DetA) and Association
Accuracy (AssA). Under the HOTA evaluation scheme InterTrack clearly
demonstrates that it outperforms the EagerMOT baseline by improving AssA by
$1.7\%$.

Additionally, we ablate our use of a feed-forward network in the Affinity Head
with traditional metrics for affinity computation in Table 6. We compare
against Cosine Similarity and Inner Product metrics computed between detection
and track feature vectors at the input to the Affinity Head. We note strong
performance of the FFN relative to traditional metrics, with improvements of
+4.00%, -0.072, and -1014 on AMOTA, AMOTP, and IDS on the nuScenes validation
set.

##  Appendix Appendix B Potential Negative Impact

We identify surveillance as the most significant negative application of
InterTrack. Our work is designed for autonomous vehicle applications, however
this technology could potentially be applied for 3D tracking in other
applications such as traffic surveillance, where the various objects such as
vehicles, pedestrians, cyclists are tracked. Our work only tracks the 3D pose
information, but one could imagine more information being tracked including
facial identities, license plates, and specific locations that are visited
frequently. We note that all 3D MOT methods have the potential for use in
surveillance.

##  Appendix Appendix C Limitations

Sensor Dependency & Distribution Shift. Our learned Interaction Transformer
introduces a dependency on the raw LiDAR data stream that does not exist for
trackers who operate solely on detector output. While this dependency enables
InterTrack to extract more powerful feature representations for objects, it
limits the flexibility of our model. The Interaction Transformer is sensitive
to the detection and LiDAR point cloud input distributions. We expect that our
model would not generalize to different LiDAR sensors without re-training.
Performance may also degrade if the tracker observes scenes with properties
significantly different from those observed during training. Neither of these
limitations have been experimentally verified, however, future work should
explore generalization of tracking methods beyond single sensor sets or
datasets.
Interactions. Our work specifically targets tracking applications for densely
occupied urban environments, in which interaction information can be helpful
due to close proximity. For sparsely distributed objects, we reason that
heuristic affinities are sufficient and including interaction information is
less beneficial (see Section 4.1). We expect interaction modeling to be useful
for 3D MOT in urban environments and less so for suburban and rural
applications.

Model |  HOTA $\uparrow$ |  DetA $\uparrow$ |  AssA $\uparrow$ |  IDS $\downarrow$
---|---|---|---|---
EagerMOT [16] | 24.9 | 13.5 | 54.5 | 899
InterTrack (Ours) | 25.4 | 13.5 | 56.2 | 1019
Table 5: Comparison of Data Association Metrics on the nuScenes [4] validation set. We compare InterTrack with the EagerMOT [16] baseline. Affinity Estimation |  AMOTA$\uparrow$ |  AMOTP $\downarrow$ |  IDS $\downarrow$
---|---|---|---
Inner Product | 66.2 | 0.689 | 3561
Cosine Similarity | 68.1 | 0.638 | 2033
FFN (Ours) | 72.1 | 0.566 | 1019
Table 6: Affinity Estimation Ablation on the nuScenes val set.