# Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey

Gaoang Wang, Shengyu Hao, Mingli Song, and Jenq-Neng Hwang  G. Wang and S. Hao
are with the Zhejiang University-University of Illinois at Urbana-Champaign
Institute, Zhejiang University, China.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>M. Song is with the
College of Computer Science and Technology, Zhejiang University, China.
E-mail<EMAIL_ADDRESS>J.-N. Hwang is with the Department of Electrical
and Computer Engineering, the University of Washington, USA.
E-mail<EMAIL_ADDRESS>
###### Abstract

Multi-object tracking (MOT) aims to associate target objects across video
frames in order to obtain entire moving trajectories. With the advancement of
deep neural networks and the increasing demand for intelligent video analysis,
MOT has gained significantly increased interest in the computer vision
community. Embedding methods play an essential role in object location
estimation and temporal identity association in MOT. Like other computer
vision tasks, such as image classification, object detection, re-
identification, and segmentation, embedding methods in MOT have large
variations. However, they have never been systematically analyzed and
summarized. In this survey, we first conduct a comprehensive overview with an
in-depth analysis of embedding methods in MOT from seven different
perspectives, including patch-level embedding, single-frame embedding, cross-
frame joint embedding, correlation embedding, sequential embedding, tracklet
embedding, and cross-track relational embedding. Then, we analyze the
advantages of existing state-of-the-art methods according to their embedding
strategies on widely used MOT benchmarks. Finally, some critical yet under-
investigated areas and future research directions are discussed.

###### Index Terms:

Multi-Object Tracking, Embedding Methods, Literature Survey, Evaluation
Metric, State-of-the-Art Analysis.

##  1 Introduction

Multi-object tracking (MOT) has been widely studied in recent years, aiming to
associate detected objects across video frames to obtain the entire moving
trajectories. With the development of tracking algorithms, MOT can be applied
in many tasks, such as traffic flow analysis [1, 2, 3, 4], human behavior
prediction and pose estimation [5, 6, 7, 8], autonomous driving assistance [9,
10], and even underwater animal abundance estimation [11, 12, 13].

The flow of MOT systems can be mainly divided into two parts, i.e., the
embedding model and the association algorithm. With multiple successive frames
as input, the object locations and track identities (IDs) are estimated via
the embedding techniques and association approaches. MOT is challenging due to
the presence of illumination changes, occlusions, complex environments, fast
camera movement, unreliable detections, and varying low-image resolutions
[14]. In addition, the tracking performance can be affected by individual
steps of tracking algorithms, such as detection, feature extraction, affinity
estimation, and association. These result in significant variations and
uncertainty. With the recent progress in representation learning with deep
neural networks, embedding methods play an essential role in object location
estimation and temporal identity association in MOT. In this survey, we focus
on the review of embedding learning rather than association methods [15, 16,
17, 18], though association methods are also important in MOT.

Like other computer vision tasks [19, 20, 21, 22], such as image
classification, object detection, object retrieval, crowd counting, visual
tracking, re-identification (Re-ID), and segmentation, embedding methods in
MOT also have large variations. However, the embedding learning approaches in
MOT have never been systematically analyzed and summarized. Some embedding
methods combine multi-task heads [23, 24, 25, 26, 27], including box
regression, object classification, and re-identification. Some embedding
methods consider spatial-temporal correlations [28, 29, 30, 31, 32], combining
both appearance and motion information. Some methods exploit the interaction
relationships among objects with correlation and attention to learn the track
embeddings [33, 34, 35]. The large deviation of embedding methods motivates us
to conduct a comprehensive survey from an embedding perspective and discuss
several under-investigated embedding areas and future directions.

Figure 1: A taxonomy of MOT embedding methods. The green and red boxes show
the categories and representative references, respectively.

There are some existing surveys [36, 37, 38, 39, 40] of MOT published in
recent years. Specifically, Xu et al. [36] summarize some deep learning-based
trackers and deep neural network structures. Emami et al. [37] focus on the
review of model-based multiple hypotheses tracking with machine learning
techniques in detection, filtering, and association. Ciaparrone et al. [38]
provide a review of how deep learning is adopted in MOT, including detection,
feature extraction, affinity computation, and association. Park et al. [39]
review the evolution of MOT in recent decades, focusing on deep learning
techniques and investigating the recent advances in MOT. Luo et al. [40]
provide a review of the MOT systems and discuss approaches from different
aspects. Unlike all existing surveys, we focus on embedding learning in MOT,
i.e., how to learn object-level representative features for the MOT task, and
comprehensively analyze state-of-the-art methods according to the embedding
strategies. The main contributions of this survey are summarized as follows:

1) We categorize and summarize existing embedding methods designed for MOT and
make an in-depth and comprehensive analysis by discussing the advantages and
limitations of the methods. The summary provides insights for future algorithm
design and new topic exploration. 2) We summarize the widely used benchmarks
and analyze the state-of-the-art approaches according to the embedding
methods. 3) We attempt to discuss several important research directions with
under-investigated issues related to embedding techniques and take a step
towards future trends in MOT.

The outline of this survey is summarized as follows. We first demonstrate
related works, including the most relevant tasks of MOT, in Section 2. A
taxonomy and detailed review of embedding methods are provided in Section 3.
We then summarize the evaluation metrics and analyze the state-of-the-art
approaches according to the embedding methods in Section 4. We discuss several
under-investigated issues and point out the trends and potential future
studies in Section 5. Conclusions are drawn in Section 6.

##  2 MOT Related Tasks

Three tasks, i.e., single object tracking (SOT), video object detection (VOD),
and re-identification (Re-ID), are highly related to MOT. Many embedding
methods in the MOT field are inspired by these tasks. A comparison and brief
review of these tasks are demonstrated in this section.

###  2.1 Single Object Tracking

Single-object tracking (SOT), also named visual object tracking (VOT), aims to
estimate an unknown visual target trajectory when only an initial state of the
target is available [41]. The tracking target is purely determined by the
first frame and does not rely on any categories. Inspired by deep learning
breakthroughs [42, 43, 44] in ImageNet large-scale visual recognition
competition (ILSVRC) [45] and visual object tracking (VOT) challenges [46,
47], embedding learning methods have attracted considerable interest in the
visual tracking community. Many works attempt the tracking by learning
discriminative target representations, such as learning distractor-aware [48]
or target-aware [49] features; leveraging different types of deep features
such as context information [50, 51] or temporal features [52, 53]; exploring
of low-level spatial features [54, 55]; employing correlation-guided attention
modules to exploit the relationship between the template and RoI feature maps
[56]; and computing correlations between attentional features [57].

Unlike SOT, the initial states of objects are unknown in the MOT task,
requiring pre-defined categories for tracking. As a result, MOT methods either
adopt off-the-shelf detection following a tracking-by-detection scheme [58,
59, 60, 61] or exploit the joint detection and tracking models [62, 28, 29,
23, 24]. Some works [63, 64, 65, 66, 67, 68, 69] incorporate embedding methods
from SOT into MOT with correlation and attention techniques.

###  2.2 Video Object Detection

Video object detection (VOD) aims to detect objects across multiple video
frames by jointly recognizing objects and estimating locations [70]. Embedding
methods are also important in the VOD task. Embedding learning can filter the
features of a video frame, select relatively representative features,
propagate key features for detection, and delineate the key areas that deserve
feature filtering in subsequent frames. For example, Guo et al. [71] propose a
special attention network known as the progressive sparse local attention
framework to propagate features between different video frames. Xiao and Lee
[72] use a module that calculates similar correlations to locally align the
feature map. Chai [73] introduces Patchwork, a method that uses the attention
mechanism to predict the position of an object in the next frame to solve the
video object detection problem. Bertasius et al. [74] propose a spatial-
temporal sampling network that introduces deformable convolution to detect
video frames.

Some embedding methods in VOD can be directly applied to MOT [75]. However,
unlike VOD, MOT usually does not require the recognition of objects unless
objects from multiple categories are tracked jointly. Besides, track ID
prediction is essential in the MOT task but not required in VOD. As a result,
the embedding methods in MOT should be discriminative among objects even in
the same category.

###  2.3 Re-Identification

Re-identification (Re-ID) aims at verifying object identity from different
collections of images, usually from varying angles, illumination, and poses in
different cameras [76, 77, 78]. With the advancement of deep neural networks
and increasing demand for intelligent video surveillance, Re-ID has
significantly increased interest in the computer vision community. Much
progress has been made in feature representation learning in Re-ID. Such
embedding methods include global feature learning that extracts global feature
representation for each image without additional annotation cues [79]; local
feature learning that aggregates part-level local features to formulate a
combined representation [80, 81, 82]; auxiliary feature learning that improves
the feature representation learning using auxiliary information like
attributes [83, 84, 85]; and video feature learning that learns video
representation for video-based Re-ID using multiple image frames and temporal
information [86, 87].

Several MOT works [63, 88, 60] employ Re-ID-based strategies to learn object
appearance embeddings. Unlike the Re-ID task that does not require temporal
information between query and gallery images, temporal consistency is a more
important cue in MOT.

##  3 A taxonomy of MOT Embedding Methods

Embedding methods are essential for object location estimation and ID
association. The definition of embedding can be described as follows. Given
the input data $\mathcal{X}$ and an embedding model
$f_{\boldsymbol{w}}(\cdot)$ parameterized by $\boldsymbol{w}$, we aim to
obtain object-level representations $\boldsymbol{z}$ from

$\boldsymbol{z}=f_{\boldsymbol{w}}(\mathcal{X}).$ (1)

For simplicity, we use $f(\cdot)$ to represent $f_{\boldsymbol{w}}(\cdot)$ in
the following sections, without loss of generality. The input data
$\mathcal{X}$ can include a combination of sequential image frames
$\boldsymbol{X}$, cropped detection images $\boldsymbol{D}$, tracklets
$\mathcal{T}$, and other types of data or modalities. The embedding models
$f(\cdot)$ focus on deep learning-based methods. Other hand-crafted
operations, such as color histograms, SIFT, HOG, and LBP, are not covered in
the paper.

Our proposed taxonomy of MOT embedding methods is shown in Fig. 1. The
representative methods shown in the figure are based on the impact, i.e.,
either received sufficient citations or very recent progress published in top
journals and conferences. We categorize the commonly used MOT embedding
methods into seven groups, including patch-level embedding, single-frame
embedding, cross-frame joint embedding, correlation-based embedding,
sequential embedding, tracklet embedding, and cross-track relational
embedding. The embedding methods are largely dependent on the type of input
data. Some methods can take advantage of temporal information in the model
design, like cross-frame joint embedding, sequential embedding, and tracklet
embedding. Some methods can utilize the spatial information of the entire
frames, like single-frame detection embedding and cross-frame joint embedding.
Some only adopt object-level Re-ID models for feature extraction, like patch-
level embedding. Different embedding strategies may not be mutually exclusive.
One method may combine multiple embedding strategies to boost the tracking
performance. For each category, we introduce the representative algorithms and
then discuss the weaknesses and strengths.

###  3.1 Patch-Level Box Image Embedding

Patch-level box image embedding attempts to extract object-level features from
cropped detection images, which is essential for the tracking-by-detection
scheme. Some approaches treat MOT as a Re-ID problem. Most of the existing
methods [58, 59, 60, 61] in this category focus on embedding individual
detections, while a few methods [89, 90, 91] try to model the relation of two
detected objects directly with pairwise embedding strategies. As a result, we
categorize the patch-level box image embedding into two subgroups, i.e., self-
embedding and pairwise embedding.

####  3.1.1 Self-Embedding

Given the cropped box image based on the detection $\boldsymbol{D}_{i}$, the
self-embedding represents the appearance feature of the detected object. This
can be formulated as follows,

$\boldsymbol{z}_{i}=f(\boldsymbol{D}_{i}),$ (2)

where $f(\cdot)$ represents the embedding network, $\boldsymbol{z}_{i}$
represents the embedding of detection $\boldsymbol{D}_{i}$.

The Re-ID framework with siamese networks is widely used for embedding to
learn discriminative features to distinguish different object identities. Some
existing approaches [58, 59, 60, 92, 63] treat the same objects across
different frames as individual classes and employ cross-entropy loss for ID
classification to learn the embedding of the cropped detection images. Such
methods follow the very conventional embedding strategy that extracts object
features by classification. Some methods [61, 93, 94, 95, 96] adopt triplet
loss and softmax-based contrastive loss to learn discriminative embeddings
within batch samples, in which detections from the same object are treated as
positive samples, and those from different objects are treated as negative
samples. For the triplet loss, tuples of anchor, positive, and negative
samples are constructed and exploited for distance metric learning, with the
same strategy used in FaceNet [97]. Compared with cross-entropy loss, triplet
loss can better distinguish the subtle difference among different IDs,
especially for objects with similar color and pose. For the softmax-based
contrastive loss, maximizing the similarity of positive pairs [98] is adopted.
Unlike triplet loss which only takes three samples each time, contrastive loss
proposed in [96] compares multiple relations among different pairs from
tracklets in the batch and shows promising performance in the embedding.

Most detection-based embedding methods are based on cropped boxes. Except for
bounding boxes, some works take advantage of segmentation masks for learning
embeddings. Segmentation-based MOT can improve the effectiveness of the
embedding, especially for partially occluded objects, and alleviate the
influence of the background regions in the bounding box. For example, based on
the instance segmentation results from [99], PointTrack [100] treats the
pixels of the cropped image as 2D point clouds, and the points from the
foreground target and background environment are jointly learned for each
cropped detection image.

There are several benefits of the patch-level self-embedding method. First, it
is more flexible with variant detection algorithms, Re-ID models, and
association methods, since each part can be trained separately. As a result,
it can always combine off-the-shelf state-of-the-art detectors and Re-ID
models. Second, since it takes cropped images as individual samples, it can
employ a large amount of data in training. Compared with fully annotated video
data, large-scale cropped image datasets are easier to collect. On the other
side, the weakness is also obvious. It takes multiple stages in training,
resulting in sub-optimal solutions. Besides, temporal information is hard to
be learned in the embedding with cropped image samples.

####  3.1.2 Pairwise Embedding

Figure 2: Pairwise patch-level embedding. $\bigoplus$ represents the
concatenation.

Rather than learning embeddings from individual detections, pairwise embedding
networks take pairs of detections and directly learn the similarity between
two detected objects, as shown in Fig. 2. A binary classifier is usually
adopted to indicate whether the two detections belong to the same target,
formulated as follows,

$\boldsymbol{z}_{i,j}=f(\boldsymbol{D}_{i},\boldsymbol{D}_{j}),$ (3)

where $\boldsymbol{z}_{i,j}$ is the pairwise embedding of two detections
$\boldsymbol{D}_{i}$ and $\boldsymbol{D}_{j}$.

Some works [89, 90, 91] follow a similar embedding framework in which
concatenations of detection pairs are regarded as input. Then binary cross-
entropy loss is adopted to distinguish whether two detections belong to the
same object. Instead of concatenating input images, some works concatenate
intermediate feature maps for embedding. For example, OTCD [101], a real-time
online tracker in the compressed domain, follows this strategy, in which an
appearance convolutional neural network is designed to take a pair of object
features from detection to ensure a reliable association. In addition, some
works also combine motion cues into embedding. Specifically, the SiameseCNN
tracker [102] concatenates both cropped image pairs and the corresponding
optical flow maps as input, followed by a siamese convolutional neural network
to learn descriptors encoding local spatial-temporal structures between the
two input image patches and aggregate pixel values and optical flow
information. FANTrack [103] is proposed to take advantage of both appearance
and motion features of paired objects with two-stage learning, where the first
stage learns a similarity function that combines visual and 3D bounding box
data to yield robust matching costs, and the second stage trains a CNN to
predict discrete target assignments from the computed pairwise similarities.
For pairwise embedding, it has similar strengths to patch-level self-
embedding. With paired input, the discrimination is easier to be learned among
different IDs.

###  3.2 Single-Frame Detection Embedding

Figure 3: Single-frame joint detection embedding.

Frame-based embedding jointly learns detection and Re-ID features in an end-
to-end manner, as shown in Fig. 3. Given an input frame $\boldsymbol{X}$, the
network learns the discriminative feature of each detection as follows,

$\\{\boldsymbol{z}_{i}|i=\\{1,2,...,|\mathcal{D}|\\}\\}=f(\boldsymbol{X}),$
(4)

where $|\mathcal{D}|$ is the number of detections of the input frame,
$\boldsymbol{z}_{i}$ is the embedding for the corresponding detection
$\boldsymbol{D}_{i}$.

For single-frame based embedding, existing works usually follow joint
detection and tracking framework and learn object-level embedding from
detector-based architectures. Similar to object detection, such methods are
designed to estimate object classes and locations. Furthermore, they are also
required to ensure that the learned object embeddings are object-sensitive,
i.e., embeddings from different objects should keep dissimilar even if they
are from the same class label. Some works use anchor-based frameworks for
embedding learning, which are built on top of state-of-the-art detectors.
Specifically, some [62, 23] adopt Faster R-CNN [104]; RetinaTrack [26] follows
RetinaNet [105]; Wang et al. [27] exploit the YOLOv3 framework with DarkNet-53
[106]. However, as mentioned in [24], anchors are not fit for Re-ID features
in nature. As a result, some works [107, 24] adopt anchor-free frameworks,
which are built upon CenterNet [108] with deep layer aggregation (DLA) [109,
29, 110] used in the backbone architecture.

One of the major challenges for jointly learning embeddings for detection and
Re-ID arises from the conflict between the two tasks. The detection task aims
to identify object categories, such as pedestrians and vehicles, from the
background, while Re-ID embeddings aim to distinguish distinct objects rather
than the class. Some works decouple the embeddings from different tasks to
address the issue with multi-task learning. JDE [27] is one of the pioneer
works that learn Re-ID embeddings built on top of detectors. The feature
pyramid network (FPN) is employed in JDE, and task-specific heads are added on
multiple FPN scales. Similarly, RetinaTrack [26] designs task-specific post-
FPN layers for classification and bounding box regression and generates
embedding vectors simultaneously. FairMOT [24] is one of the representative
methods that address the competition of multiple tasks in joint detection and
tracking, which adopts the anchor-free object detection architecture CenterNet
[108]. In FairMOT, a single image frame is fed to an encoder-decoder network
to extract high-resolution feature maps, followed by two homogeneous branches
for detecting objects and extracting re-ID features, respectively. Then the
features at the predicted object centers are used for tracking. Authors of
FairMOT find that point-based embeddings perform much better than anchor-based
embeddings in the tracking. Following the multi-task learning framework of
FairMOT, UTrack [107] designs an unsupervised Re-ID framework based on strong
and weak supervision of the association cues from two frames. RelationTrack
[25] is another method that attempts to alleviate the contradiction between
detection and Re-ID tasks. Authors devise a global context disentangling
module that decouples the learned representation into detection-specific and
Re-ID-specific embeddings. They further develop a guided Transformer encoder
module by combining the powerful reasoning ability of the Transformer encoder
and deformable attention to consider the global semantic relation in the
tracking. Moreover, CCPNet [111] decouples different tasks with multiple
decoders to learn instance segmentation and ID-based embeddings for multi-
object tracking and segmentation (MOTS) tasks. Meanwhile, a novel data
augmentation strategy named continuous copy-paste is presented to fully
exploit the pixel-wise annotations provided by MOTS to actively increase the
number of instances as well as unique instance IDs in training. Yang et al.
[23] adopt a similar embedding strategy for decoupling. In addition, they
expand the training data by using the historical positions of the target
combined with the prediction of the motion model to improve the temporal
correlation of extracted appearance features and employ the differentiable
multi-objective tracking metric, i.e., dMOTA and dMOTP [112], in training.
MTrack [113] presents a novel embedding strategy apart from the detection
head, namely multi-view trajectory contrastive learning, in which the
discriminative representations are learned based on the contrast between a
dynamically updated memory bank of trajectories and object features.

Compared with patch-level embedding, single-frame embedding methods do not
rely on pre-trained detectors and do not require extra computation and storage
to crop the detection patches. This one-stage training strategy increases the
embedding efficiency. However, frame-based methods cannot efficiently utilize
crop-based Re-ID datasets for training, causing a degradation for
generalization. Moreover, temporal consistency is still not well-exploited in
the embedding framework.

###  3.3 Cross-Frame Joint Embedding

To jointly learn the appearance and temporal features across multiple frames,
cross-frame embedding plays an important role in MOT. The embedding can be
formulated as follows,

$\\{\boldsymbol{z}_{i}^{t}|i=\\{1,2,...,|\mathcal{D}^{t}|\\}\\}=f([\boldsymbol{X}^{t-\tau};\boldsymbol{X}^{t-\tau+1};...;\boldsymbol{X}^{t}]),$
(5)

where
$[\boldsymbol{X}^{t-\tau};\boldsymbol{X}^{t-\tau+1};...;\boldsymbol{X}^{t}]$
represents the concatenation of multiple frames from $t-\tau$ to $t$. Some
approaches [29, 30, 31] adopt embedding networks, such as 3D convolution and
convolutional LSTMs, that learn spatial-temporal feature maps for tracking,
and some approaches [114, 115, 116] extract features for individual frames and
then aggregate embeddings to model temporal relations in the task-specific
heads. These two sub-categories are demonstrated in the following sub-
sections.

####  3.3.1 Multi-Frame Spatial-Temporal Embedding

Figure 4: Multi-frame spatial-temporal embedding.

The multi-frame spatial-temporal embedding can be formulated as follows,

$\displaystyle\boldsymbol{F}^{t}=f([\boldsymbol{X}^{t-\tau};\boldsymbol{X}^{t-\tau+1};...;\boldsymbol{X}^{t}]),$
(6)
$\displaystyle\\{\boldsymbol{z}_{i}^{t}|i=\\{1,2,...,|\mathcal{D}^{t}|\\}\\}=g(\boldsymbol{F}^{t}),$

where $\boldsymbol{F}^{t}$ is the intermediate feature maps obtained from the
backbone embedding network $f(\cdot)$ that learns spatial-temporal information
from sequential frames
$[\boldsymbol{X}^{t-\tau};\boldsymbol{X}^{t-\tau+1};...;\boldsymbol{X}^{t}]$,
and $g(\cdot)$ is a head network that generates the final embeddings
$\\{\boldsymbol{z}_{i}^{t}|i=\\{1,2,...,|\mathcal{D}^{t}|\\}\\}$ of objects.
This subcategory can be regarded as an early fusion of multiple frames for
embedding, as shown in Fig. 4.

TubeTK [30] is a representative method that takes advantage of spatial-
temporal information from short video clips. 3D-ResNet [117, 44] is employed
as the backbone to generate feature maps, followed by the regression of
bounding tubes in a 3D manner, trained with a combination of multiple losses,
including GIOU [118], focal loss [105], and binary cross-entropy loss.
Similarly, DMM-Net [31] employs 3D convolution to learn spatial-temporal
embeddings given multiple frames to generate tubes and predicts multi-frame
motion, classes, and visibility to generate tracklets. Wan et al. [32] take
multiple frames using an encoder-decoder architecture with temporal priors
embedding based on short connections to estimate multi-channel trajectory maps
simultaneously, including the presence map, appearance map, and motion map.
Rather than tracking objects with bounding boxes, Track R-CNN [28] is the
first work that extends the MOT task to multi-object tracking and segmentation
(MOTS), conducting tracking and instance segmentation simultaneously. The MOTS
follows the framework of Mask R-CNN [119] and aggregates the feature maps from
multiple frames using 3D convolution and convolutional LSTM [120] for temporal
context embedding, which is an end-to-end method of segmentation-based
embedding for prediction and association. Some works also combine the
prediction from previous frames to guide the embedding and association. For
example, CenterTrack [29] employs the CenterNet framework [108] and
concatenates a pair of sequential frames and the heat map of the previous
frame for joint embedding, object center location estimation, as well as the
size and offset prediction.

With the capability of learning temporal consistency with 3D neural networks
and LSTMs, motion features can be incorporated into the embedding framework.
However, current spatial-temporal embedding usually only considers a few
frames for joint embedding. As a result, the learned temporal motion features
are not robust enough to model the diverse movement of objects. Learning the
long-time dependency also needs to be further studied in future works.

####  3.3.2 Head-Level Feature Aggregated Embedding

Figure 5: Head-level aggregated embedding.

Slightly different from the multi-frame spatial-temporal embedding, methods of
this sub-category usually extract features of input frames independently and
then aggregate the feature maps in the head network for embedding, which can
be formulated as follows,

$\displaystyle\boldsymbol{F}^{t}=f(\boldsymbol{X}^{t}),$ (7)
$\displaystyle\\{\boldsymbol{z}_{i}^{t}|i=\\{1,2,...,|\mathcal{D}^{t}|\\}\\}=g([\boldsymbol{F}^{t-\tau};\boldsymbol{F}^{t-\tau+1};...;\boldsymbol{F}^{t}]),$

where $f(\cdot)$ and $g(\cdot)$ represents the backbone network and head
network, respectively. This sub-category can be regarded as the late fusion of
spatial-temporal embeddings, as shown in Fig. 5.

Dense pair aggregation. Some works aggregate embeddings for dense pairwise
matching. Specifically, DAN [114] presents a deep affinity network that
predicts the dense affinity of detection center locations between a pair of
frames using the features extracted from different layers of VGGNet [43],
leading to reliable online tracking performance. Similar to DAN [114], DEFT
[9] designs a matching head to aggregate the embeddings from pairs of frames.
Besides that, motion forecasting with LSTM is exploited in the matching head
for association in DEFT. TraDeS [121] also follows a dense pair aggregation
scheme, where the proposed cost volume-based association module extracts
point-wise re-ID embedding features by the backbone to construct a cost volume
that stores matching similarities between the embedding pairs in adjacent two
frames. Then the motion-guided feature warper module is employed to jointly
learn the 2D and 3D locations and masks based on the propagation of offsets
with motion cues using deformable convolution [122]. The dense pair
aggregation can be applied in the graph neural networks (GNN). For example,
GSDT [123] aggregates node embeddings of detected objects based on GNN using
GraphConv [124] given a pair of sequential frames. The dense pair aggregation
can also be employed in the pose embedding and tracking scenarios. For
example, POINet [125] proposes an end-to-end pose-guided insight network for
the data association in multi-person pose tracking, which jointly learns
feature extraction, similarity estimation, and identity assignment. POINet
includes two main components, i.e., a pose-guided representation network and
an ovonic insight network, where the pose-guided representation network is
used for feature extraction, and the ovonic insight network aggregates the
dense object pairs from two frames and estimates the similarity based on
forward and backward insight.

Feature map aggregation. Apart from aggregating dense embeddings, some works
concatenate feature maps from a pair of frames for prediction. Chained-tracker
[115] is one of the representative embedding approaches, which adopts the
stacked feature maps for bounding box regression and ID verification following
Faster R-CNN [104], providing a straightforward and standard framework for
feature map aggregation methods. Except for the prediction from concatenated
feature maps, Li et al. [126] also construct spatial-temporal optimization
with both motion and geometry cues from sequential stereo images in training.
A similar idea can be utilized in pose-tracking scenarios. Jin et al. [127]
propose a unified pose estimation and tracking framework with two main
components, i.e., SpatialNet, and TemporalNet. The produced heatmaps of
sequential frames from the SpatialNet are aggregated to learn the keypoint
embedding and temporal instance embedding in the TemporalNet module for
tracking. HandLer [128] is a hand tracker which takes features from two
consecutive frames and designs a forward propagation with flow estimation and
temporal feature aggregation, followed by a backward regression for offset
prediction and detection. Unicorn [129] presents a unified method that can
simultaneously solve four tracking problems (SOT, MOT, VOS, MOTS) with a
single network, which is the first work that accomplishes the great
unification of the tracking network architecture and learning paradigm. Based
on the feature maps generated from two consecutive frames, a feature
interaction module is exploited to build pixel-wise correspondence between two
frames. Based on the correspondence, a target prior is generated by
propagating the reference target to the current frame. Then the target prior
and the visual features are fused and sent to the detection head to get the
tracked objects for all tasks. Several recent works adopt Transformer
architecture for aggregating feature maps for embedding. Time3D [130] extends
the tracking to 3D for autonomous driving scenarios and introduces a spatial-
temporal information flow module that aggregates geometric and appearance
features to predict robust similarity scores across all objects in current and
past frames based on the attention mechanism of the Transformer. UTT [131]
presents a unified Transformer tracker that combines MOT and SOT in one
paradigm. After extracting frame features, the information is aggregated by a
track Transformer, which consists of one target decoder to extract target
features, one proposal decoder to produce candidate search areas, and one
target transformer to predict the target localization. P3AFormer [132] employs
a Transformer-based model that tracks objects as pixel-wise distributions.
Specifically, consecutive frame features are fed into a Transformer pixel
decoder to obtain the pixel-wise representations. Then the temporal contextual
information is extracted based on pixel-wise feature propagation, followed by
an object decoder that takes masks, image features, and queries for detection
and embedding.

Compared with multi-frame spatial-temporal embedding, head-level aggregated
embedding methods encode each frame individually with shared backbones.
However, it lacks low-level pixel-wise correlation features between frames for
detection and association.

###  3.4 Correlation-Based Embedding

Figure 6: Correlation-based embedding.

Inspired by the SOT methods, target location can be refined via the
correlation between detections and generated feature maps, as shown in Fig. 6.
The correlation-based approaches can be represented as follows,

$\boldsymbol{z}_{i}=g(\boldsymbol{F}\ast f(\boldsymbol{D}_{i})),$ (8)

where $\ast$ is denoted as the correlation operation, $g(\cdot)$ and
$f(\cdot)$ are denoted as the embedding networks. In correlation-based
embedding, features from the previous frame are usually regarded as
correlation kernels of the next frame. As a result, more accurate object
locations can be obtained based on the correlation.

Dense correlation. Some works estimate the dense correlation feature maps. For
example, D&T [75] is one of the pioneer works based on correlation-based
embedding. After obtaining the feature maps from convolutional networks, the
convolutional cross-correlation between the feature responses of adjacent
frames is computed to estimate the local displacement at different feature
scales. Meanwhile, the object class and box are estimated in a separate
detection branch. DASOT [133] uses a similar idea as D&T to compute
correlation feature maps. In their implementation, the correlation operation
is only conducted in a local square neighborhood since displacements of the
same targets across two adjacent frames are limited. In addition, DASOT
integrates data association and SOT in a unified framework, in which dense
correlation feature maps are estimated for the temporal association, built
upon truncated ResNet-50 with the feature pyramid network (FPN) [134]. Rather
than only considering the temporal correlation between a pair of frames, STLC
[135] estimates both multi-scale spatial correlation and temporal correlation
with dense feature maps. Specifically, the spatial correlation module is
established to model the topological relationship between targets and their
surrounding environment, and the temporal correlation module is proposed to
establish frame-to-frame matches over convolutional feature maps in the
different layers to align and propagate temporal context.

Surrounding search. Some works conduct correlations between individual
detections and surrounding local regions via SOT algorithms. For example,
following deep tracker SiamFC [136], UMA [65] uses the siamese network to
calculate the correlation between anchor samples with both positive and
negative local regions. Built upon lightweight AlexNet [42], the network
adopts triplet loss for discriminative learning to generate embeddings. FAMNet
[64] formulates an end-to-end model for MOT with a combination of feature
extraction, affinity estimation, and multi-dimensional assignment. The SOT is
incorporated the proposed affinity network, where the cross-correlation is
conducted between the anchor feature and local regions in adjacent frames for
each anchor object in the center frame. VAN [137] presents a versatile
affinity network that can perform the entire MOT process in a single network,
including target-specific SOT to handle incomplete detection issues, affinity
computation between target and candidates, and decision of tracking
termination. SiamMOT [66] conducts the motion modeling exploration by
leveraging a region-based Siamese MOT network. It combines a region-based
detection network with two motion models, i.e., an implicit motion model (IMM)
and an explicit motion model (EMM), inspired by the literature on siamese-
based SOT. In the proposed EMM, the cross-channel correlation is estimated
between the detected patch and local regions of the next frame, following the
Faster R-CNN detector. OMC [67] follows the joint detection and tracking
framework. After ID embedding, a re-check network is proposed to conduct
cross-correlation between previous embeddings and current feature maps. The
re-check network innovatively expands the role of ID embedding from data
association to motion forecasting by effectively propagating previous
tracklets to the current frame. QDTrack [69] models the correlation with
contrastive learning, which extracts contrastive embeddings around temporal
neighbors sampled from ROIs for both positive and negative instances,
following the Faster R-CNN framework. UAVMOT [138] presents a novel framework
for tracking unmanned aerial vehicles. An ID feature update module is proposed
in UAVMOT to calculate the correlation between the current frame and the top-k
points from the previous feature map, which addresses the irregular motion and
view change in 3D.

Attention via Transformers. With recent advances in visual Transformers, some
approaches [139, 34, 35, 140] adopt Transformer in MOT since transformers use
pairwise attention that can fuse global information in the embedding and boost
the tracking performance. The query-key mechanism plays a role as a
correlation in tracking. The prediction can be obtained with multi-head
attention [141] that measures the correlation between feature maps and track
queries. For example, TransTrack [139] presents a baseline tracker with
Transformer, which uses track queries and object queries in the Transformer
decoder for both track prediction and object detection. Similarly, Trackformer
[34] and MOTR [35] also employ the Transformer decoder to estimate the
correlation between previous tracks and current feature maps for prediction.
In Trackformer, the encoder-decoder computes attention between the input image
features and the track as well as object queries and outputs bounding boxes
with assigned IDs at each frame. Built upon DETR [142], each track query of
MOTR models the entire track of an object, which is transferred and updated
frame-by-frame to perform iterative predictions in a seamless manner.

Reconstruction-based regression. Apart from approaches that use correlation
for prediction, some works adopt reconstruction-based strategies with
regression to learn the correlations for embedding. For example, built upon a
variant of DLA-34 [109], SOTMOT [143] solves the ridge regression coefficient
from the previous frame for each object and uses the learned coefficient to
predict the label for the next frame. As a result, discriminative features can
be learned around neighbors in local regions.

Correlation-based embedding can learn the local-global correspondence. This
type of embedding also can borrow state-of-the-art methods from SOT problems.
When targets are partially occluded and lost from detectors, such trackers are
still able to continually track the objects. However, drift becomes the major
issue when long-time occlusion occurs, and trackers may gradually lose the
targets.

###  3.5 Sequential Embedding

Another commonly used strategy to model temporal information in MOT is to use
recurrent neural networks (RNNs) for sequential modeling. Such sequential
embedding methods learn the dynamic update of transformations from the
previous embedding to the current embedding. Embedding methods of this
category can be formulated as follows,

$\boldsymbol{z}_{i}^{t}=f(\boldsymbol{z}_{i}^{t-1},\boldsymbol{x}_{i}^{t}),$
(9)

where $\boldsymbol{z}_{i}^{t-1}$ represents the historical embeddings,
$\boldsymbol{x}_{i}^{t}$ represents the input to the current state.

Pure motion. Some works use sequential embedding to model motion features.
They assume object trajectories usually have non-linear patterns, which can be
modeled by variants of RNNs. Milan et al. [144] present the first end-to-end
model using RNN to predict object motion with deep learning. LM_NN [91] tests
both gated recurrent unit (GRU) and long short-term memory (LSTM) for offset
prediction of object location and size with the input of historical motions.
Jiang et al. [145] utilize historical motions to predict future motion with
LSTM units and calculate aggregated affinity with both appearance and motion
patterns using siamese networks. Besides, graph neural networks are adopted to
obtain the association matrix by embedding the affinity. Similarly, DEFT [9]
adopts a motion forecasting module with LSTM.

Appearance incorporated. Some works also take into consideration of appearance
features in the sequential embedding. TADAM [146] presents a unified model
with a synergy between position prediction and embedding association, where
the two tasks are linked by temporal-aware target attention and distractor
attention, as well as an identity-aware memory aggregation model. To better
embed the temporal and contextual information in the track memory, the
convolutional gated recurrent unit (ConvGRU) is employed in the identity-aware
memory aggregation model to aggregate spatial-temporal appearance features.
Similarly, BLSTM-MTP [147] exploits bilinear LSTM [148] to encode spatial-
temporal appearance information of the tracklet and trains a binary classifier
on the positive and hard negative tracklets to learn discriminative
embeddings. Wang et al. [149] present a multiple-node tracking (MNT) framework
in which track trees are generated to link detections and tracks. After that,
a recurrent tracking unit (RTU) is proposed to calculate scores of proposals
in the built track tree, which takes the old appearance feature template, the
appearance feature of the current node, the old hidden state, and the state
feature of the current node as input for embedding. Different from the
previous methods that use sequential embedding for object-level features,
PermaTrack [150] augments the model with a spatial-temporal recurrent memory
module, i.e., ConvGRU [151], to encode entire historical frames for feature
map estimation, which is built on top of CenterTrack [29] and extends the
architecture to an arbitrary length of the input. ArTIST [152] designs a
moving agent network, which is built upon a recurrent auto-encoder neural
network that learns to reconstruct the tracklets of all moving agents with
both appearance and motion cues.

Sequential embedding can learn the long dependency of the motion of targets.
However, the standalone sequential embedding cannot achieve high performance.
As a result, it is usually combined with other embedding methods, such as
tracklet embedding, cross-track relational embedding, and single-frame
detection embedding.

###  3.6 Tracklet Embedding

Following the tracking-by-detection paradigm, many approaches treat tracklets
as individual units rather than detections, aiming at exploiting local and
global temporal cues in the embedding. Typically, tracklet embedding can also
be divided into two sub-categories: tracklet self-embedding and pairwise
embedding.

####  3.6.1 Self-Embedding

Self-embedding can be formulated as follows,

$\boldsymbol{z}_{i}=f(\mathcal{T}_{i}),$ (10)

where
$\mathcal{T}_{i}=[\boldsymbol{D}_{i}^{t-\tau};\boldsymbol{D}_{i}^{t-\tau+1};...;\boldsymbol{D}_{i}^{t}]$
is the $i$-th tracklet, containing sequential detections of the same object
from local intermediate prediction.

Set-based. Some works treat each tracklet as a disordered set. Then learning
the tracklet embedding is similar to the set classification problem. For
example, TBooster [153] designs a tracklet refinement method with two proposed
modules, i.e., splitter, and connector. Multi-head self-attention is exploited
in the connector module to aggregate features of individual detections and
learns the tracklet embedding with the combination of triplet loss and cross-
entropy loss for the association.

Temporal-based. To better model the temporal cues, many works also take the
temporal order into the embedding. For example, TPM [154] presents a
representative-selection network with bi-directional convolutional LSTM that
encodes contextual and temporal information to select representative object
features for tracklet embedding. Hsu et al. [155] follow the temporal modeling
of video-based person Re-ID framework [156], which adopts temporal convolution
to model temporal attention for feature aggregation to obtain tracklet
embedding. Some works employ GNN for tracklet embedding. LGM [33] is a
reconstruct-to-embed framework that embeds the tracklet based on motion
features with temporal convolution and GNN to learn the interaction among
tracklets. LPC_MOT [157] presents a novel proposal-based learnable framework,
which models MOT as a proposal generation, proposal scoring, and trajectory
inference paradigm on an affinity graph. Specifically, it concatenates both
averaged appearance features, and the spatial-temporal features and then graph
convolutional neural network (GCN) [158] is further adopted for tracklet
embedding. PHALP [159] aggregates the information from the previous tracklet
for future appearance, location, and pose prediction in 3D space.
Specifically, HMAR [160] and HMMR [161] with Transformer backbone are used in
PHALP for appearance and pose embedding, respectively. MeMOT [162] introduces
a memory aggregator for tracklet embedding, which consists of three attention
modules, including one short-term attention for noise removal, one long-term
attention for long-range context encoding, and one fusion block for
aggregation.

Compared with other embedding methods, the tracklet self-embedding can
incorporate motion information from nature. As a result, the appearance and
motion features can be combined with a unified model. However, generating
tracklets is tricky. The errors from the tracklet generation can cause
degradation in the tracklet embedding.

####  3.6.2 Pairwise Embedding

Different from self-embedding, pairwise embedding jointly models the
association relationship of the input tracklet pair. This kind of approach can
be summarized as follows,

$\boldsymbol{z}_{i,j}=f(\mathcal{T}_{i},\mathcal{T}_{j}),$ (11)

where $\boldsymbol{z}_{i,j}$ is the pairwise embedding of tracklet
$\mathcal{T}_{i}$ and $\mathcal{T}_{j}$.

Tracklet-tracklet pair. Many works model the pairwise embedding of two input
tracklets. For example, CRF-RNN [163] learns unary and pairwise potentials
based on the CRF model [164, 165] using CNN and bidirectional LSTM
architecture to encode the long-term dependencies between CRF nodes, where
each graph node represents a pair of tracklets. Following the multiple
hypothesis tracking [166] framework, Zhang et al. [167] measure the input
tracklets whether they belong to the same track using CNN and LSTM to model
appearance and motion separately in the proposed appearance evaluation network
and motion evaluation network, respectively. TNT [168] takes account of the
temporal orders of the input tracklet pair based on the temporal connectivity
for pairwise embedding, where the motion and appearance features are
concatenated by temporal convolutions to learn the connectivity.

Tracklet-detection pair. Rather than tracklet-to-tracket pairs, some works
model the joint embedding between tracklet and detection. In other words,
$\mathcal{T}_{j}$ from Eq. (11) may only contain a single detection. For
example, TAMA [169] learns a pair of detection with softmax as binary
classification in the joint-inference network [102]. Then a detection is
compared with all previous detections within the tracklet, and LSTM is adopted
to measure the similarity in the association network. Inspired by the object
relation module in [170], STRN [171] presents a spatial-temporal relation
network, which is a unified framework for similarity measurement between a
tracklet and an object. Various cues, such as appearance, topology, and
location cues, are simultaneously encoded across time in STRN for calculating
the final similarity measurement. MHT-bLSTM [148] learns the embedding of
tracklet-detection pair with the proposed bilinear LSTM, where the tracklet
and detection are coupled in a multiplicative manner instead of the additive
manner in conventional LSTM approaches. Such coupling resembles an online
learned regressor at each time step, which is found to have Superior
performance for appearance modeling. Bastani et al. [172] propose a self-
supervised MOT method with cross-input consistency, which takes two distinct
inputs for the same sequence of video by hiding different information. Then a
matching network is conducted to embed the concatenated features of the
tracklet-detection pair for matching and association. DMAN [173] adopts a
spatial attention network to obtain the feature of each pair of the detection
and a frame of the tracklet. Bi-directional LSTM (Bi-LSTM) is exploited to
learn the temporal feature of the SAN in the temporal attention network. Then
the final embedding is learned for each tracklet-detection pair.

Tracklet pairwise embedding has similar properties as tracklet self-embedding
methods. Besides that, it can also learn the discriminative relations and
temporal order among different tracklets. Since each pair of tracklets needs
to be embedded separately, the computational cost is huge, which is the major
weakness.

###  3.7 Cross-Track Relational Embedding

Figure 7: Cross-track relational embedding.

Cross-track relational embedding aims at learning the object features based on
the interactions with the neighboring tracks, as shown in Fig. 7. The
embedding can be represented as follows,

$\boldsymbol{z}_{i}=f(\\{\boldsymbol{h}_{j}|j\in i\cup\mathcal{N}(i)\\}),$
(12)

where $\boldsymbol{h}_{j}$ is the intermediate embedding of a detection or a
tracklet; $\mathcal{N}(i)$ represents the connected neighbors of the $i$-th
track defined in the approach.

Learning relational embedding from attention. The attention mechanism is one
of the commonly used strategies to obtain track embeddings with the
interactions of other tracks. For example, though TADAM [146] is introduced in
sequential embedding with ConvGRU to aggregate sequential features, it also
considers cross-track relations in the embedding and presents the temporal-
aware target attention and distractor attention for object feature extraction.
Some Transformer-based trackers also exploit attention to obtain cross-track
embeddings. For example, Trackformer [34] and MOTR [35] encode the track
embeddings using the correlation with other tracks and objects via multi-head
attention.

Learning relational embedding from graph. Graph-based methods are also widely
employed for cross-track embedding. For example, GSM [174] presents a graph
representation that takes both the feature of individual objects and the
relations among objects into consideration. In GSM, local graphs are defined
for each detected object and its k-nearest neighbors. Then the graph
similarity model is designed for every two frames to measure the association
via binary classification. DeepMOT [112] proposes a deep Hungarian network
that approximates the Hungarian matching algorithm and uses bi-directional RNN
to model the association. In addition, differentiable dMOTA and dMOTP metrics
are defined for end-to-end training. Li et al. [175] design a near-online MOT
method with an end-to-end graph network that can be dynamically updated.
Specifically, it models the motion graph network and appearance graph network
separately. The graph is built for detections in adjacent two frames. The
final similarity is measured with a weighted sum of motion and appearance.
Following the message passing network (MPN) [176, 177, 178, 179], Braso et al.
[180] exploit the classical network flow formulation of MOT and define a fully
differentiable framework, which aggregates neighborhood nodes for detection
embedding. GMT [181] presents a learnable graph-matching method that models
the relationships between tracklets and the intra-frame detections as a
general undirected graph. Continuous quadratic programming is adopted for
approximation to facilitate the end-to-end training of the GNN. There are also
many works that combine other embedding strategies and cross-track relational
embedding in their methods. For example, LGM [33] not only employs tracklet
embedding but also exploits the cross-track relations in the GCN with the
proposed reconstruct-to-embed strategy to learn discriminative features from
the information of other trackers. ArTIST [152] combines sequential embedding
and cross-track relational embedding. Specifically, ArTIST presents a moving
agent network (MA-Net), where MA-Net is a recurrent auto-encoder neural
network that learns to reconstruct the tracklets of all moving agents
potentially interacting with the tracklet of interest. Based on the multi-
frame spatial-temporal embedding, GSDT [123] also exploits node feature
aggregation among different detections based on GNN given two sequential
frames, which can better extract the relational features across targets. GTR
[182] presents a global tracking Transformer that takes detections from
sequential frames and calculates the cross-attention with trajectory queries.

Unlike other embedding strategies, cross-track embedding can learn the
interaction among tracks. The discrimination between different IDs can be
significantly exploited. It can also be combined with other embedding methods,
like sequential and tracklet embedding, to enhance representation learning.
However, a thorough analysis of the effectiveness of cross-tracking embedding
has not been well studied, requiring more future research.

##  4 Benchmarks and Analysis

###  4.1 Evaluation Metrics

To evaluate the performance of MOT, CLEAR MOT [183], ID-based metrics [184],
and HOTA [185] are the widely used measurements.

CLEAR MOT [183] measures the multiple object tracking accuracy (MOTA) and
multiple object tracking precision (MOPT) between the detected boxes and
ground truth boxes. The proposed MOTA and MOTP are based on the matched pairs,
misses, false positives, and mismatches. However, MOTA overemphasizes
detection performance rather than association.

ID-based metrics [184] compute the truth-to-result match that measures the
tracking performance by how long the tracker correctly identifies targets.
Specifically, a bipartite match associates one ground-truth trajectory to
exactly one computed trajectory by minimizing the number of mismatched frames
over all the available data-true and computed. Standard measures such as
precision, recall, and F1-score are built on top of this truth-to-result
match, namely IDP, IDR, and IDF1. However, IDF1 overemphasizes association
performance rather than detection.

A higher order metric, namely higher order tracking accuracy (HOTA), is
proposed in [185] for evaluating MOT performance. HOTA explicitly balances the
effect of performing accurate detection, association, and localization into a
single unified metric for comparing trackers. HOTA decomposes into a family of
sub-metrics that are able to evaluate each of five basic error types
(detection recall, detection precision, association recall, association
precision, and localization accuracy) separately, which enables clear analysis
of tracking performance.

There are also other metrics for evaluation. For example, PR-MOTA is proposed
in the UA-Detrac dataset [186] and takes account of the detection score along
the PR-curve with the MOTA metric for MOT measurement. In the nuScenes dataset
[187], sAMOTA is adopted as the primary evaluation metric for the 3D MOT
benchmark, in which an updated formulation of MOTA is used to adjust for the
respective recall. Voigtlaender et al. [28] introduce the soft multi-object
tracking and segmentation accuracy (sMOTSA) for MOTS-based tracking
measurement. Feng et al. [188] propose the SAIDF evaluation measure that
focuses more on identity issues and fixes the insensibility and high
computational cost problems of the previous measures, such as MOTA and IDF1.
Furthermore, Valmadre et al. [189] introduce local metrics, namely LIDF1 and
ALTA, that are parametrized by a temporal horizon and thereby reveal the
temporal ranges at which association errors occur. Such metrics provide more
insights from different aspects of MOT evaluation.

###  4.2 In-Depth Analysis on State-of-the-Art Methods

TABLE I: Summary of embedding methods on MOT17 benchmark. Emb. Method |  Ref. |  Year |  Venue |  Det. |  MOTA |  IDF1 |  HOTA
---|---|---|---|---|---|---|---
Patch |  [92] |  2017 |  CVPRW |  Pub. |  50.0 |  51.3 |  41.3
|  [95] |  2018 |  CoRR |  Pub. |  51.5 |  46.9 |  38.5
|  [93] |  2018 |  ICME |  Pub. |  50.9 |  52.7 |  41.2
|  [90] |  2018 |  AVSS |  Pub. |  48.3 |  51.1 |  40.3
|  [63] |  2019 |  ArXiv |  Pub. |  54.7 |  62.3 |  47.1
|  [101] |  2019 |  Access |  Pub. |  48.6 |  47.9 |  38.4
|  [58] |  2020 |  ArXiv |  Pub. |  61.7 |  58.1 |  46.9
|  [88] |  2020 |  ICRAI |  Pub. |  49.7 |  51.5 |  39.0
|  [60] |  2021 |  J-VCIR |  Pub. |  46.8 |  54.1 |  41.5
|  [61] |  2021 |  IVC |  Priv. |  77.0 |  72.0 |  59.7
|  [96] |  2021 |  ArXiv |  Priv. |  73.3 |  73.2 |  59.8
S-Fr |  [62] |  2019 |  ICCV |  Pub. |  56.3 |  55.1 |  44.8
|  [23] |  2021 |  AI |  Pub. |  60.1 |  58.8 |  47.2
|  [24] |  2021 |  IJCV |  Priv. |  73.7 |  72.3 |  59.3
|  [25] |  2022 |  T-MM |  Priv. |  73.8 |  74.7 |  61.0
|  [107] |  2022 |  Neuroc. |  Priv. |  73.5 |  70.2 |  58.7
|  [113] |  2022 |  CVPR |  Priv. |  72.1 |  73.5 |  -
X-Fr |  [114] |  2019 |  T-PAMI |  Pub. |  52.4 |  49.5 |  39.3
|  [29] |  2020 |  ECCV |  Pub. |  61.5 |  59.6 |  48.2
|  [30] |  2020 |  CVPR |  Priv. |  63.0 |  58.6 |  48.0
|  [115] |  2020 |  ECCV |  Priv. |  66.6 |  57.4 |  49.0
|  [29] |  2020 |  ECCV |  Priv. |  67.8 |  64.7 |  52.2
|  [123] |  2021 |  ICRA |  Priv. |  73.2 |  66.5 |  55.2
|  [116] |  2021 |  ArXiv |  Priv. |  73.2 |  62.2 |  54.5
|  [140] |  2022 |  ArXiv |  Priv. |  73.6 |  65.2 |  53.9
|  [129] |  2022 |  ECCV |  Priv. |  77.2 |  75.5 |  61.7
|  [132] |  2022 |  ECCV |  Priv. |  81.2 |  78.1 |  -
Corr |  [64] |  2019 |  ICCV |  Pub. |  52.0 |  48.7 |  -
|  [133] |  2020 |  AAAI |  Pub. |  49.5 |  51.8 |  41.5
|  [137] |  2020 |  ACCV |  Pub. |  57.4 |  57.9 |  -
|  [139] |  2020 |  ArXiv |  Priv. |  75.2 |  63.5 |  54.1
|  [69] |  2021 |  CVPR |  Priv. |  68.7 |  66.3 |  53.9
|  [135] |  2021 |  CVPR |  Priv. |  76.5 |  73.6 |  60.7
Seq |  [91] |  2019 |  Neuroc. |  Pub. |  45.1 |  43.2 |  -
|  [147] |  2021 |  CVPR |  Pub. |  51.5 |  54.9 |  41.3
|  [190] |  2021 |  ArXiv |  Pub. |  67.8 |  61.4 |  50.0
|  [150] |  2021 |  ICCV |  Pub. |  73.1 |  67.2 |  54.2
|  [150] |  2021 |  ICCV |  Priv. |  73.8 |  68.9 |  55.5
|  [149] |  2021 |  ICCV |  Priv. |  74.9 |  75.0 |  62.0
Tracklet |  [191] |  2018 |  ArXiv |  Pub. |  54.1 |  48.4 |  46.8
|  [173] |  2018 |  ECCV |  Pub. |  48.2 |  55.7 |  42.5
|  [148] |  2018 |  ECCV |  Pub. |  47.5 |  51.9 |  41.0
|  [171] |  2019 |  ICCV |  Pub. |  50.9 |  56.0 |  42.6
|  [168] |  2019 |  MM |  Pub. |  51.9 |  58.1 |  44.9
|  [167] |  2020 |  T-IP |  Pub. |  54.9 |  63.1 |  48.4
|  [154] |  2020 |  PR |  Pub. |  54.2 |  52.6 |  41.5
|  [163] |  2020 |  T-CSVT |  Pub. |  53.1 |  53.7 |  42.2
|  [192] |  2021 |  J-VCIR |  Pub. |  45.4 |  39.9 |  34.0
|  [169] |  2021 |  IS |  Pub. |  50.3 |  53.5 |  42.0
|  [172] |  2021 |  NeurIPS |  Pub. |  56.8 |  58.3 |  46.4
|  [157] |  2021 |  CVPR |  Pub. |  59.0 |  66.8 |  51.5
|  [153] |  2022 |  T-MM |  Pub. |  61.5 |  63.3 |  50.5
|  [162] |  2022 |  CVPR |  Priv. |  72.5 |  69.0 |  56.9
X-Track |  [180] |  2020 |  CVPR |  Pub. |  58.8 |  61.7 |  49.0
|  [193] |  2020 |  ArXiv |  Pub. |  57.3 |  56.3 |  45.4
|  [174] |  2020 |  IJCAI |  Pub. |  56.4 |  57.8 |  45.7
|  [112] |  2020 |  CVPR |  Pub. |  53.7 |  53.8 |  42.4
|  [194] |  2020 |  ArXiv |  Priv. |  76.2 |  68.0 |  57.9
|  [123] |  2021 |  ICRA |  Priv. |  73.2 |  66.5 |  55.2
|  [182] |  2022 |  CVPR |  Priv. |  75.3 |  71.5 |  59.1
TABLE II: Summary of embedding methods on MOT20 benchmark. Emb. Method |  Ref. |  Year |  Venue |  Det. |  MOTA |  IDF1 |  HOTA
---|---|---|---|---|---|---|---
Patch |  [58] |  2020 |  ArXiv |  Pub. |  53.6 |  50.6 |  41.7
|  [60] |  2021 |  J-VCIR |  Pub. |  44.7 |  43.5 |  35.6
|  [61] |  2021 |  IVC |  Priv. |  77.4 |  73.1 |  61.2
|  [96] |  2021 |  ArXiv |  Priv. |  65.2 |  70.1 |  55.3
S-Fr |  [62] |  2019 |  ICCV |  Pub. |  52.6 |  52.7 |  42.1
|  [23] |  2021 |  AI |  Pub. |  59.3 |  59.1 |  47.1
|  [24] |  2021 |  IJCV |  Priv. |  61.8 |  67.3 |  54.6
|  [25] |  2022 |  T-MM |  Priv. |  67.2 |  70.5 |  56.5
|  [107] |  2022 |  Neuroc. |  Priv. |  68.6 |  69.4 |  56.2
|  [113] |  2022 |  CVPR |  Priv. |  63.5 |  69.2 |  -
X-Fr |  [116] |  2021 |  ArXiv |  Pub. |  61.0 |  49.8 |  43.5
|  [123] |  2021 |  ICRA |  Priv. |  67.1 |  67.5 |  53.6
|  [132] |  2022 |  ECCV |  Priv. |  78.1 |  76.4 |  -
Corr |  [139] |  2020 |  ArXiv |  Priv. |  65.0 |  59.4 |  48.9
Tracklet |  [157] |  2021 |  CVPR |  Pub. |  56.3 |  62.5 |  49.0
|  [153] |  2022 |  T-MM |  Pub. |  54.6 |  53.4 |  42.5
|  [162] |  2022 |  CVPR |  Priv. |  63.7 |  66.1 |  54.1
X-Track |  [180] |  2020 |  CVPR |  Pub. |  57.6 |  59.1 |  46.8
|  [193] |  2020 |  ArXiv |  Pub. |  54.5 |  49.0 |  40.2
TABLE III: Summary of embedding methods on KITTI MOT benchmark. Emb. Method |  Ref. |  Year |  Venue |  Mod. |  Obj. |  MOTA |  HOTA
---|---|---|---|---|---|---|---
Patch |  [103] |  2019 |  ArXiv |  V |  C |  75.8 |  60.9
|  [195] |  2020 |  IROS |  V&L |  C |  85.1 |  69.6
|  [196] |  2020 |  ACCV |  V |  C |  87.8 |  68.5
|  [196] |  2020 |  ACCV |  V |  P |  68.0 |  50.9
|  [195] |  2020 |  IROS |  V&L |  P |  45.3 |  34.2
S-Fr |  [10] |  2019 |  ICCV |  V |  C |  84.3 |  73.2
X-Fr |  [197] |  2019 |  ICCV |  V&L |  C |  83.2 |  62.1
|  [198] |  2020 |  RAL |  L |  C |  67.6 |  57.2
|  [29] |  2020 |  ECCV |  V |  C |  88.8 |  73.0
|  [199] |  2021 |  IJCAI |  L |  C |  91.7 |  80.9
|  [9] |  2021 |  CVPR |  V |  C |  88.4 |  74.2
|  [200] |  2021 |  RAL |  L |  C |  84.5 |  72.2
|  [201] |  2021 |  IROS |  V&L |  C |  85.4 |  70.7
|  [132] |  2022 |  ECCV |  V |  C |  91.2 |  78.4
|  [29] |  2020 |  ECCV |  V |  P |  53.8 |  40.4
|  [132] |  2022 |  ECCV |  V |  P |  67.7 |  49.0
Corr |  [64] |  2019 |  ICCV |  V |  C |  75.9 |  52.6
|  [202] |  2021 |  ArXiv |  V |  C |  85.9 |  72.8
|  [69] |  2021 |  CVPR |  V |  C |  84.9 |  68.5
|  [69] |  2021 |  CVPR |  V |  P |  55.6 |  41.1
|  [202] |  2021 |  ArXiv |  V |  P |  51.8 |  41.1
Seq |  [10] |  2019 |  ICCV |  V |  C |  84.3 |  73.2
|  [150] |  2021 |  ICCV |  V |  C |  91.3 |  78.0
|  [150] |  2021 |  ICCV |  V |  P |  66.0 |  48.6
Tracklet |  [33] |  2021 |  ICCV |  V |  C |  87.6 |  73.1
X-Track |  [33] |  2021 |  ICCV |  V |  C |  87.6 |  73.1
|  [203] |  2021 |  ArXiv |  V |  C |  87.3 |  72.3
|  [180] |  2020 |  CVPR |  V |  P |  46.2 |  45.3
|  [203] |  2021 |  ArXiv |  V |  P |  52.1 |  39.4
TABLE IV: SOTA embedding methods on DIVOTrack. More details of the metrics can be found in [185, 183, 184]. Emb. Method |  Tracker |  HOTA$\uparrow$ |  IDF1$\uparrow$ |  MOTA$\uparrow$ |  AssA$\uparrow$ |  sMOTA$\uparrow$ |  IDR$\uparrow$ |  IDP$\uparrow$ |  IDSW$\downarrow$ |  Frag$\downarrow$
---|---|---|---|---|---|---|---|---|---|---
Patch |  ByteTrack [16] |  55.4 |  66.1 |  61.2 |  55.7 |  46.1 |  64.7 |  67.6 |  1,470 |  4,442
S-Fr |  FairMOT [24] |  54.0 |  64.4 |  59.4 |  55.1 |  45.2 |  60.8 |  68.4 |  1,283 |  5,183
X-Fr |  CenterTrack [29] |  42.9 |  46.0 |  51.2 |  37.4 |  35.0 |  46.2 |  45.9 |  3,011 |  3,813
Corr |  QDTrack [69] |  46.8 |  52.6 |  53.4 |  43.6 |  38.3 |  51.1 |  54.1 |  2,300 |  4,288
Seq |  PermaTrack [150] |  48.2 |  53.8 |  56.2 |  44.3 |  38.8 |  55.7 |  52.1 |  2,183 |  3,109
Tracklet |  LPC_MOT [157] |  56.5 |  68.6 |  51.1 |  61.6 |  34.2 |  74.8 |  63.3 |  824 |  1,375
X-Track |  GTR [182] |  46.8 |  62.9 |  60.0 |  51.8 |  44.0 |  64.3 |  61.7 |  1,450 |  3,982

We review state-of-the-art embedding methods of MOT on widely used datasets,
including MOT17 [204], MOT20 [205], and KITTI [206]. We include methods
published in top CV venues over the past three years. We mainly focus on three
evaluation metrics: MOTA, IDF1, and HOTA. The performances on three benchmarks
are reported in Table I, Table II, and Table III. The embedding methods in the
table include “Patch-Level Box Image Embedding” (Patch), “Single-Frame
Detection Embedding” (S-Fr), “Single-Frame Detection Embedding” (X-Fr),
“Correlation-Based Embedding” (Corr), “Sequential Embedding” (Seq),
“Sequential Embedding” (Tracklet) and “Sequential Embedding” (X-Track). The
used detection can be Public (Pub.) or Private (Private). Modality (Mod.) in
the KITTI benchmark includes Vision (V) and LiDAR (L). The Object (Obj.)
category includes Car (C) and Person (P). We analyze the top performance for
each category of embedding methods as follows.

####  4.2.1 General Discussion across Embedding Methods

We provide a general discussion and comparison across different embedding
methods in this section.

Detection. Since MOT17 and MOT20 provide publicly available detections, early
works [90, 114, 168] strictly follow the rule of using public detections.
Because of the noisy results of public detection, such methods cannot achieve
very competitive performance. Then some works [207, 62, 29] employ refinement
techniques that use well-pre-trained models for bounding box regression,
significantly alleviating the drawback of noisy public detections. Recently,
more works [24, 149, 113] have focused on private detection, which can greatly
outperform trackers using public detection.

Tracking paradigm. Conventional MOT methods [95, 168] follow the tracking-by-
detection paradigm. Association is conducted after object detection (usually
in bounding box or pose format), where object detection is a separate stage.
The tracking performance is largely dependent on the detection performance.
Since the segmentation mask can provide pixel-wise semantic information,
tracking-by-segmentation [28] begins to attract attention after the MOTS
dataset is created. The methods achieve superior results compared with the
tracking-by-detection counterpart. More recently, many works [27, 24] have
been interested in end-to-end learning, which follows joint detection and
tracking/embedding paradigm. Since the detection error can be corrected
simultaneously with embedding learning, promising results are achieved. With
the fast development of visual Transformers, the tracking-by-attention
paradigm [34] shows leading performance in MOT. Moreover, some works [33]
propose a tracking-by-reconstruction scheme that can recover trajectories
directly along with the embedding.

Model architecture. CNN is widely adopted in feature extractors, backbone
networks, and Re-ID networks. Commonly used CNNs include VGGNet [114], ResNet
[95, 115, 30], and DLA [24]. Methods [29, 123, 24, 135] show that point-based
embedding can better represent object-level features with DLA architecture.
RNN is commonly used in sequential embeddings. Commonly used architectures
include LSTM [208, 9] and GRU [91, 150]. RNN alone usually cannot achieve
promising results. As a result, embedding methods with RNN also incorporate
other architectures like GNN [112, 152] and CNN [146, 150] to boost the
performance. Transformers have gained more attention in recent years. With the
global attention mechanism, Transformer-based trackers [116, 140, 132] show
great potential. Graph-based architectures are often employed in the head
networks for cross-object node embedding [180, 123, 182]. Promising results
are achieved.

Computation complexity. Generally speaking, pairwise embedding, including
patch pairs and tracklet pairs, increases the computation complexity from
$\mathcal{O}(n)$ to $\mathcal{O}(n^{2})$, where $n$ is the number of
individual objects. Computation complexity analysis is mainly for measuring
the efficiency of the association methods. For example, Ullah et al. [209]
propose a directed sparse graphical model (DSGM) to reduce computational
complexity by minimizing connectivity in the multi-target scene. Weng et al.
[210] present a baseline of a 3D MOT method with a 3D Kalman filter and the
Hungarian algorithm for efficient real-time tracking. However, the association
methods and computation analysis are well described in existing surveys [39].
As a result, the details of this part are not in the main scope of this paper.

####  4.2.2 Implementation on Extra Benchmarks

Since the embedding methods are coupled with other factors like association
methods to achieve the final performance, it is better to compare embedding
strategies alone and fix other factors the same for evaluation.

For a fair comparison, we ensure the same training and testing data are used
across different embedding methods. Since the ground truth annotation of the
testing set in MOT benchmarks is not publicly available, we use a recently
released novel MOT benchmark, DIVOTrack [211], for reporting the results.
DIVOTrack is a real-world multi-view multi-person tracking dataset with
diverse scenes and dense pedestrians. There are ten scenarios, and each
scenario is captured by three cameras. In total, 60 videos are recorded, and
each video has about one thousand frames.

We consider widely used SOTA trackers with released source code for each
embedding category in the evaluation, including ByteTrack [16], FairMOT [24],
CenterTrack [29], QDTrack [69], PermaTrack [150], LPC_MOT [157], and GTR [182]
from seven different embedding categories, as shown in Table IV. Note that
there are also other trackers without source code available that can provide
good performance in MOT benchmarks. However, it is hard to build trackers from
scratch with missing implementation details. As a result, we choose widely
known SOTA trackers for comparison.

We train the embedding models of selected trackers shown in Table IV on the
MOT17 training set. After training, all the trackers except the LPC_MOT
tracker follow the standard DeepSORT [212] association method with the
Hungarian method in the inference to decouple the influence from association
strategies. Since the LPC_MOT tracker is a tracklet-based embedding approach
and it is not suitable to use the Hungarian method for the tracklet-based
association, we use the default association strategy in LPC_MOT for reporting
the result. As for ByteTrack, we use the YOLOX model [213] for training the
detection since it is a patch-based embedding method. The detailed results are
shown in Table IV. LPC_MOT achieves the best performance in HOTA, IDF1, AssA,
IDSW, and Frag, showing the tracklet-based embedding strategy is very
effective in the ID association. ByteTrack achieves the second best in HOTA
and IDF1, and the best in MOTA, demonstrating that patch-based image embedding
can benefit from offline SOTA object detectors like YOLOX. FairMOT and GTR
also achieve relatively good performance in HOTA, IDF1, and MOTA, verifying
the effectiveness of single-frame embedding and cross-track embedding
strategies.

##  5 Future Directions of MOT Embedding

###  5.1 Under-Investigated Areas

We discuss the trends and potential future directions for MOT embedding that
are under-investigated from five different aspects, including non-fully
supervised learning, generalization, and domain adaptation, embedding for
crowded scenes, multi-view collaboration, and multi-modal MOT.

####  5.1.1 Non-Fully Supervised Learning

Most existing embedding methods in MOT use a fully supervised training
framework with object locations and track IDs. However, it is expensive to
annotate a large amount of video data from diverse scenarios. Utilizing non-
fully annotated video data is in high demand in the MOT area.

Some works try to adopt weakly supervised and semi-supervised learning
frameworks to alleviate the annotation cost. Typically, weakly supervised
learning uses the weak label in the annotation for each sample, while semi-
supervised learning combines labels and unlabeled samples in training. For
example, Ruiz et al. [214] employ weakly instance segmentation by Grad-CAM
heatmaps [215] to extract partial foreground masks for the MOTS task. In place
of tracking annotations, McKee et al. [216] first hallucinate videos from
images with bounding box annotations using zoom-in/out motion transformations
to obtain free tracking labels. Nishimura et al. [217] generate pseudo labels
for successive frames with the association for cell tracking. Li et al. [96]
adopt contrastive representation learning between detection and sub-tracklet
embeddings, combining both labeled tracks and unlabeled tracks in training.
Some works [58] claim that they propose unsupervised trackers that do not use
ground truth tracking labels. However, they assume the box annotation or
detection results are available. As a result, these are actually weakly
supervised approaches. For example, Liu et al. [107] design an unsupervised
Re-ID framework based on strong and weak supervision of the association cues
from two frames following FairMOT [24]. Wu et al. [218] propose a transductive
interactive self-training method to adapt the tracking model to unseen crowded
scenes with unlabeled testing data by means of teacher-student interactive
learning. However, these weakly supervised/semi-supervised approaches usually
generate pseudo-tracking labels with augmentation and simple trackers that
could bring much tracking error. Recently, Yoon et al. [219] propose a novel
masked warping loss that drives the network to indirectly learn how to track
objects through a video, trained only with the bounding box information. This
type of loss only considers a very short period of temporal consistency. As a
result, it cannot achieve competitive performance compared with supervised
approaches.

Some works [172, 220] follow an unsupervised learning framework in MOT. This
kind of work takes into consideration of prior consistency and constraints in
embedding learning. Specifically, Bastani et al. [172] propose a self-
supervised tracker by using cross-input consistency, in which two distinct
inputs are constructed for the same sequence of video by hiding different
information about the sequence in each input. He et al. [220] propose a
tracking-by-animation framework, where a differentiable neural model tracks
objects from input frames and then animates these objects into reconstructed
frames, which achieves both label-free and end-to-end learning of MOT.

####  5.1.2 Training with Synthetic Data

Since fully annotated real video data is hard to collect, many works [221,
222, 223] try to utilize synthetic data for training trackers. The
annotations, such as box, mask, and track ID, can be naturally obtained from
synthetic data. For example, Gaidon et al. [221] show that pre-training on
virtual data can improve performance. Liu et al. [222] introduce a large-scale
synthetic data engine named MOTX to generate synthetic data for MOT training
and achieve competitive results with the trackers trained using real data. In
addition, MOTSynth [223] shows that better performance can be achieved with
synthetic data on some widely known trackers such as Tracktor [62], Track
R-CNN [28], Lift_T [224], MPNTrack [180], and CenterTrack [29]. These works
prove the efficiency of utilizing synthetic data for the MOT task. However,
how to reduce the domain gap between synthetic and real data and design a
learning framework with the generalization for unseen diverse scenarios is
still under-explored.

####  5.1.3 Multi-View Collaborated MOT

In recent years, there have been a few works about multi-view MOT [225, 226,
227, 228, 229, 230, 231, 232, 233, 234], where multiple objects are tracked
from several different overlapping views simultaneously, aiming to address
occlusion issues with multi-view geometry and consistency. The embedding
framework is learned for both cross-frames and cross-views. The multi-view
also provides depth information, enabling tracking in the 3D space.
Specifically, Xu et al. [230] propose a hierarchical composition model and re-
formulate multi-view MOT as a problem of compositional structure optimization.
Han et al. [233] model the data similarity using appearance, motion, and
spatial reasoning and formulate the multi-view MOT as a joint optimization
problem solved by constrained integer programming. Han et al. [231] formulate
multi-view MOT as a constrained mixed-integer programming problem and
effectively measure the similarity of subjects over time and across views. Gan
et al. [227] propose a spatial-temporal association network with two designed
self-supervised learning losses, including a symmetric-similarity loss and a
transitive-similarity loss, to associate multiple humans over time and across
views. In [229], tracklets are matched to multi-camera trajectories by solving
a global lifted multi-cut formulation that incorporates short and long-range
temporal interactions on tracklets located in the same camera as well as
inter-camera ones. In addition, Chen et al. [232] present a novel solution for
multi-human 3D pose estimation and tracking simultaneously from multiple
calibrated camera views.

However, there are still unresolved issues for such settings. First, with a
limited multi-view collaborated dataset, cross-view Re-ID embedding is not
robust and causes degradation in tracking performance. Second, it is difficult
to define an end-to-end joint detection and tracking framework that can take
multiple views with variant spatial-temporal relations. Third, how to better
utilize the cross-frame and cross-view geometry consistency is still not well-
explored.

####  5.1.4 Multi-Modal MOT

Except for vision-based trackers, other modalities can also be adopted in MOT.
Since LiDAR has a good measure of depth, LiDAR-based trackers [235, 236, 199,
237, 238] have become popular in autonomous driving scenarios. As different
sensors (e.g., RGB camera, LiDAR, radar) get increasingly used together, the
multi-modal MOT [239, 197, 240, 241, 242, 201, 243, 244] starts to attract
research attention. One benefit is that multiple sensors increase the
diversity of object representations, which provides higher association
reliability across objects from different timestamps. Different modalities
have their own advantages and disadvantages. In a tracking-by-detection
framework, fusing data from multiple modalities would ideally improve tracking
performance than using a single modality.

For example, Linder et al. [239] integrate a real-time multi-modal laser/RGB-D
people tracking framework for moving platforms. Zhang et al. [197] propose a
generic sensor-agnostic multi-modality MOT framework, where each modality is
capable of performing its role independently to preserve reliability, and
could further improve its accuracy through a novel multi-modality fusion
module. In [240], features of RGB images and 3D LiDAR points are extracted
separately before they fully interact with each other through a cross-modal
attention mechanism for inter-frame proposals association. Chiu et al. [241]
learn how to fuse features from 2D images and 3D LiDAR point clouds to capture
the appearance and geometric information of an object. Huang and Hao [201]
present an efficient multi-modal MOT framework with online joint detection and
tracking schemes and robust data association for autonomous driving
applications. CFTrack [243] presents an end-to-end network for joint object
detection and tracking with a center-based radar-camera fusion algorithm.
Valverde et al. [244] propose a novel self-supervised MM-DistillNet framework
consisting of multiple teachers that leverage diverse modalities, including
RGB, depth, and thermal images, to simultaneously exploit complementary cues
and distill knowledge into a single audio student network.

However, there are still unresolved issues for multi-modal MOT. First, the
embedding strategy with multi-modality is not well explored, following the
simple fusion scheme. Second, except for LiDAR and radar, other modalities are
not frequently used. For example, how to combine sound, text, infrared
sensors, and other modalities should be explored for some special cases [245,
246]. Third, addressing missing or noisy modalities in MOT is also one of the
major challenges.

####  5.1.5 MOT with Reinforcement Learning

In recent years, deep reinforcement learning has gained significant success in
various vision applications, such as object detection [247], face recognition
[248], and image super-resolution [249]. However, deep reinforcement learning
remains under-explored in the MOT area, with a limited number of works found
[250, 251, 252, 253]. For example, in [250], a novel multi-agent reinforcement
learning formulation of MOT is presented, which treats creating, propagating,
and terminating object tracks as actions in a sequential decision-making
problem. Ren et al. [252] develop a deep prediction-decision network, which
simultaneously detects and predicts objects under a unified network via deep
reinforcement learning.

There are several challenges in deep reinforcement learning-based MOT
frameworks. First, how defining the space of state and action is crucial. The
defined states and actions should be implicit and not annotated with ground
truth. Otherwise, they may turn out to become simple supervised learning
approaches. Besides that, the defined space should assist the tracking
performance. Rather than simple states such as “tracked” and “lost”, other
more complex behaviors can be modeled, such as “leaving”, “start walking”,
etc. Second, the definition of reward functions is also important. Reward
functions should reflect the relationship between chosen actions and tracking
performance based on the defined space. Deep reinforcement learning may gain
more potential in the MOT field with carefully designed space and reward
functions.

###  5.2 Towards Future Embedding Learning in MOT

####  5.2.1 Meta-Learning

The field of meta-learning [254, 255], or learning-to-learn, has seen a
dramatic rise in interest in recent years. Contrary to conventional approaches
in machine learning, where tasks are solved from scratch using a fixed
learning algorithm, meta-learning aims to improve the learning algorithm
itself, given the experience of multiple learning episodes. This “learning-to-
learn” [256] can lead to a variety of benefits, such as improved data and
compute efficiency, and it is better aligned with human and animal learning
[257], where learning strategies improve both on a lifetime and evolutionary
timescales [258]. Meta-learning has been exploited in many computer vision
tasks, such as few-shot object detection [259, 260] and segmentation [261,
262, 263]. However, meta-learning is still under-explored in the MOT task.

Meta-learning can be a potential novel direction in MOT with the learning-to-
learn framework. First, a meta-learner can be adopted to optimize the hyper-
parameters, such as the detection and tracking threshold, multi-task loss
weighting, and pseudo-label generation. Second, meta-learning can be used for
generalization and adaptation purposes to assist MOT in diverse scenarios.

####  5.2.2 Learning with Auxiliary Tasks

Auxiliary learning [264] is a method to improve the ability of a primary task
by training on additional auxiliary tasks alongside the primary task. The
sharing of embeddings across tasks results in additional relevant features
being available, which otherwise would not have been learned from training
only on the primary task. The broader support of these features across new
interpretations of input data allows for better generalization of the primary
task. Auxiliary learning is similar to multi-task learning [265], except that
only the performance of the primary task is of importance, and the auxiliary
tasks are included purely to assist the primary task. Applying related
learning tasks is one straightforward approach to assist primary tasks.
Toshniwal et al. [266] apply auxiliary supervision with phoneme recognition at
intermediate low-level representations to improve the performance of
conversational speech recognition. In [267], the chosen auxiliary tasks can be
obtained with low efforts, such as global descriptions of a scene, to boost
the performance for single scene depth estimation and semantic segmentation.
By carefully choosing a pair of learning tasks, we may also perform auxiliary
learning without ground truth labels in an unsupervised manner. Jaderberg et
al. [268] introduce a method for improving agent learning in Atari games by
building unsupervised auxiliary tasks to predict the onset of immediate
rewards from a short historical context. Du et al. [269] propose to use cosine
similarity as an adaptive task weighting to determine when a defined auxiliary
task is useful. Jin et al. [127] explore auxiliary training in the context of
keypoint embedding representation learning.

To learn embeddings for the MOT task, the defined auxiliary tasks should help
improve the discrimination among different objects for the primary task. For
example, incorporating unlabeled videos with an unsupervised loss for
auxiliary learning, learning pose-based features to improve the discriminative
embeddings of objects, and transferring image-based Re-ID features in the
embedding models. It would largely broaden the view of researchers with such
explorations.

####  5.2.3 Large-Scale Pre-Training

Self-supervised learning has gained popularity because of its ability to avoid
the cost of annotating large-scale datasets [270]. It can adopt self-defined
pseudo labels as supervision and use the learned representations for several
downstream tasks. With recent self-supervised learning approaches, such as
SimCLR [271], SwAV [272], MoCo [273], and video-based tasks [274, 275, 276,
277], the large-scale pre-training show great power in computer vision tasks.
The pre-training aims at learning the representations with self-supervised
pretext tasks with pseudo labels. The learned model from the pretext task can
be used for downstream tasks such as classification, detection, and
segmentation.

Since unlabeled video data can be easily obtained, large-scale pre-training
can be a good opportunity for learning embeddings for MOT. However, two main
issues should be addressed in such a direction. First, defining appropriate
pretext tasks to learn object-level embeddings with spatial-temporal
information is critical for MOT. Unlike video-level classification tasks, such
as anomaly detection [278, 279] and action recognition [52, 280], object-level
embeddings play an essential role in MOT. Second, how to transfer the pre-
training model to the downstream tasks with track IDs is unclear. If these two
issues can be addressed, the pre-training approaches will be beneficial to the
MOT task with huge unlabeled video data.

####  5.2.4 Other Future Directions

There are other future directions for embedding learning methods in MOT,
summarized below.

  * •

Distilling knowledge for embedding learning from other tracking-related
models, such as image-based Re-ID models and detection models.

  * •

Learning cross-domain embeddings to bridge the training and testing
distribution differences.

  * •

Mining priors, constraints, and consistencies, such as enter-leave consistency
(counting consistency), geometry consistency, and ego-motion consistency.

  * •

Estimating implicit object behavior status to boost embedding.

  * •

Reasoning and causality learning for object trajectory estimation.

##  6 Conclusion

This paper presents a comprehensive survey with an in-depth analysis of
embedding methods in multi-object tracking (MOT). We first review the widely
used embedding methods in MOT from seven aspects and provide a detailed
summary for each embedding category. Then widely used MOT evaluation metrics
are summarized. Besides that, an in-depth analysis of state-of-the-art methods
is provided. We also point out under-investigated areas and future worth-
exploring research directions in the last section, hoping to inspire more
thinking of the embedding strategies in MOT.

## Acknowledgement

This work is supported by National Natural Science Foundation of China
(62106219).

## References

  * [1] Z. Tang, G. Wang, H. Xiao, A. Zheng, and J.-N. Hwang, “Single-camera and inter-camera vehicle tracking and 3d speed estimation based on fusion of visual and semantic features,” in _CVPR workshops_ , 2018.
  * [2] Z. Tang, M. Naphade, M.-Y. Liu, X. Yang, S. Birchfield, S. Wang, R. Kumar, D. Anastasiu, and J.-N. Hwang, “Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification,” in _CVPR_ , 2019.
  * [3] G. Wang, X. Yuan, A. Zheng, H.-M. Hsu, and J.-N. Hwang, “Anomaly candidate identification and starting time estimation of vehicles from traffic videos.” in _CVPR Workshops_ , 2019.
  * [4] H.-M. Hsu, Y. Wang, and J.-N. Hwang, “Traffic-aware multi-camera tracking of vehicles based on reid and camera link model,” in _ACM MM_ , 2020.
  * [5] Y. Wu, D. Kong, S. Wang, J. Li, and B. Yin, “An unsupervised real-time framework of human pose tracking from range image sequences,” _IEEE TMM_ , 2019.
  * [6] R. Gu, G. Wang, and J.-N. Hwang, “Efficient multi-person hierarchical 3d pose estimation for autonomous driving,” in _MIPR_ , 2019.
  * [7] R. Gu, G. Wang, Z. Jiang, and J.-N. Hwang, “Multi-person hierarchical 3d pose estimation in natural videos,” _IEEE TCSVT_ , 2019.
  * [8] A. Jalal, M. Mahmood, and A. S. Hasan, “Multi-features descriptors for human activity tracking and recognition in indoor-outdoor environments,” in _IBCAST_ , 2019.
  * [9] M. Chaabane, P. Zhang, J. R. Beveridge, and S. O’Hara, “Deft: Detection embeddings for tracking,” _arXiv preprint_ , 2021.
  * [10] H.-N. Hu, Q.-Z. Cai, D. Wang, J. Lin, M. Sun, P. Krahenbuhl, T. Darrell, and F. Yu, “Joint monocular 3d vehicle detection and tracking,” in _ICCV_ , 2019\.
  * [11] G. Wang, J.-N. Hwang, K. Williams, and G. Cutter, “Closed-loop tracking-by-detection for rov-based multiple fish tracking,” in _CVAUI_ , 2016.
  * [12] M.-C. Chuang, J.-N. Hwang, J.-H. Ye, S.-C. Huang, and K. Williams, “Underwater fish tracking for moving cameras based on deformable multiple kernels,” _IEEE TSMC_ , 2016.
  * [13] M. Dawkins, L. Sherrill, K. Fieldhouse, A. Hoogs, B. Richards, D. Zhang, L. Prasad, K. Williams, N. Lauffenburger, and G. Wang, “An open-source platform for underwater image and video analytics,” in _WACV_ , 2017.
  * [14] P. Dendorfer, A. Osep, A. Milan, K. Schindler, D. Cremers, I. Reid, S. Roth, and L. Leal-Taixé, “Motchallenge: A benchmark for single-camera multiple target tracking,” _IJCV_ , 2021.
  * [15] K. Ho, A. Kardoost, F.-J. Pfreundt, J. Keuper, and M. Keuper, “A two-stage minimum cost multicut approach to self-supervised multiple person tracking,” in _ACCV_ , 2020.
  * [16] Y. Zhang, P. Sun, Y. Jiang, D. Yu, Z. Yuan, P. Luo, W. Liu, and X. Wang, “Bytetrack: Multi-object tracking by associating every detection box,” in _ECCV_ , 2022.
  * [17] Y. Zhang, C. Wang, X. Wang, W. Zeng, and W. Liu, “Robust multi-object tracking by marginal inference,” in _ECCV_ , 2022.
  * [18] Z. Zou, J. Huang, and P. Luo, “Compensation tracker: Reprocessing lost object for multi-object tracking,” in _WACV_ , 2022.
  * [19] J. Summaira, X. Li, A. M. Shoib, S. Li, and J. Abdul, “Recent advances and trends in multimodal deep learning: A review,” _arXiv preprint arXiv:2105.11087_ , 2021.
  * [20] Y. Chen, X. Li, A. Dick, and R. Hill, “Ranking consistency for image matching and object retrieval,” _Pattern Recognition_ , 2014.
  * [21] X. Jiang, L. Zhang, P. Lv, Y. Guo, R. Zhu, Y. Li, Y. Pang, X. Li, B. Zhou, and M. Xu, “Learning multi-level density maps for crowd counting,” _TNNLS_ , 2019.
  * [22] X. Li, A. Dick, H. Wang, C. Shen, and A. van den Hengel, “Graph mode-based contextual kernels for robust svm tracking,” in _ICCV_. IEEE, 2011.
  * [23] J. Yang, H. Ge, J. Yang, Y. Tong, and S. Su, “Online multi-object tracking using multi-function integration and tracking simulation training,” _AI_ , 2021.
  * [24] Y. Zhang, C. Wang, X. Wang, W. Zeng, and W. Liu, “Fairmot: On the fairness of detection and re-identification in multiple object tracking,” _IJCV_ , 2021\.
  * [25] E. Yu, Z. Li, S. Han, and H. Wang, “Relationtrack: Relation-aware multiple object tracking with decoupled representation,” _IEEE TMM_ , 2022.
  * [26] Z. Lu, V. Rathod, R. Votel, and J. Huang, “Retinatrack: Online single stage joint detection and tracking,” in _CVPR_ , 2020.
  * [27] Z. Wang, L. Zheng, Y. Liu, Y. Li, and S. Wang, “Towards real-time multi-object tracking,” in _ECCV_ , 2020.
  * [28] P. Voigtlaender, M. Krause, A. Osep, J. Luiten, B. B. G. Sekar, A. Geiger, and B. Leibe, “Mots: Multi-object tracking and segmentation,” in _CVPR_ , 2019\.
  * [29] X. Zhou, V. Koltun, and P. Krähenbühl, “Tracking objects as points,” in _ECCV_ , 2020.
  * [30] B. Pang, Y. Li, Y. Zhang, M. Li, and C. Lu, “Tubetk: Adopting tubes to track multi-object in a one-step training model,” in _CVPR_ , 2020.
  * [31] S. Sun, N. Akhtar, X. Song, H. Song, A. Mian, and M. Shah, “Simultaneous detection and tracking with motion modelling for multiple object tracking,” in _ECCV_ , 2020.
  * [32] X. Wan, S. Zhou, J. Wang, and R. Meng, “Multiple object tracking by trajectory map regression with temporal priors embedding,” in _ACM MM_ , 2021.
  * [33] G. Wang, R. Gu, Z. Liu, W. Hu, M. Song, and J.-N. Hwang, “Track without appearance: Learn box and tracklet embedding with local and global motion patterns for vehicle tracking,” in _ICCV_ , 2021.
  * [34] T. Meinhardt, A. Kirillov, L. Leal-Taixe, and C. Feichtenhofer, “Trackformer: Multi-object tracking with transformers,” in _CVPR_ , 2022.
  * [35] F. Zeng, B. Dong, T. Wang, C. Chen, X. Zhang, and Y. Wei, “Motr: End-to-end multiple-object tracking with transformer,” in _ECCV_ , 2022.
  * [36] Y. Xu, X. Zhou, S. Chen, and F. Li, “Deep learning for multiple object tracking: a survey,” _IET CV_ , 2019.
  * [37] P. Emami, P. M. Pardalos, L. Elefteriadou, and S. Ranka, “Machine learning methods for data association in multi-object tracking,” _ACM CSUR_ , 2020\.
  * [38] G. Ciaparrone, F. L. Sánchez, S. Tabik, L. Troiano, R. Tagliaferri, and F. Herrera, “Deep learning in video multi-object tracking: A survey,” _Neurocomputing_ , 2020.
  * [39] Y. Park, L. M. Dang, S. Lee, D. Han, and H. Moon, “Multiple object tracking in deep learning approaches: A survey,” _Electronics_ , 2021.
  * [40] W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu, and T.-K. Kim, “Multiple object tracking: A literature review,” _AI_ , 2021.
  * [41] S. M. Marvasti-Zadeh, L. Cheng, H. Ghanei-Yakhdan, and S. Kasaei, “Deep learning for visual tracking: A comprehensive survey,” _IEEE TITS_ , 2021\.
  * [42] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _NeurIPS_ , 2012.
  * [43] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint_ , 2014.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [45] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _IJCV_ , 2015.
  * [46] M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, L. ˇCehovin Zajc, T. Vojir, G. Bhat, A. Lukezic, A. Eldesokey _et al._ , “The sixth visual object tracking vot2018 challenge results,” in _ECCV Workshops_ , 2018.
  * [47] M. Kristan, J. Matas, A. Leonardis, M. Felsberg, R. Pflugfelder, J.-K. Kamarainen, L. ˇCehovin Zajc, O. Drbohlav, A. Lukezic, A. Berg _et al._ , “The seventh visual object tracking vot2019 challenge results,” in _ICCV Workshops_ , 2019.
  * [48] Z. Zhu, Q. Wang, B. Li, W. Wu, J. Yan, and W. Hu, “Distractor-aware siamese networks for visual object tracking,” in _ECCV_ , 2018.
  * [49] X. Li, C. Ma, B. Wu, Z. He, and M.-H. Yang, “Target-aware deep tracking,” in _CVPR_ , 2019.
  * [50] H. Morimitsu, “Multiple context features in siamese networks for visual object tracking,” in _ECCV Workshops_ , 2018.
  * [51] J. Gao, T. Zhang, and C. Xu, “Graph convolutional tracking,” in _CVPR_ , 2019\.
  * [52] Z. Zhu, W. Wu, W. Zou, and J. Yan, “End-to-end flow correlation tracking with spatial-temporal attention,” in _CVPR_ , 2018.
  * [53] Y. Chen, L. Jing, E. Vahdani, L. Zhang, M. He, and Y. Tian, “Multi-camera vehicle tracking and re-identification on ai city challenge 2019.” in _CVPR Workshops_ , 2019.
  * [54] M. Cen and C. Jung, “Fully convolutional siamese fusion networks for object tracking,” in _ICIP_ , 2018.
  * [55] H. Fan and H. Ling, “Siamese cascaded region proposal networks for real-time visual tracking,” in _CVPR_ , 2019.
  * [56] F. Du, P. Liu, W. Zhao, and X. Tang, “Correlation-guided attention for corner detection based visual tracking,” in _CVPR_ , 2020.
  * [57] Y. Yu, Y. Xiong, W. Huang, and M. R. Scott, “Deformable siamese attention networks for visual object tracking,” in _CVPR_ , 2020.
  * [58] S. Karthik, A. Prabhu, and V. Gandhi, “Simple unsupervised multi-object tracking,” _arXiv preprint_ , 2020.
  * [59] Y. Zhang, H. Sheng, Y. Wu, S. Wang, W. Ke, and Z. Xiong, “Multiplex labeling graph for near-online tracking in crowded scenes,” _IEEE IoT Journal_ , 2020\.
  * [60] N. L. Baisa, “Occlusion-robust online multi-object visual tracking using a gm-phd filter with cnn-based re-identification,” _JVCIR_ , 2021.
  * [61] F. Yang, X. Chang, S. Sakti, Y. Wu, and S. Nakamura, “Remot: A model-agnostic refinement for multiple object tracking,” _IVC_ , 2021.
  * [62] P. Bergmann, T. Meinhardt, and L. Leal-Taixe, “Tracking without bells and whistles,” in _ICCV_ , 2019.
  * [63] W. Feng, Z. Hu, W. Wu, J. Yan, and W. Ouyang, “Multi-object tracking with multiple cues and switcher-aware classification,” _arXiv preprint_ , 2019\.
  * [64] P. Chu and H. Ling, “Famnet: Joint learning of feature, affinity and multi-dimensional assignment for online multiple object tracking,” in _ICCV_ , 2019.
  * [65] J. Yin, W. Wang, Q. Meng, R. Yang, and J. Shen, “A unified object motion and affinity model for online multi-object tracking,” in _CVPR_ , 2020.
  * [66] B. Shuai, A. Berneshawi, X. Li, D. Modolo, and J. Tighe, “Siammot: Siamese multi-object tracking,” in _CVPR_ , 2021.
  * [67] C. Liang, Z. Zhang, X. Zhou, B. Li, and W. Hu, “One more check: making “fake background” be tracked again,” in _AAAI_ , 2022.
  * [68] X. Lin, Y.-a. Guo, and J. Wang, “Global correlation network: End-to-end joint multi-object detection and tracking,” _arXiv preprint_ , 2021.
  * [69] J. Pang, L. Qiu, X. Li, H. Chen, Q. Li, T. Darrell, and F. Yu, “Quasi-dense similarity learning for multiple object tracking,” in _CVPR_ , 2021.
  * [70] L. Jiao, R. Zhang, F. Liu, S. Yang, B. Hou, L. Li, and X. Tang, “New generation deep learning for video object detection: A survey,” _IEEE TNNLS_ , 2021.
  * [71] C. Guo, B. Fan, J. Gu, Q. Zhang, S. Xiang, V. Prinet, and C. Pan, “Progressive sparse local attention for video object detection,” in _ICCV_ , 2019.
  * [72] F. Xiao and Y. J. Lee, “Video object detection with an aligned spatial-temporal memory,” in _ECCV_ , 2018.
  * [73] Y. Chai, “Patchwork: A patch-wise attention network for efficient object detection and segmentation in video streams,” in _ICCV_ , 2019.
  * [74] G. Bertasius, L. Torresani, and J. Shi, “Object detection in video with spatiotemporal sampling networks,” in _ECCV_ , 2018.
  * [75] C. Feichtenhofer, A. Pinz, and A. Zisserman, “Detect to track and track to detect,” in _ICCV_ , 2017.
  * [76] M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S. C. Hoi, “Deep learning for person re-identification: A survey and outlook,” _IEEE TPAMI_ , 2021.
  * [77] Y.-C. Chen, X. Zhu, W.-S. Zheng, and J.-H. Lai, “Person re-identification by camera correlation aware feature augmentation,” _IEEE TPAMI_ , 2017.
  * [78] L. Zheng, Y. Yang, and A. G. Hauptmann, “Person re-identification: Past, present and future,” _arXiv preprint_ , 2016.
  * [79] L. Zheng, H. Zhang, S. Sun, M. Chandraker, Y. Yang, and Q. Tian, “Person re-identification in the wild,” in _CVPR_ , 2017.
  * [80] L. Zhao, X. Li, Y. Zhuang, and J. Wang, “Deeply-learned part-aligned representations for person re-identification,” in _ICCV_ , 2017.
  * [81] H. Yao, S. Zhang, R. Hong, Y. Zhang, C. Xu, and Q. Tian, “Deep representation learning with part loss for person re-identification,” _IEEE TIP_ , 2019\.
  * [82] Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, “Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline),” in _ECCV_ , 2018.
  * [83] C. Su, S. Zhang, J. Xing, W. Gao, and Q. Tian, “Deep attributes driven multi-camera person re-identification,” in _ECCV_ , 2016.
  * [84] Y. Lin, L. Zheng, Z. Zheng, Y. Wu, Z. Hu, C. Yan, and Y. Yang, “Improving person re-identification by attribute and identity learning,” _PR_ , 2019\.
  * [85] T. Matsukawa and E. Suzuki, “Person re-identification using cnn features learned from combination of attributes,” in _ICPR_ , 2016.
  * [86] K. Liu, B. Ma, W. Zhang, and R. Huang, “A spatio-temporal appearance representation for viceo-based pedestrian re-identification,” in _ICCV_ , 2015.
  * [87] J. Dai, P. Zhang, D. Wang, H. Lu, and H. Wang, “Video person re-identification by temporal residual learning,” _IEEE TIP_ , 2018.
  * [88] Y. Ye, X. Ke, and Z. Yu, “A cost matrix optimization method based on spatial constraints under hungarian algorithm,” in _ICRAI_ , 2020.
  * [89] N. L. Baisa, “Online multi-object visual tracking using a gm-phd filter with deep appearance learning,” in _FUSION_ , 2019.
  * [90] Y.-c. Yoon, A. Boragule, Y.-m. Song, K. Yoon, and M. Jeon, “Online multi-object tracking with historical appearance matching and scene adaptive detection filtering,” in _AVSS_ , 2018.
  * [91] M. Babaee, Z. Li, and G. Rigoll, “A dual cnn–rnn for multiple people tracking,” _Neurocomputing_ , 2019.
  * [92] J. Chen, H. Sheng, Y. Zhang, and Z. Xiong, “Enhancing detection model for multiple hypothesis tracking,” in _CVPR Workshops_ , 2017.
  * [93] L. Chen, H. Ai, Z. Zhuang, and C. Shang, “Real-time multiple people tracking with deeply learned candidate selection and person re-identification,” in _ICME_ , 2018.
  * [94] E. Ristani and C. Tomasi, “Features for multi-target multi-camera tracking and re-identification,” in _CVPR_ , 2018.
  * [95] H. Shen, L. Huang, C. Huang, and W. Xu, “Tracklet association tracker: An end-to-end learning-based association approach for multi-object tracking,” _arXiv preprint_ , 2018.
  * [96] W. Li, Y. Xiong, S. Yang, M. Xu, Y. Wang, and W. Xia, “Semi-tcl: Semi-supervised track contrastive representation learning,” _arXiv preprint_ , 2021.
  * [97] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _CVPR_ , 2015.
  * [98] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” _NeurIPS_ , 2020\.
  * [99] D. Neven, B. D. Brabandere, M. Proesmans, and L. V. Gool, “Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth,” in _CVPR_ , 2019.
  * [100] Z. Xu, W. Zhang, X. Tan, W. Yang, H. Huang, S. Wen, E. Ding, and L. Huang, “Segment as points for efficient online multi-object tracking and segmentation,” in _ECCV_ , 2020.
  * [101] Q. Liu, B. Liu, Y. Wu, W. Li, and N. Yu, “Real-time online multi-object tracking in compressed domain,” _IEEE Access_ , 2019.
  * [102] L. Leal-Taixé, C. Canton-Ferrer, and K. Schindler, “Learning by tracking: Siamese cnn for robust target association,” in _CVPR Workshops_ , 2016.
  * [103] E. Baser, V. Balasubramanian, P. Bhattacharyya, and K. Czarnecki, “Fantrack: 3d multi-object tracking with feature association network,” in _IV_ , 2019\.
  * [104] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _NeurIPS_ , 2015.
  * [105] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _ICCV_ , 2017.
  * [106] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint_ , 2018.
  * [107] Q. Liu, D. Chen, Q. Chu, L. Yuan, B. Liu, L. Zhang, and N. Yu, “Online multi-object tracking with unsupervised re-identification learning and occlusion estimation,” _Neurocomputing_ , 2022.
  * [108] X. Zhou, D. Wang, and P. Krähenbühl, “Objects as points,” _arXiv preprint_ , 2019.
  * [109] F. Yu, D. Wang, E. Shelhamer, and T. Darrell, “Deep layer aggregation,” in _CVPR_ , 2018.
  * [110] T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detection and tracking,” in _CVPR_ , 2021.
  * [111] Z. Xu, A. Meng, Z. Shi, W. Yang, Z. Chen, and L. Huang, “Continuous copy-paste for one-stage multi-object tracking and segmentation,” in _ICCV_ , 2021.
  * [112] Y. Xu, A. Osep, Y. Ban, R. Horaud, L. Leal-Taixé, and X. Alameda-Pineda, “How to train your deep multi-object tracker,” in _CVPR_ , 2020.
  * [113] E. Yu, Z. Li, and S. Han, “Towards discriminative representation: Multi-view trajectory contrastive learning for online multi-object tracking,” in _CVPR_ , 2022.
  * [114] S. Sun, N. Akhtar, H. Song, A. Mian, and M. Shah, “Deep affinity network for multiple object tracking,” _IEEE TPAMI_ , 2019.
  * [115] J. Peng, C. Wang, F. Wan, Y. Wu, Y. Wang, Y. Tai, C. Wang, J. Li, F. Huang, and Y. Fu, “Chained-tracker: Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking,” in _ECCV_ , 2020\.
  * [116] Y. Xu, Y. Ban, G. Delorme, C. Gan, D. Rus, and X. Alameda-Pineda, “Transcenter: Transformers with dense queries for multiple-object tracking,” _arXiv preprint_ , 2021.
  * [117] K. Hara, H. Kataoka, and Y. Satoh, “Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?” in _CVPR_ , 2018.
  * [118] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in _CVPR_ , 2019.
  * [119] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _ICCV_ , 2017.
  * [120] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in _NeurIPS_ , 2015.
  * [121] J. Wu, J. Cao, L. Song, Y. Wang, M. Yang, and J. Yuan, “Track to detect and segment: An online multi-object tracker,” in _CVPR_ , 2021.
  * [122] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in _ICCV_ , 2017.
  * [123] Y. Wang, K. Kitani, and X. Weng, “Joint object detection and multi-object tracking with graph neural networks,” in _ICRA_ , 2021.
  * [124] C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe, “Weisfeiler and leman go neural: Higher-order graph neural networks,” in _AAAI_ , 2019.
  * [125] W. Ruan, W. Liu, Q. Bao, J. Chen, Y. Cheng, and T. Mei, “Poinet: pose-guided ovonic insight network for multi-person pose tracking,” in _ACM MM_ , 2019\.
  * [126] P. Li, J. Shi, and S. Shen, “Joint spatial-temporal optimization for stereo 3d object tracking,” in _CVPR_ , 2020.
  * [127] S. Jin, W. Liu, W. Ouyang, and C. Qian, “Multi-person articulated tracking with spatial and temporal embeddings,” in _CVPR_ , 2019.
  * [128] M. Huang, S. Narasimhaswamy, S. Vazir, H. Ling, and M. Hoai, “Forward propagation, backward regression, and pose association for hand tracking in the wild,” in _CVPR_ , 2022.
  * [129] B. Yan, Y. Jiang, P. Sun, D. Wang, Z. Yuan, P. Luo, and H. Lu, “Towards grand unification of object tracking,” in _ECCV_ , 2022.
  * [130] P. Li and J. Jin, “Time3d: End-to-end joint monocular 3d object detection and tracking for autonomous driving,” in _CVPR_ , 2022.
  * [131] F. Ma, M. Z. Shou, L. Zhu, H. Fan, Y. Xu, Y. Yang, and Z. Yan, “Unified transformer tracker for object tracking,” in _CVPR_ , 2022.
  * [132] Z. Zhao, Z. Wu, Y. Zhuang, B. Li, and J. Jia, “Tracking objects as pixel-wise distributions,” in _ECCV_ , 2022.
  * [133] Q. Chu, W. Ouyang, B. Liu, F. Zhu, and N. Yu, “Dasot: A unified framework integrating data association and single object tracking for online multi-object tracking,” in _AAAI_ , 2020.
  * [134] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _CVPR_ , 2017.
  * [135] Q. Wang, Y. Zheng, P. Pan, and Y. Xu, “Multiple object tracking with correlation learning,” in _CVPR_ , 2021.
  * [136] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in _ECCV_ , 2016\.
  * [137] H. Lee, I. Kim, and D. Kim, “Van: Versatile affinity network for end-to-end online multi-object tracking,” in _ACCV_ , 2020.
  * [138] S. Liu, X. Li, H. Lu, and Y. He, “Multi-object tracking meets moving uav,” in _CVPR_ , 2022.
  * [139] P. Sun, Y. Jiang, R. Zhang, E. Xie, J. Cao, X. Hu, T. Kong, Z. Yuan, C. Wang, and P. Luo, “Transtrack: Multiple-object tracking with transformer,” _arXiv preprint_ , 2020.
  * [140] X. Chen, S. M. Iranmanesh, and K.-C. Lien, “Patchtrack: Multiple object tracking using frame patches,” _arXiv preprint_ , 2022.
  * [141] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NeurIPS_ , 2017.
  * [142] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _ECCV_. Springer, 2020.
  * [143] L. Zheng, M. Tang, Y. Chen, G. Zhu, J. Wang, and H. Lu, “Improving multiple object tracking with single object tracking,” in _CVPR_ , 2021.
  * [144] A. Milan, S. H. Rezatofighi, A. Dick, I. Reid, and K. Schindler, “Online multi-target tracking using recurrent neural networks,” in _AAAI_ , 2017\.
  * [145] X. Jiang, P. Li, Y. Li, and X. Zhen, “Graph neural based end-to-end data association framework for online multiple-object tracking,” _arXiv preprint_ , 2019.
  * [146] S. Guo, J. Wang, X. Wang, and D. Tao, “Online multiple object tracking with cross-task synergy,” in _CVPR_ , 2021.
  * [147] C. Kim, L. Fuxin, M. Alotaibi, and J. M. Rehg, “Discriminative appearance modeling with multi-track pooling for real-time multi-object tracking,” in _CVPR_ , 2021.
  * [148] C. Kim, F. Li, and J. M. Rehg, “Multi-object tracking with neural gating using bilinear lstm,” in _ECCV_ , 2018.
  * [149] S. Wang, H. Sheng, Y. Zhang, Y. Wu, and Z. Xiong, “A general recurrent tracking framework without real data,” in _ICCV_ , 2021.
  * [150] P. Tokmakov, J. Li, W. Burgard, and A. Gaidon, “Learning to track with object permanence,” in _ICCV_ , 2021.
  * [151] N. Ballas, L. Yao, C. Pal, and A. Courville, “Delving deeper into convolutional networks for learning video representations,” _arXiv preprint_ , 2015.
  * [152] F. Saleh, S. Aliakbarian, H. Rezatofighi, M. Salzmann, and S. Gould, “Probabilistic tracklet scoring and inpainting for multiple object tracking,” in _CVPR_ , 2021.
  * [153] G. Wang, Y. Wang, R. Gu, W. Hu, and J.-N. Hwang, “Split and connect: A universal tracklet booster for multi-object tracking,” _IEEE TMM_ , 2022\.
  * [154] J. Peng, T. Wang, W. Lin, J. Wang, J. See, S. Wen, and E. Ding, “Tpm: Multiple object tracking with tracklet-plane matching,” _PR_ , 2020.
  * [155] H.-M. Hsu, T.-W. Huang, G. Wang, J. Cai, Z. Lei, and J.-N. Hwang, “Multi-camera tracking of vehicles based on deep features re-id and trajectory-based camera link models.” in _CVPR Workshops_ , 2019.
  * [156] J. Gao and R. Nevatia, “Revisiting temporal modeling for video-based person reid,” _arXiv preprint_ , 2018.
  * [157] P. Dai, R. Weng, W. Choi, C. Zhang, Z. He, and W. Ding, “Learning a proposal classifier for multiple object tracking,” in _CVPR_ , 2021.
  * [158] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint_ , 2016.
  * [159] J. Rajasegaran, G. Pavlakos, A. Kanazawa, and J. Malik, “Tracking people by predicting 3d appearance, location and pose,” in _CVPR_ , 2022.
  * [160] ——, “Tracking people with 3d representations,” in _NeurIPS_ , 2021.
  * [161] A. Kanazawa, J. Y. Zhang, P. Felsen, and J. Malik, “Learning 3d human dynamics from video,” in _CVPR_ , 2019.
  * [162] J. Cai, M. Xu, W. Li, Y. Xiong, W. Xia, Z. Tu, and S. Soatto, “Memot: Multi-object tracking with memory,” in _CVPR_ , 2022.
  * [163] J. Xiang, G. Xu, C. Ma, and J. Hou, “End-to-end learning deep crf models for multi-object tracking deep crf models,” _IEEE TCSVT_ , 2020.
  * [164] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. Torr, “Conditional random fields as recurrent neural networks,” in _ICCV_ , 2015.
  * [165] M. Larsson, F. Kahl, S. Zheng, A. Arnab, P. H. Torr, and R. I. Hartley, “Learning arbitrary potentials in crfs with gradient descent.” 2017.
  * [166] D. Reid, “An algorithm for tracking multiple targets,” _IEEE TAC_ , 1979.
  * [167] Y. Zhang, H. Sheng, Y. Wu, S. Wang, W. Lyu, W. Ke, and Z. Xiong, “Long-term tracking with deep tracklet association,” _IEEE TIP_ , 2020.
  * [168] G. Wang, Y. Wang, H. Zhang, R. Gu, and J.-N. Hwang, “Exploit the connectivity: Multi-object tracking with trackletnet,” in _ACM MM_ , 2019.
  * [169] Y.-C. Yoon, D. Y. Kim, Y.-m. Song, K. Yoon, and M. Jeon, “Online multiple pedestrians tracking using deep temporal appearance matching association,” _Information Sciences_ , 2021.
  * [170] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei, “Relation networks for object detection,” in _CVPR_ , 2018.
  * [171] J. Xu, Y. Cao, Z. Zhang, and H. Hu, “Spatial-temporal relation networks for multi-object tracking,” in _ICCV_ , 2019.
  * [172] F. Bastani, S. He, and S. Madden, “Self-supervised multi-object tracking with cross-input consistency,” _NeurIPS_ , 2021.
  * [173] J. Zhu, H. Yang, N. Liu, M. Kim, W. Zhang, and M.-H. Yang, “Online multi-object tracking with dual matching attention networks,” in _ECCV_ , 2018.
  * [174] Q. Liu, Q. Chu, B. Liu, and N. Yu, “Gsm: Graph similarity model for multi-object tracking,” in _IJCAI_ , 2020.
  * [175] J. Li, X. Gao, and T. Jiang, “Graph networks for multiple object tracking,” in _WACV_ , 2020.
  * [176] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in _ICML_ , 2017.
  * [177] M. Guo, E. Chou, D.-A. Huang, S. Song, S. Yeung, and L. Fei-Fei, “Neural graph matching networks for fewshot 3d action recognition,” in _ECCV_ , 2018.
  * [178] P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu, “Interaction networks for learning about objects, relations and physics,” _arXiv preprint_ , 2016.
  * [179] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner _et al._ , “Relational inductive biases, deep learning, and graph networks,” _arXiv preprint_ , 2018.
  * [180] G. Brasó and L. Leal-Taixé, “Learning a neural solver for multiple object tracking,” in _CVPR_ , 2020.
  * [181] J. He, Z. Huang, N. Wang, and Z. Zhang, “Learnable graph matching: Incorporating graph partitioning with deep feature learning for multiple object tracking,” in _CVPR_ , 2021.
  * [182] X. Zhou, T. Yin, V. Koltun, and P. Krähenbühl, “Global tracking transformers,” in _CVPR_ , 2022.
  * [183] K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: the clear mot metrics,” _EURASIP Journal on Image and Video Processing_ , 2008.
  * [184] E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in _ECCV_ , 2016.
  * [185] J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé, and B. Leibe, “Hota: A higher order metric for evaluating multi-object tracking,” _IJCV_ , 2021.
  * [186] L. Wen, D. Du, Z. Cai, Z. Lei, M.-C. Chang, H. Qi, J. Lim, M.-H. Yang, and S. Lyu, “Ua-detrac: A new benchmark and protocol for multi-object detection and tracking,” _CVIU_ , 2020.
  * [187] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _CVPR_ , 2020.
  * [188] W. Feng, Z. Hu, B. Li, W. Gan, W. Wu, and W. Ouyang, “Samot: Switcher-aware multi-object tracking and still another mot measure,” _arXiv preprint_ , 2020\.
  * [189] J. Valmadre, A. Bewley, J. Huang, C. Sun, C. Sminchisescu, and C. Schmid, “Local metrics for multi-object tracking,” _arXiv preprint_ , 2021.
  * [190] A. Girbau, X. Giró-i Nieto, I. Rius, and F. Marqués, “Multiple object tracking with mixture density networks for trajectory estimation,” _arXiv preprint_ , 2021.
  * [191] M. Babaee, A. Athar, and G. Rigoll, “Multiple people tracking using hierarchical deep tracklet re-identification,” _arXiv preprint_ , 2018.
  * [192] N. L. Baisa, “Robust online multi-target visual tracking using a hisp filter with discriminative deep appearance learning,” _JVCIR_ , 2021.
  * [193] I. Papakis, A. Sarkar, and A. Karpatne, “Gcnnmatch: Graph convolutional neural networks for multi-object tracking via sinkhorn normalization,” _arXiv preprint_ , 2020.
  * [194] C. Shan, C. Wei, B. Deng, J. Huang, X.-S. Hua, X. Cheng, and K. Liang, “Tracklets predicting based adaptive graph tracking,” _arXiv preprint_ , 2020.
  * [195] A. Shenoi, M. Patel, J. Gwak, P. Goebel, A. Sadeghian, H. Rezatofighi, R. Martín-Martín, and S. Savarese, “Jrmot: A real-time 3d multi-object tracker and a new large-scale dataset,” in _IROS_ , 2020.
  * [196] D. Mykheievskyi, D. Borysenko, and V. Porokhonskyy, “Learning local feature descriptors for multiple object tracking,” in _ACCV_ , 2020.
  * [197] W. Zhang, H. Zhou, S. Sun, Z. Wang, J. Shi, and C. C. Loy, “Robust multi-modality multi-object tracking,” in _ICCV_ , 2019.
  * [198] S. Wang, Y. Sun, C. Liu, and M. Liu, “Pointtracknet: An end-to-end network for 3-d object detection and tracking from point clouds,” _IEEE RAL_ , 2020.
  * [199] H. Wu, Q. Li, C. Wen, X. Li, X. Fan, and C. Wang, “Tracklet proposal network for multi-object tracking on point clouds,” in _IJCAI_ , 2021.
  * [200] S. Wang, P. Cai, L. Wang, and M. Liu, “Ditnet: End-to-end 3d object detection and track id assignment in spatio-temporal world,” _IEEE RAL_ , 2021.
  * [201] K. Huang and Q. Hao, “Joint multi-object detection and tracking with camera-lidar fusion for autonomous driving,” in _IROS_ , 2021.
  * [202] H.-N. Hu, Y.-H. Yang, T. Fischer, T. Darrell, F. Yu, and M. Sun, “Monocular quasi-dense 3d object tracking,” _arXiv preprint_ , 2021.
  * [203] A. Rangesh, P. Maheshwari, M. Gebre, S. Mhatre, V. Ramezani, and M. M. Trivedi, “Trackmpnn: A message passing graph neural architecture for multi-object tracking,” _arXiv preprint_ , 2021.
  * [204] A. Milan, L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler, “Mot16: A benchmark for multi-object tracking,” _arXiv preprint_ , 2016.
  * [205] P. Dendorfer, H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler, and L. Leal-Taixé, “Mot20: A benchmark for multi object tracking in crowded scenes,” _arXiv preprint_ , 2020.
  * [206] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _CVPR_ , 2012.
  * [207] M. Keuper, S. Tang, B. Andres, T. Brox, and B. Schiele, “Motion segmentation & multiple object tracking by correlation co-clustering,” _IEEE TPAMI_ , 2018.
  * [208] Y. Lu, C. Lu, and C.-K. Tang, “Online video object detection using association lstm,” in _ICCV_ , 2017.
  * [209] M. Ullah and F. Alaya Cheikh, “A directed sparse graphical model for multi-target tracking,” in _CVPR Workshops_ , 2018.
  * [210] X. Weng, J. Wang, D. Held, and K. Kitani, “3d multi-object tracking: A baseline and new evaluation metrics,” in _IROS_ , 2020.
  * [211] S. Hao, P. Liu, Y. Zhan, K. Jin, Z. Liu, M. Song, J.-N. Hwang, and G. Wang, “Divotrack: A novel dataset and baseline method for cross-view multi-object tracking in diverse open scenes,” _IJCV_ , 2023.
  * [212] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in _ICIP_ , 2017.
  * [213] Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, “Yolox: Exceeding yolo series in 2021,” _arXiv preprint_ , 2021.
  * [214] I. Ruiz, L. Porzi, S. R. Bulò, P. Kontschieder, and J. Serrat, “Weakly supervised multi-object tracking and segmentation.” in _WACV Workshops_ , 2021.
  * [215] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _ICCV_ , 2017.
  * [216] D. McKee, B. Shuai, A. Berneshawi, M. Wang, D. Modolo, S. Lazebnik, and J. Tighe, “Multi-object tracking with hallucinated and unlabeled videos,” _arXiv preprint_ , 2021.
  * [217] K. Nishimura, J. Hayashida, C. Wang, R. Bise _et al._ , “Weakly-supervised cell tracking via backward-and-forward propagation,” in _ECCV_ , 2020.
  * [218] A. Wu, C. Lin, B. Chen, W. Huang, Z. Huang, and W.-S. Zheng, “Transductive multi-object tracking in complex events by interactive self-training,” in _ACM MM_ , 2020.
  * [219] S. Yoon, K. Shim, K. Park, and C. Kim, “Weakly-supervised multiple object tracking via a masked center point warping loss,” in _ICIP_ , 2021.
  * [220] Z. He, J. Li, D. Liu, H. He, and D. Barber, “Tracking by animation: Unsupervised learning of multi-object attentive trackers,” in _CVPR_ , 2019\.
  * [221] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig, “Virtual worlds as proxy for multi-object tracking analysis,” in _CVPR_ , 2016.
  * [222] Y. Liu, Z. Wang, X. Zhou, and L. Zheng, “Synthetic data are as good as the real for association knowledge learning in multi-object tracking,” _arXiv preprint_ , 2021.
  * [223] M. Fabbri, G. Brasó, G. Maugeri, O. Cetintas, R. Gasparini, A. Ošep, S. Calderara, L. Leal-Taixe, and R. Cucchiara, “Motsynth: How can synthetic data help pedestrian detection and tracking?” in _ICCV_ , 2021.
  * [224] A. Hornakova, R. Henschel, B. Rosenhahn, and P. Swoboda, “Lifted disjoint paths with application in multiple object tracking,” in _ICML_ , 2020.
  * [225] D. M. Nguyen, R. Henschel, B. Rosenhahn, D. Sonntag, and P. Swoboda, “Lmgp: Lifted multicut meets geometry projections for multi-camera multi-object tracking,” in _CVPR_ , 2022.
  * [226] W. Song, S. Li, T. Chang, A. Hao, Q. Zhao, and H. Qin, “Cross-view contextual relation transferred network for unsupervised vehicle tracking in drone videos,” in _WACV_ , 2020.
  * [227] Y. Gan, R. Han, L. Yin, W. Feng, and S. Wang, “Self-supervised multi-view multi-human association and tracking,” in _ACM MM_ , 2021.
  * [228] K. G. Quach, P. Nguyen, H. Le, T.-D. Truong, C. N. Duong, M.-T. Tran, and K. Luu, “Dyglip: A dynamic graph model with link prediction for accurate multi-camera multiple object tracking,” in _CVPR_ , 2021.
  * [229] D. M. Nguyen, R. Henschel, B. Rosenhahn, D. Sonntag, and P. Swoboda, “Lmgp: Lifted multicut meets geometry projections for multi-camera multi-object tracking,” _arXiv preprint_ , 2021.
  * [230] Y. Xu, X. Liu, Y. Liu, and S.-C. Zhu, “Multi-view people tracking via hierarchical trajectory composition,” in _CVPR_ , 2016.
  * [231] R. Han, W. Feng, Y. Zhang, J. Zhao, and S. Wang, “Multiple human association and tracking from egocentric and complementary top views,” _IEEE TPAMI_ , 2021.
  * [232] L. Chen, H. Ai, R. Chen, Z. Zhuang, and S. Liu, “Cross-view tracking for multi-human 3d pose estimation at over 100 fps,” in _CVPR_ , 2020.
  * [233] R. Han, W. Feng, J. Zhao, Z. Niu, Y. Zhang, L. Wan, and S. Wang, “Complementary-view multiple human tracking,” in _AAAI_ , 2020.
  * [234] Y. Xu, X. Liu, L. Qin, and S.-C. Zhu, “Cross-view people tracking by scene-centered spatio-temporal parsing,” in _AAAI_ , 2017.
  * [235] C. Luo, X. Yang, and A. Yuille, “Exploring simple 3d multi-object tracking for autonomous driving,” in _ICCV_ , 2021.
  * [236] Z. Pang, Z. Li, and N. Wang, “Simpletrack: Understanding and rethinking 3d multi-object tracking,” _arXiv preprint_ , 2021.
  * [237] S. Shi, C. Guo, J. Yang, and H. Li, “Pv-rcnn: The top-performing lidar-only solutions for 3d detection/3d tracking/domain adaptation of waymo open dataset challenges,” _arXiv preprint_ , 2020.
  * [238] J. Sun, Y. Xie, S. Zhang, L. Chen, G. Zhang, H. Bao, and X. Zhou, “You don’t only look once: Constructing spatial-temporal memory for integrated 3d object detection and tracking,” in _ICCV_ , 2021.
  * [239] T. Linder, S. Breuers, B. Leibe, and K. O. Arras, “On multi-modal people tracking from mobile platforms in very crowded and dynamic environments,” in _ICRA_ , 2016.
  * [240] Y. Zhong, S. You, and U. Neumann, “Modeling cross-modal interaction in a multi-detector, multi-modal tracking framework,” in _ACCV_ , 2020.