# MOTS R-CNN: Cosine-margin-triplet loss for multi-object tracking

Amit Satish Unde and Renu M. Rameshan
School of Computing and Electrical Engineering, Indian Institute of
Technology, Mandi, India
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

One of the central tasks of multi-object tracking involves learning a distance
metric that is consistent with the semantic similarities of objects. Design of
an appropriate loss function that encourages discriminative feature learning
is among the most crucial challenges in deep neural network based metric
learning. Despite significant progress, slow convergence and a poor local
optimum of the existing contrastive and triplet loss based deep metric
learning methods necessitates a better solution. In this paper, we propose
cosine-margin-contrastive (CMC) and cosine-margin-triplet (CMT) loss by
reformulating both contrastive and triplet loss functions from the perspective
of cosine distance. The proposed reformulation as a cosine loss is achieved by
feature normalization which distributes the learned features on a hypersphere.
We then propose the MOTS R-CNN framework for joint multi-object tracking and
segmentation, particularly targeted at improving the tracking performance.
Specifically, the tracking problem is addressed through deep metric learning
based on the proposed loss functions. We propose a scale-invariant tracking by
using a multi-layer feature aggregation scheme to make the model robust
against object scale variations and occlusions. The MOTS R-CNN111Our code will
be made available at: https://github.com/amitsunde/MOTS-R-CNN achieves the
state-of-the-art tracking performance on the KITTI MOTS dataset. We show that
the MOTS R-CNN reduces the identity switching by $62\%$ and $61\%$ on cars and
pedestrians, respectively in comparison to Track R-CNN.

##  1 Introduction

Multi-object tracking (MOT) is a well-established problem in the field of
computer vision [7, 4, 47, 2]. It is regaining attention of the research
community owing to significant progresses in autonomous vehicles and robotics
[16, 17, 24]. The “tracking-by-detection” is the widely adopted strategy to
tackle MOT problem [47, 31, 37]. It is the process of detecting objects of
interest in each video frame and matching their identities across frames to
obtain the object trajectories over time. Given a video frame, the object
detector identifies and locates possible objects of interest and feeds them as
input to the tracking system. The tracker computes the similarity score
between the feature vectors of previously tracked targets and objects detected
in the current frame. Based on the similarity score, the tracker links
detections and targets to find optimal object trajectories.

The traditional tracking system consists of two parts: 1) robust hand-crafted
feature extraction for each detected object [35, 1, 13] and 2) distance metric
learning consistent with the semantic similarities of objects [20, 34, 27].
After an exhaustive study, it is noticed that the discrimination power of
hand-crafted features is low that results in sub-optimal performance [45, 22,
49]. With the evolution of convolutional neural networks (CNNs) in recent
years, deep CNN-based object tracking has gained increasing popularity [2, 31,
48]. Different from traditional methods, CNN-based approaches learn the
feature representation of each detected object and distance metric in an end-
to-end manner. MOT problem using CNNs has been thoroughly studied over the
past few years which in turn results in a noticeable improvement in the
tracking performance [40, 46, 41]. However, tracking remains a challenging
problem, especially in unconstrained scenarios such as autonomous driving, due
to severe occlusions, scale and appearance variation of objects, false
positives (FP) and missing detections, and illumination variations [31, 49,
29].

One possible reason that affects the tracking performance of MOT is the use of
axis-aligned rectangular bounding boxes [39, 11]. This choice of the bounding
box may contain information about the background and other objects in close
proximity due to the overlapping of bounding boxes, especially in crowded
scenes, which deteriorates the efficiency of the tracker. Hence, approaches to
improve the tracker performance is shifting from the bounding box level to the
pixel level. In this direction, Voigtlaender et al. proposed a unified
framework for multi-object tracking and segmentation (MOTS) through the
integration of the tracking head in the Mask R-CNN model [9]. Most of the
approaches thereafter reported in the literature are focused on learning
embedding representation of objects using triplet loss [37, 29, 44]. However,
existing deep metric learning frameworks aiming to learn discriminative
features based on triplet loss often suffer from slow convergence and a poor
local optimum [34, 45, 27, 25].

In this paper, we propose a cosine-margin-triplet (CMT) and cosine-margin-
contrastive (CMC) loss to enhance the feature discrimination power of the
multi-object tracking model. Since the Euclidean distance between the feature
vectors can be unbounded, we utilize the dot product between the pair of
normalized feature vectors which is equal to the cosine similarity. The
feature normalization distributes the learned feature vectors on a hypersphere
and encourages better feature learning by separating them in angular. In
addition, a margin term is introduced in the cosine metric to increase the
intra-class compactness and inter-class discrimination.

We then propose the MOTS R-CNN network to address the MOTS problem. The
proposed model extends the Mask R-CNN [9] with the addition of the tracking
head. The 3D convolutional layers are integrated into the MOTS R-CNN framework
to model the spatio-temporal dependency of objects. The tracking problem is
addressed through joint learning of feature representation and distance metric
based on the CMT loss in an end-to-end manner. The tracking capability of the
MOTS R-CNN to handle object scale variations is improved by fusing features
from different layers of the convolutional network. Further improvement in
tracking accuracy is achieved by taking advantage of the readily available
instance segmentation mask in the MOTS R-CNN to focus only on foreground
information of objects. In addition, we empirically show the inefficiency of
the softmax loss for object classification. We demonstrate that incorporating
large margin cosine loss (LMCL) [38] in the MOTS R-CNN increases the object
classification accuracy. The major contributions of our work are as follows:

  * •

We propose cosine-margin-triplet and cosine-margin-contrastive loss functions
to enforce closeness between pairs of feature vectors belonging to the same
instance while pushing dissimilar vectors far away on the embedding space. The
proposed loss functions encourage discriminative feature learning for multi-
object tracking problems.

  * •

We present the MOTS R-CNN model to address the MOTS problem. We show that
identity switching (IDS) using the proposed model for cars and pedestrians is
reduced by $62\%$ and $61\%$ respectively in comparison with Track R-CNN that
uses triplet loss for distance metric learning.

  * •

We empirically demonstrate the usefulness of large margin cosine loss over
traditional softmax loss for object classification.

The remainder of this paper is organized as follows. We discuss in section 2
the existing works related to MOT and MOTS problem. In section 3, we review
preliminaries for distance metric learning. In section 4, we propose the CMT
and CMC loss functions. We present the proposed MOTS R-CNN model in section 5.
The effectiveness of the proposed model through experimental analysis is shown
in section 6 and followed by conclusions in section 7.

##  2 Related Work

There are only few works reported in the literature to address the MOTS task
since it is introduced very recently. So, we first discuss existing literature
related to the MOT problem. Then, we briefly review the related work on the
MOTS task.

Multi-object tracking. The work in [31] combined appearance, motion, and
interaction cues using a recurrent neural network to make the network robust
against occlusions. However, a large number of IDS as compared to state-of-
the-art methods show that the combination of various cues does not guarantee
an improvement in the tracking performance. Ji Zhu et al. proposed a spatial
and temporal attention model to extract only object-specific features and to
reduce the effect of noisy observations on the tracking performance [48].
Their work was mainly focused on the extension of a single object tracker for
the MOT task. As an alternative to the traditional tracking-by-detection
approach, a joint detection and tracking method was proposed in [40] by
integrating the tracking head in the single-shot detector to develop the real-
time MOT system. While the joint paradigm speed up the overall process, the
inferior performance of the single-shot detector caused a reduction in the
tracking accuracy.

The work in [46] extended Siamese single object tracker for the MOT task
through the integration of motion estimation and data association into a
single framework. The combination of appearance and motion features from 2D
and 3D space jointly was proposed in [41] for improving the accuracy of the
tracker. Furthermore, a graph neural network was employed to promote feature
interaction of various objects aiming at more discriminative feature learning.
Sarthak Sharma et al. proposed the BeyondPixels model for MOT by taking the 3D
pose, shape, and motion information of objects into account [33]. The matching
was done only in the specific search area obtained through the propagation of
position and orientation information of objects in successive frames.

Multi-object tracking and segmentation. Paul Voigtlaender et al. proposed the
Track R-CNN framework to integrate object detection, instance segmentation,
and tracking in a unified framework [37]. The tracking head was incorporated
in parallel to the box and mask heads of the Mask R-CNN architecture. The
tracking problem was posed as a feature learning problem by minimizing the
triplet loss. It was observed that the tracking accuracy was poor both in the
case of cars and pedestrians. This can be due to the use of triplet loss for
distance metric learning and the extraction of features only from the final
convolutional layer of the backbone network. This single layer feature
extraction can make the tracking performance sensitive to the object scale
variation.

The MOTSNet proposed in [29] achieved the scale-invariant feature learning by
making use of the feature pyramid network [18] to form the backbone
convolutional network. The segmentation mask was used to improve the tracking
performance by extracting only the foreground information of objects. The
triplet loss was employed to facilitate discriminative feature learning for
object tracking. However, the overall performance improvement was largely
attributed to training the network with their automatically generated MOTS
training dataset. The tracking-by-points strategy was proposed in [43] by
interpreting image pixels as irregular 2D point clouds. The shape, color,
semantic class labels, and position information of objects was taken into
account for the object tracking. The incorporation of motion information
coupled with the recovery of missing detections in the MOTS framework was
proposed in [23]. While MOTSFusion yields promising tracking performance, the
two-stage tracking process involving the construction of short tracks using
optical flow and its projection into a 3D space makes it computationally
heavy. The computational overhead in MOTSFusion can restrict its extension to
real-time applications such as autonomous driving.

##  3 Review of distance metric learning

In this section, we review various loss functions that are used for distance
metric learning.

Softmax loss. The softmax loss is widely used for object classification, face
recognition, and person re-identification [21, 6, 42, 15]. It enables feature
learning by formulating the learning task as a multi-class classification
problem. The softmax loss is given as,

$L=\frac{1}{N}\sum_{i=1}^{N}-log\frac{e^{W_{y_{i}}^{T}x_{i}}+b_{y_{i}}}{\sum_{j=1}^{n}e^{W_{j}^{T}x_{i}}+b_{j}}$
(1)

where $(x_{i},y_{i})$ denotes the input feature vector to the softmax layer
and associated class label respectively and $N$ is the number of training
examples. The terms $W_{j}$ and $b_{j}$ represent the $j^{th}$ column of the
weight matrix $W$ and corresponding bias term respectively. In spite of its
wide use, the learned features using the softmax loss function are known to be
less discriminative [38, 6, 21].

Large margin cosine loss. The LMCL overcomes limitations of the traditional
softmax loss [38]. It forces the network to distribute learned features on a
hypersphere through $\ell_{2}$ normalization of $W_{j}$ and $x_{i}$. The LMCL
introduces a positive margin $m$ in the loss function and is expressed as,

$L=\frac{1}{N}\sum_{i=1}^{N}-log\frac{e^{s(\cos(\theta_{y_{i},i})-m)}}{e^{s(\cos(\theta_{y_{i},i})-m)}+\sum_{j\neq
y_{i}}e^{s\cos(\theta_{j,i})}}$ (2)

where $\theta_{j,i}$ is the angle between normalized $W_{j}$ and $x_{i}$. The
incorporation of LMCL in face recognition models lead to noticeable
performance improvement [38].

Contrastive loss. Given a pair of feature vectors $(f_{i},f_{j})$ together
with the binary label $(y_{ij})$, the contrastive loss minimizes the distance
between the feature vectors belonging to the same class while penalizing the
distance between negative pair if it is smaller than a specified margin $m$
[34, 45]. The contrastive loss is defined as,

$L=y_{ij}\parallel f_{i}-f_{j}\parallel_{2}^{2}+(1-y_{ij})\max(0,m-\parallel
f_{i}-f_{j}\parallel_{2})^{2}$ (3)

where $y_{ij}=1$ when $(f_{i},f_{j})$ are from the same class and $y_{ij}=0$
when $(f_{i},f_{j})$ belongs to the different class.

Triplet loss. Triplet loss has gained a lot of attention for person re-
identification and object tracking tasks due to its superior performance in
deep face recognition [32, 12, 36, 37, 29]. It is defined on triplets
$(f_{i}^{a},f_{i}^{p},f_{i}^{n})$, where $f_{i}^{a}$ is referred to as anchor
of the triplet such that positive pair $(f_{i}^{a},f_{i}^{p})$ have the same
class label and the negative pair $(f_{i}^{a},f_{i}^{n})$ have the different
label. Triplet loss distributes the learned feature vectors on an embedding
space where the distance between positive pair is larger than that of negative
pair by a specified margin $m$. It is given as,

$L=\max(0,\parallel f_{i}^{a}-f_{i}^{p}\parallel_{2}^{2}-\parallel
f_{i}^{a}-f_{i}^{n}\parallel_{2}^{2}+m)$ (4)

The performance of both contrastive and triplet loss based distance metric
learning methods depends on the hard sampling which is employed to obtain
nontrivial pairs or triplets in large mini batches. Despite their widespread
use in various applications, both the loss functions have drawbacks of slow
convergence and poor local optima [34, 45, 27, 25].

##  4 Proposed CMT and CMC loss functions

One of the important tasks of multi-object tracking involves learning a
distance metric that is consistent with the semantic similarities of objects.
Distance metric learning is the process of learning an embedded representation
of the objects to keep the distance between similar objects at the minimum
while pushing away instances of dissimilar objects far on the embedding space.
In this section, we propose a cosine-margin-triplet loss and cosine-margin-
contrastive loss to enhance the feature discrimination power of the multi-
object tracking model.

###  4.1 Cosine-margin-triplet loss

Motivated from the success of triplet loss for face recognition and person re-
identification, we design the loss function based on triplets of feature
vectors obtained using deep CNNs. Let $D$ denote the set of detections over a
batch of video frames. For each detected object $d_{i}\in D$, there is an
associated ground truth segmentation mask, class label, tracking identity, and
an association feature vector $f_{i}$. We define a loss function on the
triplets $(f_{i},f_{i}^{+},f_{i}^{-})$, where $f_{i}$ is the anchor of the
triplet that shares the same tracking identity with the anchor positive
$(f_{i}^{+})$ and has a different identity (but the same class) with the
anchor negative $(f_{i}^{-})$. The loss function is formulated as,

$L=\frac{1}{N}\sum_{i=1}^{N}-log\frac{e^{f_{i}^{T}f^{+}}}{e^{f_{i}^{T}f^{+}}+e^{f_{i}^{T}f^{-}}}$
(5)

where $N=|D|$. The loss function in Eq. (5) aims to minimize the positive
pair, $(f_{i},f_{i}^{+})$ distance, and penalize the negative pair,
$(f_{i},f_{i}^{-})$ distance. It builds upon the dot product between the pair
of feature vectors which is given as,

$f_{i}^{T}f_{i}^{+}=\parallel f_{i}\parallel\parallel
f_{i}^{+}\parallel\cos(\theta_{i}^{+})$ (6)

where $\theta_{i}^{+}$ is the angle between $f_{i}$ and $f_{i}^{+}$.

The numerical value of the dot product is influenced by both the direction and
norm of feature vectors. In order to develop effective feature learning, it is
essential to make the dot product determined only by the direction. Hence, the
feature vectors are normalized to have unit norm by $\ell_{2}$ normalization.
In order to enable sufficient angular separability, the radius of the sphere
is scaled to $s$. This step distributes the learned feature vectors on a
hypersphere of radius $s$. The cosine similarity between the normalized
feature vectors has a geometric correspondence to the geodesic distance on the
hypersphere. This correspondence can be attributed to the relationship between
the central angle and arc of a circle and also to the fact that the learned
embeddings are angular separable. The normalized loss function is expressed
as,

$\displaystyle L$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}-log\frac{e^{s(f_{i}^{T}f^{+})}}{e^{s(f_{i}^{T}f^{+})}+e^{s(f_{i}^{T}f^{-})}}$
(7)
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}-log\frac{e^{s(\cos(\theta_{i}^{+}))}}{e^{s(\cos(\theta_{i}^{+}))}+e^{s(\cos(\theta_{i}^{-}))}}$
(8)

where $f_{i}=\frac{f_{i}}{\parallel f_{i}\parallel}$,
$f_{i}^{+}=\frac{f_{i}^{+}}{\parallel f_{i}^{+}\parallel}$, and
$f_{i}^{-}=\frac{f_{i}^{-}}{\parallel f_{i}^{-}\parallel}$.

However, the features learned using the above loss function are not guaranteed
to be highly discriminative. It does not take the stringent constraints for
discrimination into account which can produce ambiguity in decision
boundaries. For example, let $\theta_{i}^{+}$ and $\theta_{i}^{-}$ denote the
angle between the positive pair (same tracking identity) and negative pair
(different tracking identity) respectively. The normalized loss function in
Eq. (8) forces $\cos(\theta_{i}^{+})>\cos(\theta_{i}^{-})$ to map similar
feature vectors close to each other.

To further enhance the intra-class compactness and inter-class discrimination,
we introduce the margin term in the cosine metric. To be more specific, the
loss function is modified to force
$\cos(\theta_{i}^{+})-m>\cos(\theta_{i}^{-})$, where the hyperparameter $m$ is
the additive cosine margin. Since $\cos(\theta_{i}^{+})-m$ is less than
$\cos(\theta_{i}^{+})$, the incorporation of the cosine margin penalty in the
loss function makes the constraints more stringent and thereby promotes
discriminative feature learning.

The cosine-margin-triplet loss that enforces the distribution of positive pair
closer on the hypersphere while pushing away negative pair is defined as,

$L_{CMT}=\frac{1}{N}\sum_{i=1}^{N}-log\frac{e^{s(\cos(\theta_{i}^{+})-m)}}{e^{s(\cos(\theta_{i}^{+})-m)}+e^{s(\cos(\theta_{i}^{-}))}}$
(9)

The tracking model is trained by sampling hard positive and hard negative for
each detected object.

Figure 1: Overview of the proposed MOTS R-CNN. The nomenclature conv-x denotes
the depth of the backbone network.

###  4.2 Cosine-margin-contrastive loss

While the CMT loss results in the relative distance, we propose the loss
function that measures the absolute distance, identical to the contrastive
loss, to answer the question “How similar/dissimilar are the feature vectors
of two objects?” The proposed cosine-margin-contrastive loss is defined as,

$L_{CMC}=\frac{1}{N}\sum_{i=1}^{N}(-loge^{\sigma(s(\cos(\theta_{i}^{+})-m))}\\\
-loge^{1-\sigma(s(\cos(\theta_{i}^{-})-m))})$ (10)

where $f_{i}=\frac{f_{i}}{\parallel f_{i}\parallel}$,
$f_{i}^{+}=\frac{f_{i}^{+}}{\parallel f_{i}^{+}\parallel}$,
$f_{i}^{-}=\frac{f_{i}^{-}}{\parallel f_{i}^{-}\parallel}$, $N=|D|$, and
$\sigma(.)$ is the sigmoid activation function. It is worth noting that the
application of the sigmoid function bounds the cosine similarity score to
$[0,1]$ and hence $1-\sigma(.)$ gives cosine distance between pair of feature
vectors. The first term $-loge^{\sigma(s(\cos(\theta_{i}^{+})-m))}$ in Eq.
(10) is responsible for mapping the positive pair closer while the second term
$-loge^{1-\sigma(s(\cos(\theta_{i}^{-})-m))})$ is responsible for pushing the
negative pair far away on the hypersphere.

The loss function in Eq. (10) can be easily reformulated as two-class softmax
loss and is given as,

$L_{CMC}=\frac{1}{N}\sum_{i=1}^{N}(-log\frac{e^{\sigma(s(\cos(\theta_{i}^{+})-m))}}{e^{\sigma(s(\cos(\theta_{i}^{+})-m))}+e^{1-\sigma(s(\cos(\theta_{i}^{+})))}}\\\
-log\frac{e^{1-\sigma(s(\cos(\theta_{i}^{-})-m))}}{e^{1-\sigma(s(\cos(\theta_{i}^{-})-m))}+e^{\sigma(s(\cos(\theta_{i}^{-})))}})$
(11)

Similar to the CMT loss, hard positive and hard negative sampling is used
during training.

##  5 Proposed MOTS R-CNN

We address the MOTS problem building on the strengths of Mask R-CNN that
extends the Faster R-CNN detector [30] with an instance segmentation (mask)
head. We propose MOTS R-CNN which is an extension of Mask R-CNN with the
addition of a tracking head. The architecture of MOTS R-CNN has 1) a bounding
box and classification head for object localization and classification, 2) a
mask head for instance segmentation of objects, and 3) a tracking head for
data association. Given an image, MOTS R-CNN outputs a set of detections with
associated bounding boxes, class labels, and instance segmentation masks
together with learned feature vectors. These learned embeddings are then used
to associate detected objects in the video sequence.

###  5.1 Overview

The proposed MOTS R-CNN architecture is shown in Fig. 1. The first 91
convolutional layers (up to conv-4) in ResNet-101 [10] is used as the
convolutional backbone network. The 3D convolutional layers are applied to the
feature maps obtained from the backbone network. The incorporation of 3D
convolutions efficiently models the spatio-temporal dependency of objects [3].
These spatio-temporal features are then shared by a region proposal network
(generating a set of object proposals) and a Fast R-CNN detection network [8].
RoI-align is performed on spatio-temporal features to extract a portion of the
feature map specified by object proposals. All convolutional layers of conv-5
in ResNet-101 are applied on RoI-aligned features which are shared by box,
classification, and mask head. Similar to the Mask R-CNN, the box head,
classification head, and mask head localizes the objects of interest, assigns
a class label, and generates binary segmentation masks respectively. The
detected objects augmented with their class labels, and segmentation masks are
feed as input to the tracking head for the association.

###  5.2 Tracking head

MOTS R-CNN extends the original implementation of Mask R-CNN through the
incorporation of the tracking head in series with the box, classification, and
mask heads. This cascade arrangement of the tracking head is encouraged to
speed up the network during inference due to the processing of the fewer
number of boxes and to improve accuracy by using more accurate bounding boxes.
The tracking head consists of fully-connected layers with bounding boxes,
class labels, and masks as inputs and an association feature vector for each
detected box as output. It is worth noting that occlusion, noisy detections,
and appearance changes pose several inevitable challenges in multi-object
tracking scenarios. Besides, the scale of various objects also affects the
tracking performance adversely. For example, the size of an occluded
pedestrian is small relative to a non-occluded car. To address the
aforementioned challenges, multi-layer feature aggregation is used in the
proposed work as described in the following.

Multi-layer feature aggregation for scale-invariant tracking. We first extract
the object-specific feature maps from spatio-temporal features using RoI-align
for each detected object. On these RoI-aligned features, all layers of conv-5
are applied. We refer to the feature map corresponding to the final
convolutional layer of 3D convolutions and conv-5 in ResNet-101 as C4 and C5
respectively. The RoI-aligned features are likely to be comprised of both
foreground and background information due to the axis-aligned rectangular
bounding boxes. It is well known from recent studies that foreground
information alone contributes significantly to object tracking. Hence, it is
highly essential to ensure that the tracking performance is largely dependent
on foreground information.

Since the MOTS framework is equipped with instance segmentation masks,
foreground-background discrimination is readily available. In view of the fact
that C4 has high spatial resolution than C5, we pixel-wise multiply RoI-
aligned features from C4 with the corresponding mask to nullify the effect of
background information. Finally, the RoI-align masked features and C5 features
are independently converted to 128-dimensional feature vectors by fully-
connected layers. These individual 128-dimensional feature vectors are then
concatenated to obtain a 256-dimensional embedding vector for each detected
object. These feature vectors are then learned using cosine-margin-triplet
loss.

Data Association. At the test time, the dot product between normalized feature
vectors of detected objects gives the similarity score which is used for data
association. Specifically, we link each detected object in the current frame
$t$ having a detection confidence score greater than a threshold $\beta$ with
detections in the previous frames. The detections from previous $t-\alpha$
frames are considered for tracking. The detections in the current frame are
compared with previously detected objects if and only if (1) corresponding
bounding box centers have Euclidean distance less than $\gamma$ and (2)
associated pairwise similarity score is larger than the threshold $\delta$.
Matching is performed using a greedy algorithm and all high confidence
detections that are not linked to any previously detected objects are assigned
to a new track.

###  5.3 Training MOTS R-CNN

The MOTS R-CNN is trained jointly in an end-to-end manner by defining multi-
task loss on each detected object. This is done through the addition of the
losses associated with the box, classification, mask, and tracking head
together. The multi-task loss is given as,

$L=L_{BH}+L_{LMC}+L_{MH}+L_{CMT}$ (12)

where $L_{BH}$ and $L_{MH}$ denote the losses corresponding to the box head
and mask head respectively as defined in [9]. Different from the Mask R-CNN,
we use LMCL and CMT loss for classification and tracking head respectively.

##  6 Experiments

In this section, we report a detailed experimental analysis to test the
effectiveness of the MOTS-RCNN and thereby the proposed loss functions on the
KITTI MOTS dataset [37]. We carry out ablation studies to signify the
contribution of each component of the MOTS R-CNN. We present the performance
of the proposed algorithm in comparison with the state-of-the-art methods for
the MOTS task. The performance is evaluated in terms of sMOTSA, MOTSA, IDS,
FP, and false negatives (FNs) metrics as described in [37]. While the sMOTSA
and MOTSA measure the overall accuracy of the MOTS network, IDS evaluates the
performance of the tracking head. The FP and FN measure the performance of the
bounding box and classification head.

Table 1: Performance of the MOTS R-CNN on KITTI validation dataset in comparison with Track R-CNN Method | sMOTSA $\uparrow$ | MOTSA $\uparrow$ | IDS $\downarrow$ | FP $\downarrow$ | FN $\downarrow$
---|---|---|---|---|---
Car | Ped | Car | Ped | Car | Ped | Car | Ped | Car | Ped
MOTS R-CNN | $\mathbf{78.2}$ | $\mathbf{49}$ | $\mathbf{90.1}$ | $\mathbf{67.3}$ | $\mathbf{35}$ | $\mathbf{30}$ | $\mathbf{130}$ | $\mathbf{208}$ | $\mathbf{629}$ | 856
Track R-CNN [37] | 76.2 | 46.8 | 87.8 | 65.1 | 93 | 78 | 134 | 267 | 753 | $\mathbf{822}$

###  6.1 Experimental Setup

For the MOTS R-CNN, the ResNet-101 is used as the backbone convolutional
network and it is pre-trained on ImageNet [5], COCO [19], and Mapillary [26]
dataset. Two depthwise separable 3D convolutional layers with filters of size
$3\times 3\times 3$ followed by ReLU activation function are applied on the
top of the backbone network. The data augmentations including random flipping
and gamma correction in the $[-0.2,0.2]$ range are used during training. We
train the MOTS R-CNN using the Adam optimizer [14] for $5$ epochs. The
learning rate of the optimizer is set to $5\times 10^{-5}$. The network is
trained with mini-batches of size $4$ which are formed by stacking adjacent
frames of the same video.

During training, the hyper-parameters of the proposed cosine-margin-triplet
loss including margin and scale $(m-s)$ are set to $0.15-8$ and $0-1$ for cars
and pedestrians respectively. Afterwards, the tracking head for pedestrians is
fine-tuned with mini-batches of size $4$ by setting $m$ and $s$ to $0.2-16$
and $0.25-8$ respectively. We then fine-tune tracking head for cars with mini-
batches of size $8$ and $16$ respectively. We set the hyper-parameters for
data association during the inference as follows: $\alpha$ equal to $10$ and
$6$ for car and pedestrian, detection confidence score $\beta$ is set to
$0.32$ and $0.345$ for car and pedestrian, $\gamma$ and $\delta$ are set to
$150$ and $0.3$ respectively. All experiments are performed on the NVIDIA
Quadro GP100 card with 16 GB of memory.

###  6.2 Main Results

We evaluate the effectiveness of the proposed MOTS R-CNN on the KITTI MOTS
validation dataset. We compare the performance of the MOTS R-CNN with the
existing Track R-CNN [37], MOTSFusion [23], BeyondPixels [33], and CIWT [28]
models. The last two approaches are primarily designed for MOT task and
extended to MOTS problem through the addition of the mask head as described in
[37]. The performance of the tracking-by-detection paradigm heavily relies on
detection results. Hence, we consider reported results with the same detection
and segmentation pipeline as MOTS R-CNN to maintain fairness in the analysis.

Comparison with Track R-CNN. The comparison of the proposed algorithm is given
in Table 1. To be more specific, our work differs from Track R-CNN in the
three aspects 1) multi-layer feature aggregation mechanism, 2) the proposed
CMT loss for the tracking head, and 3) the LMCL for object classification.
While MOTS R-CNN gives $2\%$ sMOTSA improvement in the car class, the
performance of pedestrians rises by $2.2\%$. The most encouraging finding is
that ID switching using the proposed algorithm for both cars and pedestrians
is reduced by $60\%$ in comparison with Track R-CNN which uses triplet loss
for distance metric learning. This reduction in ID switching using MOTS R-CNN
can be attributed to the proposed CMT loss and multi-layer feature aggregation
mechanism. Specifically, it signifies that the proposed CMT loss function
guides the model to converge to a better optimal minimum than the triplet
loss. Furthermore, $2.2\%$ improvement in MOTSA for both cars and pedestrians
is due to the better classification accuracy. This demonstrates the prominence
of large margin cosine loss in reducing the number of false positives.

State-of-the-art comparison. We compare in Table 2 the performance of the MOTS
R-CNN with the existing methods. While the MOTS R-CNN outperforms the
BeyondPixels and CIWT, its performance is comparable with MOTSFusion [23].
Different from MOTS R-CNN which uses appearance features, MOTSFusion exploits
motion information for object tracking. It can be seen that the MOTS R-CNN
keeps the number of IDS at the minimum, thereby attaining the state-of-the-art
tracking performance. Furthermore, MOTSFusion has a large number of parameters
since it requires an additional deep network for the computation of optical
flow. It is worth noting that the reduced number of FNs for cars using the
MOTS R-CNN $(629\;\text{vs.}\;673)$ indicates that the proposed algorithm
tracks more number of ground truth objects, which is highly desirable. This
validates the usefulness of the MOTS R-CNN model in practical applications. On
the contrary, the improvement in sMOTSA and MOTSA for pedestrians using
MOTSFusion is attributed to its ability to recover missing detections.

Table 2: Performance of the MOTS R-CNN on KITTI validation dataset in comparison with existing algorithms Method | sMOTSA $\uparrow$ | MOTSA $\uparrow$ | IDS $\downarrow$ | FP $\downarrow$ | FN $\downarrow$
---|---|---|---|---|---
Car | Ped | Car | Ped | Car | Ped | Car | Ped | Car | Ped
MOTS R-CNN | $\mathbf{78.2}$ | 49 | $\mathbf{90.1}$ | 67.3 | $\mathbf{35}$ | $\mathbf{30}$ | 130 | 208 | 629 | 856
MOTSFusion [23] | $\mathbf{78.2}$ | $\mathbf{50.1}$ | 90.0 | $\mathbf{68.0}$ | 36 | 34 | $\mathbf{94}$ | $\mathbf{181}$ | 673 | 855
BeyondPixels [33] | 76.9 | - | 89.7 | - | 88 | - | 280 | - | $\mathbf{458}$ | -
CIWT [28] | 68.1 | 42.9 | 79.4 | 61.0 | 106 | 42 | 333 | 401 | 1214 | 863
Table 3: Ablation results on the KITTI MOTS validation dataset

Method | sMOTSA $\uparrow$ | MOTSA $\uparrow$ | IDS $\downarrow$
---|---|---|---
Car | Ped | Car | Ped | Car | Ped
Multi-layer | 78.2 | 49 | 90.1 | 67.3 | $\mathbf{35}$ | $\mathbf{30}$
Single-layer | 77.8 | 47.7 | 89.8 | 66.6 | 39 | 50
(a) Feature aggregation

Method | sMOTSA $\uparrow$ | MOTSA $\uparrow$ | IDS $\downarrow$
---|---|---|---
Car | Ped | Car | Ped | Car | Ped
Greedy | 78.2 | 49 | 90.1 | 67.3 | $\mathbf{35}$ | $\mathbf{30}$
Hungarian | 78 | 48.6 | 90 | 66.8 | 47 | 46
(b) Data association mechanisms

Loss function | sMOTSA $\uparrow$ | FP $\downarrow$ | FN $\downarrow$
---|---|---|---
Car | Ped | Car | Ped | Car | Ped
LMCL | 78.2 | 49 | 130 | $\mathbf{208}$ | $\mathbf{629}$ | 856
Softmax | 76.9 | 48.2 | 125 | 232 | 835 | 850
(c) Object classification loss functions

Method | sMOTSA $\uparrow$ | MOTSA $\uparrow$ | IDS $\downarrow$
---|---|---|---
Car | Ped | Car | Ped | Car | Ped
With mask | 78.2 | 49 | 90.1 | 67.3 | $\mathbf{35}$ | $\mathbf{30}$
Without mask | 77.7 | 48.5 | 89.8 | 66.8 | 44 | 38
(d) Instance segmentation mask for feature extraction

Table 4: Ablation results for distance metric learning Loss function | sMOTSA $\uparrow$ | MOTSA $\uparrow$ | IDS $\downarrow$
---|---|---|---
Car | Ped | Car | Ped | Car | Ped
CMT | $\mathbf{78.2}$ | $\mathbf{49}$ | $\mathbf{90.1}$ | $\mathbf{67.3}$ | $\mathbf{35}$ | $\mathbf{30}$
CMC | 77.8 | 48.5 | 89.8 | 67.2 | 46 | 34

###  6.3 Ablation Studies

We perform several ablations to signify the importance of each module of MOTS
R-CNN.

Multi-layer feature aggregation. We report in Table 3(a) the effectiveness of
the multi-layer feature aggregation over the single-layer feature extraction.
More specifically, the features from the final convolutional layer of the
ResNet-101 are converted to $128$-dimensional vectors by the fully connected
layer and used for data association. The multi-layer feature aggregation
significantly reduces the number of IDS. We observed empirically that the use
of multi-layer features makes the network robust against occlusions.

Data association mechanism. The performance of MOTS R-CNN using the popular
Hungarian and greedy algorithms for data association is detailed in Table
3(b). The reduction in the number of IDS using the greedy algorithm indicates
that the learned features using the proposed method are strongly
discriminative. Also, the greedy algorithm is extremely lightweight in
comparison to the Hungarian algorithm which speeds up the tracker at test
time.

Object classification loss function. In Table 3(c), we demonstrate the
significance of LMCL over the traditional softmax loss for the task of object
classification. While the incorporation of LMCL in MOTS R-CNN reduces the
number of FNs from $835$ to $629$ for cars, it reduces the number of FPs from
$232$ to $208$ in the case of pedestrians.

Instance segmentation mask for feature extraction. Table 3(d) vividly
illustrates the usefulness of the segmentation mask for extracting features
from the foreground of objects to improve the tracking performance.

Distance metric learning. We compare in Table 4 the effect of the proposed CMT
and CMC loss functions on feature learning for multi-object tracking. From an
empirical analysis, the margin $m$ and scale factor $s$ are set to $0.35$ and
$8$ for the CMC loss. It can be noticed that the performance using the CMT
loss is better than the CMC loss. Specifically, the reduction in the number of
IDS using the CMT loss illustrates its power to encourage discriminating
feature learning. This observation is consistent with the fact that the
traditional triplet loss which results in relative distance between pair of
feature vectors is seen as an improvement over the contrastive loss.

Qualitative analysis. We display in Fig. 2 the qualitative tracking results
for visual analysis. It can be noticed that our model is robust against merge
and split problem. Furthermore, the ability of the model to handle occlusions
and to perform consistent tracking is also witnessed.

Figure 2: Qualitative analysis of tracking results on the KITTI MOTS
validation dataset (a) robustness against merge and split problem and (b)+(c)
robustness to occlusions

##  7 Conclusions

In this paper, we proposed cosine-margin-triplet and cosine-margin-contrastive
loss functions for deep metric learning. While feature normalization makes
features obtained via CNNs angular separable, the incorporation of margin term
further boosts its discriminative ability. We then propose the MOTS R-CNN
model to demonstrate the significance of the proposed loss functions for
multi-object tracking. We show the robustness of our model to occlusions and
scale variation which can be attributed to the multi-layer feature aggregation
mechanism. The significant reduction in the number of IDS validates the better
convergence of the proposed CMT loss. We believe that further improvement in
the performance is possible by using a backbone convolutional network
involving a feature pyramid network.

##

Acknowledgement: This work is funded by SERB NPDF grant (ref. no.
PDF/2019/003459), government of India.

## References

  * [1] Anton Andriyenko, Konrad Schindler, and Stefan Roth.  Discrete-continuous optimization for multi-target tracking.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1926–1933. IEEE, 2012.
  * [2] Guillem Brasó and Laura Leal-Taixé.  Learning a neural solver for multiple object tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6247–6257, 2020.
  * [3] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [4] Wongun Choi.  Near-online multi-target tracking with aggregated local flow descriptor.  In Proceedings of the IEEE international conference on computer vision, pages 3029–3037, 2015.
  * [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [6] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2019.
  * [7] Andreas Ess, Bastian Leibe, Konrad Schindler, and Luc Van Gool.  A mobile vision system for robust multi-person tracking.  In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2008.
  * [8] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [9] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [11] Roberto Henschel, Laura Leal-Taixé, Daniel Cremers, and Bodo Rosenhahn.  Fusion of head and full-body detectors for multi-object tracking.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1428–1437, 2018.
  * [12] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv preprint arXiv:1703.07737, 2017.
  * [13] Weiming Hu, Wei Li, Xiaoqin Zhang, and Stephen Maybank.  Single and multiple object tracking using a multi-feature joint sparse representation.  IEEE transactions on pattern analysis and machine intelligence, 37(4):816–833, 2014.
  * [14] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [15] Wei Li, Xiatian Zhu, and Shaogang Gong.  Harmonious attention network for person re-identification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2285–2294, 2018.
  * [16] Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, and Alexander Hauptmann.  The garden of forking paths: Towards multi-future trajectory prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10508–10518, 2020.
  * [17] Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander G Hauptmann, and Li Fei-Fei.  Peeking into the future: Predicting future person activities and locations in videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5725–5734, 2019.
  * [18] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [19] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [20] Wang G. Zuo W. Feng X. Lin, L. and L. Zhang.  Cross-domain visual matching via generalized similarity measure and feature learning.  IEEE transactions on pattern analysis and machine intelligence, 39(6):1089–1102, 2016.
  * [21] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212–220, 2017.
  * [22] Chen Long, Ai Haizhou, Zhuang Zijie, and Shang Chong.  Real-time multiple people tracking with deeply learned candidate selection and person re-identification.  In ICME, 2018.
  * [23] Jonathon Luiten, Tobias Fischer, and Bastian Leibe.  Track to reconstruct and reconstruct to track.  IEEE Robotics and Automation Letters, 5(2):1803–1810, 2020.
  * [24] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon.  It is not the journey but the destination: Endpoint conditioned trajectory prediction.  arXiv preprint arXiv:2004.02025, 2020.
  * [25] Yair Movshovitz-Attias, Alexander Toshev, Thomas K Leung, Sergey Ioffe, and Saurabh Singh.  No fuss distance metric learning using proxies.  In Proceedings of the IEEE International Conference on Computer Vision, pages 360–368, 2017.
  * [26] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder.  The mapillary vistas dataset for semantic understanding of street scenes.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4990–4999, 2017.
  * [27] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese.  Deep metric learning via lifted structured feature embedding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4004–4012, 2016.
  * [28] Aljoša Osep, Wolfgang Mehner, Markus Mathias, and Bastian Leibe.  Combined image-and world-space tracking in traffic scenes.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1988–1995. IEEE, 2017.
  * [29] Lorenzo Porzi, Markus Hofinger, Idoia Ruiz, Joan Serrat, Samuel Rota Bulo, and Peter Kontschieder.  Learning multi-object tracking and segmentation from automatic annotations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6846–6855, 2020.
  * [30] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2016.
  * [31] Amir Sadeghian, Alexandre Alahi, and Silvio Savarese.  Tracking the untrackable: Learning to track multiple cues with long-term dependencies.  In Proceedings of the IEEE International Conference on Computer Vision, pages 300–311, 2017.
  * [32] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [33] Sarthak Sharma, Junaid Ahmed Ansari, J Krishna Murthy, and K Madhava Krishna.  Beyond pixels: Leveraging geometry and shape cues for online multi-object tracking.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3508–3515. IEEE, 2018.
  * [34] Kihyuk Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In Advances in neural information processing systems, pages 1857–1865, 2016.
  * [35] Valtteri Takala and Matti Pietikainen.  Multi-object tracking using color, texture and motion.  In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–7. IEEE, 2007.
  * [36] Siyu Tang, Mykhaylo Andriluka, Bjoern Andres, and Bernt Schiele.  Multiple people tracking by lifted multicut and person re-identification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3539–3548, 2017.
  * [37] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe.  Mots: Multi-object tracking and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7942–7951, 2019.
  * [38] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.  Cosface: Large margin cosine loss for deep face recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5265–5274, 2018.
  * [39] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr.  Fast online object tracking and segmentation: A unifying approach.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1328–1338, 2019.
  * [40] Zhongdao Wang, Liang Zheng, Yixuan Liu, and Shengjin Wang.  Towards real-time multi-object tracking.  arXiv preprint arXiv:1909.12605, 2019.
  * [41] Xinshuo Weng, Yongxin Wang, Yunze Man, and Kris M Kitani.  Gnn3dmot: Graph neural network for 3d multi-object tracking with 2d-3d multi-feature learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6499–6508, 2020.
  * [42] Ancong Wu, Wei-Shi Zheng, Xiaowei Guo, and Jian-Huang Lai.  Distilled person re-identification: Towards a more scalable system.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1187–1196, 2019.
  * [43] Zhenbo Xu, Wei Zhang, Xiao Tan, Wei Yang, Huan Huang, Shilei Wen, Errui Ding, and Liusheng Huang.  Segment as points for efficient online multi-object tracking and segmentation.  In European Conference on Computer Vision, pages 264–281. Springer, 2020.
  * [44] Fan Yang, Xin Chang, Chenyu Dang, Ziqiang Zheng, Sakriani Sakti, Satoshi Nakamura, and Yang Wu.  Remots: Self-supervised refining multi-object tracking and segmentation.  arXiv e-prints, pages arXiv–2007, 2020.
  * [45] Xun Yang, Peicheng Zhou, and Meng Wang.  Person reidentification via structural deep metric learning.  IEEE Transactions on Neural Networks and Learning Systems, 30(10):2987–2998, 2018.
  * [46] Junbo Yin, Wenguan Wang, Qinghao Meng, Ruigang Yang, and Jianbing Shen.  A unified object motion and affinity model for online multi-object tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6768–6777, 2020.
  * [47] Fengwei Yu, Wenbo Li, Quanquan Li, Yu Liu, Xiaohua Shi, and Junjie Yan.  Poi: Multiple object tracking with high performance detection and appearance feature.  In European Conference on Computer Vision, pages 36–42. Springer, 2016.
  * [48] Ji Zhu, Hua Yang, Nian Liu, Minyoung Kim, Wenjun Zhang, and Ming-Hsuan Yang.  Online multi-object tracking with dual matching attention networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 366–382, 2018.
  * [49] Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu.  Distractor-aware siamese networks for visual object tracking.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 101–117, 2018.