# Discriminative Feature Representaion with Spatio-temporal Cues for Vehicle
Re-identification

Jingzheng Tu, Cailian Chen, , Xiaolin Huang, , Jianping He, , Xinping Guan J.
Tu, C. Chen, X. Huang, J. He and X. Guan are with the Department of
Automation, Shanghai Jiao Tong University, Shanghai 200240, China and also
with the Key Laboratory of System Control and Information Processing, Ministry
of Education of China, Shanghai Jiao Tong University, Shanghai 200240, China
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>xpguan@sjtu.edu.cn).

###### Abstract

Vehicle re-identification (re-ID) aims to discover and match the target
vehicles from a gallery image set taken by different cameras on a wide range
of road networks. It is crucial for lots of applications such as security
surveillance and traffic management. The remarkably similar appearances of
distinct vehicles and the significant changes of viewpoints and illumination
conditions take grand challenges to vehicle re-ID. Conventional solutions
focus on designing global visual appearances without sufficient consideration
of vehicles’ spatio-tamporal relationships in different images. In this paper,
we propose a novel discriminative feature representation with spatio-temporal
clues (DFR-ST) for vehicle re-ID. It is capable of building robust features in
the embedding space by involving appearance and spatio-temporal information.
Based on this multi-modal information, the proposed DFR-ST constructs an
appearance model for a multi-grained visual representation by a two-stream
architecture and a spatio-temporal metric to provide complementary
information. Experimental results on two public datasets demonstrate DFR-ST
outperforms the state-of-the-art methods, which validate the effectiveness of
the proposed method.

###### Index Terms:

Vehicle re-identification, computer vision, deep learning, attention
mechanism, video surveillance.

##  I Introduction

With increasing demand for public security and the rapid growth of vehicles,
vehicle re-identification (re-ID) has become one of the most pivotal
technologies for intelligent urban surveillance. It also has a wide range of
potential applications including multi-target multi-camera tracking, traffic
flow modeling [1, 2, 3]. The main task of vehicle re-ID is to locate vehicles
accurately and identify the same vehicle over multiple cameras with particular
perspectives.

A similar topic is person re-identification (re-ID), but it is totally
different from vehicle re-ID in terms of the facing challenges. Compared to
person re-ID, vehicle re-ID suffers from much smaller inter-class variations
and larger intra-class variations, as shown in Fig. 1. In the top rectangle,
each column represents a near-duplicate pair of vehicles with different
identities. Due to diversified viewpoints, illuminations, and orientations of
vehicle images, they only differ in slight as highlighted. Concretely,
vehicles with distinct labels can be of the same model and the same color. Two
different vehicles of the same model and color only differ in minor details,
e.g. brands, individual decorations and scratches. Additionally, the same
vehicle’s images with various orientations, such as front, side, and rear,
only share little visual overlap. As human sizes are smaller than vehicles,
the images of one specific person with diverse viewpoints retain more
appearance similarities.

Figure 1:  (1) Different vehicles share extremely analogous appearances. (2)
Large intra-class variations. (3) In person re-ID, the same person’s images
from different aspects share more visual overlap than those in vehicle re-ID.

Considering the sophisticated relationship between intra-class and inter-class
discrepancies, conventional methods typically focus on hand-craft features [4,
5, 6], including geometric attributes, color histograms, and textures. Their
major disadvantage lies to the insensitivity to background clutter and large
variations of light conditions and viewpoints, thus leading to restricted
applications in practical situations. Recently, deep features have dominated
many computer vision tasks, such as object detection [7, 8, 9, 10], semantic
segmentation [11] and action recognition [12, 13, 14]. Hence, researchers have
embraced deep features into vehicle re-ID to construct a more efficient
feature representation based on two strategies: 1) visual representation and
2) multi-modal representation.

The first strategy exploits the visual representation of vehicles. Refs. [15,
16, 17] devote to establish a global description of vehicle images. However,
the global features have limited performance when dealing with inevitable
visual ambiguities among different vehicles and dramatic changes of
uncontrolled variations of the same vehicle. This inspired several works [18,
19, 20] to seek helpful visual clues for distinguishing subtle differences
between vehicle images. However, these methods mainly focus on informative
region localization instead of how local regions are assigned the importances
to distinct degrees.

To solve this problem, we develop a discriminative feature representation
method to explore more fine-grained features by introducing an appearance
module with two streams, i.e., the coarse-grained and the fine-grained feature
streams respectively, to describe visual information of different
granularities. The coarse-grained feature stream extracts deep features from
the global network, presenting a macroscopic impression of images. Besides,
the fine-grained feature stream pulls the samples of the same class up
together while pushing those of different classes away in the feature
embedding.

The second strategy aims to discover and utilize multi-modal information to
improve the performance of vehicle re-ID algorithms. Because visual appearance
is not always reliable, especially in unconstrained environments with
excessively dynamical changes, including license plates [15] and vehicle
models [15, 21, 22, 23]. As aforementioned, license plate recognition is
vulnerable and involves privacy problems. Moreover, vehicle models require
manually annotated labels, which are laborious and uneconomical. Contrastly,
the spatio-temporal information of vehicle re-ID problem is usually available
due to the universalness of video monitoring systems. Refs. [17, 24, 18, 25,
26] employ the spatio-temporal information as the refinement of appearance
features. Ref. [17] defines a spatio-temporal similarity between image pairs
based on the approximate statistics of datasets. Ref. [24] uses Chain of
Markov Random Field to model visual spatio-temporal paths. However, the missed
detections of vehicles on passed spatio-temporal paths would degrade the
vehicle re-ID algorithm’s overall performance. Moreover, ref. [25] constructs
spatio-temporal constraints and refines the matching problem by a transfer
time matrix. However, the acquisition of timestamps in [25] requires the
preprocess of a multi-camera multi-target tracking task, which injures the
method’s adaptability and extendibility. Additionally, a group of works for
person re-ID also explore spatio-temporal information, which can be mainly
divided into two manners. One is to excavate implicit spatial-temporal
information in videos [27, 28]. The other is to use explicit spatio-temporal
information as physical constraints to reduce the complexity of the matching
algorithm [29, 30]. The spatio-temporal model for person re-ID can not be
straightforwardly used to vehicle re-ID due to severe performance degradation.
In particular, since vehicles move much faster than people, the assumption of
constrained location prediction [30] is not rational any more.

To solve the above problem, we propose a spatio-temporal module to obtain a
more robust model for identifying vehicles. Specifically, the distances
between camera pairs and the time intervals are modeled as a distribution
rather than a transfer matrix [25], spatio-temporal constraints [26] or visual
spatio-temporal paths [24]. By modeling the camera locations’ distance and the
discrepancy of timestamps as random variables, we formulate the spatio-
temporal relationship in a simple yet effective manner quantitively. By adding
the spatio-temporal module, we observe an evident performance improvement.

In summary, we propose a novel discriminative representation with spatio-
temporal information (DFR-ST) to establish a robust feature embedding with
multi-modal cues for vehicle re-ID. The main contributions of DFR-ST are
three-fold:

  * •

The proposed DFR-ST constructs the appearance representation by the two-stream
architecture to extract the coarse-grained and the fine-grained features.
Besides, the combination of an attention mechanism and division operations
drives the fine-grained visual representation to focus on more salient and
informative regions.

  * •

The spatio-temporal module is proposed to form a complementary representation
with the visual appearance by taking multi-modal cues into sufficient
considerations.

  * •

Extensive experiments on two large-scale benchmarks indicate the effectiveness
and robustness of the proposed DFR-ST and it achieves the state-of-the-art
performance.

The rest of this article is organized as follows. Section II refers to the
related works of vehicle re-ID and person re-ID. Section III introduces the
proposed DFR-ST method while Section IV presents the experimental results and
analyses. Section V concludes this article.

##  II Related Works

###  II-A Vehicle Re-Identification

The earliest vision-based works [4, 5, 6] design hand-craft features
scrupulously to identify vehicles. However, hand-craft features have limited
capability in practice, because heavy occlusions and drastic light changes in
unconstrained situations would destroy the performance when modeling
discriminative features. Since deep features exhibit powerful strength on
multiple vision tasks including image classification [31, 32, 33], object
detection [7, 8, 9, 10] and action recognition [12, 13, 14], researchers
exploit deep features for vehicle re-ID mainly through two approaches as
follows.

The first approach is constructing visual representation to tackle the issue
of identifying the same vehicle. Wang _et al._ [18] aggregate four local
features with different directions to describe an orientation-invariant
feature. However, this work requires a manual classification of the key points
on vehicles. He _et al._ [19] utilize an object detection method to extract
partial features for near-duplicate vehicles. Chen _et al._ [20] propose a
two-branch network with partitions on the height and the width channels to
maximally distinct local features. The experimental results illustrate that
the channel-wise partition improves most. Our work shares a similar idea with
PRN [20], which fuses the global and the local features to establish an
overall embedding. However, observing that the above methods focus on the
local region localization instead of assigning different degrees of importance
on different informative regions, we design a more discriminative
representation by an attention network and the collaboration of three-
dimension divisions. Moreover, experiments demonstrate that our algorithm
performs better than [20] on public datasets.

The second approach aims to improve vehicle re-ID performance with multi-modal
information. The most intuitive idea is the introduction of license plates
[15]. However, this unique information relies on the performance of character
recognition tasks, which inevitably suffer from low-resolution videos,
frequent occlusions and blurred motion. Besides, the model type information of
vehicles is also be explored to benefit matching vehicles. Guo _et al._ [21]
propose a structured feature embedding by defining a vehicle model
classification loss and two ranking losses with distinct granularities.
Although the fine-grained ranking loss has a huge contribution to the final
performance, acquiring vehicle models with manually annotation is uneconomical
and laborious.

Contrastly, spatio-temporal information for vehicle re-identification is
usually available because of the rapid popularization of video monitoring
systems. Hence, it is more efficient to incorporate spatio-temporal cues in
the algorithms to complement the visual appearance information with lower
costs. Wang _et al._ [18] propose the orientation-invariant visual
representation to describe the macroscopic embedding of vehicles with
constraints on spatio-temporal relationships. Shen _et al._ [24] establish
candidate paths using Chain of Markov Random Field and they employ a siamese
architecture with long short term memory (LSTM) network units to model the
visual appearance and spatio-temporal information. Lv _et al._ [25] adopt the
combination of three different features learned by various losses to identify
vehicles. The spatio-temporal constraints is realized by a transfer time
matrix, which refine the search space and reduce the computing complexity.
However, we notice that the existing models of spatio-temporal information are
relatively qualitative and intuitive, which is lack of a more precise
mathematical description. Therefore, we propose a spatio-temporal module to
construct multi-modal representation by modeling camera locations and time
intervals as random variables. Different from [18], which only focuses on
transition time intervals but neglects spatial information, we consider both
temporal and spatial clues simultaneously by a quantitative formulation.
Experiments demonstrate the effectiveness of the proposed spatio-temporal
module.

###  II-B Person Re-Identification

Most existing vision-based approaches could be classified into two categories.
The first category focuses on improving feature representation against pose
variants, clutter backgrounds, and distinct camera viewpoints. The most common
strategy is to train the deep network on multiple local regions and combine
all local branches with the global one [35, 36, 37]. The second category
considers metric learning [38, 39, 40]. Besides, some works utilize multi-
modal information including RGB-D data [41], pose estimation [42] and
segmentation annotations [43], because the collaboration of multi-modal data
can obtain further performance boosting. However, the cost of data acquisition
is an unnegligible problem. As for vehicle re-ID, camera locations and
timestamps of videos are easy to obtain because of surveillance systems’
prevalence. Hence, taking spatio-temporal information into considerations is a
reasonable proposal and can promote the performance of vehicle re-ID methods
without much higher costs. Note that our proposed spatio-temporal module does
not need camera calibration information, which is usually unavailable and
contains additional computing costs and noises.

Figure 2: An overview of our DFR-ST approach. The proposed method contains the
appearance module and the spatio-temporal module—the former aims to establish
a discriminative feature embedding through a two-stream structure by involving
multi-grained features, and the latter uses camera locations and timestamps as
the spatio-temporal cues to construct further refinement.

##  III Methodology

In this section, we design an appearance module and a spatio-temporal module,
aiming to empower the capability of DFR-ST for discriminative and robust
feature representation. Detailed descriptions of each module in the proposed
DFR-ST method is demonstrated in the following sub sections.

###  III-A System Architecture

Fig. 2 illustrates the overall framework of the proposed DFR-ST. The
appearance module is responsible for modeling visual features, in which input
images are fed into a backbone network followed by two streams: _Coarse-
Grained Feature Stream_ and _Fine-Grained Feature Stream_ , as shown in Fig.
3. Meanwhile, the spatio-temporal module establishes the spatial and temporal
distances to provide extra information for identifying the same vehicle.
Finally, the combination of visual and spatio-temporal representation measures
the similarity of vehicle images, conducting the ranking list of gallery
images.

The appearance module is composed of a coarse-grained feature stream and a
fine-grained feature stream. The coarse-grained feature stream extracts the
general feature representation $\mathbf{x}_{c}$ to deal with the complicated
relationship of the inter-class and intra-class variation. This stream
attempts to enlarge the distances of the samples with distinct identities in
the embedding space. However, only general features could fail to process the
detailed discrepancies of input images, especially dealing with the images
with only slight differences such as private decorations, irregular scratches,
and brands. Hence, $\mathbf{x}_{c}$ is replenished with $\mathbf{x}_{f}$ from
the fine-grained feature stream, which can take account of local regions and
salient parts.

Moreover, we propose a spatio-temporal module to exploit extra minutiae,
considering the images with intricate changes in unconstrained environments
degrade the visual appearance’s effectiveness in practical scenarios, In
particular, we apply the camera location and the timestamps information as the
spatio-temporal cues, which are usually easily acquired due to the prevalence
of the security video systems. Thus, the collaboration of the visual
appearance representation and the spatio-temporal clues enhances the final
vehicle re-ID performance.

Figure 3: The schematic diagram of the appearance module in DFR-ST. The
shallow representation extracted from the backbone is transmitted to the
coarse-grained feature stream for constructing a macroscopic feature embedding
and the fine-grained feature stream for a microscopic representation
simultaneously. The fine-grained feature stream contains an AttentionNet and
four branches including division operations of three dimensions to extract
high-quality part-level representations. Afterward, the aggregation of two-
stream features constructs a discriminative feature representation for
identifying the same vehicle.

###  III-B Appearance Module

####  III-B1 Coarse-Grained Feature Stream

The coarse-grained feature stream extracts a macroscopic description of input
images. Features obtained from the backbone are delivered to the coarse-
grained feature stream for further processing. Two residual blocks are placed
successively as the Global Network, followed by an average pooling operation
and a $1\times 1$ convolution operation. To promote the overall performance,
we adopt the BNNeck strategy [44] to alleviate the inconsistency of the
triplet loss and the ID loss in the embedding. Moreover, we set the stride of
down-sampling operations in the last convolutional layer to 1 to maintain more
deep information.

####  III-B2 Fine-Grained Feature Stream

Opposed to the coarse-grained stream, the fine-grained feature stream captures
the local regions’ microcosmic features to distinct challenging near-duplicate
vehicles. The AttentionNet receives the backbone network features to construct
an attentive feature representation, followed by four branches for further
processing. The first branch considers assigning reasonable weights to
different local regions. And the rest three branches conduct the divisions of
features along three dimensions, i.e., the height-wise, the width-wise, and
the channel-wise operations. The division strategy is inspired by the height
slicing strategy in the person re-ID approach [45]. But unlike the vertical
partition of a human body, a single-vehicle has semantic partitions along both
vertical and horizontal dimensions. Specifically, the vehicle is composed of
the car roof, the window, the bumper, the license plate, the chassis in the
vertical dimension, and the rearview mirror, the door, the main body in the
horizontal dimension. Hence, we design the channel, the height, and the width
division branches concurrently because different channels carry independent
semantic information.

The architecture of AttentionNet is shown in Fig. 4. The feature
$\mathbf{X}_{0}\in\mathbb{R}^{C\times H\times W}$ is firstly processed with
the channel-domain attention to learn a channel feature map
$\mathbf{g}_{c}\in\mathbb{R}^{C}$, then followed by the spatial-domain
attention with the learned attention map $\mathbf{g}_{s}\in\mathbb{R}^{1\times
H\times W}.$ $C,H,W$ denote the channel, height, and width dimensions of
$\mathbf{X}_{0}$. The above procedure is:

$\displaystyle{}\mathbf{X}_{1}(i,j,k)$
$\displaystyle=\mathbf{g}_{c}(i)\mathbf{X}_{0}(i,j,k),\forall i,$
$\displaystyle\mathbf{X}_{2}(i,j,k)$
$\displaystyle=\mathbf{g}_{s}(j,k)\mathbf{X}_{1}(i,j,k),\forall j,k.$

In particular, $\mathbf{X}_{1}$ and $\mathbf{X}_{2}$ denote the output
features after channel and spatial attention operations successively. $i,j,k$
are indexes of the channel, height, and width dimensions.

Fig. 5 illustrates the designed structure of channel attention and spatial
attention. Define the transformation $\mathcal{T}_{c}$:
$\mathbf{X}_{0}\rightarrow\mathbf{g}_{c}$. The input feature $\mathbf{X}_{0}$
is first transmitted to the squeeze operation constituted of a global average
pooling and a maximum pooling. The parallel pooling operations can aggregate
feature maps along the spatial dimension, symbolizing the overall distribution
of channel-domain responses. Second, a multi-layer perceptron machine with one
hidden layer processes the element-wise summation of two pooling operations.
Finally, a sigmoid function is applied to grasp channel-domain dependencies.
The above procedure is:

$\displaystyle{}\mathbf{g}_{c}$
$\displaystyle=\mathcal{T}_{c}(\mathbf{X}_{0})=\sigma\\{\text{MLP}[\text{P}_{\text{avg}}(\mathbf{X}_{0})]+\text{MLP}[\text{P}_{\text{max}}(\mathbf{X}_{0})]\\}$
$\displaystyle=\sigma\\{\mathbf{W}_{2}\delta[\mathbf{W}_{1}(\mathbf{x}^{\text{c}}_{\text{avg}})]+\mathbf{W}_{2}\delta[\mathbf{W}_{1}(\mathbf{x}^{\text{c}}_{\text{max}})]\\},$

where

$\displaystyle{}\mathbf{x}^{\text{c}}_{\text{avg}}(i)$
$\displaystyle=\frac{1}{W\times
H}\sum^{W}_{j=1}\sum^{H}_{k=1}\mathbf{X}_{0}(i,j,k),$
$\displaystyle\mathbf{x}^{\text{c}}_{\text{max}}(i)$
$\displaystyle=\max_{j,k}\mathbf{X}_{0}(i,j,k).$

In particular, $\sigma$ is a sigmoid function and $\delta$ represents a ReLu
function. MLP denotes a multi-layer perception machine with one hidden layer.
$\text{P}_{\text{avg}}$ and $\text{P}_{\text{max}}$ are average pooling and
maximum pooling operators. $\mathbf{W}_{1}\in\mathbb{R}^{\frac{C}{k}\times C}$
and $\mathbf{W}_{2}\in\mathbb{R}^{C\times\frac{C}{k}}$ refer to the weights of
two convolution layers in MLP, where the reduction ratio $k$ is 16 in the
experiments. $\mathbf{x}^{\text{c}}_{\text{avg}}$ and
$\mathbf{x}^{\text{c}}_{\text{max}}$ are the average and maximum pooling
results of $\mathbf{X}_{0}$.

Spatial attention concentrates on the location information of salient parts.
Define the transformation
$\mathcal{T}_{s}:\mathbf{X}_{1}\rightarrow\mathbf{g}_{s}$. The squeeze
operation processes $\mathbf{X}_{1}$ with average pooling and maximum pooling
operations across the channel dimension. Moreover, the concatenation of pooled
feature maps is passed through a single convolution layer, followed by a
sigmoid function. Mathematically,

$\mathbf{g}_{s}=\mathcal{T}_{s}(\mathbf{X}_{1})=\sigma\\{\mathbf{W}([\mathbf{x}^{\text{s}}_{\text{max}};\mathbf{x}^{\text{s}}_{\text{avg}}])\\},$

where

$\displaystyle\mathbf{x}^{\text{s}}_{\text{max}}(j,k)$
$\displaystyle=\max_{i}\mathbf{X}_{1}(i,j,k),\forall j,k,$
$\displaystyle\mathbf{x}^{\text{s}}_{\text{avg}}(j,k)$
$\displaystyle=\frac{1}{C}\sum^{C}_{i=1}\mathbf{X_{1}}(i,j,k),\forall j,k.$

Specifically, $\sigma$ denotes the sigmoid function, and $\mathbf{W}$ is the
weight of the convolution layer. $\mathbf{x}^{\text{s}}_{\text{max}}$ and
$\mathbf{x}^{\text{s}}_{\text{avg}}$ represent the output feature maps of
maximum pooling and average pooling operations across the channel axis.

####  III-B3 Object Function

The total loss is the weighted summation of a cross-entropy loss and a triplet
loss. The loss of a batch with $N$ images is written as:

$L_{\text{all}}=L_{\text{ce}}+\lambda L_{\text{tri}},$ (1)

where $L_{\text{ce}}$ is the cross-entropy loss and $L_{\text{tri}}$ is the
triplet loss with the batch hard sampling strategy. $\lambda$ is a weitht
hyperparameter. The label smoothing strategy [46] is employed with the cross-
entropy loss to alleviate the overfitting problem. Hence, the cross-entropy
loss $L_{\text{ce}}$ can be delineated as:

$L_{\text{ce}}=-\frac{1}{N}\sum_{i=0}^{N-1}\sum_{k=0}^{K-1}p_{i,k}\log
q_{i,k}$ (2)

where

$\displaystyle p_{i,k}=\left\\{\begin{aligned} &1-\varepsilon,&\text{if}\
y_{i}=k\\\ &\frac{\varepsilon}{K-1},&\text{if}\ y_{i}\neq
k\end{aligned}\right.,\quad
q_{i,k}=\frac{\exp[\Phi(\mathbf{I}_{i})]}{\sum_{k=0}^{K-1}\exp[\Phi(\mathbf{I}_{k})]}.$

In particular, $i\in\\{0,\cdots,N-1\\}$ is the index of images in the batch
and $k\in\\{0,\cdots,K-1\\}$ is the index of $K$ classes. $p_{i,k}$ denotes
the distribution after label smoothing. $y_{i}$ is the ground truth label of
the $i$th image $\mathbf{I}_{i}$. The hyperparameter $\varepsilon\in[0,1]$ is
a weight factor. $q_{i,k}$ is the network prediction probability of
$\mathbf{I}_{i}$ to the $k$th class and $\Phi(\cdot)$ means the transformation
of the appearance module. Furthermore, the triplet loss $L_{\text{tri}}$ is:

$L_{\text{tri}}=\sum_{i}^{N}[||\mathbf{x}_{i}^{a}-\mathbf{x}_{i}^{p}||_{2}^{2}-||\mathbf{x}_{i}^{a}-\mathbf{x}_{i}^{n}||_{2}^{2}+m]_{+},$
(3)

where $\mathbf{x}_{i}^{a}$, $\mathbf{x}_{i}^{p}$, $\mathbf{x}_{i}^{n}$ denote
features of anchors, positive and negative samples respectively. $[\cdot]_{+}$
represents the hinge function, and $m$ controls the margin of distances
between the positive and the negative samples to the anchors.

Figure 4: The architecture of AttentionNet. The sequential placement of the
channel and spatial attention are added in the network’s last residual block
to further learn an attentive feature representation from local regions
adaptively. Figure 5: The block diagrams of the channel and spatial attention
operations. $\oplus$ denotes the element-wise summation. $\mathbf{g}_{c}$,
$\mathbf{g}_{s}$ denote the learned channel and spatial attention maps
respectively.

####  III-B4 Backbone

The backbone network establishes a shallow representation of visual
appearance. In the proposed DFR-ST, the backbone network adopts the first
three blocks of ResNet-50 network for its flexible architecture and superior
performance. As shown in Fig. 3, we duplicate subsequent convolutional layers
after the first three residual blocks to split ResNet-50 network into two
feature branches. Different convolutional network architectures designed for
deep learning also can be adjusted as the backbone correspondingly.

###  III-C Spatio-temporal Module

In real-world applications, the appearance model is not sufficient to
construct a discriminative representation in the presence of noises from
intricate brackgrounds and severe occlusions. Therefore, camera location and
timestamp information, available in urban surveillance and intelligent
transportation systems, can provide extra content on vehicle attributes.

The fundamental assumption is that two images with smaller spatial or temporal
distances have higher possibilities to be the same identity based on the
observation in [15], and vice versa. According to this assumption, we propose
a novel spatio-temporal measurement, in which the spatial similarity $D_{s}$
and the temporal similarity $D_{t}$ between a query image $i$ from camera
$c_{i}$ and a gallery image $j$ from camera $c_{j}$ are as follows:

$\displaystyle D_{s}$
$\displaystyle=\frac{1}{1+\exp(\alpha_{1}[p(\delta|\mu_{\delta},\sigma_{\delta})-\alpha_{2}])},$
(4) $\displaystyle D_{t}$
$\displaystyle=\frac{1}{1+\exp(\beta_{1}[p(\tau|\mu_{\tau},\sigma_{\tau})-\beta_{2}])},$
(5)

where $\delta$ denotes the shortest distance on Google map between camera
$c_{i}$ and $c_{j}$. $\tau$ is the discrepancy of time stamps between $i$ and
$j$. The hyperparameters $\alpha_{1}$, $\alpha_{2}$ and $\beta_{1}$,
$\beta_{2}$ indicate that higher probilities corresponding to smaller spatio-
temporal distances. $p(\delta|\mu_{\delta},\sigma_{\delta})$ and
$p(\tau|\mu_{\tau},\sigma_{\tau})$ describe the estimation of conditional
probabilities of $\delta$ and $\tau$ with parameters
$(\mu_{\delta},\sigma_{\delta})$ and $(\mu_{\tau},\sigma_{\tau})$, which are
modeled as log-normal distributions:

$\displaystyle p(\delta|\mu_{\delta},\sigma_{\delta})$
$\displaystyle=\ln\mathcal{N}(\delta;\mu_{\delta},\sigma_{\delta})=\frac{1}{\delta\sqrt{2\pi\sigma_{\delta}^{2}}}\exp[-\frac{(\ln\delta-\mu_{\delta})^{2}}{2\sigma_{\delta}^{2}}],$
$\displaystyle p(\tau|\mu_{\tau},\sigma_{\tau})$
$\displaystyle=\ln\mathcal{N}(\tau;\mu_{\tau},\sigma_{\tau})=\frac{1}{\tau\sqrt{2\pi\sigma_{\tau}^{2}}}\exp[-\frac{(\ln\tau-\mu_{\tau})^{2}}{2\sigma_{\tau}^{2}}].$

The parameters $(\mu_{\delta},\sigma_{\delta})$ and
$(\mu_{\tau},\sigma_{\tau})$ of the distributions are estimated by maximizing
the following likelihood functions:

$\displaystyle L(\delta|\mu_{\delta},\sigma_{\delta})$
$\displaystyle=\prod_{i=1}^{N}(\frac{1}{\delta_{i}})\mathcal{N}(\ln\delta_{i};\mu_{\delta},\sigma_{\delta}),$
$\displaystyle L(\tau|\mu_{\tau},\sigma_{\tau})$
$\displaystyle=\prod_{i=1}^{N}(\frac{1}{\tau_{i}})\mathcal{N}(\ln\tau_{i};\mu_{\tau},\sigma_{\tau}).$

where $N$ is the number of images in the gallery set.

In this way, we establish the spatio-temporal module to provide an additional
refinement for vehicle re-ID. Among the existing spatio-temporal models [15,
18, 24, 26, 25], the most similar spatio-temporal model is [18]. Different
from [18], which considers the transition time intervals of vehicles but does
not model the spatial information, our DFR-ST involves both temporal and
spatial cues simultaneously. Therefore, we can explicitly obtain a
quantitative formulation of spatio-temporal relationships to promote overall
performance.

Hence, the entire retrieval process is: DFR-ST firstly extracts an appearance
representation $D_{a}$ from the appearance module, which describes the
distance of the query image $i$ and the gallery set in the feature embedding
space. Secondly, the spatio-temporal module constructs the spatio-temporal
similarity $D_{st}$ to evaluate the similarities in spatial and temporal
domains. The overall similarity between the query image $i$ and the gallery
image $j$ is calculated as:

$D(j)=D_{a}(j)+\omega[D_{s}(i,j)+D_{t}(i,j)],\forall i.$ (6)

$\omega$ is a weighted parameter. $D_{s}(i,j)$ and $D_{t}(i,j)$ model the
spatial and temporal similarities between $i$ and $j$ respectively. Finally,
the ranking list of the proposed DFR-ST is conducted by the weighted summation
of $D_{a}$ and $D_{st}$.

##  IV Experiments

In this section, the proposed DFR-ST is evaluated on public datasets and
compared with state-of-the-art methods. Extensive experiments are conducted to
demonstrate the effectiveness and the robustness of DFR-ST at the end of this
section.

###  IV-A Datasets

The experiments are executed on two public large-scale datasets, _i.e._ ,
VeRi-776 [15] and VehicleID [47]. The following subsections introduce the two
datasets and evaluation metrics.

Figure 6: Visualization of the ranking lists of DFR-ST on VeRi-776. The re-
identification results are listed in ascending order of the appearance and the
spatio-temporal distance between the query image and the gallery images. The
green (red) boxes denote the correct (wrong) re-identification.

####  IV-A1 VeRi-776

The VeRi-776 dataset contains 49,357 images of 776 different vehicles captured
by 20 cameras involving various viewpoints, illumination changes, and
background clutters. The training set consists of 37,778 images with 576
identities, and the test set receives the remaining 11,579 images with 200
identities. Moreover, the selected 1,678 images of the testing set constitute
the query set. Followed [15], the evaluation metrics of VeRi-776 are mean
Average Precision (mAP), Top-1, and Top-5 accuracy of Cumulative Match Curve
(CMC) corresponding to the image-to-track search.

####  IV-A2 VehicleID

VehicleID dataset is released after the VeRi-776, which includes 221,763
images with 26,267 identities and 250 models. Researchers annotate 90,196
images with model labels. Among the identities, the training set contains
13,164 identities, and the rest remains for the test set. Three test splits
for different gallery sizes are 800, 1,600 and 2,400, as depicted in Table I.
For each test split, the test set randomly selects images of distinct
identities to form the gallery set, and the same procedure repeats ten times.
The average results of 10-times procedures are the final performance.
Evaluation metrics on VehicleID are mAP, Top-1, and Top-5 accuracy of CMC.

TABLE I: Three test splits of VehicleID dataset. | Small | Medium | Large
---|---|---|---
Query Images | 800 | 1,600 | 2,400
Gallery Images | 6,493 | 13,377 | 19,777

###  IV-B Implementation Details

The backbone network for shallow feature extraction adopts ResNet-50 [48]. In
the coarse-grained feature stream, the average pooling operation is employed
after three residual blocks and a $1\times 1$ convolutional layer is followed.
$\lambda$ in Eq. (1) is set to 0.4 and $\varepsilon$ in Eq. (2) is set to 0.1.
$m$ in Eq. 3 is set to 1.2. Moreover, the parameters $\alpha_{1}$ and
$\alpha_{2}$ are set to 6 while $\beta_{1}$ and $\beta_{2}$ are set to 0.5 in
Eq. (4) and Eq. (5). The weighted parameter $\omega$ in Eq. (6) is set to 0.2.

As for the training procedure, we apply the adam optimizer with a weight decay
$5\emph{e}^{-4}$ and a synchronous batch normalization strategy. The initial
learning rate is $1\emph{e}^{-4}$, which is adjusted by a warmup strategy.
Moreover, we utilize the Euclidean distance to compute the similarity between
the query and gallery images during training and testing. The batch size is
32, with randomly selected eight identities and four images of each identity.
We train the appearance module of DFR-ST for 135 epochs on two Nvidia GeForce
GTX 1080 Ti GPUs. The overall DFR-ST is implemented on the PyTorch platform.
Moreover, our work adopts the re-ranking strategy [49] for further boosting
the performance. since re-ranking benefits performance promotion [50, 49, 21,
51].

TABLE II: Comparison with state-of-the-art on VeRi-776. Methods | mAP | Top-1 | Top-5
---|---|---|---
LOMO [4] | 9.64 | 25.33 | 46.48
BOW-CN [5] | 12.20 | 33.91 | 53.69
FACT [16] | 19.92 | 59.65 | 75.27
SCCN-Ft + CLBL-8-Ft [52] | 25.12 | 60.83 | 78.55
OIN [18] | 48.00 | 65.9 | 87.7
VAMI [53] | 50.13 | 77.03 | 90.82
RNN-HA (ResNet) [22] | 56.80 | 74.49 | 87.31
Hard-View-EALN [54] | 57.44 | 84.39 | 94.05
GRF + GGL [55] | 61.7 | 89.4 | 95.0
QD-DLF [56] | 61.83 | 88.50 | 94.46
SPAN w/ CPDM [57] | 68.9 | 94.0 | 97.6
SAVER [58] | 79.6 | 96.4 | 98.6
HPGN [59] | 80.18 | 96.72 | N/A
PRN [20] | 85.84 | 97.14 | $\mathbf{99.40}$
PRN + ReRanking [20] | 90.48 | 97.38 | 98.87
FACT + Plate-SNN + STR [15] | 27.77 | 61.44 | 78.78
PROVID [17] | 53.42 | 81.56 | 95.11
OIN + ST [18] | 51.42 | 68.3 | 89.7
Siamese-CNN + Path-LSTM [24] | 58.27 | 83.49 | 90.04
VAMI [53] \+ STR [16] | 61.32 | 85.92 | 91.84
ReID + query expansion [25] | 70.8 | 93.2 | 98.0
DFR-ST (ours) | 86.00 | 95.67 | $\underline{99.17}$
DFR-ST (ours) + ReRanking | $\mathbf{91.56}$ | $\mathbf{97.74}$ | 98.41

###  IV-C Performance Comparison on VeRi-776

The proposed DFR-ST is compared with state-of-the-art methods on a large-scale
public dataset, i.e., VeRi-776, and the results are presented in Table II.
Note that we directly copy state-of-the-art algorithms’ performance from the
original papers instead of reproducing all methods.

On VeRi-776, we compare our proposed DFR-ST with the following methods. Local
Maximal Occurrence Representaion (LOMO) [4] and Bag of Words with Color Name
[60] (BOW-CN) are based on hand-craft features which are first proposed for
person re-ID [4, 5]. FACT [16] is based on the fusion with colors and
attribute features. Hard-View-EALN [54] and VAMI [53] impose an adversarial
network between the generator and the discriminator to obtain more robust
cross-view features. Moreover, SCCN-Ft + CLBL-8-Ft [52] utilizes two networks
to learn the local and global multi-view features. OIN [18] produces region
masks based on the clustering of key points and establishes overall features
using these region masks and global features. SPAN w/ CPDM [57] detects
different parts of vehicle images and then generates an attentive feature
representation by aggregating the global and three-part attentive features
together. Besides, RNN-HA (ResNet) [22] employs RNN to capture hierarchical
dependencies for vehicle re-ID. HPGN [59] proposes a pyramid of the spatial
graph networks to handle multi-scale spatial features. PRN + ReRanking [20]
explores partition strategies on three dimensions of feature maps to promote
overall performance further. Moreover, GRF+GGL [55] designs an efficient
group-group loss to accelerate feature learning. QD-DLF [56] defines pooling
operations on four directions to compress basic features and concatenates them
together as deep quadruple features. SAVER [58] focuses on self-supervised
attention mechanisms to improve vehicle re-ID algorithm. As for methods using
spatio-temporal clues, ReID + query expansion [25] ensembles multiple
informative features from several existing approaches and builds an
acquisition module for vehicle locations and timestamps. FACT+Plate-SNN+STR
[15] and PROVID [17] both use a license plate recognition module and a spatio-
temporal module based on FACT [16]. OIN+ST [18] constructs a spatio-temporal
regularization module in addition to OIN [18] while VAMI [53] \+ STR [16]
integrates a spatio-temporal similarity to the original model.

Figure 7: Visualization of attention maps of DFR-ST on VehicleID. The red
regions represent the attentive areas captured by the proposed network, which
cover mostly car lights, car bumpers, sunroofs and car edges etc.

Table II illustrates the comparison results with state-of-the-art methods on
VeRi-776. The methods listed in the upper part of Table II only use visual
appearance information while those appeared in the lower part of Table II take
both appearance and spatio-temporal cues into considerations. The proposed
DFR-ST obtains the best Top-5 metric of 99.17% and achieves the highest mAP of
91.56% and Top-1 of 97.74% with the re-ranking strategy (DFR-ST + ReRanking).

Firstly, compared with the approaches only based on visual appearance
features, the proposed DFR-ST acquires the highest mAP of 91.56% and the
highest Top-1 of 97.74%. This comparison verifies the intuition on the
effectiveness of adding more information from other domains. Specifically, we
notice that the Top-5 of PRN [20] is only a bit higher (0.23%) to our proposed
DFR-ST. Although this observation indicates the advantage of the
representation from [20], we show that the unexpected noises frequent in
unconstrained environments would degrade the performance of PRN [20] while our
proposed DFR-ST maintain acceptable performance in the ablation studies. This
comparison result could validate the superiority of the proposed DFR-ST.

Secondly, among six vehicle re-ID methods based on multi-modal information
(_i.e._ , FACT + Plate-SNN + STR [15], PROVID [17], OIN + ST [18], Siamese-CNN
+ Path-LSTM [24], VAMI [53] \+ STR [16], ReID + query expansion [25]), the
proposed DFR-ST consistently outperforms FACT + Plate-SNN+STR [15], PROVID
[17], OIN + ST [18] and Siamese-CNN + Path-LSTM [24] by exceeding over 30% on
mAP, 12% on Top-1 and 5% on Top-5 at least. This comparison demonstrates that
DFR-ST can take advantage of multi-modal information exceedingly from designed
spatio-temporal scheme. Fig. 6 visualizes the vehicle re-ID results on
VeRi-776 dataset qualitatively. The bounding boxes in green describe the true
positives, and those in red are false positives.

###  IV-D Ablation Study

Extensive experiments are conducted on two large-scale datasets, i.e.,
VeRi-776 and VehicleID, to thoroughly analyze each component’s effectiveness
of the proposed approach.

TABLE III: Component effectiveness analysis on VeRi-776. Methods | mAP | Top-1 | Top-5
---|---|---|---
Baseline | 65.72 | 78.10 | 84.63
$\mathbf{x}_{c}$ | 70.53 | 84.55 | 88.76
$\mathbf{x}_{c}+\mathbf{x}_{\text{f1}}$ | 76.90 | 87.24 | 91.77
$\mathbf{x}_{c}+\mathbf{x}_{\text{f1}}+\mathbf{x}_{\text{fdiv}}$ | 84.47 | 93.02 | 97.13
$\mathbf{x}_{c}+\mathbf{x}_{\text{f1}}+\mathbf{x}_{\text{fdiv}}$ \+ ST | $\mathbf{86.00}$ | $\mathbf{95.67}$ | $\mathbf{99.17}$

####  IV-D1 Component Analysis

We conduct ablation experiments on each component in the proposed DFR-ST to
validate each component’s effectiveness in the overall architecture. Table III
shows the experimental results. The baseline adopts a ResNet-50 network with
the cross-entropy loss. $\mathbf{x}_{c}$ denotes the features extracted from
the coarse-grained feature stream. $\mathbf{x}_{\text{f1}}$ and
$\mathbf{x}_{\text{fdiv}}$ represent the features from the first branch and
the rest three-division branches of the fine-grained feature stream,
respectively. Moreover, $ST$ is the spatio-temporal module.

TABLE IV: Effectiveness of spatio-temporal information on VeRi-776. Methods | mAP | Top-1 | Top-5
---|---|---|---
FACT w/o STR [16] | 19.92 | 59.65 | 75.27
\+ STR [16] | (+6.85%) | (+1.79%) | (+1.02%)
OIN w/o ST [18] | 48.00 | 65.9 | 87.7
\+ ST [18] | (+3.42%) | (+2.40%) | (+2.00%)
VAMI [53] | 50.13 | 77.03 | 90.82
\+ STR [16] | (+11.19%) | (+8.89%) | (1.02%)
Siamese-CNN [24] | 54.21 | 79.32 | 88.92
\+ Path-LSTM [24] | (+4.06%) | (+4.17%) | (+1.12%)
DFR-ST w/o ST (ours) | 84.47 | 93.02 | 97.13
\+ STR [16] | (+1.72%) | (+1.52%) | (+0.67%)
\+ ST (ours) | (+1.53%) | (+2.65%) | (+2.04%)
TABLE V: Effectiveness of the appearance module on VehicleID. Methods | Test Size=800 | Test Size=1600 | Test Size=2400
---|---|---|---
mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5
BOW-CN [5] | N/A | 13.14 | 22.69 | N/A | 12.94 | 21.09 | N/A | 10.20 | 17.89
LOMO [4] | N/A | 19.74 | 32.14 | N/A | 18.95 | 29.46 | N/A | 15.26 | 25.63
FACT + Plate-SNN [15] | 49.2 | 43.62 | 64.84 | 44.8 | 39.94 | 62.98 | 38.6 | 35.68 | 56.24
PROVID [17] | N/A | 48.90 | 69.51 | N/A | 43.64 | 65.34 | N/A | 38.63 | 60.72
DRDL [47] | N/A | 49.0 | 73.5 | N/A | 42.8 | 66.8 | N/A | 38.2 | 61.6
DenseNet121 [61] | 68.85 | 66.10 | 77.87 | 69.45 | 67.39 | 75.49 | 65.37 | 63.07 | 72.57
TAMR [62] | N/A | 66.02 | 79.71 | N/A | 62.90 | 76.80 | N/A | 59.69 | 73.87
VAMI [53] | N/A | 63.12 | 83.25 | N/A | 52.87 | 75.12 | N/A | 47.34 | 70.29
GS-TRE [63] | 75.4 | 75.9 | 84.2 | 74.3 | 74.8 | 83.6 | 72.4 | 74.0 | 82.7
QD-DLF [56] | $76.54$ | 72.32 | 92.48 | $74.63$ | 70.66 | 88.90 | $68.41$ | 64.14 | 83.37
RNN-HA (ResNet + 672) [22] | N/A | 83.8 | 88.1 | N/A | $81.9$ | 87.0 | N/A | $81.1$ | 87.4
PRN (Single Height-Channel Branch) [20] | N/A | 78.92 | 94.81 | N/A | 74.94 | $\underline{92.02}$ | N/A | 71.58 | 88.46
GRF + GGL [55] | N/A | 77.1 | 92.8 | N/A | 72.7 | 89.2 | N/A | 70.0 | 87.1
Hard-View-EALN [54] | 77.5 | 75.11 | 88.09 | 74.2 | 71.78 | 83.94 | 71.0 | 69.30 | 81.42
OIN [18] | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 67.0 | 82.9
SAVER [58] | N/A | 79.9 | $\underline{95.2}$ | N/A | 77.6 | 91.1 | N/A | 75.3 | 88.3
HPGN [59] | $\mathbf{89.60}$ | $\mathbf{83.91}$ | N/A | $\mathbf{86.16}$ | $\mathbf{79.97}$ | N/A | $\mathbf{83.60}$ | 77.32 | N/A
DFR-ST w/o ST (ours) | 87.55 | 82.15 | $\mathbf{95.39}$ | 84.94 | 79.33 | $\mathbf{92.76}$ | 83.18 | $\mathbf{77.93}$ | $\mathbf{89.52}$
DFR-ST w/o ST (ours) + Re-Ranking | 87.76 | 82.71 | 95.02 | 83.58 | 77.66 | 91.28 | 82.28 | 76.96 | 88.48
|  |  |  |  |  |  |  |  |

As depicted in Table III, the first observation is that only utilizing
$\mathbf{x}_{c}$ for vehicle re-ID can improve the performance by 4.81% on mAP
comparing to the baseline, which confirms the useful combination of the cross-
entropy loss and the triplet loss with the batch hard sampling strategy. After
adding $\mathbf{x}_{\text{f1}}$, extracted from the first branch of the fine-
grained feature stream, the mAP metric is enhanced by an additional 6.37% and
the Top-1, Top-5 metrics are increased by 2.69%, 3.01% respectively. This
improvement indicates the aggregation of the coarse-grained features
$\mathbf{x}_{c}$ and $\mathbf{x}_{\text{f1}}$ focusing more on local regions
by AttentionNet, can provide extra useful information to recognize and
identify the same vehicle. Furthermore, the participation of
$\mathbf{x_{\text{fdiv}}}$ actively promotes the overall performance of the
proposed method. Three metrics have improved by 7.57%, 5.78%, 5.36%. This
analysis confirms the effectiveness of division operations in different
dimensions. Moreover, $ST$ brings 1.53% improvement on mAP and 2.65%, 2.04% on
Top-1, Top-5, which verifies the complementary of spatio-temporal
relationships with appearance features.

####  IV-D2 Effectiveness of the Appearance Module

To validate the effectiveness of the proposed appearance module, we further
compare our proposed DFR-ST with several current methods on VehicleID. Table V
illustrates the mAP, Top-1, and Top-5 metrics of the proposed appearance
module and state-of-the-art methods on the large-scale VehicleID dataset. The
bold value means the first, and the underlined value stands for the second.
Note that VehicleID dataset does not provide any other information besides
images, hence we can not utilize the spatio-temporal module in the performance
comparison.

As shown in Table V, the proposed appearance module outperforms all the
existing methods on the Top-5 metric with small, medium, and large test sizes.
Besides, the proposed DFR-ST also achieves the highest Top-1 of 77.93% with
the large test size, which is the most challenging setting among three test
splits. In particular, compared with traditional hand-craft methods, LOMO [4]
and BOW-CN [5], deep-learning-based approaches achieve significant
improvements which verifies the strong representation capability of deep
neural networks on the non-linear transformation.

Furthermore, our proposed appearance module beats the existing methods
including FACT [16], PROVID [17], DRDL [47], DenseNet121 [61], TAMR [62], VAMI
[53], GS-TRE [63], QD-DLF [56], PRN (Single Height-Channel Branch) [20], GRF +
GGL [55], Hard-View-EALN [54], OIN [18] and SAVER [58] on the mAP, Top-1 and
Top-5 metrics with three test splits. Although RNN-HA (ResNet+672) [22]
receives more 1.65% on Top-1 with the small test size than the proposed DFR-ST
w/o ST, the proposed appearance module outperforms RNN-HA (ResNet+672) [22] on
the other metrics with three test sizes, which show the effectiveness of the
proposed approach. Although our DFR-ST without the spatio-temporal module is a
bit inferior to HPGN [59], the proposed DFR-ST can still obtain rather
competitive performance on the challenging VehicleID dataset, which confirms
the competitiveness of our proposed appearance module.

####  IV-D3 Effectiveness of AttentionNet

We conduct qualitative experiments to examine the effectiveness of multi-
domain attention schemes in AttentionNet. Fig. 7 displays typical examples of
the attention maps obtained by the proposed DFR-ST method for interpreting the
multi-domain attention. Most of the activated regions (marked by the red
color) are the car lights, car windows, the headstocks, and car edges. These
regions can be interpreted by humans easily and have semantic meanings, which
validates the fine-grained features’ capability with the multi-domain
attention mechanism to learn salient and informative regions for identifying
near-duplicate vehicle images. This property benefits DFR-ST to maintain
acceptable performance under image contamination, which is frequent and common
in open and unconstrained environments.

(a)

(b)

(c)

Figure 8: Performance comparision of the proposed DFR-ST and PRN [20] with different percentages of image noises. With the percentage of the image contamination becomes larger, the proposed DFR-ST obtains a slower rate of performance degradation than PRN [20].  TABLE VI: Performance evaluation of DFR-ST (ours) with different contamination percentages on VehicleID. Contaminated Percentages | Test Size=800 | Test Size=1600 | Test Size=2400 | Average
---|---|---|---|---
mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5
0% | 87.76 | 82.15 | 95.39 | 84.94 | 79.33 | 92.76 | 83.18 | 77.93 | 89.52 | 85.29 | 79.80 | 92.56
5% | 84.47 | 78.83 | 91.41 | 81.50 | 75.99 | 88.32 | 80.09 | 74.77 | 85.48 | 82.02 | 76.53 | 88.40
10% | 82.55 | 76.92 | 89.71 | 79.36 | 73.72 | 86.88 | 78.72 | 73.49 | 85.04 | 80.21 | 74.71 | 87.21
20% | 69.18 | 61.13 | 78.73 | 68.35 | 61.37 | 76.98 | 64.38 | 56.63 | 73.70 | 67.30 | 59.71 | 76.47
30% | 65.75 | 58.62 | 73.69 | 61.53 | 53.59 | 71.10 | 58.24 | 50.46 | 67.29 | 61.84 | 54.22 | 70.69
40% | 61.61 | 54.12 | 70.35 | 59.77 | 51.84 | 69.21 | 54.81 | 46.84 | 63.73 | 58.73 | 50.93 | 67.76
50% | 60.66 | 52.47 | 69.93 | 56.06 | 47.75 | 65.64 | 53.70 | 45.35 | 63.24 | 56.81 | 48.52 | 66.27
|  |  |  |  |  |  |  |  |  |  |  |
TABLE VII: Performance evaluation of PRN [20] with different contamination percentages on VehicleID. Contaminated Percentages | Test Size=800 | Test Size=1600 | Test Size=2400 | Average
---|---|---|---|---
mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5
0% | 84.08 | 78.92 | 94.81 | 80.54 | 74.94 | 92.02 | 75.76 | 71.58 | 88.46 | 80.13 | 75.15 | 91.76
5% | 80.53 | 74.38 | 91.28 | 76.90 | 70.86 | 88.46 | 71.49 | 68.10 | 83.61 | 76.31 | 71.11 | 87.78
10% | 73.37 | 69.81 | 84.24 | 71.20 | 64.50 | 81.88 | 66.45 | 61.32 | 79.44 | 70.34 | 65.21 | 81.85
20% | 58.87 | 50.60 | 67.12 | 54.40 | 46.73 | 65.03 | 50.38 | 43.18 | 62.89 | 54.55 | 46.84 | 65.01
30% | 50.47 | 43.91 | 61.54 | 47.62 | 40.86 | 58.17 | 45.59 | 37.63 | 56.40 | 47.89 | 40.80 | 58.70
40% | 43.71 | 38.30 | 55.99 | 38.53 | 35.66 | 52.25 | 35.48 | 31.70 | 50.06 | 39.24 | 35.22 | 52.77
50% | 35.69 | 32.42 | 48.16 | 31.80 | 28.75 | 44.26 | 29.48 | 25.74 | 42.39 | 32.32 | 28.97 | 44.94
|  |  |  |  |  |  |  |  |  |  |  |

####  IV-D4 Performance Evaluation with Image Contamination

In real-world scenarios, images are inevitably contaminated with noises due to
unreliable sensing devices, limited network communication resources, or time-
variant transmission environments. Thus, we conduct additional experiments to
test the performance of the proposed DFR-ST under unexpected noises to
demonstrate its robustness. We simulate three main problems (_i.e._ , mosaics,
color cast, and abnormal brightness) caused by unreliable transmission as
noise categories. The percentage of noisy pixels measures the degree of image
contamination to total pixels. Note that YUV pixel matrices’ improper decoding
produces these noises instead of the decoding of elementary streams with
MPEG-2 proposal.

As shown in Table VI, firstly, the proposed DFR-ST performance only drops
5.08% on mAP, 5.09% on Top-1, and 5.35% on Top-5 with 10% noisy pixels on
average. This result confirms the robustness of the proposed DFR-ST. Note that
DFR-ST with even 10% image contamination still obtains competitive performance
compared to QD-DLF [56] and defeats GS-TRE [63], DJDL [64], VAMI [53] and
DenseNet121 [61]. This comparison validates the effectiveness of DFR-ST.

Secondly, we notice that the performance of DFR-ST gets worse with larger
image noises. However, even with the worst performance, i.e., under 50% noisy
pixels, DFR-ST still has a better performance on VehicleID than several
methods including DRDL [47], FACT [16], LOMO [4] and BOW-CN [5]. This result
verifies DFR-ST can maintain an acceptable performance with a severe degree of
image contamination, which is advantageous for real-world applications.

Besides, we reproduce PRN [20] and compare its capability to resist image
contamination with the proposed DFR-ST on VehicleID. For the fairness of the
comparison, PRN (Single height-channel branch) is evaluated thoroughly with
DFR-ST w/o ST module because PRN (Single height-channel branch) has better
performance than the complete PRN. The comparison results are shown in Fig. 8
and Table VII. With the image noises grow larger, the performance of PRN [20]
decreases faster than the proposed DFR-ST. This comparison validates the
robustness of the proposed DFR-ST to resist image noises.

##  V Conclusion

In this paper, we proposed a discriminative feature representation DFR-ST with
multi-modal cues for vehicle re-ID. The scheme consists of an appearance
module for establishing robust visual features integrating coarse-grained and
fine-grained features, and a spatio-temporal module for constructing a
quantitative description of the transition distances of camera locations and
time snapshots. Extensive experiments on two large-scale datasets validate the
superiority of the proposed DFR-ST and the complementary of appearance and
spatio-temporal information. In the future, we will exploit the improvement of
more discriminative representation for vehicle re-ID and real-time video
analysis.

## References

  * [1] Y. He, X. Wei, X. Hong _et al._ , “Multi-target multi-camera tracking by tracklet-to-target assignment,” _IEEE Trans. Image Processing_ , vol. 29, pp. 5191–5205, 2020.
  * [2] R. Bashir, M. Shahzad, and M. Fraz, “Vr-proud: Vehicle re-identification using progressive unsupervised deep architecture,” _Pattern Recognition_ , vol. 90, pp. 52–65, 2019.
  * [3] J. O. N. Castaneda, A. F. Velazquez, N. B. Bo _et al._ , “Scalable semi-automatic annotation for multi-camera person tracking,” _IEEE Trans. Image Processing_ , vol. 25, no. 5, pp. 2259–2274, 2016.
  * [4] S. Liao, Y. Hu, X. Zhu, and S. Li, “Person re-identification by local maximal occurrence representation and metric learning,” in _Proc. CVPR_ , 2015, p. 2197–2206.
  * [5] L. Zheng, L. Shen, L. Tian _et al._ , “Scalable person re-identification: A benchmark,” in _Proc. ICCV_ , 2015, pp. 1116–1124.
  * [6] D. Zapletal and A. Herout, “Vehicle re-identification for automatic video traffic surveillance,” in _Proc. CVPR_ , 2016, pp. 25–31.
  * [7] S. Chen, X. Tan, B. Wang _et al._ , “Reverse attention-based residual network for salient object detection,” _IEEE Trans. Image Processing_ , vol. 29, pp. 3763–3776, 2020.
  * [8] M. Feng, H. Lu, and Y. Yu, “Residual learning for salient object detection,” _IEEE Trans. Image Processing_ , vol. 29, pp. 4696–4708, 2020.
  * [9] H. Lee, S. Eum, and H. Kwon, “Me r-cnn: Multi-expert r-cnn for object detection,” _IEEE Trans. Image Processing_ , vol. 29, pp. 1030–1044, 2020\.
  * [10] Y. Liu, J. Han, Q. Zhang, and C. Shan, “Deep salient object detection with contextual information guidance,” _IEEE Trans. Image Processing_ , vol. 29, pp. 360–374, 2020.
  * [11] L. Pan, Y. Dai, M. Liu _et al._ , “Joint stereo video deblurring, scene flow estimation and moving object segmentation,” _IEEE Trans. Image Processing_ , vol. 29, pp. 1748–1761, 2020.
  * [12] C. Dhiman and D. K. Vishwakarma, “View-invariant deep architecture for human action recognition using two-stream motion and shape temporal dynamics,” _IEEE Trans. Image Processing_ , vol. 29, pp. 3835–3844, 2020.
  * [13] Y. Ji, Y. Zhan, Y. Yang _et al._ , “Deep image-to-video adaptation and fusion networks for action recognition,” _IEEE Trans. Image Processing_ , vol. 29, pp. 3168–3182, 2020.
  * [14] H. Yang, C. Yuan, and L. Zhang, “Sta-cnn: Convolutional spatial-temporal attention learning for action recognition,” _IEEE Trans. Image Processing_ , vol. 29, pp. 5783–5793, 2020.
  * [15] X. Liu, W. Liu, T. Mei _et al._ , “A deep learning-based approach to progressive vehicle re-identification for urban surveillance,” in _Proc. ECCV_ , 2016, pp. 869–884.
  * [16] X. Liu, W. Liu, and H. Ma, “Large-scale vehicle re-identification in urban surveillance videos,” in _Proc. ICME_ , 2016, pp. 1–6.
  * [17] X. Liu, W. Liu, T. Mei, and H. Ma, “Provid: Progressive and multimodal vehicle reidentification for large-scale urban surveillance,” _IEEE Trans. Multimedia_ , vol. 20, no. 3, pp. 645–658, Mar. 2018.
  * [18] Z. Wang, L. Tang, X. Liu _et al._ , “Orientation invariant feature embedding and spatial temporal regularization for vehicle re-identification,” in _Proc. ICCV_ , 2017, pp. 379–387.
  * [19] B. He, J. Li, Y. Zhao _et al._ , “Part-regularized near-duplicate vehicle re-identification,” in _Proc. CVPR_ , 2019, pp. 3997–4005.
  * [20] H. Chen, B. Lagadec, and F. Bremond, “Partition and reunion: A two-branch neural network for vehicle re-identification,” in _Proc. CVPR_ , 2019, pp. 184–192.
  * [21] H. Guo, C. Zhao, Z. Liu _et al._ , “Learning coarse-to-fine structured feature embedding for vehicle re-identification,” in _Proc. AAAI_ , 2018, pp. 6853–6860.
  * [22] X. S. Wei, C. L. Zhang, L. Liu _et al._ , “Coarse-to-fine: A rnn-based hierarchical attention model for vehicle re-identification,” in _Proc. ACCV_ , 2018.
  * [23] G. Rajamanoharan, A. Kanaci, M. Li _et al._ , “Multi-task mutual learning for vehicle re-identification,” in _Proc. CVPR_ , 2019, pp. 62–70.
  * [24] Y. Shen, T. Xiao, H. Li _et al._ , “Learning deep neural networks for vehicle re-id with visual-spatio-temporal path proposals,” in _Proc. ICCV_ , 2017, pp. 1900–1909.
  * [25] K. Lv, H. Du, Y. Hou _et al._ , “Vehicle re-identification with location and time stamps,” in _Proc. CVPR_ , 2019, pp. 399–406.
  * [26] X. Tan, Z. Wang, M. Jiang _et al._ , “Multi-camera vehicle tracking and re-identification based on visual and spatial-temporal features,” in _Proc. ICCV_ , 2019, pp. 275–284.
  * [27] G. Wang, J. Lai, and X. Xie, “P2snet: Can an image match a video for person re-identification in an end-to-end way?” _IEEE Trans. Circuits Syst. Video Technol. (TCSVT)_ , vol. 28, no. 10, pp. 2777–2787, 2018.
  * [28] S. Li, S. Bak, P. Carr, and X. Wang, “Diversity regularized spatio-temporal attention for video-based person re-identification,” in _Proc. CVPR_ , 2018, pp. 369–378.
  * [29] Y. Cho, S. Kim, J. Park _et al._ , “Joint person re-identification and camera network topology inference in multiple cameras,” _Comput. Vis. Image Underst._ , vol. 180, pp. 34–46, 2019.
  * [30] J. L. abd W. Chen, Q. Li, and C. Yang, “Unsupervised cross-dataset person re-identification by transfer learning of spatial-temporal patterns,” in _Proc. CVPR_ , 2018, pp. 7984–7956.
  * [31] D. Chang, Y. Ding, J. Xie _et al._ , “The devil is in the channels: Mutual-channel loss for fine-grained image classification,” _IEEE Trans. Image Processing_ , vol. 29, pp. 4683–4695, 2020.
  * [32] M. Meng, M. Lan, J. Yu _et al._ , “Constrained discriminative projection learning for image classification,” _IEEE Trans. Image Processing_ , vol. 29, pp. 186–198, 2020.
  * [33] X. Sun, N. M. Nasrabadi, and T. D. Tran, “Supervised deep sparse coding networks for image classification,” _IEEE Trans. Image Processing_ , vol. 29, pp. 405–418, 2020.
  * [34]
  * [35] H. Luo, W. Jiang, X. Zhang _et al._ , “Alignedreid++: Dynamically matching local information for person re-identification,” _Pattern Recognition_ , vol. 94, pp. 53–61, 2019.
  * [36] K. Wang, C. Ding, S. J. Maybank, and D. Tao, “Cdpm: Convolutional deformable part models for semantically aligned person re-identification,” _IEEE Trans. Image Processing_ , vol. 29, pp. 3416–3428, 2020.
  * [37] W. Zhang, X. He, X. Yu _et al._ , “A multi-scale spatial-temporal attention model for person re-identification in videos,” _IEEE Trans. Image Processing_ , vol. 29, pp. 3365–3373, 2020.
  * [38] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proc. CVPR_ , 2015, pp. 815–823.
  * [39] X. Zhu, X. Y. Jing, X. You _et al._ , “Video-based person re-identification by simultaneously learning intra-video and inter-video distance metrics,” _IEEE Trans. Image Processing_ , vol. 27, no. 11, pp. 5683–5695, 2018.
  * [40] Q. Zhu, B. Zhong, X. Lan _et al._ , “Fine-grained spatial alignment model for person re-identification with focal triplet loss,” _IEEE Trans. Image Processing_ , vol. 29, pp. 7578–7589, 2020.
  * [41] L. Ren, J. Lu, J. Feng, and J. Zhou, “Uniform and variational deep learning for rgb-d object recognition and person re-identification,” _IEEE Trans. Image Processing_ , vol. 28, no. 10, pp. 4970–4982, 2019.
  * [42] Y. J. Cho and K. J. Yoon, “Pamm: Pose-aware multi-shot matching for improving person re-identification,” _IEEE Trans. Image Processing_ , vol. 27, no. 8, pp. 3739–3752, 2018.
  * [43] H. Huang, W. Yang, L. J _et al._ , “Improve person re-identification with part awareness learning,” _IEEE Trans. Image Processing_ , vol. 29, pp. 7468–7481, 2020.
  * [44] X. L. H. Luo, Y. Gu _et al._ , “Bag of tricks and a strong baseline for deep person re-identification,” in _Proc. CVPR_ , 2019.
  * [45] H. Luo, W. Jiang, X. Zhang _et al._ , “Alignedreid++: Dynamically matching local information for person re-identification,” _Pattern Recognition_ , vol. 94, pp. 53–61, 2020.
  * [46] C. Szegedy, V. Vanhoucke, S. Ioffe _et al._ , “Rethinking the inception architecture for computer vision,” _CoRR_ , vol. abs/1512.00567, 2015. [Online]. Available: http://arxiv.org/abs/1512.00567
  * [47] H. Liu, Y. Tian, Y. Wang _et al._ , “Deep relative distance learning: Tell the difference between similar vehicles,” in _Proc. CVPR_ , 2016, pp. 2167–2175.
  * [48] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. CVPR_ , 2016, pp. 770–778.
  * [49] Z. Zhong, L. Zheng, D. Cao, and S. Li, “Re-ranking person re-identification with k-reciprocal encoding,” in _Proc. CVPR_ , 2017, p. 3652–3661.
  * [50] W. Li, Y. Wu, M. Mukunoki, and M. Minoh, “Common-near-neighbor analysis for person re-identification,” in _Proc. ICIP_ , 2019, pp. 1621–1624.
  * [51] A. Shankar, A. Poojary, V. Kollerathu _et al._ , “Comparative study of various losses for vehicle re-identification,” in _Proc. CVPR_ , 2019, pp. 256–264.
  * [52] Y. Zhou, L. Liu, and L. Shao, “Vehicle re-identification by deep hidden multi-view inference,” _IEEE Trans. Image Processing_ , vol. 27, no. 7, pp. 3275–3284, 2018.
  * [53] Y. Zhou and L. Shao, “Viewpoint-aware attentive multi-view inference for vehicle re-identification,” in _Proc. CVPR_ , 2018, pp. 6489–6498.
  * [54] Y. Lou, Y. Bai, J. Liu _et al._ , “Embedding adversarial learning for vehicle re-identification,” _IEEE Trans. Image Processing_ , vol. 28, no. 8, pp. 3794–3807, 2019.
  * [55] X. Liu, S. Zhang, X. Wang _et al._ , “Group-group loss-based global-regional feature learning for vehicle re-identification,” _IEEE Trans. Image Processing_ , vol. 29, pp. 2638–2652, 2020.
  * [56] J. Zhu, H. Zeng, J. Huang _et al._ , “Vehicle re-identification using quadruple directional deep learning features,” _IEEE Trans. on Intell. Transp. Syst._ , vol. 21, no. 1, pp. 410–420, Jan. 2020.
  * [57] T. Chen, C. Liu, C. Wu, and S. Chien, “Orientation-aware vehicle re-identification with semantics-guided part attention network,” in _Proc. ECCV_ , 2020.
  * [58] P. Khorramshahi, N. Peri, J. Chen, and R. Chellappa, “The devil is in the details: Self-supervised attention for vehicle re-identification,” in _Proc. ECCV_ , 2020.
  * [59] F. Shen, J. Zhu, X. Zhu _et al._ , “Exploring spatial significance via hybrid pyramidal graph network for vehicle re-identification,” _CoRR_ , vol. abs/2005.14684, 2020. [Online]. Available: https://arxiv.org/abs/2005.14684
  * [60] J. V. D. Weijer, C. Schmid, J. Verbeek, and D. Larlus, “Learning color names for real-world applications,” _IEEE Trans. Image Processing_ , vol. 18, no. 7, pp. 1512–1523, 2009.
  * [61] G. Huang, Z. Liu, L. Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proc. CVPR_ , 2017, pp. 2261–2269.
  * [62] H. Guo, K. Zhu, M. Tang, and J. Wang, “Two-level attention network with multi-grain ranking loss for vehicle re-identification,” _IEEE Trans. Image Processing_ , vol. 28, no. 9, pp. 4328–4338, 2019.
  * [63] Y. Bai, Y. Lou, F. Gao _et al._ , “Group-sensitive triplet embedding for vehicle reidentification,” _IEEE Trans. Multimedia_ , vol. 20, no. 9, pp. 2385–2399, Sept. 2018.
  * [64] C. T. Liu, M. Y. Lee, and C. W. Wu, “Supervised joint domain learning for vehicle re-identification,” in _Proc. CVPR_ , 2019, pp. 45–52.

##  Appendix A Effect of the Division Branches

Table VIII presents the evaluation results of variants on the division
branches on the VeRi-776 dataset. The first variant refers to the appearance
module without three division branches, which only contains a dual path of the
coarse-grained features stream and the first branch of the fine-grained
feature stream.

The first observation of this comparison is that the division operations along
three dimensions contribute to the overall performance, and the channel
division branch brings the most significant improvement among these three
categories of division branches. This result is also observed in work PRN
[20], which only uses a single channel branch can get the highest performance
on VehicleID dataset than the fusion of three branches from different
dimensions. Different from PRN [20], we discover that the combination of
multiple primary division branches can promote the performance further.
Besides, we achieve the best performance by employing these three division
strategies on VeRi-776. It indicates the complementary of the information
provided from the division strategies along different axes.

##  Appendix B Effect of placements in AttentionNet

We conduct extensive experiments to investigate the effect on different
placements of the multi-domain attention scheme in the fine-grained feature
stream of the proposed DFR-ST on VeRi-776 dataset. Table IX shows the
performance with different orders of the channel attention and spatial
attention. Note that the experiments neglect the spatio-temporal module for
simplicity. As observed in Table IX, the consolidation of the awareness in
different domains (i.e., the spatial domain and the channel domain) can
promote the overall performance rather than single-domain attention from the
experimental results. Moreover, we adopt the placement of the sequential
channel attention and spatial attention for better performance in further
experiments. However, the empirical consequences reveal that the order of the
channel attention and spatial attention is insubstantial to the overall
performance.

TABLE VIII: Performance analysis of the division branches on VeRi-776. Variants | mAP | Top-1 | Top-5
---|---|---|---
w/o division | 76.90 | 87.24 | 91.77
Height division | 77.81 | 85.29 | 89.59
Width division | 78.28 | 87.43 | 91.31
Channel division | 82.92 | 90.46 | 94.06
Height + Width division | 82.67 | 92.14 | 94.89
Height + Channel division | 84.30 | 92.52 | 96.10
Width + Channel division | 83.82 | 92.35 | 96.96
Height + Width + Channel division | $\mathbf{84.47}$ | $\mathbf{93.02}$ | $\mathbf{97.13}$
TABLE IX: The Effect of different placements of attention sub-modules in AttentionNet on VeRi-776. Placements | mAP | Top-1 | Top-5
---|---|---|---
Channel Attention | 83.69 | 90.17 | 95.04
Spatial Attention | 82.55 | 90.76 | 94.48
Sequential Channel + Spatial Attention | $\mathbf{84.47}$ | $\mathbf{93.02}$ | $\mathbf{97.13}$
Sequential Spatial + Channel Attention | 83.74 | 92.81 | 96.35
Parallel Spatial + Channel Attention | 83.51 | 92.38 | 95.86

##  Appendix C Parameter Setting

We first investigate the influence of $\lambda$ in Eq. (1). As shown in Table
X, we set $\lambda=0.4$ in all experiments because adding the triplet loss can
boost the overall performance further.

TABLE X: Evaluation on the influence of $\lambda$ on VeRi-776. $\lambda$ | mAP | Top-1 | Top-5
---|---|---|---
0 | 83.04 | 91.55 | 96.18
0.1 | 83.56 | 92.32 | 96.70
0.2 | 84.08 | 92.64 | 96.82
0.3 | 84.41 | $\mathbf{93.05}$ | 97.00
0.4 | $\mathbf{84.47}$ | 93.02 | $\mathbf{97.13}$
0.5 | 84.43 | 92.94 | 96.89
0.6 | 84.30 | 92.96 | 96.80
0.7 | 84.15 | 92.74 | 96.53
0.8 | 84.07 | 92.43 | 96.45
0.9 | 83.69 | 92.16 | 96.39
1.0 | 83.35 | 91.86 | 96.24
TABLE XI: Evaluation on the influence of $\omega$ on VeRi-776. $\omega$ | mAP | Top-1 | Top-5
---|---|---|---
0 | 84.47 | 93.02 | 97.13
0.1 | 85.73 | 94.60 | 98.52
0.2 | $\mathbf{86.00}$ | $\mathbf{95.67}$ | 99.17
0.3 | 85.86 | 95.60 | $\mathbf{99.23}$
0.4 | 85.56 | 95.12 | 98.35
0.5 | 83.40 | 93.66 | 96.67
0.6 | 81.58 | 92.31 | 94.83
0.7 | 80.09 | 91.24 | 93.37
0.8 | 78.35 | 88.28 | 92.54
0.9 | 75.70 | 84.35 | 89.11
1.0 | 73.61 | 82.33 | 86.64

(a)

(b)

Figure 9: Visualization the effect of $\alpha$ and $\beta$ on the distribution
shapes. (a) The effect on the distribution shapes of the parameter $\alpha$.
The curves are $D=\frac{1}{1+\exp(\alpha(x-0.5))}$ as
$\alpha=0.5,1,2,\cdots,6$ based on the ascending order of the intersections
across the y axis. (b) The effect on the distribution shapes of the parameter
$\beta$. The curves are $D=\frac{1}{1+\exp(6(x-\beta))}$ as
$\beta=0,0.1,0.2,\cdots,1$ from left to right.

Besides, we study the impact of $\omega$ in Eq. (6), i.e., the weight of the
spatio-temporal similarity between the image pairs in the distance measurement
of Algorithm LABEL:alg. Table XI shows the experimental results. We first
observe that a moderate $\omega$ can increase the performance, which validates
the complementary of the spatio-temporal information and the appearance
representation. Second, the performance becomes worse quickly as $\omega$
becomes larger. It shows that too large weights on spatio-temporal cues can
bring unexpected degradation, and the appearance features account for more
contributions to the proposed vehicle re-ID approach. This result has
following the human perception of the world, in which the vision accounts for
over 80% information. Therefore, we set $\omega=0.2$ for all experiments.

Moreover, we investigate the effect of different distribution shapes
controlled by $\alpha_{1}$, $\alpha_{2}$ and $\beta_{1}$, $\beta_{2}$ in the
spatio-temporal module. The distribution in Eq. 5 delineates the declining
speeds of the spatio-temporal similarity with the differences of the camera
locations and the timestamps between two vehicle images. The intuition is that
we hope to maintain more plausible samples of high spatio-temporal
similarities for the appearance module to make decisions and get rid of those
relatively dissimilar images to reduce the search complexity. Thus, we set the
the parameters $\alpha_{1}$ and $\alpha_{2}$ to be 6 and $\beta_{1}$,
$\beta_{2}$ to be 0.5 based on Fig. 9.