# Exploring Modality-shared Appearance Features and Modality-invariant
Relation Features for Cross-modality Person Re-Identification

Nianchang Huang, Jianan Liu, Qiang Zhang*, Jungong Han* Nianchang Huang,
Jianan Liu, Qiang Zhang are with Center for Complex Systems, School of
Mechano-Electronic Engineering, Xidian University, Xi’an, Shaanxi 710071,
China. Email<EMAIL_ADDRESS><EMAIL_ADDRESS>and
<EMAIL_ADDRESS>Han is with Computer Science Department,
Aberystwyth University, SY23 3FL, UK. Email<EMAIL_ADDRESS>*Corresponding authors: Qiang Zhang and Jungong Han.

###### Abstract

Most existing cross-modality person re-identification works rely on
discriminative modality-shared features for reducing cross-modality variations
and intra-modality variations. Despite some initial success, such modality-
shared appearance features cannot capture enough modality-invariant
discriminative information due to a massive discrepancy between RGB and
infrared images. To address this issue, on the top of appearance features, we
further capture the modality-invariant relations among different person parts
(referred to as modality-invariant relation features), which are the
complement to those modality-shared appearance features and help to identify
persons with similar appearances but different body shapes. To this end, a
Multi-level Two-streamed Modality-shared Feature Extraction (MTMFE) sub-
network is designed, where the modality-shared appearance features and
modality-invariant relation features are first extracted in a shared 2D
feature space and a shared 3D feature space, respectively. The two features
are then fused into the final modality-shared features such that both cross-
modality variations and intra-modality variations can be reduced. Besides, a
novel cross-modality quadruplet loss is proposed to further reduce the cross-
modality variations. Experimental results on several benchmark datasets
demonstrate that our proposed method exceeds state-of-the-art algorithms by a
noticeable margin.

###### Index Terms:

Cross-modality person re-identification, visible images, thermal infrared
images, modality-shared appearance features, modality-invariant relation
features.

##  I Introduction

Person Re-IDentification (Re-ID) aims to match a given pedestrian from
disjointed camera views, which plays an important role in intelligent video
surveillance [1] and people tracking [2]. Recently, visible image based person
Re-ID models (_i.e.,_ RGB-RGB images matching) have attracted wide attention
and made great progress [3, 4, 5, 6, 7, 8]. However, visible cameras are
sensitive to light conditions, which cannot capture informative images under
inadequate illumination (_e.g.,_ at night). In such cases, the visible image
based person Re-ID methods may show a dramatic performance degradation.
Compared with visible cameras, infrared cameras are less dependent on light
conditions and may capture more informative infrared (IR) images in
challenging illuminations. Considering that, to capture extensive information,
many surveillance cameras support an automatic switch from the visible camera
to the infrared camera when the light condition is worse [9]. Accordingly,
cross-modality person Re-ID (_i.e.,_ RGB-IR image matching) has received
increasing interests recently [9, 10, 11, 12, 13]. This further promotes the
real-world applications of person Re-ID.

Figure 1: Examples of cross-modality person Re-ID. (a) Query of IR images. (b)
and (c) Matched RGB images corresponding to (a). (d) Query of RGB images. (e)
and (f) Matched IR images corresponding to (d). Images marked by green boxes
denote the right matches, while those marked by red boxes denote the wrong
matches.

As shown in Fig. 1, the difficulties of cross-modality person Re-ID mainly lie
in the following two aspects, _i.e.,_ cross-modality variations and intra-
modality variations [13, 14, 15, 16]. Cross-modality variations are caused by
the modality differences between visible (RGB) and infrared (IR) images, which
may lead to diverse distributions of single-modality RGB and IR features.
Intra-modality variations may result from many factors, _e.g.,_ different
viewpoints, human poses changing and self-occlusions, which further bring
difficulties in cross-modality person Re-ID. Most existing cross-modality
person Re-ID models [17, 18, 19, 20] follow the idea of modality-shared
feature learning, which aims at extracting discriminative modality-shared
features (or modality-invariant features) to simultaneously reduce cross-
modality variations and intra-modality variations. Concretely, these models
first separately extract modality-specific features from the input RGB and IR
images. Then, the modality-specific features are projected into a shared
feature space, on which modality-shared features are extracted for cross-
modality person Re-ID.

Figure 2: Illustrations of modality-invariant relations of different human
body parts. (a) and (b) The RGB and IR images of a person. (c) and (d) The RGB
and IR images of another person. The yellow lines mean the ratios of the
distance of the head to the shoulder with that of the shoulder to the elbow,
while the red lines mean the ratio of the distance of the hip to the knee with
that of the knee to the foot. By virtue of these modality-invariant relations,
the persons in the images of different modalities (_e.g.,_ (a) and (d), (b)
and (c), respectively) can be identified as different ones. Figure 3:
Visualization of some appearance features. (a) Input RGB or IR image. (b)-(e)
The appearance features and corresponding heat maps in input images. The heat
maps are obtained by mapping those appearance features into input images,
which reflect the regions of corresponding features’ focus. It can be seen
that different channels of appearance features mainly focus on different
person parts.

Similar to single-modality person Re-ID [21], most existing cross-modality
person Re-ID models extract appearance features from the shared feature space
of input RGB and IR images as the final modality-shared features. However, the
appearances of the same person vary significantly from RGB images to IR images
due to the large modality discrepancies caused by different imaging
mechanisms. As a result, only extracting modality-shared appearance features
may not capture enough discriminative information for identifying different
persons in cross-modality person Re-ID. For examples, Fig. 1(a) and (b) are
the same identity in the RGB and IR images, respectively, but Fig. 1(c) is
another identity in an RGB image. Unfortunately, Fig. 1(c) is still wrongly
identified as the same person as in Fig. 1(a) due to the large modality
differences between RGB and IR images. Similarly, Fig. 1(f) is also wrongly
matched as the person in Fig. 1(d).

Although those appearance features may be different in the images of different
modalities, the relations among different person parts are invariant to
modalities, which may help to identify persons with different body shapes,
especially when their appearance features are similar in cross-modality person
Re-ID. For examples, as shown in Fig. 2, the two persons in the images of
different modalities (_i.e.,_ persons in Fig. 2(a) and (d), respectively) can
be easily identified as different ones via the ratios of the distance of the
hip to the knee with that of the knee to the foot. Differently, the two
persons in Fig. 2(c) and (d) have similar ratios of the distance of the hip to
the knee with that of the knee to the foot, since they are from the same
identity. Therefore, on the top of appearance features, extracting modality-
invariant relation features may further enhance the discriminability of
modality-shared features, and reduce cross-modality variations and intra-
modality variations for cross-modality person Re-ID. Here, we refer to the
relations among different person parts as relation features.

Meanwhile, as shown in Fig. 3, we also find that one channel of appearance
features extracted by using a Re-ID model from a person may mainly focus on
describing one certain part of the person. For examples, the channel of
appearance features in Fig. 3(b) mainly focuses on the head of a person, and
the channel of appearance features in Fig. 3(c) mainly focuses on the body of
a person. Motived by such observations, the relation features about a person
can be obtained by capturing the relations among different channels of the
appearance features of the person. To this end, a novel cross-modality person
Re-ID model is presented in this paper for cross-modality person Re-ID, where
the modality-shared appearance features and their relations among different
channels are simultaneously adopted to boost performance.

Concretely, a novel Multi-level Two-streamed Modality-shared Feature
Extraction (MTMFE) sub-network is designed to simultaneously extract modality-
shared appearance features and modality-invariant relation features. In MTMFE
sub-network, modality-shared appearance features are first extracted by
employing a shared 2D CNN based sub-network. Then, inspired by [22, 23],
modality-invariant relation features are extracted by using a shared 3D CNN,
which has proven capability in capturing the relations of different frames in
the 3D feature space [24, 25, 26]. In light of that, in MTMFE sub-network, the
extracted appearance features, including modality-specific ones and modality-
shared ones, are considered as different ‘frames’ and fed into the shared 3D
CNN based sub-network to capture their relations. More specifically, the
extracted modality-specific features and modality-shared appearance features
are first projected into a shared 3D feature space, followed by a shared 3D
CNN based sub-network to capture the modality-invariant relation features.
With the proposed MTMFE subnetwork, the discriminative of the extracted
modality-shared features, including modality-shared appearance features and
modality-invariant relation features, will be significantly enhanced for
reducing the cross-modality variations and intra-modality variations, which
will further boosts the performance of cross-modality person Re-ID.

Besides, many works employ the Bi-directional Dual-constrained Top-Ranking
(BDTR) loss or its variants [13, 17, 27, 28] to train their cross-modality
person Re-ID models. The underlying constraint imposed by this sort of losses
is that the distance of an anchor sample to its farthest cross-modality
positive sample in the feature space should be smaller than the anchor sample
to its nearest cross-modality negative sample by a predefined margin. However,
such constraints cannot prevent the undesired situation that the distance of
an anchor sample to its farthest cross-modality positive sample in the feature
space is larger than the anchor sample to its nearest intra-modality negative
sample. To address such issue, a novel Cross-modality Quadruplet (CQ) loss is
designed, which simultaneously considers the following two aspects. Firstly,
the distance of an anchor sample to its farthest cross-modality positive
sample should be smaller than the anchor sample to its nearest cross-modality
negative sample by a predefined margin. Secondly, the distance of an anchor
sample to its farthest cross-modality positive sample should also be smaller
than the anchor sample to its nearest intra-modality negative sample by a
predefined margin. By virtue of the proposed CQ loss, more cross-modality
constraints are introduced to train the proposed model, thus leading to better
person Re-ID results.

In summary, the main contributions of this paper are as follows.

(1) As a departure from most existing models that rely only on the modality-
shared appearance features, our new end-to-end cross-modality person Re-ID
model makes use of both modality-shared appearance features and modality-
invariant relation features to boost performance.

(2) An MTMFE sub-network is designed to extract the modality-shared appearance
features and the modality-invariant relation features in a shared 2D feature
space and a shared 3D feature space, respectively. Doing so aims to reduce the
cross-modality variations as well as the intra-modality variations.

(3) A novel CQ loss is proposed to get the cross-modality variations further
decreased by introducing more cross-modality constraints. On top of the
constraints among an anchor sample from one modality and its positive as well
as negative samples from another modality, our proposed CQ loss further
imposes the constraints among an anchor sample from one modality, its positive
samples from another modality and its negative sample from the same modality.

The rest of this paper is organized as follows. We briefly describe some
previous works related to the visible image based person Re-ID and cross-
modality person Re-ID first in Section II, followed by the details of our
newly proposed method in Section III. Several experiments are conducted to
validate the proposed model in Section IV, Finally, in Section V, a brief
conclusion is made.

Figure 4: Illustration of the proposed model. A query RGB or IR image is first
fed into the modality-specific feature extraction to extract corresponding
modality-specific features. Then, the extracted modality-specific features are
fed into the modality-shared feature extraction to extract modality-shared
features, which contain the modality-shared appearance features and modality-
invariant relation features. Finally, the extracted modality-shared features
are fed into a part-aligned block to obtain the person features from different
person parts.

##  II Related Work

###  II-A Single-modality Person Re-ID

Conventional person Re-ID models are mainly based on handcrafted features
(_e.g.,_ color, textures, edges and shapes), followed by supervised distance
metric learning [29, 30]. Recently, deep learning based person Re-ID models
have become the mainstream and achieved great improvements over conventional
ones. Generally, most existing deep learning based person Re-ID models can be
divided into two categories: representation learning based models [3, 4, 5,
31, 32] and metric learning based models [6, 7, 8, 33, 34].

Representation learning based models [3, 4, 5, 31] try to obtain
discriminative and robust person features from the input images. The extracted
person features should be invariant to variations in illuminations, poses and
viewpoints. For examples, a deep representation learning procedure, named as
part loss network, was presented in [32] to learn discriminative
representations for unseen person images by employing a novel part loss.
Concretely, this part loss enforces their network to learn representations
from different body parts and further gains the discriminative power on unseen
persons. In [3], a Pose-Invariant Embedding (PIE) was presented to align
pedestrians to address the issue of pedestrian misalignment in person Re-ID.
Specifically, it employs a pose estimator based PoseBox structure to produce
well-aligned pedestrian images, so that the learned features can find the same
person under intensive pose changes.

Metric learning based models [6, 7, 8, 33, 34] determine whether two images
are captured from the same pedestrian by measuring the similarity between
their corresponding image features. For example, in [8], a dedicated variant
of the triplet loss was presented to perform end-to-end deep metric learning,
providing guidance for triplet loss training. In [33], a quadruplet loss was
designed to improve the triplet loss, which would obtain features with a
larger inter-class variations and a smaller intra-class variations.

###  II-B Cross-modality Person Re-ID

Cross-modality person Re-ID aims to match the queries from one modality
against a gallery set from another modality, which is very important for video
surveillance in real-world scenarios.

Previous methods can be summarized into two major categories: modality-shared
feature learning and modality-specific feature compensation. Modality-shared
feature learning based models project modality-specific features into the same
feature space [17, 18, 19, 20, 27, 16, 15, 27, 35, 36], where cross-modality
variations and intra-modality variations are simultaneously addressed by
extracting discriminative modality-shared features. For example, in [17], a
dual-path network was presented, which first employs two AlexNet [37] based
sub-networks to extract the modality-specific features from the input RGB and
IR images, respectively. Then, they employ a shared sub-network with two
stacked fully connected layers to transfer the extracted modality-specific
features into a shared feature embedding for cross-modality person Re-ID. In
[15], besides a cross-modality triplet loss, a cutting-edge generative
adversarial training based discriminator was also employed to learn more
discriminative modality-shared feature representations from different
modalities. Recently, apart from modality-shared features, modality-specific
features are also employed to boost the performance of cross-modality person
Re-ID. In [27], a novel cross-modality shared-specific feature transfer
algorithm was presented to explore the potential of both the modality-shared
information and the modality-specific characteristics for boosting cross-
modality person Re-ID.

Modality-specific feature compensation based models aim to generate the
missing specific information from the existing ones [38, 36], which address
cross-modality variations by generating the images of missed modality and
address cross-modality variations by extracting discriminative cross-modality
features from paired RGB and IR images. For example, in [38], a Alignment
Generative Adversarial Network (AlignGAN) was designed, which consists of a
pixel generator, a feature generator and a joint discriminator to jointly
exploit pixel alignment and feature alignment. In [36], a GAN based cross-
modality person Re-ID model was presented to generate the cross-modality
paired-images by disentangling features and decoding from exchanged features.
It also proposes a new variation module to map modality-invariant features to
a latent manifold feature space for boosting cross-modality person Re-ID.

###  II-C 3D Convolutional Neural Network

3D Convolutional Neural Networks (3D CNNs) have been widely used in many
computer vision tasks, _e.g.,_ video classification [39, 40], action
recognition [41, 26, 24, 23] and medical image processing [42, 43]. In video
analysis tasks, 3D CNN can capture the relations among different video frames
in the 3D feature space. In [24], 3D CNN was first introduced to learn
discriminative features along both spatial and temporal dimensions for action
recognition. Then, in [26], a 3D CNN, named as C3D, was designed to extract
features for video processing, which achieves state-of-the-art performance on
multiple video analysis tasks. In medical image processing, 3D CNN can capture
the relations among different slices of the CT or MR images. For example, in
[42], a 3D CNN based Multiple Sclerosis (MS) lesion segmentation model was
designed, which consists of two stages for automatically segmenting MS
lesions. The alternative lesion voxels were selected in the first stage, while
in the second stage, the final lesion voxels were segmented from the lesion
voxels.

In this paper, inspired by [22, 23], 3D CNN is employed to extract the spatial
and channel-wise relations among different appearance features. For that, the
extracted appearance features are projected into a shared 3D feature space,
where different appearance features are considered as different ‘video frames’
or ‘CT/MR image slices’ and fed into a shared 3D CNN based sub-network to
extract the spatial and channel-wise relations among different appearance
features.

##  III Proposed Model

As shown in Fig. 4, the proposed model mainly contains three stages, _i.e.,_
modality-specific feature extraction, modality-shared feature extraction and
part-aligned block. In modality-specific feature extraction, a two-streamed
sub-network is first employed to extract modality-specific features from the
input RGB image and IR images, respectively. Then, in modality-shared feature
extraction, a Multi-level Two-streamed Modality-shared Feature Extraction
(MTMFE) sub-network is presented to extract the modality-shared features,
including modality-shared appearance features and modality-invariant relation
features, for cross-modality person Re-ID. Finally, a part-aligned block is
employed to extract final person features from different parts. More details
about these three components will be discussed in the following contents.

###  III-A Modality-Specific Feature Extraction

There are essential distinctions between RGB images and IR images due to the
fact that RGB images and IR images are captured in different spectrums. As a
result, the features extracted from RGB images and IR images have large
variations. Considering that, two sub-networks with the same structures but
different parameters are employed to extract modality-specific features from
the given RGB images and IR images, respectively. One of the two sub-networks
aims to extract modality-specific features from RGB images (denoted as
$\mathbf{F}_{RGB}$) and the other is used to extract modality-specific
features from IR images (denoted as $\mathbf{F}_{IR}$). Meanwhile, features at
different levels have different properties. Low-level features contain more
spatial or local-context information (_e.g.,_ colors, textures, edges, and
contours), while high-level features contain more semantic information
(_e.g.,_ objects and human parts). Furthermore, compared with high-level
semantic information, low-level spatial information is more modality-related.
For example, the semantic information of person bodies can be simultaneously
contained in the high-level features of RGB images and IR images, while the
color information only exists in the low-level features from RGB images.
Therefore, in this paper, the two sub-networks employ a relatively shallower
CNN for modality-specific feature extraction. Specifically, the two sub-
networks follow the same structures as the first three convolutional blocks in
ResNet50 [44] for modality-specific feature extraction, due to the fact that
the first three convolutional blocks of ResNet50 mainly extract low-level
features from the input images. In this way, the extracted modality-specific
features can well exploit those single-modality information from the input
images. Mathematically, this process is expressed by:

$\mathbf{F}_{RGB}=\operatorname{Conv}(X_{RGB},\theta_{RGB}),\mathbf{F}_{IR}=\operatorname{Conv}(X_{IR},\theta_{IR}),$
(1)

where $\operatorname{Conv}(*,\theta_{RGB})$ and
$\operatorname{Conv}(*,\theta_{IR})$ denote the convolutional blocks with
their corresponding parameters $\theta_{RGB}$ and $\theta_{IR}$, respectively.
$X_{IR}$ and $X_{RGB}$ denote the input RGB image and IR image, respectively.

###  III-B Modality-shared Feature Extraction

Modality-shared feature extraction aims to extract discriminative modality-
shared features to reduce cross-modality variations and intra-modality
variations. However, as discussed in Section I, most existing models only
extract modality-shared appearance features [17, 18] for cross-modality person
Re-ID, which may not capture enough modality-invariant and discriminative
information from RGB and IR images due to their large modality differences.
Furthermore, the relations among different person parts (_i.e.,_ modality-
invariant relation features) may be complementary to the modality-shared
appearance features. Therefore, jointly capturing modality-shared appearance
features and modality-invariant relation features may enhance the
discriminability of the final modality-shared features. This further reduces
cross-modality variations and intra-modality variations for cross-modality
person Re-ID. To this end, a novel Multi-level Two-streamed Modality-shared
Feature Extraction (MTMFE) sub-network is designed.

As shown in Fig. 4, the proposed MTMFE sub-network also contains two shared
sub-networks. One is a 2D CNN based sub-network to extract the modality-shared
appearance features by projecting the extracted modality-specific features
into a shared 2D feature space. The other is a 3D CNN based sub-network to
capture the modality-invariant relation features by projecting the extracted
modality-specific features and the modality-shared appearance features into a
shared 3D feature space. It should be noted that the modality-specific
features for modality-shared feature extraction may be extracted from the RGB
image (_i.e.,_ $\mathbf{F}_{RGB}$) or the IR image (_i.e.,_
$\mathbf{F}_{IR}$). For simplicity, the modality-specific features (_i.e.,_
$\mathbf{F}_{RGB}$ or $\mathbf{F}_{IR}$) for modality-shared feature
extraction are denoted by $\mathbf{F}\in R^{C\times H\times W}$ in this
section. Here, $C,H$ and $W$ denote its channels, height and width.

####  III-B1 Modality-shared Appearance Feature Extraction

As shown in Fig. 4, on the top of modality-specific feature extraction, a
shared 2D CNN based sub-network is first employed to extract modality-shared
appearance features by projecting the modality-specific features into a shared
feature space. Specifically, given the modality-specific features
$\mathbf{F}$, the shared sub-network employs two convolutional blocks to
extract two levels of modality-shared appearance features (denoted as
$\mathbf{F}_{a}^{1}$ and $\mathbf{F}_{a}^{2}$, respectively). Mathematically,
this process is expressed by:

$\mathbf{F}_{a}^{1}=\operatorname{Conv}(\mathbf{F},\theta_{a1}),\mathbf{F}_{a}^{2}=\operatorname{Conv}(\mathbf{F}_{a}^{1},\theta_{a2}),$
(2)

where $\operatorname{Conv}(*,\theta_{a1})$ and
$\operatorname{Conv}(*,\theta_{a2})$ denote two convolutional blocks with
their corresponding parameters $\theta_{a1}$ and $\theta_{a2}$, respectively.
In our model, the shared sub-network follows the structure of the last two
convolutional blocks of ResNet50 [44]. In our model, both the two levels of
modality-shared appearance features ($\mathbf{F}_{a}^{1}$ and
$\mathbf{F}_{a}^{2}$) are employed for cross-modality person Re-ID, due to the
fact that the modality-shared appearance features in the two levels contain
varieties of semantic information. This may further enhance the
discriminability and robustness of the final person features.

####  III-B2 Modality-shared relation Feature Extraction

Figure 5: Illustration of the projection operation. (a) Project the 2D
appearance features into 3D feature space; (b) Project the modality-invariant
relation features into 2D feature space. In (a), the input appearance features
with the size of $C\times H\times W$ are projected as 3D features with the
size of $\frac{C}{D}\times D\times H\times W$. In (b), the input relation
feature with the size of $\frac{C}{D}\times D\times H\times W$ are projected
as 2D features with the size of $C\times H\times W$. Here, $C,H,W$ and $D$
denote channels, height, width and depth of corresponding features.

As discussed in Section I, extracting modality-invariant relation features
will further enhance the discriminability of modality-shared features and
reduce cross-modality variations as well as intra-modality variations for
cross-modality person Re-ID. Furthermore, different channels of appearance
features mainly contain discriminative information at different person parts.
Therefore, relation features can be obtained by capturing the relations among
different channels of appearance features. Considering that, in our proposed
model, the next step is to capture modality-invariant relation features from
different types of appearance features (_i.e.,_ modality-specific features and
modality-shared appearance features).

As shown in Fig. 4, in the proposed MTMFE sub-network, two levels of the
relation features (_i.e.,_ $\mathbf{F}_{I}^{1}$ and $\mathbf{F}_{I}^{2}$) are
extracted. Moreover, the first-level modality-invariant relation features
$\mathbf{F}_{I}^{1}$ are composed of two sub-parts (_i.e.,_
$\mathbf{F}_{I}^{1\\_1}$ and $\mathbf{F}_{I}^{1\\_2}$). Here,
$\mathbf{F}_{I}^{1\\_1}$ is extracted from the modality-specific features
$\mathbf{F}$, while $\mathbf{F}_{I}^{1\\_2}$ is extracted from the first level
of modality-shared appearance features $\mathbf{F}_{a}^{1}$. The second-level
modality-invariant relation features $\mathbf{F}_{I}^{2}$ are directly
extracted from the first level of modality-invariant relation features
$\mathbf{F}_{I}^{1}$.

Figure 6: Illustration of the extraction of the modality-invariant relation
features. (a) A 3D features transferred from the modality-shared appearance
features. (b) Corresponding heat maps of the modality-shared appearance
features. (c) A 3D convolutional kernel. (d)-(f) Capturing the modality-
invariant relation features at different spatial locations by using the 3D
convolutional kernel. It can be seen that the spatial-wise and channel-wise
relation of the different appearance features are simultaneously captured by
using the 3D convolutional kernel. As a result, the relation of those spatial
information of different person parts are effectively extracted to reduce the
cross-modality variants and further enhance the discriminability of the
modality-shared features for cross-modality person Re-ID.

Specifically, for the extraction of $\mathbf{F}_{I}^{1\\_1}$, inspired by [22,
23], the modality-specific features $\mathbf{F}$ are first projected into a
shared 3D feature spaces by using the proposed Projecting features inTo 3D
feature space (PT3D) operation. Then, a 3D convolutional block is employed to
extract the relations among the modality-specific features. Here, as shown in
Fig. 5(a), the PT3D operation is implemented by first splitting the extracted
modality-specific features $\mathbf{F}$ into multiple groups along its channel
dimension, where each group contains $D$ channels of the features (_i.e.,_
total $\frac{C}{D}$ groups). Then, each group is constructed as a 3D feature
of size $D\times H\times W$. In this way, the extracted modality-specific
features are projected into a shared 3D feature space. Here, as in [22, 23],
the 3D feature space is actually a pseudo 3D feature space due to the fact
that the features are not extracted from the real 3D data, liking videos or
medical images, but projected from the extracted 2D features.

In this shared 3D feature space, the spatial- and channel-wise relations among
the modality-specific features are simultaneously established for cross-
modality person Re-ID by using two stacked 3D convolutional layers with kernel
size of $3\times 3\times 3$. For better understanding, Fig. 6(a) can be seen
as a 3D feature transferred from a group of modality-shared appearance
features. Fig. 6(b) demonstrates that the modality-shared appearance features
contain the spatial information of different person parts. As shown in Fig.
6(d)-(f), the relations of the modality-shared appearance features
corresponding to different person parts can be captured by performing a
3$\times$3$\times$3 3D convolutional kernel (_e.g.,_ Fig. 6(c)) in the shared
3D feature space. Furthermore, more modality-invariant relation features can
be extracted by employing more 3D convolutional kernels. Mathematically, this
process is expressed by:

$\mathbf{F}_{I}^{1\\_1}=\operatorname{Conv3d}(\operatorname{PT3D}(\mathbf{F}),\delta_{1}),$
(3)

where $\operatorname{Conv3d}(*,\delta_{1})$ denotes two stacked 3D
convolutional layers with their corresponding parameters $\delta_{1}$.
$\operatorname{PT3D}(*)$ denotes the projection operation.

Then, considering that the extracted modality-specific features and modality-
shared appearance features have different properties, the relations of the
modality-shared appearance features are also captured to increase the
diversity of the modality-invariant relation features and further enhance the
discriminability of modality-shared features for cross-modality person Re-ID
in this paper. For that, the second part of the modality-invariant relation
features in the first level (_i.e.,_ $\mathbf{F}_{I}^{1\\_2}$) are extracted
from the first level of modality-shared appearance features (_i.e.,_
$\mathbf{F}_{a}^{1}$) in the same way as in the previous step. Concretely,
$\mathbf{F}_{a}^{1}$ is first projected into the shared 3D feature space by
using the proposed PT3D operation and then fed into another two stacked 3D
convolutional layers to extract $\mathbf{F}_{I}^{1\\_2}$. Mathematically, this
process is expressed by:

$\mathbf{F}_{I}^{1\\_2}=\operatorname{Conv3d}(\operatorname{PT3D}(\mathbf{F}_{a}^{1}),\delta_{2}),$
(4)

where $\operatorname{Conv3d}(*,\delta_{2})$ denotes the two stacked 3D
convolutional layers with its corresponding parameters $\delta_{2}$.

Finally, the first level of modality-invariant relation features
$\mathbf{F}_{I}^{1}$ are obtained by the element-wise summation of
$\mathbf{F}_{I}^{1\\_1}$ and $\mathbf{F}_{I}^{1\\_2}$, _i.e.,_

$\mathbf{F}_{I}^{1}=\mathbf{F}_{I}^{1\\_1}+\mathbf{F}_{I}^{1\\_2}.$ (5)

Here, the element-wise summation is employed for its relatively lower
computational complexity and memory usage. Other fusion strategies (_e.g.,_
concatenation) may also be used.

The second level of modality-invariant relation features $\mathbf{F}_{I}^{2}$
are obtained by feeding $\mathbf{F}_{I}^{1}$ into the second 3D convolutional
block, which is expressed by

$\mathbf{F}_{I}^{2}=\operatorname{Conv3d}(\mathbf{F}_{I}^{1},\delta_{3}),$ (6)

where $\operatorname{Conv3d}(*,\delta_{3})$ denotes the 3D convolutional block
with its corresponding parameters $\delta_{3}$. Furthermore, the relations
among the second-level modality-shared appearance features
$\mathbf{F}_{a}^{2}$ may also be extracted by projecting the second-level
modality-shared appearance features into the shared 3D feature space. However,
our experimental results show that the performance of cross-modality person
Re-ID may not be further improved even if more modality-invariant relation
features are extracted. Therefore, the modality-invariant relation features
are not further extracted from the second-level of appearance features
$\mathbf{F}_{a}^{2}$ in our proposed model.

In this way, the relations of the modality-specific features and modality-
shared appearance features at different person parts are captured in a shared
3D features space, respectively. By virtue of modality-shared appearance
features and modality-invariant relation features, the diversity and
discriminability of the final modality-shared features are significantly
increased, thus effectively reducing cross-modality variations and intra-
modality variations for cross-modality person Re-ID.

Finally, the extracted modality-invariant relation features are projected back
into the 2D feature space and concatenated with the modality-shared appearance
features in the same levels to facilitate the subsequent steps and meet the
requirement of the loss function. In this way, the modality-shared features in
different levels are thus obtained, _i.e.,_

$\mathbf{F}_{s}^{1}=\operatorname{Cat}(\mathbf{F}_{a}^{1},\operatorname{PT2D}(\mathbf{F}_{I}^{1})),\mathbf{F}_{s}^{2}=\operatorname{Cat}(\mathbf{F}_{a}^{2},\operatorname{PT2D}(\mathbf{F}_{I}^{2})),$
(7)

where $\mathbf{F}_{s}^{1}$ and $\mathbf{F}_{s}^{2}$ denote the final modality-
shared features. $\operatorname{PT2D}(*)$, as shown in Fig. 5(b), denotes the
projection operation that projects the 3D features back into the 2D feature
space, which is the reverse operation of $\operatorname{PT3D}(*)$.

###  III-C Part-Aligned Block

Given the extracted modality-shared features $\mathbf{F}_{s}^{1}$ and
$\mathbf{F}_{s}^{2}$, a part-aligned block is employed to extract the final
person features from different person parts. Specifically, as shown in Fig. 4,
following the PCB-based model [45, 12], the extracted modality-shared features
$\mathbf{F}_{s}^{1}$ and $\mathbf{F}_{s}^{2}$ are separated into six parts,
respectively. For the features within each part, a Global Average Pooling
(GAP) is employed to obtain the global information of each part, Then, a fully
connected layer is further employed to obtain the final feature embedding for
representing the corresponding part of a given image. Mathematically, this
process is expressed by:

$\mathbf{\hat{F}}_{p,1}^{i},...,\mathbf{\hat{F}}_{p,6}^{i}=\operatorname{Part}(\mathbf{F}_{s}^{i}),$
(8)
$\mathbf{F}_{p,k}^{i}=\operatorname{FC}(\operatorname{GAP}(\mathbf{\hat{F}}_{p,k}^{i}),\vartheta_{k}),$
(9)

where $k$=1,2,3,4,5,6 denotes different parts and $i$=1,2 denotes different
levels. $\operatorname{Part}(*)$ denotes the separation operation.
$\operatorname{FC}(*,\vartheta_{k})$ denotes the fully connected layers with
corresponding parameters $\vartheta_{k}$. $\operatorname{GAP}(*)$ denotes the
global average pooling. $\mathbf{F}_{p,k}^{i}$ and
$\mathbf{\hat{F}}_{p,k}^{i}$ denote the features from $k$-th part of the
$i$-th level features, respectively.

Furthermore, the features $\mathbf{F}_{p,k}^{i}$ from different person parts
should be discriminative for different persons. For that, the features
$\mathbf{F}_{p,k}^{i}$ are employed to predict the corresponding person
identification. Concretely, for the features within each part, an FC layer is
performed to predict which person identification it belongs to.
Mathematically, this process is expressed by:

$cls_{p,k}^{i}=\operatorname{FC}(\mathbf{F}_{p,k}^{i},\vartheta_{c_{k}^{i}}),$
(10)

where $cls_{p,k}^{i}$ denotes the person identification classification score
derived from the $k$-th part in the $i$-th level of features.
$\operatorname{FC}(*,\vartheta_{c_{k}^{i}})$ denotes the fully connected
layers with the corresponding parameters $\vartheta_{c_{k}^{i}}$.

###  III-D Loss Function

Figure 7: Illustration of the proposed cross-modality quadruplet loss. (a) The
proposed cross-modality quadruplet loss; (b) Single-modality triplet loss.
Here, $d_{i},i=1,2,3,4$ denotes the distance of two features. For example,
$d_{1}$ represents the distance of the features extracted from the images of
anchor-positive pairs in different modalities. Meanwhile, the smaller value of
$d_{i}$ represents that the two features become more similar.

After extracting person features from the input visible or thermal images by
employing the proposed network, a cross-modality quadruplet loss is further
designed to supervise the feature learning objectives. Furthermore, the intra-
modality variations are also reduced by employing a single-modality triplet
loss in this paper.

Suppose that, given an anchor sample, $\mathbf{F}_{an}^{R}$ and
$\mathbf{F}_{an}^{I}$ denote its features extracted from the corresponding RGB
and thermal images, respectively. Similarly, $\mathbf{F}_{pt}^{R}$ and
$\mathbf{F}_{pt}^{I}$ denote the features extracted from corresponding
positive RGB and thermal image samples, while $\mathbf{F}_{ng}^{R}$ and
$\mathbf{F}_{ng}^{I}$ denote the features extracted from corresponding
negative RGB and thermal image samples. The details of our losses are
discussed as follows.

####  III-D1 Cross-modality Quadruplet Loss

Many existing works employ the Bi-directional Dual-constrained Top-Ranking
(BDTR) loss or its variants [13, 17, 27, 28] to train their cross-modality
person Re-ID models. Following the BDTR loss (_i.e.,_ shown in Eq. 11), these
losses only constrain that the distance of an anchor sample to its farthest
cross-modality positive sample should be smaller than the anchor sample to its
nearest cross-modality negative sample by a predefined margin (_i.e.,_
$d_{1}<d_{2}$ in Fig. 7(a)).

$\begin{split}{\zeta}_{bi}=&\sum\operatorname{max}[\rho_{1}+\operatorname{D}(\mathbf{F}_{an}^{R},\mathbf{F}_{pt}^{I})-\operatorname{D}(\mathbf{F}_{an}^{R},\mathbf{F}_{ng}^{I})]\\\
&+\sum\operatorname{max}[\rho_{1}+\operatorname{D}(\mathbf{F}_{an}^{I},\mathbf{F}_{pt}^{R})-\operatorname{D}(\mathbf{F}_{an}^{I},\mathbf{F}_{ng}^{R})],\end{split}$
(11)

where $\rho_{1}$ denotes the corresponding margin. $\operatorname{D}(*)$
denotes the distance function, which default is the squared Euclidean
distance, _i.e.,_

$\operatorname{D}(\mathbf{X},\mathbf{Y})=\frac{1}{2}\parallel\mathbf{X}-\mathbf{Y}\parallel^{2}_{2},$
(12)

where $\mathbf{X}$ and $\mathbf{Y}$ denote the input features.

However, as shown in Fig. 7(a), the cross-modality discrepancy may not be
fully reduced by only employing the BDTR loss or its variants. They may ignore
another aspect that is also caused by modality differences, _i.e.,_ the
distance of an anchor sample to its farthest cross-modality positive sample
may also be larger than the anchor sample to its nearest intra-modality
negative sample (_i.e.,_ $d_{1}>d_{3}$ in 7(a)). Considering that, a novel
Cross-modality Quadruplet (CQ) loss is designed to further reduce the cross-
modality discrepancy by simultaneously considering the following two aspects.
(1) The distance of an anchor sample to its farthest cross-modality positive
sample should be smaller than the anchor sample to its nearest cross-modality
negative sample by a predefined margin (_i.e.,_ $d_{1}<d_{2}$). (2) The
distance of an anchor sample to its farthest cross-modality positive sample
should also be smaller than the anchor sample to its nearest intra-modality
negative sample by a predefined margin (_i.e.,_ $d_{1}<d_{3}$). This is
expressed by:

$\begin{split}{\zeta}_{cq}=&\sum\operatorname{max}[\rho_{2}+\operatorname{D}(\mathbf{F}_{an}^{R},\mathbf{F}_{pt}^{I})-\operatorname{D}(\mathbf{F}_{an}^{R},\mathbf{F}_{ng}^{I})]\\\
&+\sum\operatorname{max}[\rho_{2}+\operatorname{D}(\mathbf{F}_{an}^{I},\mathbf{F}_{pt}^{R})-\operatorname{D}(\mathbf{F}_{an}^{I},\mathbf{F}_{ng}^{R})]\\\
&+\sum\operatorname{max}[\rho_{2}+\operatorname{D}(\mathbf{F}_{an}^{R},\mathbf{F}_{pt}^{I})-\operatorname{D}(\mathbf{F}_{an}^{R},\mathbf{F}_{ng}^{R})]\\\
&+\sum\operatorname{max}[\rho_{2}+\operatorname{D}(\mathbf{F}_{an}^{I},\mathbf{F}_{pt}^{R})-\operatorname{D}(\mathbf{F}_{an}^{I},\mathbf{F}_{ng}^{I})].\end{split}$
(13)

Here, $\rho_{2}$ denotes the corresponding margin. Note that all the input
features are $l_{2}$ normalized for stable convergence.

####  III-D2 Single-modality Triplet Loss

Besides the proposed CQ loss, single-modality triplet loss [8, 33] is also
employed to reduce the large intra-modality variations. As shown in Fig. 7(b),
single-modality triple loss tries to let the distance of an anchor sample to
its farthest intra-modality positive sample smaller than the anchor sample to
its nearest intra-modality negative sample by a predefined margin. This is
expressed by

$\begin{split}{\zeta}_{st}=&\sum\operatorname{max}[\rho_{3}+\operatorname{D}(\mathbf{F}_{an}^{R},\mathbf{F}_{pt}^{R})-\operatorname{D}(\mathbf{F}_{an}^{R},\mathbf{F}_{ng}^{R})]\\\
&+\sum\operatorname{max}[\rho_{3}+\operatorname{D}(\mathbf{F}_{an}^{I},\mathbf{F}_{pt}^{I})-\operatorname{D}(\mathbf{F}_{an}^{I},\mathbf{F}_{ng}^{I})].\end{split}$
(14)

Here, $\rho_{3}$ denotes the corresponding margin.

####  III-D3 Identification loss

Finally, an identification loss is employed to make sure that those extracted
features are discriminative for pedestrians, which is expressed by:

${\zeta}_{id}=-\frac{1}{N_{id}}\sum_{c=1}^{N_{id}}q_{c}\operatorname{log}(p_{c}),$
(15)

where $N_{id}$ denotes the total numbers of person identifications. $q_{c}$
and $p_{c}$ denote the ground truth and the predicted probability of being the
$c$-th identification, respectively. In our proposed model, for features of
each part in each level, the identification loss is employed to facilitate the
proposed model to learn more discriminative person features.

Total Loss for Training: In the training process, the anchor samples
($I_{an}^{R}$ and $I_{an}^{I}$), positive samples ($I_{pt}^{R}$ and
$I_{pt}^{I}$) and negative samples ($I_{ng}^{R}$ and $I_{ng}^{I}$) are
sampled. After feeding these samples into the proposed model, corresponding
person part features are obtained, _i.e.,_ $\mathbf{F}_{an,k}^{R,i}$,
$\mathbf{F}_{an,k}^{I,i}$, $\mathbf{F}_{pt,k}^{R,i}$,
$\mathbf{F}_{pt,k}^{I,i}$, $\mathbf{F}_{ng,k}^{R,i}$ and
$\mathbf{F}_{ng,k}^{I,i}$. Here, $k$=1,2,3,4,5,6 denotes different parts and
$i$=1,2 denotes different levels. Meanwhile, the classification scores for the
anchor images (_i.e.,_ $p_{an,k}^{R,i}$ and $p_{an,k}^{I,i}$) are also
obtained. Based on that, the total loss ${\zeta}$ for the anchor images in the
training process is expressed by

$\begin{split}{\zeta}=\sum^{2}_{i=1}&\sum^{6}_{k=1}({\zeta}_{cq}(\mathbf{F}_{an,k}^{R,i},\mathbf{F}_{an,k}^{I,i},\mathbf{F}_{pt,k}^{R,i},\mathbf{F}_{pt,k}^{I,i},\mathbf{F}_{ng,k}^{R,i},\mathbf{F}_{ng,k}^{I,i})\\\
&+{\zeta}_{st}(\mathbf{F}_{an,k}^{R,i},\mathbf{F}_{an,k}^{I,i},\mathbf{F}_{pt,k}^{R,i},\mathbf{F}_{pt,k}^{I,i},\mathbf{F}_{ng,k}^{R,i},\mathbf{F}_{ng,k}^{I,i})\\\
&+{\zeta}_{id}(p_{an,k}^{R,i},q_{id})+{\zeta}_{id}(p_{an,k}^{I,i},q_{id})),\end{split}$
(16)

where $q_{id}$ denotes the ground truth person identification.

##  IV Experiments

###  IV-A Datasets and Evaluation Metrics

Datasets: To evaluate the performance of our proposed network, we conduct
evaluations on two public cross-modality person Re-ID benchmarks: SYSU-MM01
[46] and RegDB [47].

SYSU-MM01 is a large-scale and widely used cross-modality person Re-ID dataset
[46]. Four visible cameras and two infrared cameras are employed to collect
RGB images and IR images from both indoor and outdoor environments. Its
training set contains 395 person identities, including totally 22,258 RGB
images and 11,909 IR images. Furthermore, the testing set of SYSU-MM01
contains 96 person identities with 3,803 IR images for query and 301/3010
(one-shot/multi-shot) randomly selected RGB images as the gallery. There are
two evaluation modes in SYSU-MM01 for cross-modality person Re-ID: indoor-
search and all-search [46].

RegDB is another widely used cross-modality person Re-ID dataset [47]. It
totally contains 412 person identities and 8240 images collected by dual
camera systems, where 206 identities are employed for training and the rest of
206 identities are employed for testing. For each person identity, 10 RGB
images and 10 IR images under different poses, positions and illuminations are
available. There are also two evaluation modes for RegDB. One is RGB-IR
matching for searching IR images from a given RGB images. The other mode is
IR-RGB matching for searching RGB images from a given IR image.

Evaluation metrics: Following existing works [12, 13, 15, 48], Cumulated
Matching Characteristics (CMC) and mean Average Precision (mAP) are adopted as
the evaluation metrics. Here, CMC (_i.e.,_ Rank-r accuracy) measures the
probability of a correct cross-modality person image occurs in the top-r
retrieved results. mAP measures the retrieval performance when multiple
matching images occur in the gallery set.

###  IV-B Online Batch Sampling Strategy

Our online batch sampling strategy is discussed in this subsection.
Specifically, for each batch in a training iteration, $N$ person identities
are first selected from training sets randomly. Then, for each selected person
identity, $K$ corresponding RGB images and $K$ IR images are randomly
selected. As a result, each training batch contains $N\times K$ RGB images and
$N\times K$ IR images (_i.e.,_ totally $2\times N\times K$ images) for
training. Here, we set $N=8$ and $K=4$ during our training process.

In the testing strategy, the features of all the query images are first
obtained by feeding all the query images into our model. Similarly, the
features of all the gallery images are then obtained by feeding all the
gallery images into our model. After that, the pairwise distances between the
query and gallery images are calculated and the corresponding quantitative
results are then obtained.

###  IV-C Implementation details

The proposed model is implemented on an NVIDIA 1080Ti GPU by using the PyTorch
repository [49]. The parameters of the modality-specific extraction sub-
networks and the 2D CNN of the modality-shared feature extraction sub-networks
are initialized by using a pre-trained ResNet50 [44] on ImageNet [50]. Other
parameters are randomly initialized by using the Xavier initialization [51].
All the experiments are performed by using the SGD optimizer with the weight
decay of 0.0005. The initial learning rates for the parameters of modality-
specific features extraction sub-networks are set to 0.01 and for those of
modality-shared feature extraction sub-networks are set to 0.1. The learning
rates are decreased by a factor of 0.1 for every 7 epochs. In the training
process, all the sampled images are resized to the $288\times 144$ by
employing the $\operatorname{bilinear}$ operation. Meanwhile, some data
augmentations, such as random flipping and cropping, are also employed.

Figure 8: The top-10 retrieved results of some example queries with the
proposed method on the SYSU-MM01 and RegDB datasets. The green bounding boxes
indicate the correct matchings and red bounding boxes represent wrong
matchings (best viewed in color). Examples in the first two rows are sampled
from SYSU-MM01 dataset which takes the IR images as queries to search from the
RGB images. Examples in the third and fourth rows are sampled from RegDB
dataset which takes RGB images as queries to search from IR images. The last
two rows show some failure cases.

###  IV-D Visualization

The top ten retrieved results of some randomly selected query examples on the
SYSU-MM01 and RegDB datasets are shown in Fig. 8. We can see that cross-
modality person Re-ID is an extremely challenging task and it is even hard for
human to tell which person is the correct matching of the query. Then, there
are some wrongly retrieved examples in the ranking list, but the top-ranked
images are usually quite close to the query image due to similar appearances
and relation information. Meanwhile, as shown in the last two rows of Fig. 8,
for those failure cases, the top matches in the ranking list are indeed very
similar to the query person.

###  IV-E Ablation study

TABLE I: quantitative results of different ablation experiments. Methods | ML | P | RF-only | RF | CQ | r1 | r10 | MAP
---|---|---|---|---|---|---|---|---
1 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | 16.43 | 51.84 | 18.50
2 | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | 18.25 | 57.38 | 19.81
3 | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | 45.36 | 85.22 | 45.28
4 | ✓ | ✓ | $\times$ | $\times$ | $\times$ | 48.15 | 77.78 | 48.35
5 | ✓ | ✓ | ✓ | $\times$ | $\times$ | 47.17 | 88.25 | 46.29
6 | ✓ | ✓ | $\times$ | ✓ | $\times$ | 60.66 | 91.61 | 58.28
7 | ✓ | ✓ | $\times$ | ✓ | ✓ | 62.56 | 93.85 | 60.57
TABLE II: Comparison with the state-of-the-art models on SYSU-MM01 dataset.

- | All-Search | Indoor-Search
---|---|---
- | Single-shot | Multi-shot | Single-shot | Multi-shot
Methods | R1 | R10 | R20 | mAP | R1 | R10 | R20 | mAP | R1 | R10 | R20 | mAP | R1 | R10 | R20 | mAP
HOG [52] | 2.76 | 18.25 | 31.91 | 4.24 | 3.82 | 22.77 | 37.63 | 2.16 | 3.22 | 24.68 | 44.52 | 7.25 | 4.75 | 29.06 | 49.38 | 3.51
LOMO [53] | 3.64 | 23.18 | 37.28 | 4.53 | 4.70 | 28.22 | 43.05 | 2.28 | 5.75 | 34.35 | 54.90 | 10.19 | 7.36 | 40.38 | 60.33 | 5.64
One-stream Network [36] | 12.04 | 49.68 | 66.74 | 13.67 | 16.26 | 58.14 | 75.05 | 8.59 | 16.94 | 63.55 | 82.10 | 22.95 | 22.62 | 71.74 | 87.82 | 15.04
Two-stream Network [36] | 11.65 | 47.99 | 65.50 | 12.85 | 16.33 | 58.35 | 74.46 | 8.03 | 15.60 | 61.18 | 81.02 | 21.49 | 22.49 | 72.22 | 88.61 | 13.92
Zero-padding [36] | 14.80 | 54.12 | 71.33 | 15.95 | 19.13 | 61.40 | 78.41 | 10.89 | 20.58 | 68.38 | 85.79 | 26.92 | 24.43 | 75.86 | 91.32 | 18.64
MLBP [54] | 2.1 | 16.2 | 28.3 | 3.9 | - | - | - | - | - | - | - | - | - | - | - | -
TONE[20] | 12.5 | 50.7 | 68.6 | 14.4 | - | - | - | - | - | - | - | - | - | - | - | -
HCML[20] | 14.3 | 53.2 | 69.2 | 16.2 | - | - | - | - | - | - | - | - | - | - | - | -
BDTR [17] | 17.0 | 55.4 | 72.0 | 19.7 | - | - | - | - | 26.8 | 73.2 | 87.6 | 37.6 | - | - | - | -
cmGAN [15] | 27.0 | 67.5 | 80.6 | 27.8 | 31.7 | 77.2 | 89.2 | 42.4 | - | - | - | - | - | - | - | -
GSM [55] | 5.29 | 33.71 | 52.95 | 8.00 | 6.19 | 37.15 | 55.66 | 4.38 | 9.46 | 48.98 | 72.06 | 15.57 | 11.36 | 51.34 | 73.41 | 9.03
D-HSME [19] | 20.7 | 62.8 | 78.0 | 23.2 | - | - | - | - | - | - | - | - | - | - | - | -
IPVT+MSR[56] | 23.2 | 51.2 | 61.7 | 22.5 | - | - | - | - | - | - | - | - | - | - | - | -
D2RL [57] | 28.9 | 70.6 | 82.4 | 29.2 | - | - | - | - | - | - | - | - | - | - | - | -
DBT [18] | 29.05 | 74.71 | 87.16 | 30.94 | 35.40 | 81.02 | 91.85 | 24.12 | 32.74 | 82.40 | 93.35 | 44.26 | 40.41 | 86.83 | 96.27 | 33.93
CC-S [10] | 33.4 | 78.6 | 89.4 | 37.2 | - | - | - | - | - | - | - | - | - | - | - | -
CC-F [10] | 35.1 | 77.6 | 88.9 | 37.4 | - | - | - | - | - | - | - | - | - | - | - | -
DGD_MSR[9] | 37.35 | 83.40 | 93.34 | 38.11 | 43.86 | 86.94 | 95.68 | 30.48 | 39.64 | 89.29 | 97.66 | 50.88 | 46.56 | 93.57 | 98.80 | 40.08
AlignGAN[38] | 42.4 | 85.0 | 93.7 | 40.7 | 51.5 | 89.4 | 95.7 | 33.9 | 45.9 | 87.6 | 94.4 | 54.3 | 57.1 | 92.7 | 97.4 | 45.3
eBDTR [13] | 27.8 | 67.3 | 81.3 | 28.4 | - | - | - | - | 32.4 | 77.4 | 89.6 | 42.4 | - | - | - | -
Hi-CMD[58] | 34.94 | 77.58 | - | 35.94 | - | - | - | - | - | - | - | - | - | - | - | -
EDFL [16] | 36.9 | 84.5 | 93.2 | 40.7 | - | - | - | - | - | - | - | - | - | - | - | -
CMPG [36] | 38.1 | 80.7 | 89.9 | 36.9 | 45.1 | 85.7 | 93.8 | 29.5 | 43.8 | 86.2 | 94.2 | 52.9 | 52.7 | 91.1 | 96.4 | 42.7
ECMC [59] | 30.26 | 75.59 | 88.13 | 33.38 | - | - | - | - | - | - | - | - | - | - | - | -
BEAT [60] | 38.57 | 76.64 | 86.39 | 38.61 | 44.71 | 69.82 | 77.87 | 32.20 | - | - | - | - | - | - | - | -
HPILN[28] | 41.36 | 84.78 | 94.51 | 42.95 | 47.56 | 88.13 | 95.98 | 36.08 | 45.77 | 91.82 | 98.46 | 56.52 | 53.05 | 93.71 | 98.93 | 47.48
XM [11] | 49.9 | 89.7 | 95.9 | 50.7 | - | - | - | - | - | - | - | - | - | - | - | -
TSGAN(w/o rerank) [61] | 49.8 | 87.3 | 93.8 | 47.4 | 56.1 | 90.2 | 96.3 | 38.5 | 50.4 | 90.8 | 96.8 | 63.1 | 59.3 | 91.2 | 97.8 | 50.2
TSGAN(w/ rerank) [61] | 58.9 | 87.8 | 94.1 | 55.1 | 55.9 | 91.2 | 96.6 | 39.7 | 62.1 | 90.8 | 96.4 | 71.3 | 59.7 | 91.8 | 97.9 | 50.9
ABP [48] | 51.56 | 75.65 | 81.69 | 32.50 | - | - | - | - | - | - | - | - | - | - | - | -
HATML [62] | 55.29 | 92.41 | 97.36 | 53.89 | - | - | - | - | 62.10 | 95.75 | 99.20 | 69.37 | - | - | - | -
HC [12] | 56.96 | 91.50 | 96.82 | 54.95 | 62.09 | 93.74 | 97.85 | 48.02 | 59.74 | 92.07 | 96.22 | 64.91 | 69.76 | 95.85 | 98.90 | 57.81
DG-VAE [63] | 59.49 | 93.77 | - | 58.46 | - | - | - | - | - | - | - | - | - | - | - | -
our | 62.56 | 93.85 | 97.63 | 60.57 | 68.09 | 95.73 | 98.30 | 54.33 | 65.06 | 95.17 | 98.17 | 73.86 | 69.81 | 96.86 | 99.30 | 64.84

TABLE III: Comparison with the state-of-the-art models on RegDB dataset. - | Visible to Thermal | Thermal to Visible
---|---|---
Methods | R1 | R10 | R20 | mAP | R1 | R10 | R20 | mAP
HOG [52] | 13.5 | 33.2 | 43.7 | 10.3 | - | - | - |
MLBP [54] | 2.0 | 7.3 | 10.9 | 6.8 | - | - | - |
LOMO [53] | 0.9 | 2.5 | 4.1 | 2.3 | - | - | - |
GSM [55] | 17.3 | 34.5 | 45.3 | 15.1 | - | - | - |
One-stream Network [36] | 13.1 | 33.0 | 42.5 | 14.0 | - | - | - | -
Two-stream Network [36] | 12.4 | 30.4 | 41.0 | 13.4 | - | - | - | -
Zero-padding [36] | 17.8 | 34.2 | 44.4 | 18.9 | 16.63 | 34.68 | 44.25 | 17.82
TONE[20] | 16.9 | 34.0 | 44.1 | 14.9 | 13.86 | 30.08 | 40.05 | 16.98
HCML[20] | 24.4 | 47.5 | 56.8 | 20.8 | 21.70 | 45.02 | 55.58 | 22.24
D-HSME [19] | 50.85 | 73.36 | 81.66 | 47.00 | 50.15 | 72.40 | 81.07 | 46.16
BDTR [17] | 33.5 | 58.4 | 67.5 | 31.8 | 32.92 | 58.46 | 68.43 | 31.96
eBDTR [13] | 31.8 | 56.1 | 66.8 | 33.2 | 34.21 | 58.74 | 68.64 | 32.49
IPVT+MSR[56] | 58.76 | 85.75 | 90.27 | 47.85 | - | - | - | -
D2RL [57] | 43.4 | 66.1 | 76.3 | 44.1 | - | - | - | -
DBT [18] | 38.64 | 60.18 | 69.81 | 38.08 | - | - | - | -
CC-S [10] | 53.1 | 72.3 | 80.2 | 53.5 | - | - | - | -
CC-F [10] | 30.26 | 75.59 | 88.13 | 60.0 | - | - | - | -
ECMC [59] | 39.75 | 61.26 | 70.10 | 40.79 | - | - | - | -
BEAT [60] | 67.45 | - | - | 66.51 | 66.48 | - | - | 67.31
Hi-CMD[58] | 70.93 | 86.39 | - | 66.04 | - | - | - | -
EDFL [16] | 52.58 | 72.10 | 81.47 | 52.98 | - | - | - | -
DGD_MSR[9] | 48.43 | 70.32 | 79.59 | 48.67 | - | - | - | -
AlignGAN[38] | 56.3 | 53.4 | 57.9 | 53.6 |  |  |  |
XM [11] | 62.21 | 83.13 | 91.72 | 60.18 | 57.9 | - | - | 53.6
ABP [48] | 56.35 | 80.87 | 87.96 | 48.58 | 54.03 | 78.69 | 85.83 | 47.60
HATML [62] | 71.83 | 87.16 | 92.16 | 67.56 | 70.02 | 86.45 | 91.61 | 66.30
DG-VAE [63] | 72.97 | 86.89 | - | 71.78 | - | - | - | -
our | 76.10 | 88.86 | 92.41 | 74.39 | 72.18 | 87.06 | 92.38 | 71.04

The effectiveness of each component in our proposed model is studied in this
subsection. As shown in Table I, different versions of our proposed model are
designed for comparisons. Specifically, the modality-invariant relation
features, multi-level features and PCB-based part-aligned block are first
removed from the proposed model as the Baseline model (_i.e.,_ the first line
of Table I). Then, the multi-level features are represented as ‘ML’ and the
PCB-based part-aligned block is represented as ‘P’. ‘RF’ denotes that the
modality-invariant relation features are employed for cross-modality person
Re-ID. ‘RF-only’ represents that only modality-invariant relation features are
employed, while modality-shared appearance features are not extracted for
cross-modality person Re-ID. Finally, the proposed cross-modality quadruplet
loss is represented as ‘CQ’. It should be noted that these models are trained
by using the joint loss of classification loss, bi-directional dual-
constrained top-ranking loss [13] and single-modality triplet loss, except for
the last model in Table I, in which the proposed cross-modality quadruplet
loss, instead of the bi-directional dual-constrained top-ranking loss, is
adopted for training.

As shown in the second and third rows of Table I, the performance of cross-
modality person Re-ID are boosted by employing the multi-level features and
the PCB-based algorithm. This results from that both multi-level features and
PCB-based algorithm can enhance the discriminability of the extracted
modality-shared features. Furthermore, the results shown in the fourth row
indicate that only employing the modality-shared appearance features for
cross-modality person Re-ID cannot achieve the optimal results. This may be
due to the fact that using modality-shared appearance features only cannot
capture enough discriminative information for identifying different persons
since the modality differences between RGB and IR images are large. Meanwhile,
the results in the fifth row of Table I indicate that employing modality-
invariant relation features only also obtains suboptimal results. This may
result from the fact the modality-shared appearance features also contain lots
of valuable information for identifying different persons. The modality-
invariant relation features can be employed as important complementary
information for modality-shared appearance features but cannot fully replace
the modality-shared appearance features. Furthermore, as shown in the sixth
row of Table I, the performance of cross-modality person Re-ID is
significantly boosted by simultaneously capturing the modality-shared
appearance features and modality-invariant relation features. This means that,
by virtue of modality-shared appearance features and modality-invariant
relation features, our proposed MTMFE sub-network can extract more modality-
invariant and discriminative modality-shared features to reduce cross-modality
variations and intra-modality variations, thus boosting the cross-modality
person Re-ID. As shown in the last row of Table I, the performance of our
model is further improved by employing the proposed cross-modality quadruplet
loss. This results from the fact that the proposed cross-modality quadruplet
loss provides stronger constraints to force our model to learn more
discriminative features in the training process.

###  IV-F Comparison with State-of-the-Art Methods

In this subsection, the proposed model is compared with some of the State-Of-
The-Art (SOTA) methods, including: HOG [52], MLBP [54], LOMO [53], GSM [55],
One-stream Network[36], Two-stream Network [36], Zero-padding [36], TONE[20],
HCML[20], D-HSME [19], BDTR [17], eBDTR [13], cmGAN [15], IPVT+MSR[56], D2RL
[57], DBT [18], CC [10], Hi-CMD[58], EDFL [16], DGD_MSR[9], CMPG [36],
AlignGAN[38], TSGAN[61], BEAT [60], ECMC [59], HPILN[28], XM [11], HC [12],
ABP [48] and HATML [62].

The quantitative results of these SOTA models on SYSU-MM01 [46] dataset are
shown in Table II. It can be seen that the proposed model outperforms other
SOTA methods by a large margin. Specifically, in all-search mode, the proposed
model surpasses SOAT models by 3.07% on Rank-1 accuracy and 2.11% on mAP in
the single-shot setting. Meanwhile, the multi-shot setting exhibits a similar
phenomenon. This indicates that the proposed model can extract highly
discriminative person features by jointly capturing modality-shared appearance
features and modality-invariant relation features. Furthermore, our proposed
model can obtain higher recall values than other SOTA models when the gallery
size increases. Meanwhile, the proposed model also obtains the best
performance in the indoor-search mode, which further demonstrates the
effectiveness and robustness of our proposed model.

The quantitative results of these SOTA models on RegDB dataset are shown in
Table III. It can be seen that, for both the visible-to-thermal mode and the
thermal-to-visible mode, the proposed model surpasses others by a large
margin. Meanwhile, the proposed model obtains similar mAP values in the
visible-to-thermal mode and thermal-to-visible mode. This also indicates that,
with the collaboration of modality-shared appearance features, modality-
invariant relation features and cross-modality quadruplet loss, the proposed
model can significantly reduce the cross-modality variations and intra-
modality variations for cross-modality person Re-ID. As a result, more
discriminative modality-shared features are extracted to boost the performance
of cross-modality person Re-ID.

##  V Conclusion

A novel cross-modality person Re-ID model has been presented in this paper.
Specifically, on the top of modality-shared appearance features, our proposed
MTMFE sub-network enhances the discriminability of the extracted modality-
shared features by further extracting modality-invariant relation features. By
virtue of the extracted modality-invariant relation features, the cross-
modality variations as well as the intra-modality variations are significantly
reduced. As a result, the performance of cross-modality person Re-ID is
greatly improved. After that, the cross-modality variations are further
reduced by employing the proposed cross-modality quadruplet loss. With the
collaboration of modality-shared appearance features, modality-invariant
relation features and cross-modality quadruplet loss, the proposed model
achieves new state-of-the-art experimental results on several benchmarks,
which validates the superiorities of our model over others.

## Acknowledgment

This work is supported by the National Natural Science Foundation of China
under Grant No. 61773301 and 61876140, and the China Postdoctoral Support
Scheme for Innovative Talents under Grant No. BX20180236.

## References

  * [1] X. Wang, “Intelligent multi-camera video surveillance: A review,” _Pattern Recognition Letters_ , vol. 34, pp. 3–19, 2013.
  * [2] N. McLaughlin, J. M. del Rincon, and P. Miller, “Video person re-identification for wide area tracking based on recurrent neural networks,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, pp. 2613–2626.
  * [3] L. Zheng, Y. Huang, H. Lu, and Y. Yang, “Pose-invariant embedding for deep person re-identification,” _IEEE Transactions on Image Processing_ , vol. 28, no. 9, pp. 4500–4509, 2019.
  * [4] Z. Feng, J. Lai, and X. Xie, “Learning view-specific deep networks for person re-identification,” _IEEE Transactions on Image Processing_ , vol. 27, no. 7, pp. 3472–3483, 2018.
  * [5] Z. Zheng, L. Zheng, and Y. Yang, “Pedestrian alignment network for large-scale person re-identification,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 10, pp. 3037–3045, 2018.
  * [6] B. Nguyen and B. De Baets, “Kernel distance metric learning using pairwise constraints for person re-identification,” _IEEE Transactions on Image Processing_ , vol. 28, no. 2, pp. 589–600, 2018.
  * [7] Y.-J. Cho and K.-J. Yoon, “Pamm: Pose-aware multi-shot matching for improving person re-identification,” _IEEE Transactions on Image Processing_ , vol. 27, no. 8, pp. 3739–3752, 2018.
  * [8] A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” _ArXiv_ , vol. abs/1703.07737, 2017.
  * [9] Z. Feng, J. Lai, and X. Xie, “Learning modality-specific representations for visible-infrared person re-identification,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 579–590, 2019.
  * [10] S. Zhang, Y. Yang, P. Wang, X. Zhang, and Y. Zhang, “Attend to the difference: Cross-modality person re-identification via contrastive correlation,” _arXiv preprint arXiv:1910.11656_ , 2019.
  * [11] D. Li, X. Wei, X. Hong, and Y. Gong, “Infrared-visible cross-modal person re-identification with an X modality,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2020, pp. 4610–4617.
  * [12] Y. Zhu, Z. Yang, L.-C. Wang, S. Zhao, X. Hu, and D. Tao, “Hetero-center loss for cross-modality person re-identification,” _Neurocomputing_ , vol. 386, pp. 97–109, 2020.
  * [13] M. Ye, X. Lan, Z. Wang, and P. C. Yuen, “Bi-directional center-constrained top-ranking for visible thermal person re-identification,” _IEEE Transactions on Information Forensics and Security_ , vol. 15, pp. 407–419, 2020\.
  * [14] E. Basaran, M. Gökmen, and M. E. Kamasak, “An efficient framework for visible-infrared cross modality person re-identification,” _Signal Processing: Image Communication_ , p. 115933, 2020.
  * [15] P. Dai, R. Ji, H. Wang, Q. Wu, and Y. Huang, “Cross-modality person re-identification with generative adversarial training,” in _Proceedings of the International Joint Conference on Artificial Intelligence_ , 2018, pp. 677–683.
  * [16] H. Liu, J. Cheng, W. Wang, Y. Su, and H. Bai, “Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification,” _Neurocomputing_ , vol. 398, pp. 11–19, 2020.
  * [17] M. Ye, Z. Wang, X. Lan, and P. C. Yuen, “Visible thermal person re-identification via dual-constrained top-ranking,” in _Proceedings of the International Joint Conference on Artificial Intelligence_ , 2018, pp. 1092–1099.
  * [18] N. Tekeli and A. B. Can, “Distance based training for cross-modality person re-identification,” in _Proceedings of the IEEE International Conference on Computer Vision Workshop_ , 2019, pp. 4540–4549.
  * [19] Y. Hao, N. Wang, L. Jie, and X. Gao, “HSME: Hypersphere manifold embedding for visible thermal person re-identification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2019, pp. 8385–8392.
  * [20] M. Ye, X. Lan, J. Li, and P. C. Yuen, “Hierarchical discriminative learning for visible thermal person re-identification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018, pp. 7501–7508.
  * [21] J. Zhou, B. Su, and Y. Wu, “Easy identification from better constraints: Multi-shot person re-identification from reference constraints,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5373–5381.
  * [22] Z. Qiu, T. Yao, and T. Mei, “Learning spatio-temporal representation with pseudo-3d residual networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 5533–5541.
  * [23] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6450–6459.
  * [24] S. Ji, W. Xu, M. Yang, and K. Yu, “3D convolutional neural networks for human action recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 1, pp. 221–231, 2013.
  * [25] M. Sabokrou, M. Fayyaz, M. Fathy, and R. Klette, “Deep-cascade: Cascading 3D deep neural networks for fast anomaly detection and localization in crowded scenes,” _IEEE Transactions on Image Processing_ , vol. 26, no. 4, pp. 1992–2004, 2017.
  * [26] D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 4489–4497.
  * [27] Y. Lu, Y. Wu, B. Liu, T. Zhang, B. Li, Q. Chu, and N. Yu, “Cross-modality person re-identification with shared-specific feature transfer,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 379–13 389.
  * [28] Y.-B. Zhao, J.-W. Lin, Q. Xuan, and X. Xi, “Hpiln: a feature learning framework for cross-modality person re-identification,” _IET Image Processing_ , vol. 13, no. 14, pp. 2897–2904, 2019.
  * [29] S. Karanam, M. Gou, Z. Wu, A. Rates-Borras, O. I. Camps, and R. J. Radke, “A systematic evaluation and benchmark for person re-identification: Features, metrics, and datasets,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, pp. 523–536, 2019.
  * [30] M. A. Saghafi, A. Hussain, H. B. Zaman, and M. H. M. Saad, “Review of person re-identification techniques,” _IET Computer Vision_ , vol. 8, no. 6, pp. 455–474, 2014.
  * [31] Z. Feng, J. Lai, and X. Xie, “Learning view-specific deep networks for person re-identification,” _IEEE Transactions on Image Processing_ , vol. 27, no. 7, pp. 3472–3483, 2018.
  * [32] H. Yao, S. Zhang, R. Hong, Y. Zhang, C. Xu, and Q. Tian, “Deep representation learning with part loss for person re-identification,” _IEEE Transactions on Image Processing_ , vol. 28, no. 6, pp. 2860–2871, 2019.
  * [33] W. Chen, X. Chen, J. Zhang, and K. Huang, “Beyond triplet loss: A deep quadruplet network for person re-identification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1320–1329.
  * [34] J. Chen, Z. Zhang, and Y. Wang, “Relevance metric learning for person re-identification by exploiting listwise similarities,” _IEEE Transactions on Image Processing_ , vol. 24, no. 12, pp. 4741–4755, 2015.
  * [35] M. Ye, X. Lan, and Q. Leng, “Modality-aware collaborative learning for visible thermal person re-identification,” in _Proceedings of the ACM International Conference on Multimedia_ , 2019, pp. 347–355.
  * [36] Y. Yang, T. Zhang, J. Cheng, Z. Hou, P. Tiwari, H. M. Pandey _et al._ , “Cross-modality paired-images generation and augmentation for RGB-Infrared person re-identification,” _Neural Networks_ , vol. 128, pp. 294–304, 2020\.
  * [37] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _The proceedings of the Neural Information Processing Systems Conference_ , 2012, pp. 1097–1105.
  * [38] G. Wang, T. Zhang, J. Cheng, S. Liu, Y. Yang, and Z. Hou, “RGB-Infrared cross-modality person re-identification via joint pixel and feature alignment,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3622–3631.
  * [39] Z. Sun, X. Wang, Q. Zhang, and J. Jiang, “Real-time video saliency prediction via 3D residual convolutional neural network,” _IEEE Access_ , vol. 7, pp. 147 743–147 754, 2019.
  * [40] T. M. Lee, J.-C. Yoon, and I.-K. Lee, “Motion sickness prediction in stereoscopic videos using 3D convolutional neural networks,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 25, no. 5, pp. 1919–1927, 2019.
  * [41] S. Ji, W. Xu, M. Yang, and K. Yu, “3D convolutional neural networks for human action recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 1, pp. 221–231, 2012.
  * [42] Y. Xiang, H. Liu, S. Wang, L. Ma, X. Xiong, C. Xu, and D. Shao, “Segmentation method of multiple sclerosis lesions based on 3d-cnn networks,” _IET Image Processing_ , pp. 1806–1812, 2020.
  * [43] Q. Dou, H. Chen, L. Yu, L. Zhao, J. Qin, D. Wang, V. C. Mok, L. Shi, and P.-A. Heng, “Automatic detection of cerebral microbleeds from mr images via 3d convolutional neural networks,” _IEEE Transactions on Medical Imaging_ , vol. 35, no. 5, pp. 1182–1195, 2016.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 770–778.
  * [45] Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, “Beyond part models: Person retrieval with refined part pooling and a strong convolutional baseline,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 501–518.
  * [46] A. Wu, W.-S. Zheng, H.-X. Yu, S. Gong, and J.-H. Lai, “RGB-Infrared cross-modality person re-identification,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 5390–5399.
  * [47] D. T. Nguyen, H. G. Hong, K. W. Kim, and K. R. Park, “Person recognition system based on a combination of body images from visible light and thermal cameras,” _Sensors_ , vol. 17, no. 3, p. 605, 2017.
  * [48] Z. Wei, X. Yang, N. Wang, B. Song, and X. Gao, “ABP: Adaptive body partition model for visible infrared person re-identification,” in _Proceedings of the IEEE International Conference on Multimedia and Expo_ , 2020, pp. 1–6.
  * [49] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Proceedings of the Neural Information Processing Systems_ , 2019, pp. 8026–8037.
  * [50] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, “ImageNet: A large-scale hierarchical image database,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [51] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _Proceedings of the International Conference on Artificial Intelligence and Statistics_ , 2010, pp. 249–256.
  * [52] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , vol. 1, 2005, pp. 886–893.
  * [53] S. Liao, X. Zhu, Z. Lei, L. Zhang, and S. Z. Li, “Learning multi-scale block local binary patterns for face recognition,” in _Proceedings of the International Conference on Biometrics_ , 2007, pp. 828–837.
  * [54] S. Liao and S. Z. Li, “Efficient PSD constrained asymmetric metric learning for person re-identification,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 3685–3693.
  * [55] L. Lin, G. Wang, W. Zuo, X. Feng, and L. Zhang, “Cross-domain visual matching via generalized similarity measure and feature learning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, pp. 1089–1102, 2017.
  * [56] J. K. Kang, T. M. Hoang, and K. R. Park, “Person re-identification between visible and thermal camera images based on deep residual CNN using single input,” _IEEE Access_ , vol. 7, pp. 57 972–57 984, 2019.
  * [57] Z. Wang, Z. Wang, Y. Zheng, Y.-Y. Chuang, and S. Satoh, “Learning to reduce dual-level discrepancy for infrared-visible person re-identification,” in _Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 618–626.
  * [58] S. Choi, S. Lee, Y. Kim, T. Kim, and C. Kim, “Hi-CMD: Hierarchical cross-modality disentanglement for visible-infrared person re-identification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 257–10 266.
  * [59] D. Cheng, X. Li, M. Qi, X. Liu, C. Chen, and D. Niu, “Exploring cross-modality commonalities via dual-stream multi-branch network for infrared-visible person re-identification,” _IEEE Access_ , vol. 8, pp. 12 824–12 834, 2020\.
  * [60] H. Ye, H. Liu, F. Meng, and X. Li, “Bi-directional exponential angular triplet loss for rgb-infrared person re-identification,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 1583–1595, 2020.
  * [61] Z. Zhang, S. Jiang, C. Huang, Y. Li, and R. Y. Da Xu, “RGB-IR cross-modality person reid based on teacher-student GAN model,” _arXiv preprint arXiv:2007.07452_ , 2020.
  * [62] M. Ye, J. Shen, and L. Shao, “Visible-infrared person re-identification via homogeneous augmented tri-modal learning,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 728–739, 2020.
  * [63] N. Pu, W. Chen, Y. Liu, E. M. Bakker, and M. S. Lew, “Dual gaussian-based variational subspace disentanglement for visible-infrared person re-identification,” in _Proceedings of the ACM International Conference on Multimedia_ , 2020, pp. 2149–2158.