# Shape-centered Representation Learning for Visible-Infrared Person Re-
identification

Shuang Li, Jiaxu Leng, Ji Gan, Mengjingcheng Mo, and Xinbo Gao This work was
supported in part by the National Natural Science Foundation of China under
Grants No. 62102057, No. U22A2096, and No. 62221005, in part by the Nat- ural
Science Foundation of Chongqing under Grand No. CSTB2022NSCQ-MSX1024, in part
by the Science and Tech- nology Research Program of Chongqing Municipal
Education Commission under Grant No. KJZD-K202300604, in part by the Chongqing
Postdoctoral Innovative Talent Plan under Grant No. CQBX202217, in part by the
Postdoctoral Science Foundation of China under Grant No. 2022M720548, in part
by the Chongqing Institute for Brain and Intelligence, and in part by
Chongqing Excellent Scientist Project under Grant No. cstc2021ycjh-
bgzxm0339.(Corresponding author: X. Gao.)S. Li, J. Leng, J Gan, M Mo and X.
Gao, are with the Chongqing University of Posts and Telecommunications,
Chongqing, China (E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>gaoxb@cqupt.edu.cn).

###### Abstract

Current Visible-Infrared Person Re-Identification (VI-ReID) methods prioritize
extracting distinguishing appearance features, ignoring the natural resistance
of body shape against modality changes. Initially, we gauged the
discriminative potential of shapes by a straightforward concatenation of shape
and appearance features. However, two unresolved issues persist in the
utilization of shape features. One pertains to the dependence on auxiliary
models for shape feature extraction in the inference phase, along with the
errors in generated infrared shapes due to the intrinsic modality disparity.
The other issue involves the inadequately explored correlation between shape
and appearance features. To tackle the aforementioned challenges, we propose
the Shape-centered Representation Learning framework (ScRL), which focuses on
learning shape features and appearance features associated with shapes.
Specifically, we devise the Shape Feature Propagation (SFP), facilitating
direct extraction of shape features from original images with minimal
complexity costs during inference. To restitute inaccuracies in infrared body
shapes at the feature level, we present the Infrared Shape Restitution (ISR).
Furthermore, to acquire appearance features related to shape, we design the
Appearance Feature Enhancement (AFE), which accentuates identity-related
features while suppressing identity-unrelated features guided by shape
features. Extensive experiments are conducted to validate the effectiveness of
the proposed ScRL. Achieving remarkable results, the Rank-1 (mAP) accuracy
attains 76.1%, 71.2%, 92.4% (72.6%, 52.9%, 86.7%) on the SYSU-MM01, HITSZ-VCM,
RegDB datasets respectively, outperforming existing state-of-the-art methods.

###### Index Terms:

person re-identification, logical relation inference, multi-view information
interaction, domain adaptation.

##  I Introduction

Person Re-Identification (ReID) serves the purpose of identifying specific
individuals across non-overlapping camera views, carrying substantial
significance in the domain of intelligent surveillance systems. Consequently,
it has garnered considerable attention from a multitude of researchers,
undergoing rapid advancements in recent times [1, 2, 3, 4, 5, 6, 7]. However,
the majority of existing methods are confined to scenarios where pedestrians
are solely present during daylight hours[8, 9, 10, 11, 12, 13, 14].
Consequently, the efficacy of these approaches hinges on visible appearances.
This limitation has resulted in notable performance degradation when
attempting to match pedestrians captured by both visible (VIS) and infrared
(IR) cameras. To effectively tackle this issue, the Visible-Infrared Person
Re-Identification (VI-ReID) task [15] was introduced. This task aims to enable
the retrieval of pedestrians across the distinct spectra of IR and VIS [16,
17].

Figure 1: The visible (infrared) images and their corresponding body shapes
and the orange box indicate an incorrect area of the infrared body shape.

In contrast to the extensively explored realm of ReID within the visible
spectrum, the domain of VI-ReID presents notably greater challenges. This is
primarily attributed to substantial intra- and inter-modality variations that
exist between images captured in the visible (VIS) and infrared (IR) spectra.

A prevailing trend among existing VI-ReID methods is the zealous pursuit of
mining modality-shared appearance cues to enhance the discriminability of
features but ignoring the learning of body shape clues. Nevertheless, three
compelling reasons underscore the importance of not disregarding body shape
cues. 1) The natural resistance of body shape to modality changes stands as
the first reason. As illustrated in Fig. 1, no modality discrepancy in the
body shape corresponding to infrared and visible images. 2) The second
rationale stems from the identity-discriminative nature of body shape. As
depicted in Fig. 1, one can observe that pedestrian A slightly fatter than
pedestrian B in terms of their global body shapes, with distinctions extending
to local characteristics such as facial shape, hair shape, limb shape, etc.
Thus, the identification of pedestrians becomes feasible through body shape
analysis, even when changes in modality render color texture features
unreliable. 3) Body shape estimation can be achieved through pre-trained human
parsing models, obviating the necessity for human annotation. Moreover,
single-modality ReID methods have demonstrated successful instances in this
regard.

Nevertheless, when attempting to apply body shape estimates to the VI-ReID
task, as depicted in Fig. 1, certain inaccuracies become evident in the
infrared body shapes associated with infrared images. These inaccuracies
manifest as discrepancies such as missing or inaccurately represented local
shapes, with a predominant focus on limbs. This occurrence stems from the skin
color of pedestrians bearing a striking resemblance to the background color
present in infrared images. Consequently, the human parsing model erroneously
categorizes exposed arms and legs as the background. Although the identity-
related information carried by body shapes indeed falls within the realm of
modality-shared cues, the presence of incorrect infrared body shapes limits
the effective utilization of these cues. Furthermore, we acknowledge that the
discriminative cues offered by body shape possess limitations. Depending
solely on body shape for the identification of all pedestrians proves
inadequate. Compared to the body shape, VIS (IR) pedestrian images contain
richer identity clues, including clothing style, hairstyle, and more [18].
However, the direct extraction of the aforementioned appearance features from
images presents challenges due to the interference caused by modality-specific
information (color features in visible images, temperature in infrared images)
and intricate backgrounds. Based on our observations, these appearance
features exhibit a strong correlation with body shape, while the modality-
specific information and background are unrelated to the body shape. So
effectively utilizing body shape not only requires extracting discriminant
shape features from the visible (infrared) images but also requires
effectively exploring the correlation between shape features and pedestrian
appearance features.

Figure 2: Framework comparison of VI-ReID methods that explore the utilization
of body shape. (a) Learning the features associated with shapes through multi-
task learning. (b) Learning diverse appearance features through decoupling and
discarding shape features. (c) Learning shape features and enhancing
appearance features through shape features.

Within the realm of VI-ReID, two methods closely aligned with body shape
considerations are CMMTL [18] and SEFL [19]. Illustrated in Fig. 2, CMMTL
undertakes the implicit acquisition of shape features, leveraging human
parsing as an auxiliary task. However, it falls short in addressing the
potential adverse effects of incorrect infrared shape representation and fails
to elucidate the manner in which shape features contribute to enhancement,
thus lacking interpretability. On the other hand, SEFL adopts the stance that
body shape cues are unreliable and subsequently seeks to acquire diverse
modality-shared features by disentangling and discarding potentially
unreliable shape features. In contrast, our approach diverges from SEFL’s
viewpoint. We contend that body shape features possess intrinsic
discriminative qualities and are adept at withstanding modality variations.
Consequently, our focus centers on devising effective strategies for acquiring
robust body shape features and delving into the intricate relationship between
these features and appearance features.

Based on the aforementioned insights and analysis, we introduce the Shape-
centered Representation Learning (ScRL) framework, which facilitates the
direct acquisition of modality-shared shape features and shape-associated
appearance features from visible (infrared) images. Illustrated in Fig. 3,
this proposed framework comprises three key components: Infrared Shape
Restitution (ISR), Shape Feature Propagation (SFP), and Appearance Feature
Enhancement (AFE). Specifically, ISR is designed to restitute IR shapes by
capturing missing infrared shape features from appearance features during the
training phase. SFP is designed to make the model can directly extract shape
features from pedestrian images without the participation of additional
auxiliary networks during the inference phase by transferring the ability of
the shape features extraction to the appearance feature extraction network in
the training phase. AFE is aimed at mining appearance features associated with
body shape by introducing a two-stage cascade attention mechanism that
directly and indirectly emphasizes identity-related features and suppresses
identity-unrelated features.

Our main contributions are summarized as follows:

  * •

We propose a novel deep-learning framework that prioritizes learning shape-
centered features without the necessity of auxiliary networks during the
inference phase. Notably, we are the first to effectively incorporate explicit
shape features into VI-ReID.

  * •

We design the Infrared Shape Resolution (ISR), which effectively restitutes
erroneous infrared body shapes generated by human parsing networks and
improves the discriminability of shape features.

  * •

We design the Appearance Feature Enhancement (AFE) to elevate appearance
features by harnessing the inherent relationship between shape and appearance
features.

  * •

Extensive experimental results on SYSU-MM01, RegDB, and HITSZ-VCM datasets
show the proposed ScRL achieves a new state-of-the-art performance.

The rest of this paper is organized as follows. Section II introduces related
work; Section III elaborates the proposed method; Section IV analyzes the
comparative experimental results; and Section V concludes this paper.

##  II Related Work

###  II-A Visible Person Re-Identification

Visible Person Re-Identification (ReID) aims to match visible pedestrian
images with the same identity under non-overlapping cameras. With the
introduction of large-scale datasets [20],[21],[22], Visible person ReID based
on deep learning has rapidly developed. In order to directly conduct end-to-
end training in the expected embedding space, [23] improved the hard sample
mining of the classic triplet loss, which improved the discriminability of
pedestrian features. In order to obtain fine-grained discriminative features,
PCB[24] proposes to horizontally divide pedestrian feature maps into 6 parts
to learn part-level features. MGN[25] proposes an end-to-end multi-branch deep
network to learn pedestrian feature representations of different granularity.
To overcome the information loss caused by convolution and pooling operations,
TransReid[26] proposes a transformer-based reid framework. In addition,
changes in illumination, pose, and perspective also pose challenges for
extracting discriminative pediatric features In response to the issue of
illumination changes, IID[27] proposes to eliminate the adverse effects of
illumination changes by decoupling illumination features and identity
features. In order to align with the standard posture of pedestrians, PIE[28]
introduces the PoseBox structure to obtain pose-invariant embedded features.
In response to the adverse impact of camera style changes on pedestrian
matching, Camstyle[29] proposed using CycleGAN[30] to achieve transfer between
different camera styles, smoothing out camera style differences at the data
level.

###  II-B Visible-Infrared Person Re-Identification

In order to match pedestrians from different modalities, researchers have
contributed a lot of excellent work, VI-ReID methods can be roughly divided
into two main categories: feature-level modality alignment and image-level
modality alignment methods.

The feature-level modality alignment methods aim to learn modality-shared
features by aligning IR and VIS features at the feature level. HSME [31] maps
features of different modalities to a unified hypersphere. MBCE [32] proposes
a memory-based prototype feature learning method to suppress the modality
discrepancy. MRCN [33] reduces the modality discrepancy by decoupling the
modality-relevant and modality-irrelevant features. To more effectively mine
diverse cross-modality clues, MPANet [34] be designed to discover the nuanced
modality-shared features. DEEN [35] enhances the embedding representation in
the embedding space by generating diverse embeddings.

The image level modality alignment methods alleviate modality differences by
generating images with the target or intermediate modality styles. D2RL [36]
transfers the style of IR (VIS) images to VIS (IR) images through the GAN
network, compensating for the missing modality information. XIV [37] transform
IR and VIS images into auxiliary X-modality images respectively, and perform
X-IR-VIS three-mode learning. SMCL [38] generates syncretic modality images
that contain information from both modalities to steer modality-invariant
feature learning. However, the modality-shared clues (like shape-centered
clues) have not been fully explored, which limits the discriminability of
features.

###  II-C Semantic Parsing for Person Re-Identification

SPReID [39] first introduced human semantic parsing into the ReID to obtain
refined local features. MGCAM [40] introduces human semantic parsing for the
first time to construct RGB mask pairs to eliminate the adverse effects of
background noise. EaNet [41] guides the ReID model through human parsing as
labels for multitasking learning. In addition to the human parts captured by
human parsing results, there are also some other useful clues (such as
personal belongings.). P2Net [42] and ISP [43] explore more comprehensive
features through accurate human parts and non-human parts. Considering that
widely studied methods overly rely on color appearance and fail to apply to
cloth changing ReID, FSAM [44] transfers knowledge from shape stream to
appearance stream by interactive mutual learning. In the VI-ReID task, CMMTL
[18] and SEFL [19] involve human semantic parsing. CMMTL uses the body shape
as the semantic label guidance model to perform both VI-ReID and human
semantic segmentation tasks. SEFL [19] discards the shape feature and focuses
on learning identity-related modality-shared features through disentanglement
learning.

Compared to their work, our method aims to learn shape-centered feature
representation guided by body shape, while considering the unreliable
situation of IR human semantic parsing.

Figure 3: Overview of our method. This framework mainly consists of Shape
Feature Propagation (SFP), Infrared Shape Restitution (ISR), and Appearance
Feature Enhancement (AFE), which can also be divided into appearance stream
and shape stream. SFP is used to transfer the shape feature extraction ability
to the appearance stream, enabling the model to directly extract shape
features from the original image.

##  III The Proposed Method

###  III-A Preliminaries and Baseline

Preliminaries. Let
$X=\\{(x^{vis}_{i},x^{ir}_{i},x^{vis}_{s,i},x^{ir}_{s,i},y_{i})|_{i=1}^{N}\\}$
represent the training dataset, where $N$ represents the total number of
pedestrian images, $y_{i}\in\\{1,2,\cdots,K\\}$ represents its corresponding
identity label, $K$ represents the total number of identities.
$x^{vis}_{i}$($x^{ir}_{i}$) represents the $i$-th VIS(IR) image,
$x^{vis}_{s,i}$($x^{ir}_{s,i}$) represents the body shape corresponding to
$x^{vis}_{i}$($x^{ir}_{i}$), which is obtained by feeding
$x^{vis}_{i}$($x^{ir}_{i}$) into the pre-trained Self-Correction Human Parsing
(SCHP) [45] model.

Baseline: Appearance Feature Learning. Referring to the ReID framework named
AGW [46], we adopted a dual stream network that removed the Non-local
Attention as the Baseline. Specifically, a dual stream network consists of two
parallel convolutional layers with unshared parameters, four blocks with
shared parameters, and a Generalized-mean (GeM) Pooling layer, named
$\bm{F}_{a}$. Two parallel convolutional layers independently process VIS and
IR images to extract low-level features, then the features are fed to the
subsequent four blocks to extract high-level features $\bm{f}_{i}^{m}$. The
above process can be formalized as:

$\displaystyle\bm{f}_{i}^{m}$ $\displaystyle=\bm{F}_{a}(x^{m}_{i})\quad\quad
m=\\{vis,ir\\}.$ (1)

To ensure the identity discriminability of $\bm{f}_{i}^{m}$, Cross Entropy
(CE) loss $L_{id}$ and Weighted Regularization Triplet (WRT)[47] loss
$L_{wrt}$ was adopt to constrain it as follows:

$\begin{aligned}
\bm{L}_{id}&=-\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}q_{i}\log(\bm{W}_{id}(\bm{f}_{i}^{m}))\end{aligned},$
(2)

where $n_{b}$ represents the batch size, $\bm{W}_{id}$ represents the shared
identity classifier for IR and VIS pedestrian features,
$q_{i}\in\mathbb{R}^{K\times 1}$ is a one-hot vector, and only the element at
$y_{i}$ is $1$.

$\begin{aligned}
\hfil\displaystyle\begin{split}L_{wrt}=\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}\log(1+\exp(\sum_{i,j}w_{i,j}^{p}\bm{d}_{i,j}^{p}-\sum_{i,k}w_{i,k}^{n}\bm{d}_{i,k}^{n}),\\\
w_{i,j}^{p}=\frac{\exp(\bm{d}_{i,j}^{p})}{\sum_{\bm{d}_{i,j}\in{P_{i}}}\exp(\bm{d}_{i,j}^{p})},w_{i,k}^{n}=\frac{\exp(-\bm{d}_{i,k}^{n})}{\sum_{\bm{d}_{i,k}\in{N_{i}}}\exp(-\bm{d}_{i,k}^{n})}\end{split}\end{aligned},$
(3)

where $j,k$ ($P_{i},N_{i}$) represents the index of the positive and negative
samples(set) corresponding to the anchor sample $x_{i}$ within a batch size,
respectively. And $d_{i,j}$ represents the euclidean distance between two
features: $\bm{d}_{i,j}=\bm{\|}\bm{f}_{i}^{m}-\bm{f}_{j}^{m}\|_{2}$.

###  III-B Shape Feature Learning

Although the design of the dual stream network is aimed at extracting
modality-shared features, it focuses on learning pedestrian appearance
features and lacks the ability to learn pedestrian body shape features.
Compared to appearance features, the body shape features of pedestrians are
more robust against modality changes. Therefore, in this section, we aim to
learn the body shape features by the SCHP and a shape feature learning network
$\bm{F}_{s}$. In order to obtain discriminative body shape features, we feed
the body shape $x^{vis}_{s,i}$ and $x^{ir}_{s,i}$ generated by the SCHP to
$\bm{F}_{s}$ to extract the shape features $\bm{f}_{s,i}=\bm{F}_{s}(x_{s,i})$.
Then, $\bm{f}_{s,i}$ is into the shape identity classifier $\bm{W}_{s}$ for
identity classification.

Infrared Shape Restitution: as discussed in the introduction and shown in
Fig.1, inherent errors are present within the IR body shapes, which not only
leads to the inability of $\bm{F}_{s}$ to derive adequately discriminative
shape features from input IR shapes but also limit the ability of $\bm{F}_{s}$
to extract VIS shape features due to pulling the distance between incorrect IR
shape features and VIS shape features.

To address this issue, we propose to obtain absent shape details of the IR
body shape from the corresponding IR image to restitute the IR shape at the
feature level, as shown in Fig. 4. This module is motivated by the fact that,
despite certain errors in the infrared shape generated by SCHP, the original
infrared pedestrian image still contains the complete shape information.
Specifically, let the IR shape feature map and IR feature map output from the
$j$-th block of $\bm{F}_{s}$ and $\bm{F}_{a}$ be represented as
$\bm{f}^{ir,j}_{s,i}$ and $\bm{f}^{ir,j}_{i}$, respectively. In order to
search for features related to incorrect IR shape in IR feature map
$\bm{f}^{ir,j}_{i}$, we design a cross-attention consisting of query $\bm{Q}$,
key $\bm{K}$, value $\bm{V}$, inspired by self-attention[48]. A noteworthy
distinction from self-attention is that our query is obtained by adding the IR
shape feature map $\bm{f}^{ir,j}_{i}$ and IR feature map
$\bm{f}^{ir,j}_{s,i}$:

$\displaystyle\bm{Q}$
$\displaystyle=\bm{W}_{q}(\bm{f}^{ir,j}_{i}+\bm{f}^{ir,j}_{s,i}),$ (4)

Where $\bm{W}_{q}$ represents the 2D convolution layer with kernel sizes of
$1\times 1$. The query generated by this operation contains the information
required to restitute the body shape, so it can be used to search for valid
information in the value. And the key and value are represented as:

$\displaystyle\bm{K}=\bm{W}_{k}(\bm{f}^{ir,j}_{i}),\bm{V}=\bm{W}_{v}(\bm{f}^{ir,j}_{i}),$
(5)

Where $\bm{W}_{k}$ and $\bm{W}_{v}$ represent the 2D convolution layer with
kernel sizes of $1\times 1$, respectively. Then we obtain the required feature
from $\bm{V}$ and the corrected IR shape feature map
$\hat{\bm{f}}^{ir,j}_{s,i}$ can be obtained as follows:

$\displaystyle\hat{\bm{f}}^{ir,j}_{s,i}$
$\displaystyle=\bm{W}_{v2}(\bm{BN}(\bm{Norm}(\bm{Q}\bm{K}^{\text{T}})\bm{V}))+\bm{f}^{ir,j}_{s,i},$
(6)

Where $\bm{W}_{v2}$ represents the 2D convolution layer with kernel sizes of
$1\times 1$, $\bm{BN}$, $\bm{Norm}$ and ${(\cdot)}^{\textbf{T}}$ represents
the batch normalization layer, the normalization operation and the transpose
operation.

Figure 4: Illustration of our proposed ISR that is used to obtain missing
shape features from appearance features for restituting IR shapes.

Following the above steps, we incorporated the infrared shape restitution
attention following the first and second blocks of $\bm{F}_{s}$. In order to
guide the training of $\bm{F}_{s}$ and ISR, we introduce CE loss and WRT loss
to constrain the shape features $\hat{\bm{f}}_{s,i}$ output from $\bm{F}_{s}$
with GeM pooling layer:

$\begin{aligned}
\bm{L}^{s}_{id}&=-\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}q_{i}\log(\bm{W}_{id}^{s}(\bm{\hat{}}{\bm{f}}_{s,i}))\end{aligned},$
(7)

where $\bm{W}_{id}^{s}$ represents the shared identity classifier for IR and
VIS pedestrian shape features.

$\begin{aligned}
\hfil\displaystyle\begin{split}L^{s}_{wrt}=\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}\log(1+\exp(\sum_{i,j}w_{i,j}^{s,p}\bm{d}_{i,j}^{s,p}-\sum_{i,k}w_{i,k}^{s,n}\bm{d}_{i,k}^{s,n}),\\\
w_{i,j}^{s,p}=\frac{\exp(\bm{d}_{i,j}^{s,p})}{\sum_{\bm{d}_{i,j}\in{P_{i}^{s}}}\exp(\bm{d}_{i,j}^{s,p})},w_{i,k}^{s,n}=\frac{\exp(-\bm{d}_{i,k}^{s,n})}{\sum_{\bm{d}_{i,k}\in{N_{i}^{s}}}\exp(-\bm{d}_{i,k}^{s,n})}\end{split}\end{aligned},$
(8)

where $j,k$ ($P_{i}^{s},N_{i}^{s}$) represents the index of the positive and
negative samples(set) corresponding to the anchor shape feature
$\bm{\hat{}}{\bm{f}}_{s,i}$ within a batch size, respectively. And
$d_{i,j}^{s}$ represents the euclidean distance between two shape features:
$\bm{d}_{i,j}^{s}=\bm{\|}\bm{\hat{}}{\bm{f}}_{s,i}-\bm{\hat{}}{\bm{f}}_{s,j}\|_{2}$.
Importantly, it should be highlighted that the VIS shape features do not
require restitution and we mix them with corrected IR shape features in batch
size to participate in the loss calculation. Therefore, the VIS shape features
can guide the learning of the ISR at the loss level.

Shape Feature Propagation: we can obtain shape features through the
collaboration of appearance feature extraction network $\bm{F}_{a}$, shape
feature learning network $\bm{F}_{s}$, and ISR. However, this poses challenges
to model deployment due to the increased parameters and computational
complexity. Therefore, it is crucial to transfer the ability of extracting
shape features to the appearance stream network, so that the testing phase
does not require the participation of the shape stream network.

Towards this objective, we replicate the fourth block of $\bm{F}_{a}$ as the
shape subnetwork $\bm{F}_{\tilde{s}}$ with GeM pooling layer and apply it to
the output features of the third block of $\bm{F}_{a}$ to obtain the shape
features $\tilde{\bm{f}}_{s,i}$ under the guidance of the shape features
$\hat{\bm{f}}_{s,i}$ output from the $\bm{F}_{s}$ at instance (Eq.9) and
prototype level (Eq.10).

$\displaystyle\bm{L}_{kd}=\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}\left|\left|\tilde{\bm{f}}_{s,i}-\hat{\bm{f}}_{s,i}\right|\right|_{2},$
(9)

where $n_{b}$ represents the batch size, and
$\left|\left|\cdot\right|\right|_{2}$ represents $l_{2}$-norm.

$\displaystyle\bm{L}_{kd\\_ce}$
$\displaystyle=-\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}\bm{q}_{i}\log(\bm{\tilde{}}{f}_{s,i}\bm{\Theta}^{\textbf{T}}),$
(10)

where $\bm{\Theta}\in\mathbb{R}^{C\times K}$ represents the class prototype
from the classifier $\bm{W}_{id}^{s}$ .

###  III-C Appearance Feature Enhancement

In addition to shape features, pedestrian appearance contains essential
identity cues like clothing style, hairstyle and etc. Extracting these
features from the original image is complex due to modality(background)
changes. It is worth noting that these clues are tied to body shape—hairstyle
features relate to head shape. Therefore, we expect to mine these appearance
clues mentioned above and suppress identity-unrelated modality-specific
features, such as color, temperature, etc. Due to the limited appearance
features directly associated with shape features, it is necessary to first
mine the appearance features directly associated with shape and then use these
appearance features to further explore the appearance features indirectly
associated with body shape.

So we devised a cascading two-stage attention mechanism that systematically
extracts appearance features related to body shape—both directly and
indirectly, as shown in Fig.5. The mechanism operates as follows: in the first
stage, let the shape feature map $\bar{\bm{f}}_{s,i}$ output by
$\bm{F}_{\tilde{s}}$ serve as query, and the appearance feature map
$\bm{f}_{i}$ extracted by $\bm{F}_{a}$ serve as key and value:

$\displaystyle\bm{Q}=\bm{W}_{q}(\bar{\bm{f}}_{s,i}),\bm{K}=\bm{W}_{k}(\bm{f}_{i}),\bm{V}=\bm{W}_{v}(\bm{f}_{i}),$
(11)

Then, similar to ISR, the correlation score between $\bm{Q}$ and $\bm{K}$ is
used to search features directly related to shape in $\bm{V}$, and fused with
shape feature map $\bar{\bm{f}}_{s,i}$ to obtain $\tilde{\bm{f}}^{fuse}_{i}$.

$\displaystyle\tilde{\bm{f}}^{fuse}_{i}$
$\displaystyle=\bm{W}_{v2}(\bm{BN}(\bm{Norm}(\bm{Q}\bm{K}^{\text{T}})\bm{V}))+\bar{\bm{f}}_{s,i},$
(12)

Considering $\tilde{\bm{f}}^{fuse}_{i}$ contains shape discriminative
information and appearance discriminative information directly related to body
shape, $\tilde{\bm{f}}^{fuse}_{i}$ can be effectively employed as a query
during the second stage of attention, which facilitates the acquisition of
appearance features that are both directly and indirectly associated with the
body shape. As the query feature of the second-stage attention,
$\tilde{\bm{f}}^{fuse}_{i}$ plays a pivotal role in determining whether
modality-invariant identity-discriminative features, both directly and
indirectly associated with shape, can be effectively extracted from the
appearance features. To ensure the discriminability of the query
$\tilde{\bm{f}}^{fuse}_{i}$, we also employ CE loss and WRT loss to jointly
constrain $GeM(\tilde{\bm{f}}^{fuse}_{i})$, $GeM$ represents the GeM pooling
layer.

Figure 5: Illustration of our proposed AFE, it can mine shape-centered
appearance features guided by shape features.

$\begin{aligned}
\bm{L}^{q}_{id}&=-\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}q_{i}\log(\bm{W}_{id}^{q}(GeM(\tilde{\bm{f}}^{fuse}_{i})))\end{aligned},$
(13)

where $\bm{W}_{id}^{q}$ represents the shared identity classifier for infrared
and visible pedestrian query features.

$\begin{aligned}
\hfil\displaystyle\begin{split}L^{q}_{wrt}=\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}\log(1+\exp(\sum_{i,j}w_{i,j}^{q,p}\bm{d}_{i,j}^{q,p}-\sum_{i,k}w_{i,k}^{q,n}\bm{d}_{i,k}^{q,n}),\\\
w_{i,j}^{q,p}=\frac{\exp(\bm{d}_{i,j}^{q,p})}{\sum_{\bm{d}_{i,j}\in{P_{i}^{q}}}\exp(\bm{d}_{i,j}^{q,p})},w_{i,k}^{q,n}=\frac{\exp(-\bm{d}_{i,k}^{q,n})}{\sum_{\bm{d}_{i,k}\in{N_{i}^{q}}}\exp(-\bm{d}_{i,k}^{q,n})}\end{split}\end{aligned},$
(14)

where $j,k$ ($P_{i}^{q},N_{i}^{q}$) represents the index of the positive and
negative samples(set) corresponding to the anchor shape feature
$\bm{\hat{}}{\bm{f}}_{s,i}$ within a batch size, respectively. And
$d_{i,j}^{q}$ represents the euclidean distance between two query features:
$\bm{d}_{i,j}^{q}=\bm{\|}(GeM(\tilde{\bm{f}}^{fuse}_{i})-(GeM(\tilde{\bm{f}}^{fuse}_{j})\|_{2}$.

In the second stage, we employed the output feature
$\tilde{\bm{f}}^{fuse}_{i}$ of the attention of the first stage as a query to
emphasize the appearance features directly and indirectly related to the body
shape in $\bm{V}$ as follows:

$\displaystyle\tilde{\bm{f}}_{i}$
$\displaystyle=\bm{W}_{v2}(\bm{BN}(\bm{Norm}(\bm{W}_{q}(\tilde{\bm{f}}^{fuse}_{i})\bm{K}^{\text{T}})\bm{V}))+\bm{f}_{i},$
(15)

Similar to the baseline method, we also utilized a combination of CE loss and
WRT loss to jointly constrain appearance features $GeM(\tilde{\bm{f}}_{i})$
that are closely linked to shape. This constraint serves to enhance their
identity discrimination and modality invariance. As follows:

$\begin{aligned}
\bm{L}^{a}_{id}&=-\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}q_{i}\log(\bm{W}_{id}(GeM(\tilde{\bm{f}}_{i})))\end{aligned},$
(16) $\begin{aligned}
\hfil\displaystyle\begin{split}L^{a}_{wrt}=\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}\log(1+\exp(\sum_{i,j}w_{i,j}^{a,p}\bm{d}_{i,j}^{a,p}-\sum_{i,k}w_{i,k}^{a,n}\bm{d}_{i,k}^{a,n}),\\\
w_{i,j}^{a,p}=\frac{\exp(\bm{d}_{i,j}^{a,p})}{\sum_{\bm{d}_{i,j}^{a}\in{P_{i}}}\exp(\bm{d}_{i,j}^{a,p})},w_{i,k}^{a,n}=\frac{\exp(-\bm{d}_{i,k}^{a,n})}{\sum_{\bm{d}_{i,k}\in{N_{i}}}\exp(-\bm{d}_{i,k}^{a,n})}\end{split}\end{aligned},$
(17)

where $j,k$ ($P_{i}^{a},N_{i}^{a}$) represents the index of the positive and
negative samples(set) corresponding to the anchor shape feature
$\tilde{\bm{f}}_{i}$ within a batch size, respectively. And $d_{i,j}^{a}$
represents the euclidean distance between two features:
$\bm{d}_{i,j}^{a}=\bm{\|}GeM(\tilde{\bm{f}}_{i})-GeM(\tilde{\bm{f}}_{j})\|_{2}$.

TABLE I: The Settings of different infrared visible re-id datasets. Datasets | Type | #Identites | #RGB cam. | #IR cam. | #Images&BBoxes | #Tracklets | #Evaluation
---|---|---|---|---|---|---|---
SYSU-MM01[15] | Image | 412 | 1 | 1 | 8240 | - | CMC+mAP
RegDB[49] | Image | 491 | 4 | 2 | 303420 | - | CMC+mAP
HITSZ-VCM [50] | Video | 927 | 12 | 12 | 463259 | 21863 | CMC+mAP
TABLE II: Results of mAP and CMC (%) obtained by our proposed method and the state-of-the-art Re-ID methods on SYSU-MM01.‘R1’, ‘R10’ and ’R20’ denote Rank-1,Rank-10 and Rank-20, respectively. These results are copied from their papers. Methods |  Reference | All search | Indoor search
---|---|---|---
|  R1 |  R10 |  R20 |  mAP |  mINP |  R1 |  R10 |  R20 |  mAP |  mINP
Zero-Pad[15] |  ICCV’2017 |  14.8 |  54.1 |  71.3 |  16.0 |  - |  20.6 |  68.4 |  85.8 |  26.9 |  -
HCML[51] |  AAAI’2018 |  14.3 |  53.2 |  69.2 |  16.2 |  - |  24.5 |  73.3 |  86.7 |  30.1 |  -
HSME[31] |  AAAI’2019 |  20.7 |  32.7 |  78.0 |  23.1 |  - |  - |  - |  - |  - |  -
D2RL[36] |  CVPR’2019 |  28.9 |  70.6 |  82.4 |  29.2 |  - |  - |  - |  - |  - |  -
eBDTR[52] |  TIFS’2019 |  27.8 |  67.3 |  81.3 |  28.4 |  - |  32.5 |  77.4 |  89.6 |  42.5 |  -
X-Modal[37] |  AAAI’2020 |  49.9 |  89.8 |  96.0 |  50.7 |  - |  - |  - |  - |  - |  -
Hi-CMD[53] |  CVPR’2020 |  34.9 |  77.6 |  - |  35.9 |  - |  - |  - |  - |  - |  -
DDAG[54] |  ECCV’2020 |  54.8 |  90.4 |  95.8 |  53.0 |  39.6 |  61.0 |  94.1 |  98.4 |  68.0 |  62.6
HAT[55] |  TIFS’2021 |  55.3 |  92.1 |  97.4 |  53.9 |  - |  62.1 |  95.8 |  99.2 |  69.4 |  -
MCLNet[56] |  ICCV’2021 |  65.4 |  93.3 |  97.1 |  62.0 |  - |  72.6 |  96.9 |  99.2 |  76.6 |  -
CM-NAS[57] |  CVPR’2021 |  62.0 |  92.9 |  97.3 |  60.0 |  - |  67.0 |  97.0 |  99.3 |  73.0 |  -
MPANet[34] |  CVPR’2021 |  70.6 |  96.2 |  98.8 |  68.2 |  - |  76.7 |  98.2 |  99.6 |  81.0 |  -
CAJ[47] |  ICCV’2021 |  69.9 |  95.7 |  98.5 |  66.9 |  - |  76.3 |  97.9 |  99.5 |  80.4 |  -
AGW[3] |  TPAMI’2022 |  47.5 |  84.4 |  92.1 |  47.7 |  35.3 |  54.2 |  91.1 |  96.0 |  63.0 |  59.2
FMCNet[58] |  CVPR’2022 |  66.3 |  - |  - |  62.5 |  - |  68.2 |  - |  - |  74.1 |  -
CMMTL[18] |  PR’2022 |  67.3 |  95.4 |  98.5 |  64.3 |  - |  69.6 |  96.7 |  99.0 |  74.4 |  -
PMT[59] |  AAAI’2022 |  67.5 |  95.4 |  98.6 |  65.0 |  51.9 |  71.7 |  96.7 |  99.3 |  76.5 |  72.7
DSCNet[60] |  TIFS’2022 |  73.9 |  96.3 |  98.8 |  69.5 |  - |  79.4 |  98.3 |  99.8 |  82.7 |  -
MRCN[33] |  AAAI’2023 |  68.9 |  95.2 |  98.4 |  65.5 |  - |  76.0 |  98.3 |  99.7 |  79.8 |  -
DEEN[35] |  CVPR’2023 |  74.7 |  97.6 |  99.2 |  71.8 |  - |  80.3 |  99.0 |  99.8 |  83.3 |  -
SEFL[19] |  CVPR’2023 |  75.2 |  96.9 |  99.1 |  70.1 |  - |  78.4 |  97.5 |  98.9 |  81.2 |  -
ScRL(our) |  |  76.1 |  97.6 |  99.4 |  72.6 |  59.8 |  82.4 |  98.8 |  99.8 |  85.4 |  82.2

###  III-D Training and Inference

In the training process, we employ the appearance flow network $\bm{F}_{a}$
for extracting appearance features and the shape flow network $\bm{F}_{s}$ for
extracting shape features. To further enhance this process, we introduce the
ISR module to restitute inaccuracies in infrared shape features. Additionally,
the SFP module is integrated to impart the shape feature extraction
capabilities of the shape stream network $\bm{F}_{s}$ to the appearance stream
network $\bm{F}_{a}$. Lastly, the AFE module is introduced to mine appearance
features that have both direct and indirect associations with the shape
features. The complete training process is executed in an end-to-end fashion,
as illustrated in Algorithm 1.

During the testing process, we solely utilize the appearance stream network
$\bm{F}_{a}$, $\bm{F}_{\tilde{s}}$ and the AFE module, without involving the
shape stream network $\bm{F}_{s}$ and the human parsing network SCHP.

Algorithm 1 Training process of the proposed ScRL

0: A mini-batch that consists of $n_{b}$ visible(infrared) images
$x^{vis}_{i}$($x^{ir}_{i}$) and visible(infrared) shapes
$x^{vis}_{s,i}$($x^{ir}_{s,i}$).

0: Trained ScRL model.

1: for $i=1$; $i<iteration$; $i++$ do

2: Extract $\bm{f}^{ir,j}_{i}$,$\bm{f}_{i}$ by inputting $x^{vis}_{i}$,
$x^{ir}_{i}$ into the appearance stream network $F_{a}$.

3: Extract $\bm{f}^{ir,j}_{i,s}$ by inputting $x^{vis}_{i,s}$, $x^{ir}_{i,s}$
into the shape stream network $F_{s}$.

4: Send $\bm{f}^{ir,j}_{i}$,$\bm{f}^{ir,j}_{i,s}$ to the ISR module to obtain
the repaired infrared shape feature $\hat{\bm{f}}^{ir,j}_{s,i}$.

5: Send $\hat{\bm{f}}^{ir,j}_{s,i}$ to the subsequent network of the shape
flow $F_{s}$ to obtain the final shape features $\hat{\bm{f}}_{s,i}$.

6: Extract $\tilde{\bm{f}}_{s,i}$ by feeding the output features of the third
block of $F_{a}$ into the shape subnetwork $\bm{F}_{\tilde{s}}$.

7: Transfer the capability to extract shape features from the shape stream
network $\bm{F}_{\tilde{s}}$ to the appearance stream network $\bm{F}_{a}$,
$\bm{F}_{\tilde{s}}$ by the SFP module.

8: Extract appearance features $\tilde{\bm{f}}^{fuse}_{i}$ directly related to
body shape by inputting $\bar{\bm{f}}_{s,i}$, ${\bm{f}}_{i}$ into stage 1 of
the AFE module.

9: Extract appearance features $\tilde{\bm{f}}_{i}$ both directly and
indirectly related to body shape by inputting $\tilde{\bm{f}}^{fuse}_{i}$,
${\bm{f}}_{i}$ into stage 2 of the AFE module.

10: Optimize $\bm{F}_{a}$, $\bm{F}_{s}$, $\bm{F}_{\tilde{s}}$, ISR module and
AFE module through loss functions $\bm{L}_{kd}$(Eq.7) , $\bm{L}_{kd\\_ce}$
(Eq.8), $\bm{L}_{kd}$(Eq.9), $\bm{L}_{kd\\_ce}$ (Eq.10),
$\bm{L}^{q}_{id}$(Eq.13), $\bm{L}^{q}_{wrt}$(Eq.14), $\bm{L}^{a}_{id}$(Eq.16)
and $\bm{L}^{a}_{wrt}$(Eq.17).

11: end for

##  IV Experiments

###  IV-A Datasets

SYSU-MM01[15] is a large-scale dataset with complex environments. The training
set consists of 11909 IR (22258 VIS) images of 395 identities captured across
2 IR (4 VIS) cameras. For the testing set, there are 96 pedestrians, with a
total of 3,803 IR pedestrian images and 301 randomly selected VIS images.

RegDB[49] is a small dataset consisting of 8420 images of 421 identities
captured by a single VIS(IR) camera. Each pedestrian has 10 VIS(IR) images. We
followed BDTR[61] and randomly divided the dataset into training and testing
sets for training and evaluation.

HITSZ-VCM[50] is a video-based VI-ReID dataset that contains 251452 VIS images
and 211807 IR images of 927 identities, with each track containing 24
consecutive images. The training(testing) set encompasses 11061(10802) tracks
of 500(427) identities.

Evaluation Metrics We employ Cumulative Matching Characteristics (CMC), mean
Average Precision (mAP), and mean Inverse Negative Penalty (mINP)[3] as
evaluation metrics to assess the performance of the ScRL and the methods
compared in this paper.

TABLE III: Results of mAP and CMC (%) obtained by our proposed method and the state-of-the-art Re-ID methods on RegDB.‘R1’, ‘R10’ and ’R20’ denote Rank-1,Rank-10 and Rank-20, respectively. These results are copied from their papers. Methods |  Reference | Visible to Infrared | Infrared to Visible
---|---|---|---
|  R1 |  R10 |  R20 |  mAP |  mINP |  R1 |  R10 |  R20 |  mAP |  mINP
Zero-Pad[15] |  ICCV’2017 |  17.8 |  34.2 |  44.4 |  18.9 |  - |  16.6 |  34.7 |  44.3 |  17.8 |  -
HCML[51] |  AAAI’2018 |  24.4 |  47.5 |  56.8 |  20.1 |  - |  21.7 |  45.0 |  55.6 |  22.2 |  -
HSME[31] |  AAAI’2019 |  50.9 |  73.4 |  81.7 |  47.0 |  - |  50.2 |  72.4 |  81.1 |  46.2 |  -
D2RL[36] |  CVPR’2019 |  43.4 |  66.1 |  76.3 |  44.1 |  - |  - |  - |  - |  - |  -
eBDTR[52] |  TIFS’2019 |  34.6 |  59.0 |  68.7 |  33.5 |  - |  34.2 |  58.7 |  68.6 |  32.5 |  -
XModal[37] |  AAAI’2020 |  62.2 |  83.1 |  91.7 |  60.2 |  - |  - |  - |  - |  - |  -
Hi-CMD[53] |  CVPR’2020 |  70.9 |  86.4 |  - |  66.0 |  - |  - |  - |  - |  - |  -
DDAG[54] |  ECCV’2020 |  69.3 |  86.2 |  91.5 |  63.5 |  49.2 |  68.1 |  85.2 |  90.3 |  61.8 |  48.6
HAT[55] |  TIFS’2021 |  71.8 |  87.2 |  92.2 |  67.6 |  - |  70.0 |  86.5 |  91.6 |  66.3 |  -
MCLNet[56] |  ICCV’2021 |  80.3 |  92.7 |  96.0 |  73.1 |  57.3 |  75.9 |  90.9 |  94.6 |  69.5 |  52.6
CM-NAS[57] |  ICCV’2021 |  84.5 |  95.2 |  97.9 |  80.3 |  - |  82.6 |  94.5 |  97.4 |  78.3 |  -
MPANet[34] |  CVPR’2021 |  83.7 |  - |  - |  80.9 |  - |  82.8 |  - |  - |  80.7 |  -
CAJ[47] |  ICCV’2021 |  85.0 |  95.5 |  97.5 |  79.1 |  65.3 |  84.8 |  95.3 |  97.5 |  77.8 |  61.6
AGW[3] |  TPAMI’2022 |  70.1 |  86.2 |  91.6 |  66.4 |  50.2 |  70.5 |  87.2 |  91.8 |  65.9 |  51.2
FMCNet[58] |  CVPR’2022 |  89.1 |  - |  - |  84.4 |  - |  88.4 |  - |  - |  83.9 |  -
CMMTL[18] |  PR’2022 |  89.9 |  96.6 |  98.3 |  85.6 |  - |  88.3 |  96.2 |  98.0 |  84.1 |  -
PMT[59] |  AAAI’2022 |  84.8 |  - |  - |  76.6 |  - |  84.2 |  - |  - |  75.1 |
DSCNet[60] |  TIFS’2022 |  85.4 |  - |  - |  77.3 |  - |  83.5 |  9- |  - |  75.2 |  -
MRCN[33] |  AAAI’2023 |  91.4 |  98.0 |  99.0 |  84.6 |  - |  88.3 |  96.7 |  98.5 |  81.9 |  -
DEEN[35] |  CVPR’2023 |  91.1 |  97.8 |  98.9 |  85.1 |  |  89.5 |  96.8 |  98.4 |  83.4 |  -
SEFL[19] |  CVPR’2023 |  92.2 |  - |  - |  86.6 |  - |  91.1 |  - |  - |  85.2 |  -
ScRL(our) |  |  92.4 |  98.1 |  99.1 |  86.7 |  73.6 |  91.8 |  98.0 |  99.1 |  85.3 |  70.9
TABLE IV: Results of mAP and CMC (%) obtained by our proposed method and the state-of-the-art Re-ID methods on HITSZ-VCM.‘R1’, ‘R5’ and ’R10’ denote Rank-1,Rank-5 and Rank-10, respectively. These results are copied from their papers. Strategy |  Methods |  Reference | Infrared to Visible | Visible to Infrared
---|---|---|---|---
|  |  |  R1 |  R5 |  R10 |  mAP |  R1 |  R5 |  R10 |  mAP
Video |  MITML[50] |  CVPR’22 |  63.7 |  76.9 |  81.7 |  45.3 |  64.5 |  79.0 |  83.0 |  47.7
|  IBAN[62] |  TCSVT’23 |  65.0 |  78.3 |  83.0 |  48.8 |  69.6 |  81.5 |  85.4 |  51.0
|  SADSTRM[63] |  Arxiv’23 |  65.3 |  77.9 |  82.7 |  49.5 |  67.7 |  80.7 |  85.1 |  51.8
Image |  Lba[64] |  ICCV’21 |  46.4 |  65.3 |  72.2 |  30.7 |  49.3 |  69.3 |  75.9 |  32.4
|  MPANet[34] |  CVPR’21 |  46.5 |  63.1 |  70.5 |  35.3 |  50.3 |  67.3 |  73.6 |  37.8
|  VSD[65] |  CVPR’21 |  54.5 |  70.0 |  76.3 |  41.2 |  57.5 |  73.7 |  79.4 |  43.5
|  CAJ[47] |  ICCV’21 |  56.6 |  73.5 |  79.5 |  41.5 |  60.1 |  74.6 |  79.9 |  42.8
|  SEFL[19] |  CVPR’23 |  67.7 |  80.3 |  84.7 |  52.3 |  70.2 |  82.2 |  86.1 |  52.5
|  ScRL(our) |  |  71.2 |  83.2 |  87.4 |  52.9 |  73.3 |  84.4 |  87.7 |  53.0

###  IV-B Implementation Details

Similar to DEEN [35], we adopted ResNet50 pre-trained on imagenet as the
backbone and replaced the average pooling layer with the GEM pooling layer,
with all input image sizes resized to $384\times 144$. In the training phase,
we adopted Random Crop, Random Horizontal Flip, Channel Random Erasing and
Channel AdapGray [47] to enhance the IR(VIS) images, and we adopt Random Crop
and Random Horizontal Flip for the shape image. We adopt the Adam optimizer
for optimization, the learning rate of the classifier $\bm{W}_{s}$,
$\bm{W}_{a}$ was set to 0.0007, and the learning rate of other networks was
set to 0.00035. The model trained a total of 120 epochs, in the first 10
epochs, the learning rate is dynamically adjusted through the warmup strategy,
in the 40th and 60th epochs, the learning rate decreases by 10%. At every
batch size, we randomly sample 64 images from 8 identities, with 4 VIS (IR)
images for each identity and we follow the sampling settings of MITML for
HITSZ-VCM[50].

###  IV-C Comparison with State-of-the-Art Methods

In this section, we present a comprehensive comparison between the proposed
ScRL and other state-of-the-art methods across the SYSU-MM01, RegDB, and
HITSZ-VCM datasets. Specifically, the state of the art methods we compared
include Zero-Pad[15], HCML[51], HSME[31], D2RL[36], eBDTR[52], X-Modal[37],
Hi-CMD[53], DDAG[54], HAT[55], MCLNet[56], CM-NAS[57], MPANet[34], CAJ[47],
AGW[3], FMCNet[58], CMMTL[18], PMT[59], DSCNet[60], MRCN[33], DEEN[35],
SEFL[19].

SYSU-MM01. As shown in Tab.II, the proposed method exhibits strong
competitiveness on SYSU-MM01, particularly in terms of Rank-1, mAP, and mINP.
Specifically, in the “all search” mode, the proposed method attained an
accuracy of 76.1%, 72.6%, and 59.8% for the Rank-1, mAP, and mINP indicators,
respectively. Similarly, in the “indoor search” mode, the proposed method
delivered an accuracy of 82.4%, 85.4%, and 82.2% for the Rank-1, mAP, and mINP
indicators, respectively. Furthermore, our method outperforms the suboptimal
method by 0.9%(2.1%), 0.8%( 2.1%), and 7.9% (9.5%) on Rank-1, mAP, and mINP in
“all search”(“indoor search”) mode, respectively. The results suggest that the
proposed method, which focuses on pedestrian features with shape as the
central component, effectively mitigates modality changes and enhances the
accuracy of cross-modality pedestrian matching.

RegDB. As illustrated in Tab. III, the proposed approach has showcased
commendable performance even on the limited-scale dataset RegDB. Notably,
within the “IR to VIS” mode, our method achieved 91.8% and 85.3% accuracy in
Rank-1 and mAP indicators, respectively, and has demonstrated superiority over
the suboptimal SEFL approach by 0.7% (0.1%) in terms of Rank-1 (mAP).
Similarly, in the “VIS to IR” mode, our method attained an accuracy of 92.4%
in Rank-1 and 86.7% in mAP, and our method outperformed SEFL in both Rank-1
and mAP metrics. These experimental results demonstrate that our method also
achieves significant effectiveness with small-scale datasets.

TABLE V: Ablation studies of the proposed ScRL. B: Baseline. S: SFP. I: ISR. A:AFE.  Setting | Component | All search | Indoor search
---|---|---|---
|  S |  I |  A |  R1 |  R10 |  R20 |  mAP |  mINP |  R1 |  R10 |  R20 |  mAP |  mINP
B |  |  |  |  70.2 |  95.7 |  98.6 |  66.9 |  52.9 |  77.6 |  97.8 |  99.5 |  81.1 |  77.4
+S |  ✓ |  |  |  73.6 |  97.1 |  99.2 |  70.0 |  56.5 |  79.0 |  98.2 |  99.4 |  82.8 |  79.4
+I |  ✓ |  ✓ |  |  74.6 |  97.4 |  99.3 |  71.0 |  57.7 |  79.4 |  98.4 |  99.3 |  83.2 |  79.8
+A |  ✓ |  ✓ |  ✓ |  76.1 |  97.6 |  99.4 |  72.6 |  59.8 |  82.4 |  98.8 |  99.8 |  85.4 |  82.2
TABLE VI: Ablation studies of the proposed AFE. S1: the attention of Stage 1 . S2: the attention of Stage 2. Setting | Component | All search | Indoor search
---|---|---|---
|  S2 |  S1 |  R1 |  R10 |  R20 |  mAP |  mINP |  R1 |  R5 |  R10 |  mAP |  mINP
B+S+I |  |  |  74.6 |  97.4 |  99.3 |  71.0 |  57.7 |  79.4 |  98.4 |  99.3 |  83.2 |  79.8
+S2 |  ✓ |  |  75.3 |  97.6 |  99.3 |  71.9 |  58.9 |  80.4 |  98.6 |  99.7 |  84.0 |  80.7
+S1 |  ✓ |  ✓ |  76.1 |  97.6 |  99.4 |  72.6 |  59.8 |  82.4 |  98.8 |  99.8 |  85.4 |  82.2
TABLE VII: Comparison of performance and computational costs on different settings in the inference stage. Setting 1: Baseline. Setting 2: Two independent networks extract shape and appearance features respectively, and concatenate the two features for inference. Setting 3: Baseline + SFP. Set 4: the ScRL. HP: human parsing network.  Setting |  SYSU-MM01(All search) | Params/M (FLOPs/G)
---|---|---
|  R1 |  mAP |  mINP |  HP |  $\bm{F}_{s}$ |  $\bm{F}_{a}$ |  Total
1 |  70.2 |  66.9 |  52.9 |  ✗ |  ✗ |  23.5(6.9) |  23.5(6.9)
2 |  71.9 |  67.5 |  52.7 |  66.6(76.5) |  23.5(6.9) |  23.5(6.9) |  113.6(90.3)
|  |  |  |  HP |  $\bm{F}_{s}$ |  $\bm{F}_{a}$,$\bm{F}_{\tilde{s}}$ |
3 |  73.6 |  70.0 |  56.5 |  ✗ |  ✗ |  38.5(10.1) |  38.5(10.1)
4 |  76.1 |  72.6 |  59.8 |  ✗ |  ✗ |  38.5(10.1) |  38.5(10.1)

HITSZ-VCM. To verify the scalability of the proposed method, we conducted
experiments on the video-based VI-ReID. Specifically, similar to SEFL, we can
obtain sequence-level feature by applying an average pooling layer to the
frame level. As showcased in Tab. IV, our method has exhibited superior
performance compared to all methods grounded in video (image) strategies and
achieved 71.2%(73.3%) and 52.9%(53.0%) accuracy in Rank-1 and mAP in the “VIS
to IR”(“IR to VIS”) mode, respectively. In comparison to the video-based
approach, our method has demonstrated superior performance by exceeding the
suboptimal SADSTRM approach by 5.9% and 3.4% in terms of Rank-1 and mAP,
respectively, within the “IR to VIS” mode. Additionally, our method has also
outperformed the suboptimal IBAN approach by 3.7% and 2% in terms of Rank-1
and mAP in the “VIS to IR” mode. Furthermore, when compared to the image-based
approach, our method has achieved remarkable results in the “VIS to IR” (“IR
to VIS”) mode, surpassing the suboptimal SEFL approach by 3.5% (3.1%) in
Rank-1 and 0.6% (0.5%) in mAP. The experimental results suggest that our
method can extract shape-centered features that remain insensitive to modality
variations at the frame level, ensuring robustness in pedestrian feature
representation at the sequence level. As a result, it achieves the best
performance.

The comparative experimental results have confirmed the effectiveness and
advancements of our method for image-based VI-Reid tasks on both the large-
scale dataset SYSU-MM01 and the small-scale dataset RegDB. Moreover, our
method has demonstrated its effectiveness and advancements in video-based VI-
Reid tasks on the large-scale video dataset MITML. These findings underscore
the significance of learning shape-centered features, offering tangible
benefits for cross-modality pedestrian retrieval.

###  IV-D Ablation Studies

In this subsection, we analyze the contribution of different components
including SFP (Shape Feature Propagation), ISR (Infrared Shape Restitution),
AFE (Appearance Feature Enhancement) based on the SYSU-MM01 dataset.

Effectiveness of SFP. To verify the effectiveness of SFP, we integrated the
SFP into the Baseline, resulting in “B+S”. This augmentation empowers the
model to autonomously acquire shape features from the original image during
inference, eliminating the need for shape feature learning and human parsing
networks. Specifically, as shown in Tab. V, in “All search”(“Indoor search”)
mode, “B+S” has demonstrated improvements of 3.4%(1.4%) in Rank-1, 3.1%(1.7%)
in mAP, and 3.6%(2.0%) in mINP when contrasted with the Baseline. These
improvements are attributed to the fact that shape features, which remain
unaffected by modality changes, complement appearance features, thereby
enhancing the overall performance.

Figure 6: Retrieval results under different ablation settings on SYSU-MM01. a,
b, c, and d denote “B”, “B+S”, “B+S+I”, and “B+S+I+A”, respectively. The green
(red) border represents the same (different) identity. It is evident that the
incorporation of various modules has led to a notable enhancement in the hit
rate for pedestrian retrieval.

Furthermore, we conducted comprehensive experiments on both model and
computational complexity. As shown in Tab. VII, we validated the
complementarity of shape features to appearance features through experiments
in Setting 2, specifically, we utilized two independent networks to extract
shape and appearance features and then concatenated these two features for
inference. Compared to the Baseline, the Rank-1 (mAP) of Setting 2 increased
by 1.7% (0.6%) by increasing the model parameter (computational complexity) by
90.1M (83.4G). In contrast, “B+S” (Setting 3) achieved an even greater
improvement of 3.4% in Rank-1 (and 3.1% in mAP) with a relatively modest
increment of 15M parameters and 3.2G in computational complexity. This
suggests that propagating shape features to the appearance stream network can
enhance the acquisition of shape-related features, thus improving the model’s
performance. In comparison to Setting 2, The introduction of SFP has also
reduced the computational complexity of the inference process, as it
eliminates the need for the involvement of shape stream networks and human
parsing networks.

Figure 7: Visualization of regions that the model interests. The first row
depicts the original input pedestrian images, with four pedestrians, each
having both an infrared and a visible image. The second row illustrates the
region where the Baseline method captures features, while the third row
showcases the region where the proposed ScRL method captures features. It’s
evident that the ScRL method outperforms the Baseline significantly in
capturing these features.

Effectiveness of ISR. Incorporating SFP extends the shape stream’s
capabilities to the appearance stream. However, the incorrect IR shape
restricts the potential of the shape stream, subsequently curbing the efficacy
of the appearance stream. To tackle this challenge, ISR was introduced to the
“B+S”, yielding“B+S+I”. As highlighted in Tab. V, the Rank-1, mAP, and mINP
metrics have collectively improved by 1.0%(0.4%), 1.0%(0.4%), and 1.2%(0.4%)
in “All search”(“Indoor search”) mode respectively. The performance
improvement suggests the feasibility of extracting pertinent information for
repairing infrared shape features from the intermediate features of the
appearance stream network and underscores ISR’s prowess in restituting
inaccurate IR shapes.

Effectiveness of AFE. In order to accentuate appearance features linked with
shape, we incorporated the AFE into the “B+S+I”. The corresponding results, as
depicted in Tab. V, reflect an enhancement in Rank-1, mAP, and mINP, in “All
search” mode, these metrics increased from 74.6%, 71.0%, and 57.7% to 76.1%,
72.6%, and 59.8%, respectively, in “Indoor search” mode, the metrics improved
from 79.4%, 83.2%, and 79.8% to 82.4%, 85.4%, and 82.2%, respectively. This
suggests that in contrast to typical appearance features, appearance features
centered on shape are better at emphasizing identity-discriminative
information related to individuals while minimizing the influence of
background and other irrelevant noise features. Furthermore, as delineated in
Tab. VII, the computational overhead imposed by AFE is negligibly minimal
after rounding, rendering it inconsequential during the inference phase. This
ensures that the introduction of AFE hardly imposes any burden on the model’s
inference process.

AFE effectively learns appearance features that have both direct and indirect
connections to shape. To confirm its ability to capture features directly
associated with shape, we replaced the original query feature of Stage 2 (S2)
with shape features. The results in Table VI show that adding S2 to “B+S+I” to
create “B+S+I+S2” resulted in improvements across Rank-1, mAP, and mINP.
Specifically, when compared to “B+S+I,” there were increases of 0.7%(1.0%),
0.9%(0.8%), and 1.2%(0.9%) in “All search”(“Indoor search”) mode,
respectively. Furthermore, when incorporating S1 into the “B+S+I+S2”
configuration to create “B+S+I+S2+S1,” the metrics exhibited further
improvements. In the “All search” mode, Rank-1, mAP, and mINP increased from
75.3%, 71.9%, and 58.9% to 76.1%, 72.6%, and 59.8%, respectively. Similarly,
in the Indoor search” mode, the metrics improved from 80.4%, 84.0%, and 80.7%
to 82.4%, 85.4%, and 82.2%, respectively. The experimental results affirm that
the strategy of initially learning appearance features directly associated
with shape and subsequently those indirectly linked is effective. This is due
to the fact that appearance features, whether directly or indirectly connected
to shape, are modality-insensitive discriminative features pertaining to human
subjects.

###  IV-E Visualization

This section encompasses visualization experiments on retrieval results and
model interest.

Retrieval result. In order to further analyze the effectiveness of different
ablation settings, as shown in Fig. 6, we visualized the retrieval results of
different ablation settings. Moving left to right, the degree of similarity
with the query image progressively decreases. Compared to the Baseline, it is
clear that the addition of each module yields improvements in retrieval
performance.

Model interest. The proposed ScRL primarily learns modality-invariant
pedestrian feature representations with a focus on shape. To illustrate the
advantages of our method more vividly, we have visualized the heatmap, as
displayed in Fig. 7. It is evident that, in comparison to the Baseline method,
the proposed ScRL excels in emphasizing more discriminative information about
the human body area. Furthermore, it effectively suppresses interference
stemming from background noise. This is evident in Fig. 7, where the baseline
method concentrates on background objects (the trash cans in columns 5 of Fig.
7, as well as the patterns on the wall in column 6, etc), whereas our proposed
method exclusively emphasizes the body area.

##  V Conclusion and Future Works

This paper introduces the innovative ScRL framework designed to enhance the
effective utilization of body shape in VI-ReID. Firstly, we introduce Shape
Feature Propagation (SFP), which directly extracts shape features from
visible(infrared) pedestrian images. The distinctive characteristic of SFP
lies in its ability to avoid the need for the auxiliary model, resulting in a
significant reduction in computational complexity. Additionally, the proposed
Infrared Shape Restitution (ISR) restitutes errors in the infrared shape at
the feature level. This correction enhances the discriminative capacity of
infrared shape features. Furthermore, we present the Appearance Feature
Enhancement (AFE), which focuses on learning shape-centered appearance
features while concurrently suppressing irrelevant identity-unrelated
information. The synergy of these techniques culminates in the ScRL framework.
Extensive experiments have validated the superiority of ScRL on the SYSU-MM01,
RegDB image datasets, and HITSZ-VCM video datasets.

Nonetheless, our proposed method has not taken into account the significant
variations in shape features arising from different camera perspectives. This
oversight may result in inadequate discrimination of shape features,
subsequently impacting the learning of appearance features related to shape,
especially when dealing with modality changes. Therefore, in future research,
we will delve deeper into the exploration of methods for learning camera-
invariant shape features, with the ultimate goal of enhancing the learning of
appearance features related to shape.

## References

  * [1] Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, “Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline),” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [2] H. Luo, Y. Gu, X. Liao, S. Lai, and W. Jiang, “Bag of tricks and a strong baseline for deep person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2019.
  * [3] M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S. C. H. Hoi, “Deep learning for person re-identification: A survey and outlook,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 6, pp. 2872–2893, 2022.
  * [4] S. He, H. Luo, P. Wang, F. Wang, H. Li, and W. Jiang, “Transreid: Transformer-based object re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021, pp. 15 013–15 022.
  * [5] H. Tang, J. Liu, S. Yan, R. Yan, Z. Li, and J. Tang, “M3net: Multi-view encoding, matching, and fusion for few-shot fine-grained action recognition,” in _ACM international conference on Multimedia (MM)_ , 2023.
  * [6] N. Dong, L. Zhang, S. Yan, H. Tang, and J. Tang, “Erasing, transforming, and noising defense network for occluded person re-identification,” _arXiv_ , 2023.
  * [7] H. Li, S. Yan, Z. Yu, and D. Tao, “Attribute-identity embedding and self-supervised learning for scalable person re-identification,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 10, pp. 3472–3485, 2019.
  * [8] S. Yan, H. Tang, L. Zhang, and J. Tang, “Image-specific information suppression and implicit local alignment for text-based person search,” _IEEE Transactions on Neural Networks and Learning Systems_ , pp. 1–14, 2023.
  * [9] S. Yan, N. Dong, J. Liu, L. Zhang, and J. Tang, “Learning comprehensive representations with richer self for text-to-image person re-identification,” in _ACM international conference on Multimedia (MM)_ , 2023.
  * [10] S. Yan, Y. Zhang, M. Xie, D. Zhang, and Z. Yu, “Cross-domain person re-identification with pose-invariant feature decomposition and hypergraph structure alignment,” _Neurocomputing_ , vol. 467, pp. 229–241, 2022.
  * [11] S. Yan, N. Dong, L. Zhang, and J. Tang, “Clip-driven fine-grained text-image person re-identification,” _arXiv_ , 2022.
  * [12] Y. Wang, G. Qi, S. Li, Y. Chai, and H. Li, “Body part-level domain alignment for domain-adaptive person re-identification with transformer framework,” _IEEE Transactions on Information Forensics and Security_ , vol. 17, pp. 3321–3334, 2022.
  * [13] Y. Zhang, Y. Wang, H. Li, and S. Li, “Cross-compatible embedding and semantic consistent feature construction for sketch re-identification,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, pp. 3347–3355.
  * [14] S. Li, F. Li, J. Li, H. Li, B. Zhang, D. Tao, and X. Gao, “Logical relation inference and multiview information interaction for domain adaptation person re-identification,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [15] A. Wu, W.-S. Zheng, H.-X. Yu, S. Gong, and J. Lai, “Rgb-infrared cross-modality person re-identification,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5380–5389.
  * [16] B. Yang, M. Ye, J. Chen, and Z. Wu, “Augmented dual-contrastive aggregation learning for unsupervised visible-infrared person re-identification,” in _ACM MM_ , 2022, p. 2843–2851.
  * [17] B. Yang, J. Chen, X. Ma, and M. Ye, “Translation, association and augmentation: Learning cross-modality re-identification from single-modality annotation,” _IEEE Transactions on Image Processing_ , vol. 32, pp. 5099–5113, 2023.
  * [18] N. Huang, K. Liu, Y. Liu, Q. Zhang, and J. Han, “Cross-modality person re-identification via multi-task learning,” _Pattern Recognition_ , vol. 128, p. 108653, 2022.
  * [19] J. Feng, A. Wu, and W.-S. Zheng, “Shape-erased feature learning for visible-infrared person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 22 752–22 761.
  * [20] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian, “Scalable person re-identification: A benchmark,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1116–1124.
  * [21] Z. Zheng, L. Zheng, and Y. Yang, “Unlabeled samples generated by gan improve the person re-identification baseline in vitro,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 3754–3762.
  * [22] L. Wei, S. Zhang, W. Gao, and Q. Tian, “Person transfer gan to bridge domain gap for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 79–88.
  * [23] A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” _arXiv preprint arXiv:1703.07737_ , 2017.
  * [24] Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, “Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline),” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 480–496.
  * [25] G. Wang, Y. Yuan, X. Chen, J. Li, and X. Zhou, “Learning discriminative features with multiple granularities for person re-identification,” in _Proceedings of the 26th ACM international conference on Multimedia_ , 2018, pp. 274–282.
  * [26] S. He, H. Luo, P. Wang, F. Wang, H. Li, and W. Jiang, “Transreid: Transformer-based object re-identification,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 15 013–15 022.
  * [27] Z. Zeng, Z. Wang, Z. Wang, Y. Zheng, Y.-Y. Chuang, and S. Satoh, “Illumination-adaptive person re-identification,” _IEEE Transactions on Multimedia_ , vol. 22, no. 12, pp. 3064–3074, 2020.
  * [28] L. Zheng, Y. Huang, H. Lu, and Y. Yang, “Pose-invariant embedding for deep person re-identification,” _IEEE Transactions on Image Processing_ , vol. 28, no. 9, pp. 4500–4509, 2019.
  * [29] Z. Zhong, L. Zheng, Z. Zheng, S. Li, and Y. Yang, “Camera style adaptation for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5157–5166.
  * [30] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [31] Y. Hao, N. Wang, J. Li, and X. Gao, “Hsme: Hypersphere manifold embedding for visible thermal person re-identification,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 33, no. 01, 2019, pp. 8385–8392.
  * [32] D. Cheng, X. Wang, N. Wang, Z. Wang, X. Wang, and X. Gao, “Cross-modality person re-identification with memory-based contrastive embedding,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 1, 2023, pp. 425–432.
  * [33] Y. Zhang, Y. Yan, J. Li, and H. Wang, “Mrcn: A novel modality restitution and compensation network for visible-infrared person re-identification,” _arXiv preprint arXiv:2303.14626_ , 2023.
  * [34] Q. Wu, P. Dai, J. Chen, C.-W. Lin, Y. Wu, F. Huang, B. Zhong, and R. Ji, “Discover cross-modality nuances for visible-infrared person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4330–4339.
  * [35] Y. Zhang and H. Wang, “Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 2153–2162.
  * [36] Z. Wang, Z. Wang, Y. Zheng, Y.-Y. Chuang, and S. Satoh, “Learning to reduce dual-level discrepancy for infrared-visible person re-identification,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 618–626.
  * [37] D. Li, X. Wei, X. Hong, and Y. Gong, “Infrared-visible cross-modal person re-identification with an x modality,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 04, 2020, pp. 4610–4617.
  * [38] Z. Wei, X. Yang, N. Wang, and X. Gao, “Syncretic modality collaborative learning for visible infrared person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 225–234.
  * [39] M. M. Kalayeh, E. Basaran, M. Gökmen, M. E. Kamasak, and M. Shah, “Human semantic parsing for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1062–1071.
  * [40] C. Song, Y. Huang, W. Ouyang, and L. Wang, “Mask-guided contrastive attention model for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1179–1188.
  * [41] H. Huang, W. Yang, X. Chen, X. Zhao, K. Huang, J. Lin, G. Huang, and D. Du, “Eanet: Enhancing alignment for cross-domain person re-identification,” _arXiv preprint arXiv:1812.11369_ , 2018.
  * [42] J. Guo, Y. Yuan, L. Huang, C. Zhang, J.-G. Yao, and K. Han, “Beyond human parts: Dual part-aligned representations for person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3642–3651.
  * [43] K. Zhu, H. Guo, Z. Liu, M. Tang, and J. Wang, “Identity-guided human semantic parsing for person re-identification,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_. Springer, 2020, pp. 346–363.
  * [44] P. Hong, T. Wu, A. Wu, X. Han, and W.-S. Zheng, “Fine-grained shape-appearance mutual learning for cloth-changing person re-identification,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 10 513–10 522.
  * [45] P. Li, Y. Xu, Y. Wei, and Y. Yang, “Self-correction for human parsing,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 6, pp. 3260–3271, 2020.
  * [46] M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao, and S. C. Hoi, “Deep learning for person re-identification: A survey and outlook,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 6, pp. 2872–2893, 2021.
  * [47] M. Ye, W. Ruan, B. Du, and M. Z. Shou, “Channel augmented joint learning for visible-infrared recognition,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 13 567–13 576.
  * [48] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.
  * [49] D. T. Nguyen, H. G. Hong, K. W. Kim, and K. R. Park, “Person recognition system based on a combination of body images from visible light and thermal cameras,” _Sensors_ , vol. 17, no. 3, p. 605, 2017.
  * [50] X. Lin, J. Li, Z. Ma, H. Li, S. Li, K. Xu, G. Lu, and D. Zhang, “Learning modal-invariant and temporal-memory for video-based visible-infrared person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 20 973–20 982.
  * [51] M. Ye, X. Lan, J. Li, and P. Yuen, “Hierarchical discriminative learning for visible thermal person re-identification,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [52] M. Ye, X. Lan, Z. Wang, and P. C. Yuen, “Bi-directional center-constrained top-ranking for visible thermal person re-identification,” _IEEE Transactions on Information Forensics and Security_ , vol. 15, pp. 407–419, 2019.
  * [53] S. Choi, S. Lee, Y. Kim, T. Kim, and C. Kim, “Hi-cmd: Hierarchical cross-modality disentanglement for visible-infrared person re-identification,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 257–10 266.
  * [54] M. Ye, J. Shen, D. J. Crandall, L. Shao, and J. Luo, “Dynamic dual-attentive aggregation learning for visible-infrared person re-identification,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16_. Springer, 2020, pp. 229–247.
  * [55] M. Ye, J. Shen, and L. Shao, “Visible-infrared person re-identification via homogeneous augmented tri-modal learning,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 728–739, 2020.
  * [56] X. Hao, S. Zhao, M. Ye, and J. Shen, “Cross-modality person re-identification via modality confusion and center aggregation,” in _Proceedings of the IEEE/CVF International conference on computer vision_ , 2021, pp. 16 403–16 412.
  * [57] C. Fu, Y. Hu, X. Wu, H. Shi, T. Mei, and R. He, “Cm-nas: Cross-modality neural architecture search for visible-infrared person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 823–11 832.
  * [58] Q. Zhang, C. Lai, J. Liu, N. Huang, and J. Han, “Fmcnet: Feature-level modality compensation for visible-infrared person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 7349–7358.
  * [59] H. Lu, X. Zou, and P. Zhang, “Learning progressive modality-shared transformers for effective visible-infrared person re-identification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 2, 2023, pp. 1835–1843.
  * [60] Y. Zhang, Y. Kang, S. Zhao, and J. Shen, “Dual-semantic consistency learning for visible-infrared person re-identification,” _IEEE Transactions on Information Forensics and Security_ , vol. 18, pp. 1554–1565, 2022.
  * [61] M. Ye, Z. Wang, X. Lan, and P. C. Yuen, “Visible thermal person re-identification via dual-constrained top-ranking.” in _IJCAI_ , vol. 1, 2018, p. 2.
  * [62] H. Li, M. Liu, Z. Hu, F. Nie, and Z. Yu, “Intermediary-guided bidirectional spatial-temporal aggregation network for video-based visible-infrared person re-identification,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [63] H. Li, L. Xu, Y. Zhang, D. Tao, and Z. Yu, “Adversarial self-attack defense and spatial-temporal relation mining for visible-infrared video person re-identification,” _arXiv preprint arXiv:2307.03903_ , 2023.
  * [64] H. Park, S. Lee, J. Lee, and B. Ham, “Learning by aligning: Visible-infrared person re-identification using cross-modal correspondences,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 12 046–12 055.
  * [65] X. Tian, Z. Zhang, S. Lin, Y. Qu, Y. Xie, and L. Ma, “Farewell to mutual information: Variational distillation for cross-modal person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1522–1531.