(cvpr) Package cvpr Warning: Package ‘hyperref’ is not loaded, but highly
recommended for camera-ready version

# Cloth-Changing Person Re-identification from A Single Image
with Gait Prediction and Regularization

Xin Jin1,2, Tianyu He2, Kecheng Zheng1, Zhiheng Yin3, Xu Shen2, Zhen Huang1,
Ruoyu Feng1,
Jianqiang Huang2, Zhibo Chen1, Xian-Sheng Hua222footnotemark: 2
1University of Science and Technology of China, 2Alibaba Cloud Computing Ltd.
3University of Michigan
{jinxustc,zkcys001,hz13,ustcfry}@mail.ustc.edu.cn,yzhiheng<EMAIL_ADDRESS>
<EMAIL_ADDRESS>This work
was done when he was visiting Alibaba as a research intern.Corresponding
author.

###### Abstract

Cloth-Changing person re-identification (CC-ReID) aims at matching the same
person across different locations over a long-duration, e.g., over days, and
therefore inevitably has cases of changing clothing. In this paper, we focus
on handling well the CC-ReID problem under a more challenging setting, i.e.,
just from a single image, which enables an efficient and latency-free person
identity matching for surveillance. Specifically, we introduce Gait
recognition as an auxiliary task to drive the Image ReID model to learn cloth-
agnostic representations by leveraging personal unique and cloth-independent
gait information, we name this framework as GI-ReID. GI-ReID adopts a two-
stream architecture that consists of an image ReID-Stream and an auxiliary
gait recognition stream (Gait-Stream). The Gait-Stream, that is discarded in
the inference for high efficiency, acts as a regulator to encourage the ReID-
Stream to capture cloth-invariant biometric motion features during the
training. To get temporal continuous motion cues from a single image, we
design a Gait Sequence Prediction (GSP) module for Gait-Stream to enrich gait
information. Finally, a semantics consistency constraint over two streams is
enforced for effective knowledge regularization. Extensive experiments on
multiple image-based Cloth-Changing ReID benchmarks, e.g., LTCC, PRCC, Real28,
and VC-Clothes, demonstrate that GI-ReID performs favorably against the state-
of-the-art methods.

##  1 Introduction

Person re-identification (ReID) aims at identifying a specific person across
cameras, times, and locations. Abundant approaches have been proposed to
address the challenging geometric misalignment among person images caused by
diversities of human poses [58, 78, 52], camera viewpoints [74, 60, 26], and
style/scales [28, 27]. These methods usually inadvertently assume that both
query and gallery images of the same person have the _same clothing_. In
general, they perform well on the trained short-term datasets but suffer from
signiﬁcant performance degradations when testing on a long-term collected ReID
dataset [69, 54, 71, 63]. Because large clothing variations occur over long-
duration among these datasets, which seriously hinders the accuracy of ReID.
For example, Figure 1(a) shows a realistic wanted case 111Information comes
from https://www.wjr.com/2016/01/06/woman-wanted-in-southwest-detroit-bank-
robbery/ where a suspect that captured by surveillance devices at different
times/locations changed her coat from black to white, which makes ReID
difficult, especially when she wears a mask and the captured images are of low
quality.

Figure 1: (a) shows a realistic wanted case that a suspect changed her coat
from black to white for hiding. (b) reveals that the gait of person could help
ReID, especially when the identity matching meets the cloth-changing challenge
(All faces in the images are masked for anonymization).

In recent years, to handle the cloth-changing ReID (CC-ReID) problem, some
studies have contributed some new datasets where clothing changes are
commonplace (e.g., Celebrities-reID [22, 20], PRCC [69], LTCC [54], Real28 and
VC-Clothes [63]). They also propose some new algorithms that could learn
cloth-agnostic representations for CC-ReID. For instance, Yang  _et al_. [69]
propose a contour-sketch-based network to overcome the moderate cloth-changing
problem. Similarly, Qian  _et al_. [54], Li  _et al_. [38], and Hong  _et al_.
[18] all use body shape to tackle the CC-ReID problem. However, no matter of
using a contour sketch or body shape, all these methods are prone to suffer
from the estimation error problem. Because the single-view contour/shape
inference (from 2D image) is extremely difficult due to the vast range of
possible situations, especially when people wear thick clothes in winter.
Besides, these contour-sketch-based or shape-based methods only focus on
extracting _static spatial cues_ from persons as extra cloth-agnostic
representations, the rich _dynamic motion information_ (e.g., gait, implied
motion [33]) are often ignored.

In this paper, we explore to leverage the unique gait features that imply
dynamic motion cues of a pedestrian to drive a model to learn cloth-agnostic
and discriminative ReID representations. As shown in Figure 1(b), although it
is hard to identify the same person when he/she wears different clothes, or to
distinguish the different persons when they wear similar/same clothes, we can
still leverage their unique/discriminative gaits to achieve correct identity
matching. It is because that gait, as a unique biometric feature, has the
superior invariance compared with other easy-changing appearance
characteristics, e.g., face, body shape, contour [43, 75]. Besides, gait can
be authenticated at a long distance even with low-quality camera imaging.

Unfortunately, existing gait-related studies mainly rely on large video
sequences [3, 9]. Capturing videos requires time latency and saving videos
needs a large hardware storage cost, which are both undesirable for the real-
time ReID applications. Even the recent work [67] first attempts to achieve
gait recognition from a single image, how to leverage gait feature to handle
CC-ReID problem from a single image is still under-studied and this task is
more challenging due to the potential viewpoint-variations and occlusions.

In this paper, we propose a Gait-assisted Image-based ReID framework, termed
as GI-ReID, which could learn cloth-agnostic ReID representations from a
single image with the gait feature assistance. GI-ReID consists of a main
image-based ReID-Stream and an auxiliary gait recognition stream (Gait-
Stream). Figure 2 shows the entire framework. The Gait-Stream aims to
regularize the ReID-Stream to learn cloth-agnostic features from a single RGB
image for effective CC-ReID. It is discarded in the inference for the high
efficiency. Since the comprehensive gait features extraction typically needs a
gait video sequence as input [3, 9], we introduce a new Gait Sequence
Prediction (GSP) module for Gait-Stream to approximately forecast continuous
gait frames from a single input query image, which enriches the learned gait
information. Finally, to encourage the main ReID-Stream’s efficient learning
from Gait-Stream, we further enforce a high-level Semantics Consistency (SC)
constraint for the same person over two streams’ features. We summarize our
main contributions as follows:

  * •

We specially aimed at handling the challenging cloth-changing issue for image
ReID to promote practical applications. A Gait-assisted Image-based cloth-
changing ReID (GI-ReID) framework is proposed. As a regulator, the Gait-Stream
in GI-ReID can be removed in the inference without sacrificing ReID
performance. This reduces the dependency on the accuracy of gait recognition,
making our method computationally efficient and robust.

  * •

A well-designed Gait Sequence Prediction (GSP) module makes our method
effective in the challenging image-based ReID scenarios. And, a high-level
semantics consistency (SC) constraint enables an effective regularization over
two streams, enhancing the distinguishing power of ReID-Stream under the
cloth-changing setting.

With the gait prediction and regularization, GI-ReID achieves a state-of-the-
art performance on the image-based cloth-changing ReID. It is also general
enough to be compatible with the existing ReID-specific networks, except
ResNet-50 [14], we also use OSNet [81], LTCC-shape [54], and PRCC-contour [69]
as our baselines for evaluation.

Figure 2: Overview of the proposed GI-ReID, which consists of _ReID-Stream_
and _Gait-Stream_ , they are jointly trained with a high-level Semantics
Consistency (SC) constraint. The _Gait-Stream_ plays the role of a regulator
to drive _ReID-Stream_ to learn cloth-agnostic representations from a single
image, and it is discarded in the inference for computational efficiency. Gait
Sequence Prediction (GSP) module aims at predicting gait frames from an image.
GaitSet [3] is responsible for extracting discriminative gait features.

##  2 Related Work

###  2.1 Person Re-identification

General ReID. Without cloth-changing cases, the general ReID has achieved a
great success with the deep learning. It includes exploring fine-grained
pedestrian feature descriptions [61, 64, 10, 81], and addressing spatial
misalignment caused by (a) different camera viewpoints [59, 26], (b) different
poses [58, 11, 52], (c) semantics inconsistency [77, 28], (d)
occlusion/partial-observation [82, 46, 80, 15], etc. These methods rely
substantially on static spatial texture information. However, when person ReID
meets changing clothes, the texture information is not so reliable since it
changes significantly even for the same person. Compared to static texture,
the gait information, as a discriminative biometric modality, is more
consistent and reliable.

Cloth-Changing ReID. Considering the wider application range and greater
practical value of Cloth-Changing ReID (CC-ReID), more and more studies pay
their attention to solve this challenging problem. Huang  _et al_. [22, 20]
propose to use vector-neuron capsules [55] to perceive cloth changes of the
same person. Yang  _et al_. [69], Qian  _et al_. [54]/ Li  _et al_. [38], Yu
_et al_. [71]/Wan  _et al_. [63] propose to leverage contour sketch, body
shape, face/hairstyle to assist ReID under the cloth-changing setting,
respectively. Nevertheless, these methods usually suffer from estimation error
due to the difficulty of obtaining external cues (e.g., body shape, face,
etc). Besides, they also ignore the exploration of discriminative dynamic
motion cues, like gait.

FITD [75] solves the cloth-changing ReID problem based on true motion cues of
videos. Our work differs from FITD for at least three perspectives: 1). FITD
uses motion information derived from dense trajectories (optical flow), which
requires continuous video sequences. Our GI-ReID handles cloth-changing ReID
from a single image with gait prediction and regularization, which is more
challenging and practical. 2). FITD directly uses human motion cues to
complete ReID, which relies on the accurate motion prediction and may suffer
from estimation errors. Our GI-ReID just takes the gait recognition task as a
regulator to drive the main ReID model to learn cloth-independent features,
which makes our method less sensitive to gait estimation errors. 3). FITD only
characterizes temporal motion patterns for ReID, ignoring other
distinguishable local spatial cues, like personal belongings (e.g.,
backpacks). Our GI-ReID not only explores dynamic gait cues, but also learns
from raw RGB images, leading more comprehensive features.

Table 1: Differences between Gait Recognition and CC-ReID. Task | Gait Recognition | Cloth-Changing Person ReID
---|---|---
Data format |  | Gait Energy Image (GEI) /
---
Sequence set of silhouette /
Video sequences
|  Discontinuous RGB images
---
across cameras
Datasets |  | USF, CASIA-B,
---
OU-ISIR, OU-MVLP, etc.
| COCAS, PRCC, LTCC,
---
Real28, VC-Clothes, etc.
| Unsolved
---
problems
|  1) Viewing angles (e.g., frontal view);
---
2) Occlusion, body incompleteness;
3) Cluttered/complex background;
Clothes variation

###  2.2 Gait Recognition and Prediction

Gait recognition [47, 43, 44, 3, 2, 36, 9, 8, 67] directly uses gait sequence
for identity matching, which is also cloth-independent, but different from our
work and cannot be directly applied into image-based cloth-changing ReID. We
clarify the differences between two tasks in detail in Table 1: this paper
focuses on image-based cloth-changing ReID where large viewpoint variations,
occlusion, and complex environments make gait recognition failed. And, these
gait sequence based methods are not optimal for the image-based CC-ReID. Thus,
we just take the gait recognition as an auxiliary regularization to drive ReID
model to learn cloth-agnostic representations, which makes our method robust
to the recognition errors. Moreover, the gait representations can be grouped
into model-based [50, 39, 37] and appearance-based [2, 3, 9]. The first one
relies on human pose, while the latter relies on silhouettes. We use
silhouette as gait representation for simplicity and robustness.

Gait Prediction from a single frame, or said, the field of video frame
prediction (i.e., motion prediction) has been widely studied and achieved a
great success [13, 19, 42, 49, 67], which verifies the feasibility of our
work. This task is very challenging, that’s why we carefully design the gait
sequence prediction module while indirectly using the prediction results in a
robust regularization manner to help cloth-changing ReID.

##  3 Proposed GI-ReID Framework

GI-ReID framework aims to fully exploit the unique human gait to handle the
cloth-changing challenge of ReID just depending on a single image. Figure 2
shows the flowchart of the entire framework. Given a single person image, its
silhouette (i.e., mask) will be first extracted as input to the Gait-Stream
using semantic segmentation methods, such as PointRend [30]. With the proposed
gait sequence prediction (GSP) module, we could predict a gait sequence with
more comprehensive gait information, which is then fed into the subsequent
recognition network (GaitSet [3]) to extract discriminative gait features.
Through a high-level semantics consistency (SC) constraint, the cloth-
independent Gait-Stream acts as a regulator to encourage the main ReID-Stream
to capture cloth-agnostic features from a single RGB image. We discuss the
details of each component in the following sections.

###  3.1 The Auxiliary Gait-Stream

Gait-Stream is composed of two parts: Gait Sequence Prediction (GSP) module
and the pre-trained gait recognition network (GaitSet [3]). GSP is designed
for gait information augmentation. Then, GaitSet extracts cloth-independent
and discriminative motion feature cues from augmented gait to guide/regularize
ReID-Stream’s training.

Gait Sequence Prediction (GSP) Module: GSP module aims to predict a gait
sequence that contains continuous gait frames. This module is related to the
general video frame prediction task (i.e., frame interpolation and
extrapolation studies [49, 19, 42, 13]), and gait sequence prediction can be
deemed as a “gait frame synthesis” process.

As shown in Figure 2, GSP is based on an auto-encoder architecture [7] with
feature encoder $E$ and decoder $D$. In order to reduce the prediction
ambiguity and difficulty (e.g., given a dangling arm, it is hard to guess
whether it will rise or fall in the next frame), we manually integrate an
extra prior information of middle frame index into the inner learned feature
through a position embedder $P$ and a feature aggregator $A$. Intuitively, the
middle frame index means that the input gait silhouette corresponds to the
middle result of the predicted gait sequence. Such prior knowledge aims to
drive GSP module to predict the adjacent walking statuses _before and after_
the current input walking status so as to reduce prediction ambiguity.

(1). Encoder. Given a silhouette input $S$, the encoder $E$ aims to extract a
dimension-shrinked compact feature:

$\displaystyle f_{S}=E(S).$ (1)

Specific/detailed network structures (including other components in Gait-
Stream) can be found in Supplementary.

(2). Position Embedder and Feature Aggregator. Considering the prediction
ambiguity [45], we introduce a _middle frame input principle_ , which assumes
that the input silhouette always corresponds to the middle one of the
predicted gait sequence. During the GSP training, we take the gait frame in
the middle position of the ground truth gait sequence as input to GSP, and use
a one-dimensional vector $p\in\mathbb{R}^{1}$, to denote such _position
label_. Given a ground truth gait sequence with $N$ frames, the position label
$p_{mid}\in\mathbb{R}^{1}$ of the input middle gait is defined as
$p_{mid}=N//2$ which indicates the relative position relationship of input
frame to the entire sequence. For convenience, we convert position label to
one-hot vector to calculate loss. In formula, the position embedder $P$ works
as:

$\displaystyle\widetilde{p}=P(S)\hskip 1.42262pt\in\hskip
1.42262pt\mathbb{R}^{1},\hskip
8.53581pt\mathcal{L}_{position}=||\widetilde{p}-p_{mid}||_{2}^{2},$ (2)

where we compare the embedded position output $\widetilde{p}$ with the ground
truth $p_{mid}$ to construct a _position_ loss $\mathcal{L}_{position}$. $P$
is to build a mapping between input and middle position.

Feature aggregator $A$, implemented by a fully connected layer, is inserted
between the encoder and the decoder to convert the raw encoded features
$f_{S}$ into _middle-position-aware_ features $f_{S}^{\widetilde{p}}$ by
taking the embedded middle position information $\widetilde{p}$ into account
for the following decoder, which explicitly tells the decoder that we need to
predict the gait statuses _before and after_ the current input middle gait
status, and thus reduces prediction ambiguity for the predicted results. This
feature aggregation process is formulated as:

$\displaystyle f_{S}^{\widetilde{p}}=A([\hskip 1.42262ptf_{S},\hskip
1.42262pt\widetilde{p}\hskip 1.42262pt]),$ (3)

where $[\cdot]$ means a simple concatenation.

(3). Decoder. We feed the aggregated feature $f_{S}^{\widetilde{p}}$ into the
decoder $D$, which has a symmetrical structure to that of the encoder $E$, to
predict the gait sequence with a pre-defined fixed number of frames $N$. Such
process is formulated as,

$\displaystyle\widetilde{R}=D(f_{S}^{\widetilde{p}})\hskip 1.42262pt\in\hskip
1.42262pt\mathbb{R}^{N*h*w},\hskip
8.53581pt\mathcal{L}_{pred.}=||\widetilde{R}-GT||_{2}^{2},$ (4)

where $(h,w)$ denotes the $(height,width)$ of predicted gait frames, same as
the input silhouette image. A prediction loss $\mathcal{L}_{pred.}$ is
calculated to ensure the predicted gait sequence results is consistent with
the ground truth (GT).

Gait Feature Extraction: The predicted gait sequence $\widetilde{R}$ is fed
into the pre-trained GaitSet [3] to learn discriminative and cloth-independent
gait feature $g$. GaitSet is a set-based gait recognition model that takes a
set of silhouettes as an input and aggregates features over frames into a set-
level feature, which is formulated as $g=GaitSet(\widetilde{R})$. More details
are presented in Supplementary.

###  3.2 The Main ReID-Stream

The backbone of the ReID-Stream could be any of the off-the-shelf networks,
such as commonly-used ResNet-50 [14], ReID-specific PCB [61], MGN [64], and
OSNet [81]. And, we use the widely-adopted classification loss [61, 10], and
triplet loss with batch hard mining [16]) on the ReID feature vector $r$ as
basic optimization objectives for training. The feature $r$ is finally used
for reference.

###  3.3 Joint Learning of Two Streams

Due to the potential rough silhouette extraction and the gait sequence
prediction errors of GSP module, it is very difficult to directly exploit the
gait information alone to complete effective ReID. Experimentally, we have
attempted to conduct CC-ReID with only the predicted gait sequence
$\widetilde{R}$ as input, and found this scheme failed to deliver good results
(see ablation study for more details). Therefore, to exploit the cloth-
independent merits of the gait information while avoiding the above-mentioned
issues, we propose to jointly train Gait-Stream and ReID-Stream through a
high-level semantics consistency (SC) constraint, where gait characteristics
is taken as a regulator to drive the cloth-agnostic feature learning of ReID-
Stream. Note that the SC constraint is also not needed in the inference.

Semantics Consistency (SC) Constraint. SC constraint is essentially related to
the common feature learning works, such as knowledge distillation [17], mutual
learning [76], and knowledge amalgamation [70]. Our SC constraint differs from
them mainly in two perspectives: 1). SC is to encourage a high-level common
feature learning from two modalities (dynamic gait and static RGB image). 2).
SC ensures information integrity for each stream/modality.

The details of the SC constraint are shown in Figure 2. The learned gait
feature $g$ of Gait-Stream and ReID feature $r$ of ReID-Stream are first
transformed to a common and interactable space, via an embedding layer:
$\hat{r}=Emb.(r)$ and $\hat{g}=Emb.(g)$, where $\hat{r}$ and $\hat{g}$ have
the same feature dimensions. Then, we enforce the transformed features
$\hat{r}$ and $\hat{g}$ to be closed to each other by minimizing the Maximum
Mean Discrepancy (MMD) [12]. MMD is a distance metric to measure the domain
mismatch for probability distributions. We use it to measure the high-level
semantics discrepancy between the transformed features $\hat{r}$ and
$\hat{g}$, and minimize it to drive ReID-Stream to pay more attention to
cloth-independent gait biometric. An empirical approximation to the MMD
distance of $\hat{r}$ and $\hat{g}$ is simplified as follows:

$\displaystyle\mathcal{L}_{MMD}=\|{\mu}(\hat{g})-{\mu}(\hat{r})\|_{2}^{2}+\|{\sigma}(\hat{g})-{\sigma}(\hat{r})\|_{2}^{2},$
(5)

where $\mu(\cdot),\sigma(\cdot)$ denotes the mean, variance calculation
functions for the transformed features $\hat{r}$ and $\hat{g}$.

To avoid the information lost caused by feature regularization with SC
constraint, we further enforce a reconstruction penalty to ensure that the
transformed features $\hat{g}$ and $\hat{r}$ could be recovered to original
versions. Specifically, we reconstruct the original output features through a
_Recon._ layer (implemented by FC layer): $\widetilde{r}=Recon.(\hat{r})$ and
$\widetilde{g}=Recon.(\hat{g})$, and calculate the corresponding
reconstruction loss as follows:

$\displaystyle\mathcal{L}_{recon.}=\|\widetilde{g}-g\|_{2}^{2}+\|\widetilde{r}-r\|_{2}^{2}.$
(6)

Training Pipeline. The whole training process of the proposed GI-ReID consists
of three stages: 1). Pre-training GaitSet [3] for gait feature extraction. 2).
Joint Training for the proposed gait sequence prediction (GSP) module and
GaitSet in Gait-Stream on gait-related datasets. 3). Joint Training for Gait-
Stream and ReID-Stream on CC-ReID-related datasets. More details are provided
in Supplementary, including pseudo code and loss balance strategy.

##  4 Experiment

###  4.1 Datasets, Metric and Experimental Setups

Datasets Details. We use four recent cloth-changing ReID datasets Real28 [63],
VC-Clothes [63], LTCC [54], PRCC [69], and one general video ReID dataset MARS
[79] (to highlight the difficulty and necessity of image-based CC-ReID) to
perform experiments. Table 9 gives a brief information and comparison of these
ReID datasets. More detailed introductions can be found in Supplementary.

Table 2: Brief introduction and comparison of datasets. | MARS | Real28 | VC-Clothes | LTCC | PRCC
---|---|---|---|---|---
Category | Video | Image | Image | Image | Image
Photo Style | Real | Real | Synthetic | Real | Real
Scale | Large | Small | Large | Large | Large
Cloth Change | No | Yes | Yes | Yes | Yes
Identities | 1,261 | 28 | 512 | 152 | 221
Samples | 20,715 | 4,324 | 19,060 | 17,138 | 33,698
Cameras | 6 | 4 | 4 | N/A | 3
Usage | Train&Test | Test | Train&Test | Train&Test | Train&Test

Evaluation Metrics. We use the cumulative matching characteristics (CMC) at
Rank-1/-10/-20, and mean average precision (mAP) to evaluate the performance.

Experimental Setups. We build three kinds of different experiment settings to
comprehensively validate the effectiveness of gait biometric for person ReID,
and also validate the rationality/superiority of the proposed gait prediction
and regularization in our GI-ReID framework: (1) Real Cloth-Changing Image
ReID, (2) General Video ReID, and (3) Imitated Cloth-Changing Video ReID. In
the main manuscripts, to save space and highlight core contributions of our
paper, we only present the results related to the most challenging setting of
(1) Real Cloth-Changing Image ReID. The rest results about (2)(3) are in
Supplementary.

For (1) Real Cloth-Changing Image ReID, we employ real image-based cloth-
changing datasets Real28 [63], VC-Clothes [63], LTCC [54], and PRCC [69] for
experiments to validate the effectiveness of GSP module, SC constraint, and
also compare our GI-ReID with SOTA cloth-changing ReID methods. In this
setting, GSP module and GaitSet are both first pre-trained on gait-specific
datasets CASIA-B [3] and then fine-tuned on the CC-ReID datasets with the SC
constraint $\mathcal{L}_{MMD}\&\mathcal{L}_{recon.}$ and the ReID
supervisions. ResNet-50 [14], OSNet [81], LTCC-shape [54], and PRCC-contour
[69] are taken as ReID backbone for comparisons.

###  4.2 Ablation Study

Baseline means the model that only ingests RGB images.

Results of Real Cloth-Changing Image ReID. We conduct ablation experiments on
three cloth-changing datasets Real28, VC-Clothes, and LTCC. Real28 is too
small for training, so we train model on VC-Clothes and only test on Real28
[63]. In Table 3, we see that 1) All Gait-Stream (GS) related schemes achieve
obvious gains (over 2.7% in mAP) over _Baseline_ , which demonstrates the
effectiveness of using gait to handle cloth-changing issue. 2) With the well-
designed GSP module, _Baseline+ GS-GSP (concat)_ outperforms the ablated
scheme _Baseline+GS (concat)_ by 3.3%/6.7%/2.3% in mAP on Real28/VC-
Clothes/LTCC, which demonstrates the effectiveness of gait sequence prediction
(GSP) on gait information augmentation. Note that _Baseline+GS (concat)_ just
uses Gait-Stream (GS) but removes GSP, where we duplicate the only available
single person silhouette as input to GaitSet. 3) Semantics consistency (SC)
performs well in the cloth-changing settings, it helps our scheme GI-ReID
achieve the best performance on the most evaluation cases while saving
computational cost by discarding Gait-Stream in the inference.

Table 3: Performance (%) comparison on the real image-based cloth-changing datasets Real28, VC-Clothes, LTCC. GS-GSP means Gait-Stream (GS) with gait sequence prediction (GSP) module. The ReID backbone is ResNet-50. ‘Standard’ is the setting where the images in the test set with the same identity and camera view are discarded when computing mAP/Rank-1 [54]. Methods | Real28 | VC-Clothes | LTCC (Standard)
---|---|---|---
mAP | Rank-1 | mAP | Rank-1 | mAP | Rank-1
Baseline | 4.1 | 6.7 | 49.1 | 53.7 | 23.2 | 55.1
\+ GS (concat) | 6.8 | 7.9 | 52.3 | 58.9 | 26.5 | 60.0
\+ GS-GSP (concat) | 10.1 | 10.8 | 59.0 | 63.7 | 28.8 | 64.5
\+ GS-GSP + SC (ours) | 10.4 | 11.1 | 57.8 | 64.5 | 29.4 | 63.2
Table 4: Performance (%) comparison on the cloth-changing dataset LTCC. Such experiment aims to show that our GI-ReID can bring gains because of the exploration of gait information, rather than simply introducing silhouettes (i.e., human masks). The ReID backbone is ResNet-50. Methods |  LTCC (Cloth-Changing)
---|---
mAP |  Rank-1
Baseline |  8.10 |  19.58
Silhouette-ReID |  7.04 |  17.92
GI-ReID (ours) |  10.38 |  23.72

Improvement Comes From Gait Prediction, Not Silhouettes Usage. We believe that
our GI-ReID could successfully address the cloth-changing ReID problem from a
single image is indeed because it effectively leverages the gait prediction,
instead of the introduction of human silhouettes (i.e., masks). To prove that,
we additionally design a scheme of _Silhouette-ReID_ that directly takes the
person RGB-Silhouette pair as input to ReID model (following [57, 4]), and
compare it with our GI-ReID on the cloth-changing ReID dataset LTCC. ResNet-50
is taken as ReID backbone for all schemes for comparison fairness. As shown in
Table 4, we found that _Silhouette-ReID_ is even inferior to the baseline
scheme _Baseline (ResNet-50)_ by 1.06% in mAP under the cloth-changing
setting. We analyze that directly using silhouette to remove the background
clutters in pixel-level will make ReID model pay more attention on the
foreground objects’ appearance/clothes color information, which is unexpected
and unreliable for cloth-changing ReID, and thus leads to a performance drop.

Study on Directly Using Gait Recognition Methods for Cloth-Changing ReID. As
we have discussed in the related work, directly using the algorithms of gait
recognition for solving cloth-changing ReID problem is not optimal, especially
in the image-based CC-ReID scenarios. Experimentally, we compare the proposed
GI-ReID with two popular pure gait recognition works, GaitSet [3] and PA-GCR
[67]. GaitSet needs a set/sequence of person silhouettes as input, but
recently-released cloth-changing ReID datasets are image datasets that lack of
continuous frames for the same person. Thus, we duplicate the only available
single one person silhouette to a set as input to approximately apply GaitSet
into image-based CC-ReID task. As shown in Table 5, these pure gait
recognition works of GaitSet [3] and PA-GCR [67] are both inferior to the
baseline scheme _Baseline (ResNet-50)_ in mAP under the cloth-changing
setting, which indicates that simply using gait biometric for person matching
can not work well for cloth-changing ReID, our gait prediction and
regularization idea performs better for handling CC-ReID, especially for the
image-based CC-ReID.

Table 5: Performance (%) comparison on the cloth-changing dataset LTCC. Such experiment aims to show that these pure gait recognition works can not work well for cloth-changing ReID. The ReID backbone is ResNet-50. Methods |  LTCC (Cloth-Changing)
---|---
mAP |  Rank-1
Baseline |  8.10 |  19.58
GaitSet [3] |  2.14 |  7.22
PA-GCR [67] |  3.36 |  9.01
GI-ReID (ours) |  10.38 |  23.72
Table 6: Study on the different design choices in the (a)(b) GSP module, and
(c) SC constraint of our GI-ReID framework. ‘Cloth-Changing’ setting means
that the images with same identity, camera view and clothes are discarded
during the testing.

Methods | LTCC
---|---
Standard | Cloth-Changing
mAP | Rank-1 | mAP | Rank-1
Baseline | 23.2 | 55.1 | 8.1 | 19.6
N=4 | 26.9 | 59.2 | 8.9 | 21.7
N=6 | 28.2 | 61.9 | 9.8 | 22.6
N=8 (ours) | 29.4 | 63.2 | 10.4 | 23.7
N=10 | 28.4 | 63.1 | 10.4 | 22.8
N=12 | 27.7 | 60.8 | 10.0 | 22.5
(a)

Methods | LTCC
---|---
Standard | Cloth-Changing
mAP | Rank-1 | mAP | Rank-1
Baseline | 23.2 | 55.1 | 8.1 | 19.6
Arb. | 27.1 | 59.5 | 9.2 | 20.5
BEGN | 28.4 | 61.2 | 9.8 | 22.0
END | 28.1 | 61.5 | 9.5 | 22.4
Mid. (ours) | 29.4 | 63.2 | 10.4 | 23.7
(b)

Methods | LTCC
---|---
Standard | Cloth-Changing
mAP | Rank-1 | mAP | Rank-1
Baseline | 23.2 | 55.1 | 8.1 | 19.6
w/ $\mathcal{L}_{MSE}$ | 27.5 | 61.0 | 9.0 | 21.4
w/o $\mathcal{L}_{recon.}$ | 28.3 | 62.7 | 9.6 | 22.9
ours | 29.4 | 63.2 | 10.4 | 23.7
(c)

Table 7: Study on the different ReID inference strategies. Methods | LTCC
---|---
Standard | Cloth-Changing
mAP | Rank-1 | mAP | Rank-1
Baseline | 23.2 | 55.1 | 8.1 | 19.6
$\widetilde{R}$ | 8.6 | 21.1 | 4.3 | 9.9
$\hat{r}+\hat{g}$ | 29.8 | 64.0 | 10.9 | 24.4
$\widetilde{r}+\widetilde{g}$ | 28.9 | 63.2 | 9.7 | 23.1
$\widetilde{r}$ | 28.1 | 60.8 | 9.1 | 21.3
$r$ (ours) | 29.4 | 63.2 | 10.4 | 23.7

###  4.3 Design Choices in Our GI-ReID Framework

We study the different design choices in our GI-ReID framework. We train and
test model on the real large-scale cloth-changing ReID dataset LTCC [54].

Influence of the Length $N$ of Predicted Gait Sequence. As shown in Eq-(4) of
Sec. 3.1, the output of GSP $\widetilde{R}\in\mathbb{R}^{N*h*w}$ is a sequence
with $N$ predicted gait frames. We study the influence of length $N$ w.r.t the
ReID performance. Table LABEL:tab:GSP_len shows that when $N=8$, our GI-ReID
gets the best performance, achieving a good trade-off between gait prediction
error and gait information augmentation.

Is ‘Middle Frame Input Principle’ Necessary? As described in Eq-(2) of GSP in
Sec. 3.1, we employ a position embedder $P$ and a feature aggregator $A$ to
set up a _middle frame input principle_ to reduce the gait prediction
ambiguity and difficulty. Here we compare several schemes to show the
necessity of such design. _Arb._ : we remove position embedder $P$, feature
aggregator $A$, position loss $\mathcal{L}_{position}$ for GSP, and take the
gait silhouette at _arbitrary_ position as input for training. _BEGN_ and
_END_ : we respectively take the gait stance at the _beginning_ and the _end_
position as input to predict gait sequence during the GSP training. In Table
LABEL:tab:GSP_position, the scheme  _Mid. (ours)_ that uses the gait frame at
middle position for gait sequence prediction achieves the best performance,
outperforming _Arb._ by 2.3% in mAP in the standard setting, which reveals
that predicting the gait statuses before and after the input middle gait
status indeed could reduce prediction difficulty/ambiguity.

Why Use MMD for Regularization? For the SC constraint, we shrink the gap
between the embeded ReID vector $\hat{r}$ and gait vector $\hat{g}$ by
minimizing MMD through $\mathcal{L}_{MMD}$. We study this design in Table
LABEL:tab:SC_Loss and find that when replacing $\mathcal{L}_{MMD}$ with
$\mathcal{L}_{MSE}$, the performance of _w/ $\mathcal{L}_{MSE}$_ drops nearly
2.0% in mAP. That’s because MMD loss is a distribution-level constraint and
could better enforce the high-level semantics consistency between dynamic
motion gait features and static spatial ReID features. MSE loss is an element-
wise constraint, and not so suitable to coordinate two modalities of motion
gait and RGB feature.

Is Reconstruction Penalty Necessary? When removing $\mathcal{L}_{recon.}$ in
Eq-(6), as shown in Table LABEL:tab:SC_Loss, the shceme _w/o
$\mathcal{L}_{recon.}$_ is inferior to ours by 1.1%/0.8% in mAP in the two
settings, which demonstrates that avoiding information lost caused by feature
regularization could enhance the final ReID performance of our GI-ReID
framework.

Which One for ReID Inference? We compare several cases of using (1) predicted
gait sequence $\widetilde{R}$, (2) aligned features fusion $\hat{r}$ \+
$\hat{g}$, (3) reconstructed features fusion $\widetilde{r}$ \+
$\widetilde{g}$, and (4) reconstructed ReID vector $\widetilde{r}$ for ReID
inference. Table 7 shows that 1) Directly using the predicted gait sequence
$\widetilde{R}$ for CC-ReID failed to get satisfactory results, this also
indicates that these gait recognition works [3, 9, 8] are not optimal for CC-
ReID. 2) Using the well-aligned features fusion $\hat{r}+\hat{g}$ achieves the
best performance, outperforming ours by 0.4%/0.5% in mAP in the two settings,
but this scheme still needs Gait-Stream in the inference. 3) Using the
reconstructed ReID vector $\widetilde{r}$ for inference suffers from
information lost and is inferior to ours by 1.3% in mAP in the both two
settings. 4) Our scheme that using the regularized ReID vector $r$ achieves
the second best performance while saving the computation costs brought by
Gait-Stream.

Figure 3: Six predicted gait sequences vs. realistic gait samples.

###  4.4 More Analysis, Visualization and Insights

To further prove that the proposed gait sequence prediction (GSP) module can
actually predict unique human motion features, and the achieved improvements
of GI-ReID indeed come from gait information, not from using additional gait-
related datasets or person silhouette images, here we provide more analysis
and visualization results. For example, when testing the gait-based
recognition performance using GaitSet [3] on the predicted human gait
sequences $\widetilde{R}$ generated by GSP module (see Supplementary for
details), it can achieve a competitive 62.4% in Rank-1 on CASIA-B.

Table 8: Performance (%) comparisons of our GI-ReID and other competitors on
the cloth-changing datasets LTCC [54] and PRCC [69]. ‘${\dagger}$’ means that
only identities with clothes changing are used for training. More results are
presented in Supplementary.

Methods | Standard | Cloth-changing |  Standard† |  Cloth-changing†
---|---|---|---|---
Rank-1 | mAP | Rank-1 | mAP | Rank-1 | mAP | Rank-1 | mAP
LOMO [40] \+ NullSpace [73] | 34.83 | 11.92 | 16.45 | 6.29 | 27.59 | 9.43 | 13.37 | 5.34
ResNet-50 + Face [68] | 60.44 | 25.42 | 22.10 | 9.44 | 55.37 | 22.23 | 20.68 | 8.99
PCB [61] | 65.11 | 30.60 | 23.52 | 10.03 | 59.22 | 26.61 | 21.93 | 8.81
HACNN [35] | 60.24 | 26.71 | 21.59 | 9.25 | 57.12 | 23.48 | 20.81 | 8.27
MuDeep [53] | 61.86 | 27.52 | 23.53 | 10.23 | 56.99 | 24.10 | 18.66 | 8.76
Baseline (ResNet-50) | 55.14 | 23.21 | 19.58 | 8.10 | 54.27 | 21.98 | 19.14 | 7.74
GI-ReID (ResNet-50, ours) | 63.21 | 29.44 | 23.72 | 10.38 | 61.39 | 27.88 | 22.59 | 9.87
Baseline (OSNet) | 66.07 | 31.18 | 23.43 | 10.56 | 61.22 | 27.41 | 22.97 | 9.74
GI-ReID (OSNet, ours) | 73.59 | 36.07 | 28.11 | 13.17 | 66.94 | 33.04 | 26.71 | 12.69
Baseline (LTCC-shape [54]) | – | – | 26.15 | 12.40 | – | – | 25.15 | 11.67
LTCC-shape + Gait-Stream (ours) | – | – | 28.86 | 14.19 | – | – | 26.41 | 13.26
(a)

Methods | Cross-clothes
---|---
Rank-1 | Rank-10 | Rank-20
Shape [1] | 11.48 | 38.66 | 53.21
LNSCT [66] | 15.33 | 53.87 | 67.12
HACNN [35] | 21.81 | 59.47 | 67.45
PCB [61] | 22.86 | 61.24 | 78.27
SketchNet [72] | 17.89 | 43.70 | 58.62
Deformable [5] | 25.98 | 71.67 | 85.31
STN [24] | 27.47 | 69.53 | 83.22
RCSANet [21] | 31.60 | – | –
PRCC-contour [69] | 34.38 | 77.30 | 88.05
\+ Gait-Stream (ours) | 36.19 | 79.93 | 91,67
Baseline (ResNet-50) | 22.23 | 61.08 | 76.44
GI-ReID (ResNet-50) | 33.26 | 75.09 | 87.44
Baseline (OSNet) | 28.70 | 72.34 | 85.89
GI-ReID (OSNet) | 37.55 | 82.25 | 93.76
(b)

Gait Sequence Prediction Visualization. Figure 3 further shows 6 groups of
gait prediction results (left) and 2 groups of realistic gait samples from
CASIA-B dataset [3] (right). Compared to the realistic gait samples, the
predicted gait results (i.e., the outputs of GSP) have the reasonable
continuous movements, e.g., swing arms and opening/closing legs. The gait-
stream could learn the discriminative dynamic clues from these predicted gait
results, like the walking stride, the left-right swinging range of arms, the
opening/closing angle of legs, etc, (see red circles in Figure 3).

Figure 4: Left: three examples of activation maps comparison between baseline
and our GI-ReID, which shows GI-ReID not only focuses on people’s clothes, but
also pay attention to the holistic human gait and local face; Right: Top-3
ranking list of GI-ReID for two query images on Real28. GI-ReID could identify
the same person with different clothes based on the assistance of gait.

Feature Map Visualization. To better understand how our GI-ReID works, we
visualize the intermediate activation feature maps of _Baseline_ and our GI-
ReID for comparison following [81, 25, 27]. On the left of Figure 4, we show
three examples of activation maps on Real28, and we observe that the feature
maps of _Baseline_ have high response mainly on person’s clothes. In contrast,
the activation features of our GI-ReID not only have high response on person’s
clothes, but also cover the holistic human body structure (gait) and local
face information (robust to cloth changing).

###  4.5 Comparison with State-of-the-Arts

The study on cloth-changing ReID is relatively rare [22, 20, 69, 54, 38, 71,
63], and most of them have not released source codes, even the dataset [71].
We compare our GI-ReID with multiple general ReID algorithms, including PCB
[61], HACNN [35], MuDeep [53], and specific cloth-changing ReID methods LTCC-
shape [54], PRCC-contour [69], RCSANet [21]. In Table 3(b), we observe that 1)
Thanks to the cloth-independent gait characteristics, our scheme GI-ReID
(OSNet) achieves the best performance on PRCC, outperforming the second best
PRCC-contour [69] by 3.17% in Rank-1 in the cross-clothes setting. 2) The
proposed Gait-Stream, as a kind of regularization, could benefit other
methods, e.g., LTCC-shape [54]. We find that the scheme of _LTCC-shape + Gait-
Stream_ could further obtain 1.79%/1.59% gain in mAP on LTCC. 3) For two
cloth-changing settings of LTCC, our scheme GI-ReID (ResNet-50) both achieve
obvious gains (2.28%/2.13% in mAP) over the Baseline (ResNet-50), which
totally-fair results indicate that GI-ReID could handle clothes changes and
learn identity-relevant features. 4) Our method is compatible with existing
ReID networks, e.g., built upon the strong ReID-specific network OSNet [81],
GI-ReID (OSNet) further achieves gains than GI-ReID (ResNet-50).

###  4.6 Failure Cases Analysis

Due to the large difference on the capture viewpoints and environments between
gait and ReID training data, the predicted gait results of GSP are not always
perfect (see Supplementary) when occlusion, partial, multi-person, etc,
existed in the person images, which may affect the CC-ReID performance. That
is why we indirectly use gait predictions in a knowledge regularization
manner, which makes GI-ReID robust and not sensitive to these failure cases.

##  5 Conclusion

In this paper, we propose to utilize human unique gait to address the cloth-
changing ReID problem from a single image. A novel gait-involved two-stream
framework GI-ReID is introduced, which takes gait as a regulator with a Gait-
Stream (discarded in the inference), to encourage the cloth-agnostic
representation learning of image-based ReID-Stream. To facilitate the gait
utilization, a gait sequence prediction (GSP) module and a high-level
semantics consistency (SC) constraint are further designed. Extensive
experiments on multiple CC-ReID benchmarks demonstrate the effectiveness and
superiority of GI-ReID.

##  6 Acknowledgements

This work was (partially) supported by the National Key R&D Program of China
under Grant 2020AAA0103902, Alibaba Innovative Research (AIR) program, and
NSFC under Grant U1908209, 61632001, 62021001.

Supplementary

##  Appendix 1 Detailed Network Structures of GI-ReID

GI-ReID, as a image-based cloth-changing ReID framework, with gait information
as assistance, consists of an auxiliary Gait-Stream and a mainstream ReID-
Stream. ReID-Stream can be arbitrary commonly-used network architectures, such
as ResNet [14], and also can be some ReID-specific network architectures, such
as PCB [61], OSNet [81]. Thus, in this section, we mainly introduce/describe
the detailed network architecture of Gait-Stream, which contains two key
parts, GSP module for gait information prediction/augmentation and GaitSet [3]
for gait features extraction.

Architecture of Gait Sequence Prediction (GSP) Module: The proposed GSP module
consists of a feature encoder $E$, a decoder $D$, a position embedder $P$, and
a feature aggregator $A$.

(1). Encoder $E$. The encoder $E$ is a CNN with four Conv. layers (filter size
= $4\times 4$ and stride = 2). The number of filters is increased from 64
$\rightarrow$ 512\. Each Conv. layer is followed by a batch-normalization (BN)
layer [23] and a rectified linear unit (ReLU) activation function [48]. In the
end, a 100-dimensional feature is obtained through a fully connected (FC)
layer.

Note that, when pre-training GSP module on the gait-specific datasets
following [3], the input gait silhouette of encoder $E$ has a size of $1\times
64\times 64$ (height-width ratio is 1:1). We use CASIA-B [3] as training
dataset. On the ReID-specific datasets, since the input person images usually
have a height-width ratio of 2:1 (e.g., height-256, width-128), we need
leverage an operation of “resize+zero_padding” to handle such training data
gap, which is pivotal for GSP’s accurate gait sequence prediction. For better
understanding, we vividly visualize such process in Figure 5.

Figure 5: We apply “resize+zero_padding” in the person masks (right) when
fine-tuning GSP module on the ReID-specific datasets, because the raw gait
training data (left) typically have the height-width ratio of (1:1), which is
important/necessary for training GSP to get satisfactory gait prediction
results.

(2). Position Embedder $P$ and Feature Aggregator $A$. To reduce the gait
prediction ambiguity and difficulty of GSP, a position embedder $P$ and a
feature aggregator $A$ are introduced to integrate the prior information of
input middle frame index into the prediction process of GSP. The position
embedder $P$ has a similar structure to that of the encoder $E$, but with one
more FC layer to regress the 1D position label $\widetilde{p}$. The feature
aggregator $A$ is inserted between the encoder and the decoder to convert the
raw encoded features $f_{S}$ into middle-position-aware features
$f_{S}^{\widetilde{p}}$ by taking the embedded middle position information
$\widetilde{p}$ into account. With respect to the architecture of $A$, it is
implemented only by a FC layer, which aims to regress to the aggregated
100-dimension feature $f_{S}^{\widetilde{p}}\in\mathbb{R}^{100}$ from the
101-dimension concatenated vector of the raw encoded feature
$f_{S}\in\mathbb{R}^{100}$ and the embedded middle position prior vector
$\widetilde{p}\in\mathbb{R}^{1}$.

(3). Decoder $D$. The structure of the decoder $D$ is symmetrical to that of
the encoder $E$. A FC layer along with reshaping is first employed to convert
the input 100D feature into the same size as the last feature output of the
encoder $E$, and then four DeConv. layers are used for up-sampling. A sigmoid
activation function is applied in the end, and outputs the gait predictions
with a size of $N\times 64\times 64$, where each channel indicates a predicted
gait frame of final results.

Architecture of GaitSet: GaitSet [3] is a classic set-based gait recognition
network, which takes a set of silhouettes/gait frames as input. After
obtaining features from each input silhouette independently using a CNN, _set
pooling_ is applied to merge features over frames into a set-level feature.
This set-level feature is then used for discrimination learning via
_horizontal pyramid mapping_ (HPM), which aims to extract features of
different spatial locations on different scales. We recommend seeing more
details from their original paper [3].

##  Appendix 2 Training Details of our GI-ReID

Phase-1: Pre-training for GaitSet. The input is a set of aligned silhouettes
in size of $64\times 44$. The silhouettes are directly provided by the
datasets and are aligned based on methods in [62]. The set cardinality in the
training is set to be 30. Adam is chosen as an optimizer. The number of scales
$S$ in HPM is set as $5$. The margin in separate triplet loss
$\mathcal{L}_{tri}^{sep}$ [3] is set as $0.2$. The mini-batch is composed of
$P=16$ and $N=8$ ($P,N$ respectively mean the number of person identities and
input gait frames). We set the number of channels in $C1$ and $C2$ as $32$, in
$C3$ and $C4$ as $64$ and in $C5$ and $C6$ as 128 (following [3]). The
learning rate is set to be $1\times$10-4, and the model is trained for 80
epochs.

Phase-2: Joint Training for GSP module and GaitSet. After pre-training
GaitSet, we jointly train the proposed gait sequence prediction (GSP) module
and GaitSet for Gait-Stream. Specifically, during the joint-training, we also
re-use CASIA-B dataset for effective gait prediction training. Following [16],
a batch is formed by first randomly sampling $P$ identities. For each
identity, we sample $N$ continuous gait frames as the ground-truth gait
sequence. Then the batch size is $B=P\times N$. We set $P=4$ and $N=8$ (i.e.,
batch size $B=P\times N=32$. As presented in the main manuscript, we use the
middle one of the ground-truth gait sequence (i.e., the fourth one when $N=8$)
as input for GSP training. We first optimize GSP with the proposed position
loss $\mathcal{L}_{position}$ and prediction loss $\mathcal{L}_{pred}$ (loss
balance is set as 1:1) for 80 epochs, which enables GSP to output reasonable
predicted gait sequence results. We train GSP with Adam optimizer [29] with a
initial learning rate of 5$\times$10-4. We optimize the Adam optimizer with a
weight decay of 1$\times$10-4. The learning rate is decayed by a factor of 0.1
at 40 epoch.

Algorithm 1 Training Process of GI-ReID

1: Input: gait dataset $\mathcal{G}$ (e.g., CASIA-B [3]), ReID dataset
$\mathcal{R}$ (e.g., LTCC [54]). Learning rate is simply denoted as $\eta$.
The entire GI-ReID framework consists of GSP module $GSP_{\theta}$, GaitSet
(GS) $GS_{\phi}$, SC constraints related FC layers $SC_{\psi}$, and ReID-
Stream backbone $ReID_{\omega}$.

2: Output: inference ReID vector $r$.

3: ### Phase-1: Pre-training for GaitSet

4: for $epoch=1$ to $80$ do

5: Sample $P\times N=16\times 8$ samples from gait training set $\mathcal{G}$.

6: $\mathcal{L}_{total}=\mathcal{L}_{tri}^{sep}$ // Use the separate triplet
loss as objective function [3].

7: $\phi=\phi-\eta\nabla_{\phi}\mathcal{L}_{tri}^{sep}$ // Update GaitSet (GS)
$GS_{\phi}$.

8: end for

9: ### Phase-2: Joint Training for GSP module and GaitSet

10: for $epoch=1$ to $80$ do

11: Sample $P\times N=4\times 8$ samples from gait training set $\mathcal{G}$.

12: $\mathcal{L}_{total}=\mathcal{L}_{position}+\mathcal{L}_{pred}$ // Use the
proposed position loss and prediction loss as objective functions.

13: $\theta=\theta-\eta\nabla_{\theta}\mathcal{L}_{total}$ // Warm up GSP
module $GSP_{\theta}$.

14: end for

15: for $epoch=1$ to $160$ do

16: Sample $P\times N=4\times 8$ samples from gait training set $\mathcal{G}$.

17:
$\mathcal{L}_{total}=\mathcal{L}_{position}+\mathcal{L}_{pred}+\mathcal{L}_{tri}^{sep}$
// Use the position loss, prediction loss , and separate triplet loss as
objective functions.

18:
$(\theta,\phi)=(\theta,\phi)-\eta\nabla_{(\theta,\phi)}\mathcal{L}_{total}$ //
Jointly update GSP module $GSP_{\theta}$ and GaitSet (GS) $GS_{\phi}$.

19: end for

20: ### Phase-3: Joint Training for Gait-Stream and ReID-Stream

21: for $epoch=1$ to $240$ do

22: Sample $P\times N=10\times 8$ samples from ReID training set
$\mathcal{R}$.

23:
$\mathcal{L}_{total}=0.1*\mathcal{L}_{position}+0.1*\mathcal{L}_{pred}+0.1*\mathcal{L}_{tri}^{sep}+\mathcal{L}_{cla}+\mathcal{L}_{tri}^{HM}+0.5*\mathcal{L}_{MMD}+0.5*\mathcal{L}_{recon.}$//
Total objective functions consists of the position loss, prediction loss,
separate triplet loss (for Gait-Stream), and the classification loss, triplet
loss (with hard-mining, HM) (for ReID-Stream), and the MMD loss,
reconstruction loss (SC constraints).

24:
$(\theta,\phi,\psi,\omega)=(\theta,\phi,\psi,\omega)-\eta\nabla_{(\theta,\phi,\psi,\omega)}\mathcal{L}_{total}$
// Jointly update GSP module $GSP_{\theta}$, GaitSet (GS) $GS_{\phi}$, SC
constraints related FC embedding layers $SC_{\psi}$, and ReID-Stream backbone
$ReID_{\omega}$.

25: end for

After warming up the GSP module for 80 epochs, we jointly train GSP and
GaitSet for extra 160 epochs with initial learning rate as 5$\times$10-4. We
also use Adam optimizer [29] for optimization with a weight decay of
1$\times$10-4, the learning rate is decayed by a factor of 0.5 at 40, 80, and
120 epochs. When jointly training GSP and GaitSet, excluding the GSP-related
position loss $\mathcal{L}_{position}$ and prediction loss
$\mathcal{L}_{pred}$, we further use _separate triplet loss_
$\mathcal{L}_{tri}^{sep}$ for training, which is introduced in details in
GaitSet [3], and we also set the loss weight as 1.0 for this supervision.

Phase-3: Joint Training for Gait-Stream and ReID-Stream. When we jointly
training Gait-Stream and ReID-Stream on the ReID datasets, Gait-Stream is also
fine-tuned/learnable. Since the full gait sequence ground truth are not
available for ReID-specific datasets, we adjust the original prediction loss
$\mathcal{L}_{pred}$ in GSP by only calculating L1 distance between the single
input person mask and the middle frame result of the entire predicted gait
sequence.

On the large-scale cloth-changing datasets VC-Clothes [63], LTCC [54], and
PRCC [69], we set training batch size as $B=80=P\times N=10\times 8$. Both of
Gait-Stream (including GSP and GaitSet) and ReID-Stream use Adam optimizer
[29] for optimization, where the initial learning rate for Gait-Stream is
1$\times$10-5, for ReID-Stream is 5$\times$10-4. We optimize two Adam
optimizers for Gait-Stream and ReID-Stream with a weight decay of
1$\times$10-5 for total 240 epochs. The learning rate is decayed by a factor
of 0.1 at 80 and 160 epochs for ReID-Stream, while no learning rate decay for
Gait-Stream. For the losses usage, we adopt the widely-adopted classification
loss $\mathcal{L}_{cla}$ [61, 10], and triplet loss with batch hard mining
$\mathcal{L}_{tri}^{HM}$[16]) as basic optimization objectives for ReID-Stream
training, and we set these two loss weights as 1.0. Besides, for the Gait-
Stream related losses, including
$\mathcal{L}_{position},\mathcal{L}_{pred},\mathcal{L}_{tri}^{sep}$, we set
all their loss weights as 0.1. For the semantics consistency (SC) constraints
related FC embedding layers, we merge their learnable parameters into ReID-
Stream’s optimization, and set the balance weights for MMD loss
$\mathcal{L}_{MMD}$ and reconstruction penalty $\mathcal{L}_{recon.}$ as 0.5.
The pseudo code of the entire training process of our GI-ReID is given in
Algorithm 1.

##  Appendix 3 Details of Datasets

We use one widely-used video ReID dataset MARS [79], and four image-based
cloth-changing ReID datasets Real28 [63], VC-Clothes [63], LTCC [54], PRCC
[69] to perform experiments. In Table 9, we present the detailed information
about these ReID datasets.

Table 9: Brief introduction/comparison of datasets. | MARS | Real28 | VC-Clothes | LTCC | PRCC
---|---|---|---|---|---
Category | Video | Image | Image | Image | Image
Photo Style | Real | Real | Synthetic | Real | Real
Scale | Large | Small | Large | Large | Large
Cloth Change | No | Yes | Yes | Yes | Yes
Identities | 1,261 | 28 | 512 | 152 | 221
Samples | 20,715 | 4,324 | 19,060 | 17,138 | 33,698
Cameras | 6 | 4 | 4 | N/A | 3
Usage | Train&Test | Test | Train&Test | Train&Test | Train&Test

_MARS_ is a popular dataset for video-based person ReID. There are 20,715
track-lets come from 1,261 pedestrians who are captured by at least 2 cameras.
We use the train/test split protocol defined in [79].

_Real28, VC-Clothes, LTCC and PRCC_ are all newly released image datasets for
cloth-changing ReID [63, 54, 69].

_Real28_ is a small real-scenario dataset, which is collected in 3 different
days (with different clothing) by 4 cameras. It consists of totally 4,324
images from 28 different identities with 2 indoor scenes and 2 outdoors.
Similar to [63], since the size of Real28 is not big enough for training deep
learning models, we just use it for evaluation. There are totally 336 images
in the query and 3,988 images in the gallery.

_VC-Clothes_ is a synthetic dataset where images are rendered by the Grand
Theft Auto V (GTA5). It has 512 identities, 4 scenes (cameras) and on average
9 images/scenes for each identity and a total number of 19,060 images.
Following [63], we equally split the dataset by identities: 256 identities for
training and the other 256 for testing. We randomly chose 4 images per person
from each camera as query, and have the other images serve as gallery images.
Eventually, we get totally 9,449 images in the training, 1,020 images as
queries and 8,591 others in the gallery.

_LTCC_ is a large-scale real-scenario cloth-changing dataset, which contains
17,138 person images of 152 identities. On average, there are 5 different
clothes for each cloth-changing person, with the numbers of outfit changes
ranging from 2 to 14. Following [54], we split the LTCC dataset into training
and testing sets. The training set consists of 77 identities, where 46 people
have cloth changes and the rest of 31 people wear the same outfits during the
recording. Similarly, the testing set contains 45 people with changing clothes
and 30 people wearing the same outfits.

_PRCC_ is also a large-scale real-scenario cloth-changing dataset, recently
published by Yang  _et al_. [69]. It consists of 221 identities with three
camera views _Camera A_ , _Camera B_ , and _Camera C_. Each person in Cameras
A and B is wearing the same clothes, but the images are captured in different
rooms. For Camera C, the person wears different clothes, and the images are
captured in a different day. The images in the PRCC dataset include not only
clothing changes for the same person across different camera views but also
other variations, e.g. changes in illumination, occlusion, pose and viewpoint.
In summary, nearly 50 images exists for each person in each camera view.
Therefore, approximately 152 images of each person are included in the
dataset, for 33,698 images in total.

Following [69], we split the PRCC dataset into a training set and a testing
set. The training set consists of 150 people, and the testing set consists of
71 people, with no overlap between them in terms of identities. The testing
set was further divided into a gallery set and a probe set. For each identity
in the testing set, we chose one image in _Camera view A_ to form the gallery
set for a single-shot matching. All images in _Camera views B_ and _Camera C_
were used for the probe set. Specifically, the person matching between Camera
views A and B was performed without clothing changes, whereas the matching
between Camera views A and C was cross-clothes matching. The results were
assessed in terms of the cumulated matching characteristics, specifically, the
Rank-1, Rank-10, and Rank-20 matching accuracy.

##  Appendix 4 Experimental Results of Different Settings

Experimental Setups. As we described in the main manuscript, we build three
kinds of different experiment settings to comprehensively validate the
effectiveness of gait biometric for person ReID, and also validate the
rationality/superiority of the proposed gait prediction and regularization in
our GI-ReID framework: (1) Real Cloth-Changing Image ReID, (2) General Video
ReID, (3) Imitated Cloth-Changing Video ReID. In the main manuscripts, we have
presented all the results related to the most challenging setting of (1) real
cloth-changing image ReID. The rest results about (2)(3) are shown here.
Baseline means the model that only ingests RGB images.

2) General Video ReID. In this setting, we use a general video ReID dataset
MARS for experiments. This dataset has no cloth-changing cases. This group of
experiments aims to verify two things: 1) gait could benefit ReID even without
clothes variations. 2) extracting gait feature from video is easier than that
from image, or said, exploiting gait feature in the image-based CC-ReID is
more challenging. Since MARS itself contains continuous video frames/clips and
human gait masks 222https://pan.baidu.com/s/16ZrlM1f_1_T-eZHmQTTkYg., we don’t
need GSP to additionally predict gait sequence, so we discard it for
simplicity.

3) Imitated Cloth-Changing Video ReID. We still use MARS as dataset to perform
experiments in this setting. But the difference is that we _imitate_ cloth-
changing cases for the images with the same identity through a data
augmentation strategy—body-wise color jitter (i.e., randomly change the
brightness, contrast and saturation of the human body region in an person
image) for training. This group of experiments aims to show that gait
information could alleviate the ReID interference caused by clothes changing.
GSP module is also removed in this setting.

Results of General Video ReID. Table 10 shows the results. We observe that: 1)
Thanks to the leverage of gait characteristics through the proposed Gait-
Stream (GS), _Baseline + GS (concat)_ and _Baseline + GS + SC_ outperform
_Baseline_ by 1.07%/1.29% in mAP respectively, which demonstrates that gait
information indeed benefits ReID. 2) We find that _Baseline + GS + SC_ further
outperforms _Baseline + GS (concat)_ by 0.22% in mAP. This result validates
the superiority of our gait utilization manner (i.e., regularization), which
makes ReID-Stream not only robust to the gait estimation error, but also
computationally efficient (Gait-Stream is not needed in the inference).

Table 10: Performance (%) comparison on the general video ReID dataset MARS [79]. GS refers to Gait-Stream and SC refers to semantics consistency constraints. Note that ‘concat’ means concatenating ReID vector $r$ and gait vector $g$ together for ReID. The backbone is ResNet-50. Methods |  MARS
---|---
mAP |  Rank-1
Baseline |  79.12 |  87.34
Baseline + GS (concat) |  80.19 |  88.16
Baseline + GS + SC (ours) |  80.41 |  88.32

Results of Imitated Cloth-Changing Video ReID. To prove that gait indeed could
alleviate clothes variation issue, we _imitate_ cloth-changing cases for MARS
(denoted as CC-MARS). In Table 11, we observe that 1) Disturbed by the
synthetic clothing change, _Baseline_ suffers from large degradation, 68.52%
on CC-MARS vs. 79.12% on raw MARS in mAP. 2) With the assistance of gait,
_Baseline+GS (concat)_ and _Baseline+GS+SC_ improve _Baseline_ near 5.0% in
mAP. 3) On CC-MARS, the gait ‘concat’ scheme shows a little superiority than
ours. We analyse that’s because the ‘concat’ could help ReID more explicitly,
especially when meeting changing clothes. But, the ‘concat’ scheme needs
maintain Gait-Stream in the inference, leading extra computational cost. 4) As
video ReID datasets, it is relatively easy to extract gait features on
MARS/CC-MARS.

Table 11: Performance (%) comparison on the imitated (using color jitter) cloth-changing video ReID dataset, termed as CC-MARS. The ReID backbone is ResNet-50. Methods |  CC-MARS
---|---
mAP |  Rank-1
Baseline |  68.52 |  72.31
Baseline + GS (concat) |  73.46 |  79.34
Baseline + GS + SC (ours) |  73.13 |  79.15

##  Appendix 5 Comparison with State-of-the-Arts (Complete version)

Table 12:  Performance (%) comparisons of our GI-ReID and other competitors on the cloth-changing dataset LTCC [54]. ‘Standard’ and ‘Cloth-changing’ respectively mean the standard setting and cloth-changing setting as mentioned in our main manuscript. ‘(Image)’ or ‘(Parsing)’ represents that the input data is the person image or the body parsing image. ‘${\dagger}$’ means the setting that only identities with clothes changing are used for training. Methods | Standard | Cloth-changing |  Standard† |  Cloth-changing†
---|---|---|---|---
Rank-1 | mAP | Rank-1 | mAP | Rank-1 | mAP | Rank-1 | mAP
LOMO [40] \+ KISSME [31] | 26.57 | 9.11 | 10.75 | 5.25 | 19.47 | 7.37 | 8.32 | 4.37
LOMO [40] \+ XQDA [40] | 25.35 | 9.54 | 10.95 | 5.56 | 22.52 | 8.21 | 10.55 | 4.95
LOMO [40] \+ NullSpace [73] | 34.83 | 11.92 | 16.45 | 6.29 | 27.59 | 9.43 | 13.37 | 5.34
ResNet-50 (Image) [14] | 58.82 | 25.98 | 20.08 | 9.02 | 57.20 | 22.82 | 20.68 | 8.38
ResNet-50 (Parsing) [14] | 19.87 | 6.64 | 7.51 | 3.75 | 18.86 | 6.16 | 6.28 | 3.46
PCB (Parsing) [61] | 27.38 | 9.16 | 9.33 | 4.50 | 25.96 | 7.77 | 10.54 | 4.04
ResNet-50 + Face [68] | 60.44 | 25.42 | 22.10 | 9.44 | 55.37 | 22.23 | 20.68 | 8.99
PCB [61] | 65.11 | 30.60 | 23.52 | 10.03 | 59.22 | 26.61 | 21.93 | 8.81
HACNN [35] | 60.24 | 26.71 | 21.59 | 9.25 | 57.12 | 23.48 | 20.81 | 8.27
MuDeep [53] | 61.86 | 27.52 | 23.53 | 10.23 | 56.99 | 24.10 | 18.66 | 8.76
Face [68] | 60.44 | 25.42 | 22.10 | 9.44 | 55.37 | 22.23 | 20.68 | 8.99
Baseline (ResNet-50) | 55.14 | 23.21 | 19.58 | 8.10 | 54.27 | 21.98 | 19.14 | 7.74
GI-ReID (ResNet-50, ours) | 63.21 | 29.44 | 23.72 | 10.38 | 61.39 | 27.88 | 22.59 | 9.87
Baseline (OSNet) | 66.07 | 31.18 | 23.43 | 10.56 | 61.22 | 27.41 | 22.97 | 9.74
GI-ReID (OSNet, ours) | 73.59 | 36.07 | 28.11 | 13.17 | 66.94 | 33.04 | 26.71 | 12.69
Baseline (LTCC-shape [54]) | – | – | 26.15 | 12.40 | – | – | 25.15 | 11.67
LTCC-shape + Gait-Stream (ours) | – | – | 28.86 | 14.19 | – | – | 26.41 | 13.26
Table 13: Performance (%) comparisons of our GI-ReID and other competitors on
the cloth-changing dataset PRCC [69]. “RGB” means the inputs of the model are
RGB images; “Sketch” means the inputs of the model are contour sketch images

Methods |  Cameras A and C (Cross-Clothes) |  Cameras A and B (Same Clothes)
---|---|---
Rank-1 |  Rank-10 |  Rank-20 |  Rank-1 |  Rank-10 |  Rank-20
LBP [51] \+ KISSME [32] |  18.71 |  58.09 |  71.40 |  39.03 |  76.18 |  86.91
HOG [6] \+ KISSME [32] |  17.52 |  49.52 |  63.55 |  36.02 |  68.83 |  80.49
LBP [51] \+ HOG [6] \+ KISSME [32] |  17.66 |  54.07 |  67.85 |  47.73 |  81.88 |  90.54
LOMO [41] \+ KISSME [32] |  18.55 |  49.81 |  67.27 |  47.40 |  81.42 |  90.38
LBP [51] \+ XQDA [41] |  18.25 |  52.75 |  61.98 |  40.66 |  77.74 |  87.44
HOG [6] \+ XQDA [41] |  22.11 |  57.33 |  69.93 |  42.32 |  75.63 |  85.38
LBP [51] \+ HOG [6] \+ XQDA [41] |  23.71 |  62.04 |  74.49 |  54.16 |  84.11 |  91.21
LOMO [41] \+ XQDA [41] |  14.53 |  43.63 |  60.34 |  29.41 |  67.24 |  80.52
Shape [1] |  11.48 |  38.66 |  53.21 |  23.87 |  68.41 |  76.32
LNSCT [66] |  15.33 |  53.87 |  67.12 |  35.54 |  69.56 |  82.37
Alexnet [34] (RGB) |  16.33 |  48.01 |  65.87 |  63.28 |  91.70 |  94.73
VGG16 [56] (RGB) |  18.21 |  46.13 |  60.76 |  71.39 |  95.89 |  98.68
HA-CNN [35] (RGB) |  21.81 |  59.47 |  67.45 |  82.45 |  98.12 |  99.04
PCB [61] (RGB) |  22.86 |  61.24 |  78.27 |  86.88 |  98.79 |  99.62
Alexnet [34] (Sketch) |  14.94 |  57.68 |  75.40 |  38.00 |  82.15 |  91.91
VGG16 [56] (Sketch) |  18.79 |  66.01 |  81.27 |  54.00 |  91.33 |  96.73
HA-CNN [35] (Sketch) |  20.45 |  63.87 |  79.58 |  58.63 |  90.45 |  95.78
PCB [61] (Sketch) |  22.48 |  61.07 |  77.05 |  57.36 |  92.12 |  96.72
SketchNet [72] (Sketch+RGB) |  17.89 |  43.70 |  58.62 |  64.56 |  95.09 |  97.84
Face [65] |  2.97 |  9.85 |  13.52 |  4.75 |  13.40 |  45.54
Deformable Conv. [5] |  25.98 |  71.67 |  85.31 |  61.87 |  92.13 |  97.65
STN [24] |  27.47 |  69.53 |  83.22 |  59.21 |  91.43 |  96.11
RCSANet [21] |  31.60 |  – |  – |  – |  – |  –
PRCC-contour [69] |  34.38 |  77.30 |  88.05 |  64.20 |  92.62 |  96.65
\+ Gait-Stream (ours) |  36.19 |  79.93 |  91,67 |  – |  – |  –
Baseline (ResNet-50) |  22.23 |  61.08 |  76.44 |  75.81 |  97.34 |  98.95
GI-ReID (ResNet-50) |  33.26 |  75.09 |  87.44 |  78.95 |  97.89 |  99.11
Baseline (OSNet) |  28.70 |  72.34 |  85.89 |  83.68 |  98.24 |  99.26
GI-ReID (OSNet) |  37.55 |  82.25 |  93.76 |  85.97 |  98.82 |  99.72

To save space, we only present the latest approaches in the main manuscripts,
and here we show comparisons with more approaches and more evaluation settings
on LTCC (Table 12) and PRCC datasets (Table 13).

From the comparison results on PRCC that are shown in Table 13, we observe
that 1) Although person ReID with no clothing change (i.e.“Same Clothes” in
the Table 13) is not the purpose in our work, our method GI-ReID can still
achieve an accuracy of 85.97% in Rank-1, which is better than that of all
hand-crafted features with metric learning methods and most deep learning
methods. 2) When the input images are RGB images without clothing changes,
Alexnet [34], VGG16 [56], HA-CNN [35], and PCB [61] all achieve good
performance, but they have a sharp performance drop when a clothing change
occurs, illustrating the challenge of person ReID when a person dresses
differently. Therefore, the application of existing person ReID methods is not
straightforward in this scenario. In contrast, our GI-ReID that leverages gait
information is beneficial to learn the clothing invariant feature, which makes
our method achieve satisfactory performance 37.55% in Rank-1 even under the
cloth-changing scenario.

##  Appendix 6 Study on Failure Cases (Limitations)

As we described in the main manuscript, since the existed large difference on
the capture viewpoint and environment between gait and ReID training data, the
predicted results of gait sequence prediction (GSP) module are not so accurate
when occlusion, partial, multi-person, etc, existed in the person images. As
shown in Figure 6, GSP gives unsatisfactory gait prediction results, where
large estimation errors exist in the predicted gait frames, which will hurt
the ReID performance. That is why we indirectly use gait prediction results in
a two-stream knowledge regularization manner, which makes our GI-ReID
robust/less sensitive to these failure cases.

Figure 6: Failure cases of gait sequence prediction (GSP).

##  Appendix 7 Social Impact

Positive. In this paper, we propose to utilize human unique gait to address
the cloth-changing ReID (CC-ReID) problem from a single image. A novel gait-
involved two-stream framework GI-ReID is introduced for image-based CC-ReID.
To our best knowledge, this paper is the first attempt to take gait as a
regulator with a Gait-Stream (discarded in the inference), to encourage the
cloth-agnostic representation learning of image-based ReID-Stream. This is
very important for both of academic community and industry, and it is also
valuable and meaningful to bridge the gap between the fast-developing ReID
algorithms and practical applications.

Negative. Due to the urgent demand of public safety and increasing number of
surveillance cameras, person ReID is imperative in intelligent surveillance
systems with significant research impact and practical importance, but this
task also might raise questions about the risk of leaking private information.
On the other hand, the data collected from the surveillance equipments or
downloaded from the internet may violate the privacy of human beings.
Therefore, we appeal and encourage research that understands and mitigates the
risks arising from surveillance applications.

## References

  * [1] Serge Belongie, Jitendra Malik, and Jan Puzicha.  Shape matching and object recognition using shape contexts.  IEEE TPAMI, 24(4):509–522, 2002.
  * [2] Cassandra Carley, Ergys Ristani, and Carlo Tomasi.  Person re-identification from gait using an autocorrelation network.  In CVPRW, pages 0–0, 2019.
  * [3] Hanqing Chao, Yiwei He, Junping Zhang, and Jianfeng Feng.  Gaitset: Regarding gait as a set for cross-view gait recognition.  In AAAI, volume 33, pages 8126–8133, 2019.
  * [4] Di Chen, Shanshan Zhang, Wanli Ouyang, Jian Yang, and Ying Tai.  Person search via a mask-guided two-stream cnn model.  In ECCV, pages 734–750, 2018.
  * [5] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In ICCV, pages 764–773, 2017.
  * [6] Navneet Dalal and Bill Triggs.  Histograms of oriented gradients for human detection.  2005\.
  * [7] Carl Doersch.  Tutorial on variational autoencoders.  arXiv preprint arXiv:1606.05908, 2016.
  * [8] Omar Elharrouss, Noor Almaadeed, Somaya Al-Maadeed, and Ahmed Bouridane.  Gait recognition for person re-identification.  The Journal of Supercomputing, pages 1–20, 2020.
  * [9] Chao Fan, Yunjie Peng, Chunshui Cao, Xu Liu, Saihui Hou, Jiannan Chi, Yongzhen Huang, Qing Li, and Zhiqiang He.  Gaitpart: Temporal part-based model for gait recognition.  In CVPR, June 2020.
  * [10] Yang Fu, Yunchao Wei, Yuqian Zhou, Honghui Shi, Gao Huang, Xinchao Wang, Zhiqiang Yao, and Thomas Huang.  Horizontal pyramid matching for person re-identification.  In AAAI, volume 33, pages 8295–8302, 2019.
  * [11] Yixiao Ge, Zhuowan Li, Haiyu Zhao, et al.  Fd-gan: Pose-guided feature distilling gan for robust person re-identification.  In NeurIPS, 2018.
  * [12] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  The Journal of Machine Learning Research, 13(1):723–773, 2012.
  * [13] Vincent Le Guen and Nicolas Thome.  Disentangling physical dynamics from unknown factors for unsupervised video prediction.  In CVPR, pages 11474–11484, 2020.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, et al.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [15] Lingxiao He, Jian Liang, Haiqing Li, and Zhenan Sun.  Deep spatial feature reconstruction for partial person re-identification: Alignment-free approach.  In CVPR, 2018.
  * [16] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv preprint arXiv:1703.07737, 2017.
  * [17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  NeurIPS Workshop, 2015.
  * [18] Peixian Hong, Tao Wu, Ancong Wu, Xintong Han, and Wei-Shi Zheng.  Fine-grained shape-appearance mutual learning for cloth-changing person re-identification.  In CVPR, pages 10513–10522, 2021.
  * [19] Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles.  Learning to decompose and disentangle representations for video prediction.  In NeurIPS, pages 517–526, 2018.
  * [20] Yan Huang, Qiang Wu, Jingsong Xu, and Yi Zhong.  Celebrities-reid: A benchmark for clothes variation in long-term person re-identification.  In IJCNN, pages 1–8. IEEE, 2019.
  * [21] Yan Huang, Qiang Wu, JingSong Xu, Yi Zhong, and ZhaoXiang Zhang.  Clothing status awareness for long-term person re-identification.  In ICCV, pages 11895–11904, 2021.
  * [22] Yan Huang, Jingsong Xu, Qiang Wu, Yi Zhong, Peng Zhang, and Zhaoxiang Zhang.  Beyond scalar neuron: Adopting vector-neuron capsules for long-term person re-identification.  TCSVT, 2019.
  * [23] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  ICML, 2015.
  * [24] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  In NeurIPS, pages 2017–2025, 2015.
  * [25] Xin Jin, Cuiling Lan, Wenjun Zeng, and Zhibo Chen.  Global distance-distributions separation for unsupervised person re-identification.  ECCV, 2020.
  * [26] Xin Jin, Cuiling Lan, Wenjun Zeng, and Zhibo Chen.  Uncertainty-aware multi-shot knowledge distillation for image-based object re-identification.  In AAAI, 2020.
  * [27] Xin Jin, Cuiling Lan, Wenjun Zeng, Zhibo Chen, and Li Zhang.  Style normalization and restitution for generalizable person re-identification.  In CVPR, pages 3143–3152, 2020.
  * [28] Xin Jin, Cuiling Lan, Wenjun Zeng, Guoqiang Wei, and Zhibo Chen.  Semantics-aligned representation learning for person re-identification.  In AAAI, 2020.
  * [29] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2014.
  * [30] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  Pointrend: Image segmentation as rendering.  In CVPR, pages 9799–9808, 2020.
  * [31] Aniket Kittur, Ed H. Chi, and Bongwon Suh.  Crowdsourcing user studies with mechanical turk.  2008\.
  * [32] Martin Koestinger, Martin Hirzer, Paul Wohlhart, Peter M Roth, and Horst Bischof.  Large scale metric learning from equivalence constraints.  In CVPR, pages 2288–2295. IEEE, 2012.
  * [33] Zoe Kourtzi and Nancy Kanwisher.  Activation in human mt/mst by static images with implied motion.  Journal of cognitive neuroscience, 12(1):48–55, 2000.
  * [34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [35] Wei Li, Xiatian Zhu, and Shaogang Gong.  Harmonious attention network for person re-identification.  In CVPR, 2018.
  * [36] Xiang Li, Yasushi Makihara, Chi Xu, Yasushi Yagi, and Mingwu Ren.  Gait recognition via semi-supervised disentangled representation learning to identity and covariate features.  In CVPR, pages 13309–13319, 2020.
  * [37] Xiang Li, Yasushi Makihara, Chi Xu, Yasushi Yagi, Shiqi Yu, and Mingwu Ren.  End-to-end model-based gait recognition.  In ACCV, 2020.
  * [38] Yu-Jhe Li, Zhengyi Luo, Xinshuo Weng, and Kris M Kitani.  Learning shape representations for clothing variations in person re-identification.  arXiv preprint arXiv:2003.07340, 2020.
  * [39] Rijun Liao, Shiqi Yu, Weizhi An, and Yongzhen Huang.  A model-based gait recognition method with body pose and human prior knowledge.  Pattern Recognition, 98:107069, 2020.
  * [40] S. Liao, Y. Hu, X. Zhu, and S. Z. Li.  Person re-identification by local maximal occurrence representation and metric learning.  In CVPR, 2015.
  * [41] Shengcai Liao, Yang Hu, Xiangyu Zhu, and Stan Z Li.  Person re-identification by local maximal occurrence representation and metric learning.  In CVPR, 2015.
  * [42] Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang.  Deep video frame interpolation using cyclic frame generation.  In AAAI, volume 33, pages 8794–8802, 2019.
  * [43] Zheng Liu, Zhaoxiang Zhang, Qiang Wu, and Yunhong Wang.  Enhancing person re-identification by integrating gait biometric.  Neurocomputing, 168:1144–1156, 2015.
  * [44] Yasushi Makihara, Atsuyuki Suzuki, Daigo Muramatsu, Xiang Li, and Yasushi Yagi.  Joint intensity and spatial metric learning for robust gait recognition.  In CVPR, pages 5705–5715, 2017.
  * [45] Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung.  Phase-based frame interpolation for video.  In CVPR, pages 1410–1418, 2015.
  * [46] Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang.  Pose-guided feature alignment for occluded person re-identification.  In ICCV, pages 542–551, 2019.
  * [47] Daigo Muramatsu, Akira Shiraishi, Yasushi Makihara, Md Zasim Uddin, and Yasushi Yagi.  Gait-based person recognition using arbitrary view transformation model.  TIP, 24(1):140–154, 2014.
  * [48] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In ICML, 2010.
  * [49] Simon Niklaus and Feng Liu.  Context-aware synthesis for video frame interpolation.  In CVPR, pages 1701–1710, 2018.
  * [50] Mark Nixon et al.  Model-based gait recognition.  2009\.
  * [51] Timo Ojala, Matti Pietikäinen, and David Harwood.  A comparative study of texture measures with classification based on featured distributions.  Pattern recognition, 29(1):51–59, 1996.
  * [52] Xuelin Qian, Yanwei Fu, Wenxuan Wang, et al.  Pose-normalized image generation for person re-identification.  In ECCV, 2018.
  * [53] Xuelin Qian, Yanwei Fu, Tao Xiang, Yu-Gang Jiang, and Xiangyang Xue.  Leader-based multi-scale attention deep architecture for person re-identification.  TPAMI, 2019.
  * [54] Xuelin Qian, Wenxuan Wang, Li Zhang, Fangrui Zhu, Yanwei Fu, Tao Xiang, Yu-Gang Jiang, and Xiangyang Xue.  Long-term cloth-changing person re-identification.  WACV, 2020.
  * [55] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton.  Dynamic routing between capsules.  In NeurIPS, pages 3856–3866, 2017.
  * [56] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [57] Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang.  Mask-guided contrastive attention model for person re-identification.  In CVPR, 2018.
  * [58] Chi Su, Jianing Li, Shiliang Zhang, et al.  Pose-driven deep convolutional model for person re-identification.  In ICCV, 2017.
  * [59] Xiaoxiao Sun and Liang Zheng.  Dissecting person re-identification from the viewpoint of viewpoint.  arXiv preprint arXiv:1812.02162, 2018.
  * [60] Xiaoxiao Sun and Liang Zheng.  Dissecting person re-identification from the viewpoint of viewpoint.  In CVPR, pages 608–617, 2019.
  * [61] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang.  Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline).  In ECCV, pages 480–496, 2018.
  * [62] Noriko Takemura, Yasushi Makihara, Daigo Muramatsu, Tomio Echigo, and Yasushi Yagi.  Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition.  IPSJ Transactions on Computer Vision and Applications, 10(1):4, 2018\.
  * [63] Fangbin Wan, Yang Wu, Xuelin Qian, Yixiong Chen, and Yanwei Fu.  When person re-identification meets changing clothes.  In CVPRW, pages 830–831, 2020.
  * [64] Guanshuo Wang, Yufeng Yuan, Xiong Chen, et al.  Learning discriminative features with multiple granularities for person re-identification.  In ACM MM, pages 274–282, 2018.
  * [65] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao.  A discriminative feature learning approach for deep face recognition.  In ECCV, pages 499–515. Springer, 2016.
  * [66] Xiaohua Xie, Jianhuang Lai, and Wei-Shi Zheng.  Extraction of illumination invariant facial features from a single image using nonsubsampled contourlet transform.  Pattern Recognition, 43(12):4177–4189, 2010.
  * [67] Chi Xu, Yasushi Makihara, Xiang Li, Yasushi Yagi, and Jianfeng Lu.  Gait recognition from a single image using a phase-aware gait cycle reconstruction network. eccv 2020.  2020\.
  * [68] Jia Xue, Zibo Meng, Karthik Katipally, Haibo Wang, and Kees van Zon.  Clothing change aware person identification.  In CVPR Workshops, pages 2112–2120, 2018.
  * [69] Qize Yang, Ancong Wu, and Wei-Shi Zheng.  Person re-identification by contour sketch under moderate clothing change.  TPAMI, 2019.
  * [70] Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, and Mingli Song.  Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more.  In CVPR, pages 2829–2838, 2019.
  * [71] Shijie Yu, Shihua Li, Dapeng Chen, Rui Zhao, Junjie Yan, and Yu Qiao.  Cocas: A large-scale clothes changing person dataset for re-identification.  In CVPR, pages 3400–3409, 2020.
  * [72] Hua Zhang, Si Liu, Changqing Zhang, Wenqi Ren, Rui Wang, and Xiaochun Cao.  Sketchnet: Sketch classification with web images.  In CVPR, pages 1105–1113, 2016.
  * [73] Li Zhang, Tao Xiang, and Shaogang Gong.  Learning a discriminative null space for person re-identificatio.  In CVPR, 2016.
  * [74] Li Zhang, Tao Xiang, and Shaogang Gong.  Learning a discriminative null space for person re-identification.  In CVPR, 2016.
  * [75] Peng Zhang, Qiang Wu, Jingsong Xu, and Jian Zhang.  Long-term person re-identification using true motion from videos.  In WACV, pages 494–502. IEEE, 2018.
  * [76] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu.  Deep mutual learning.  In CVPR, pages 4320–4328, 2018.
  * [77] Zhizheng Zhang, Cuiling Lan, Wenjun Zeng, et al.  Densely semantically aligned person re-identification.  In CVPR, 2019.
  * [78] Haiyu Zhao, Maoqing Tian, Shuyang Sun, et al.  Spindle net: Person re-identification with human body region guided feature decomposition and fusion.  In CVPR, 2017.
  * [79] Liang Zheng, Zhi Bie, Yifan Sun, Jingdong Wang, Chi Su, Shengjin Wang, and Qi Tian.  Mars: A video benchmark for large-scale person re-identification.  In ECCV, pages 868–884. Springer, 2016.
  * [80] Wei-Shi Zheng, Xiang Li, Tao Xiang, Shengcai Liao, Jianhuang Lai, and Shaogang Gong.  Partial person re-identification.  In ICCV, 2015.
  * [81] Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, et al.  Omni-scale feature learning for person re-identification.  ICCV, 2019.
  * [82] Jiaxuan Zhuo, Zeyu Chen, Jianhuang Lai, and Guangcong Wang.  Occluded person re-identification.  In ICME, pages 1–6. IEEE, 2018.