# Spatiotemporal Transformer for Video-based Person Re-identification

Tianyu Zhang,1 Longhui Wei,5 Lingxi Xie,4 Zijie Zhuang,4 Yongfei Zhang,1,2,3
Bo Li,1,2,3 Qi Tian 6
1Beijing Key Laboratory of Digital Media, School of Computer Science and
Engineering, Beihang University,
2State Key Laboratory of Virtual Reality Technology and Systems, Beihang
University,
3Pengcheng Laboratory, 4Tsinghua University, 5University of Science and
Technology of China, 6Xidian University
{tianyu1949, weilh2568, 198808xc, jayzhuang42<EMAIL_ADDRESS>{yfzhang,
<EMAIL_ADDRESS>
###### Abstract

Recently, the Transformer module has been transplanted from natural language
processing to computer vision. This paper applies the Transformer to video-
based person re-identification, where the key issue is to extract the
discriminative information from a tracklet. We show that, despite the strong
learning ability, the vanilla Transformer suffers from an increased risk of
over-fitting, arguably due to a large number of attention parameters and
insufficient training data. To solve this problem, we propose a novel pipeline
where the model is pre-trained on a set of synthesized video data and then
transferred to the downstream domains with the perception-constrained
Spatiotemporal Transformer (STT) module and Global Transformer (GT) module.
The derived algorithm achieves significant accuracy gain on three popular
video-based person re-identification benchmarks, MARS, DukeMTMC-VideoReID, and
LS-VID, especially when the training and testing data are from different
domains. More importantly, our research sheds light on the application of the
Transformer on highly-structured visual data.

##  1 Introduction

With increasing concerns on public security, person re-identification (ReID),
which aims to re-identify certain pedestrians in a disjoint camera network,
has drawn more and more attention from researchers. In past decades, image-
based ReID tasks, which rely on stationary pedestrian appearance from isolated
and noncontinuous frames, have been studied widely, and lots of novel works
[55, 46, 37, 38, 19, 20] have been proposed.

However, image-based ReID tasks neglect several vital clues. Compared to
isolated frames, continuous frames from surveillance videos provide rich
spatiotemporal information that reveals the causality between motions/viewing-
angle-changes and identity-appearance-variations. Aggregating such
spatiotemporal information properly into ReID tasks benefits deep ReID models
in extracting discriminative and stable features against the misalignment of
both body parts and pedestrian locations inside each frame. Meanwhile, the
redundancy of pedestrian appearance in continuous frames helps to overcome
temporary occlusion naturally. Thus, video-based ReID tasks, more precisely
tracklet-based ReID tasks, have recently become more attractive for the ReID
community [43, 29, 39, 11].

Figure 1: Illustrations of the strengths and weaknesses of the Transformer.
(a) Spatial attention is suitable for misaligned bounding boxes or occluded
persons. (b) Temporal attention is good at excluding misdetection. (c) The
vanilla Transformer learns biased attention. Minimal regions of the images are
involved in the attention map.

The key for video-based ReID tasks is to efficiently aggregate the spatial and
temporal knowledge provided by tracklets into one representation learning
framework. The Transformer [35] originated from natural language processing
(NLP) tasks sheds light on long sequence modeling. Encouraged by its success
in NLP tasks [8, 1], there are several attempts to utilize the Transformer in
computer vision tasks [3, 44]. Most of them focus on static and noncontinuous
images. Images are manually cropped as patches, and Transformers are used to
model the relations of these patches. Benefited from a strong capability of
“attention and then aggregation”, Transformer-based networks learn
discriminative representations from all image patches and thus achieve notable
success.

Such a capability meets the demands of video-based ReID tasks perfectly, where
the learning procedure is required in not only the spatial dimension of each
frame, e.g., relations between human body parts but also the temporal
dimension across continuous frames, e.g., appearance variations through time.
Thus, we propose a two-stage Spatiotemporal Transformer (STT) module, where a
Spatial Transformer handles the information in image patches and another
subsequent Temporal Transformer focuses on the video sequence. As shown in
Fig. 1(a) and Fig. 1(b), the Spatial Transformer is capable of attending to
human regions from various backgrounds, while the Temporal Transformer
excludes “noisy” frames in the given tracklet. However, such a vanilla
assembly of two Transformer modules suffers from the over-fitting risk due to
the insufficient training data in video-based ReID tasks. Fig. 1(c)
demonstrates a typical failure case. Although the Transformer surpasses
convolutional neural networks (CNN) in excluding background areas, it also
tends to over-focusing on local salient body parts while ignoring other common
but useful body parts information. Without any restrictions on learning
generic pedestrian appearance knowledge, the performance of the vanilla
Transformer on video-based ReID tasks is much worse than previous CNN-based
methods,  _e.g._ , $11.8\%$ rank-1 accuracy drop on MARS dataset.

To address the over-fitting issue mentioned above, we further propose the
constrained attention learning scheme upon the STT module, where multiple
prior constraints are applied on the Spatial Transformer and Temporal
Transformer, respectively. Meanwhile, we introduce a global attention learning
branch as the supplement to exploit the relationships between patches from
different frames. Finally, since the insufficient training data hinders us
from fully validating the Transformer in video ReID, we resort to the
synthesized data and propose a novel pre-training pipeline that relieves the
over-fitting problem. Extensive experiments demonstrate that, with our
proposed pipeline, on three popular benchmarks,  _i.e._ , MARS, DukeMTMC-
VideoReID, and LS-VID, our STT outperforms both CNN baselines and the vanilla
Transformer by a large margin. Our contributions are summarized as follows:

  * •

We propose a simple yet effective framework with the Spatiotemporal
Transformer for video-based person ReID. To the best of our knowledge, it is
the first to evaluate the effectiveness of the Transformer in video-based ReID
tasks, which paves a new way for further study.

  * •

To reduce the over-fitting risk caused by the Transformer and further enhance
the diversity of learned representation, we propose a constrained attention
learning scheme upon the STT module and an extra Global Transformer module as
a supplement. Moreover, the synthesized video data are generated for pre-
training.

  * •

Extensive experiments on two large-scale video-based ReID datasets have
validated the effectiveness of the proposed approaches. Compared with the
previous state-of-the-art methods with pure CNN architectures, our
Transformer-based approach significantly improves the final performance in
multiple datasets.

##  2 Related Work

###  2.1 Person Re-identification

Existing image-based ReID methods mainly focus on discriminative feature
extraction from images. Human body parts are essential for image-based ReID.
Many study how to learn feature embeddings from detected body parts [38, 30],
or fixed image areas [36, 33]. Some works learn the relationship between body
parts. For example, Su  _et al._ [32] align image patches that contain the
same body part; Xia  _et al._ [2] utilize Non-local module to explore the
importance of different parts; He  _et al._ [16] adopt the Vision Transformer
[9] for image-based ReID feature extraction. These methods show excellent
performance on public datasets, such as Market-1501 [50] and MSMT17 [37].
However, the temporal information from video tracklets also provides helpful
information to re-identify a person. Therefore, researchers pay more and more
attention to video-based ReID methods recently. Early works introduce the
optical flow and design two-stream networks to process optical flow and RGB
frames, respectively. Due to the computational complexity of the optical flow,
many works turn to more direct ways of modeling image sequences, such as RNN
[27, 54, 41] and LSTM [5, 42]. Graph neural networks are also effective to
mine the relationship of continuous frames [29, 43]. Although these methods
achieve promising performance, the extra heavy modules mentioned above are
time-consuming. Compared to these methods, the Transformer enjoys the benefits
of time-efficiency and is more suitable for video-based ReID tasks.

Figure 2: The overall framework with the Spatiotemporal Transformer. (a) The
constrained attention learning is adopted on the Spatial Transformer and
Temporal Transformer. (b) A global attention learning branch is proposed to
supplement STT. (c) The synthesized video pre-training plays an important role
in avoiding over-fitting.

###  2.2 Transformer for Vision Tasks

Transformer [35] is a self-attention-based architecture proposed for modelling
sequence data. In natural language processing (NLP), it has become the basic
module for state-of-the-art methods [8, 1]. Attention mechanism has also been
studied in computer vision and plays an important role in many computer vision
tasks. More recently, researchers apply the Transformer directly on image
patches, like Vit [9] and DeiT [34]. With large-scale pre-training [9], or
distillation techniques [34], these methods achieve competitive results on
image classification tasks. The application of Transformer modules on video
tasks has also arisen. For example, Gavrilyuk  _et al._ [12] propose the
actor-transformer for activity recognition. To better encode the point cloud
for the 3D video object detection task, Yin  _et al._ [45] propose a Spatial
Transformer to distinguish background and a Temporal Transformer to integrate
motion information. In this paper, we propose a hybrid architecture of CNN and
Transformer encoders for extracting features from each frame and aggregating
appearance information from the whole tracklet. To the best of our knowledge,
this is the first work that applies the Transformer to video-based person
ReID.

##  3 Methods

###  3.1 Formulation

Given an annotated video dataset
$\mathcal{S}=\\{(\boldsymbol{T}_{1},\boldsymbol{y}_{1}),(\boldsymbol{T}_{2},\boldsymbol{y}_{2}),...,(\boldsymbol{T}_{N},\boldsymbol{y}_{N})\\}$,
where each $\boldsymbol{T}_{i}$ denote a tracklet, and $\boldsymbol{y}_{i}$ is
the ground truth label of the corresponding identity, the goal of video-based
ReID tasks is to learn a feature embedding function
$\boldsymbol{f}(\theta;\boldsymbol{T}_{i})$ that maps tracklets into a feature
space
$\mathcal{X}=\\{\boldsymbol{x}_{i}|\boldsymbol{x}_{i}=\boldsymbol{f}(\theta;\boldsymbol{T}_{i}),1\leq
i\leq N\\}$, where tracklets of the same identity are closer than that of
different identity. To achieve this goal, researchers propose two types of
objective functions, i.e., classification loss and distance metric learning
loss. The classification loss classifies features $\boldsymbol{x}_{i}$ of the
same person into the same category, which can be implemented by the cross
entropy loss:

$\mathcal{L}_{\mathrm{xent}}=\sum^{N}_{i=1}-\log(\frac{\exp(\boldsymbol{h}(\boldsymbol{x}_{i})^{c})}{\sum_{j}\exp(\boldsymbol{h}(\boldsymbol{x}_{i})^{j})}),$
(1)

where $\boldsymbol{h}$ is a classifier, consisting of a linear layer and
softmax layer that maps $\boldsymbol{x}$ into a one-hot classification vector.
For distance metric learning losses, the triplet loss with online hard sample
mining [17] is a popular choice. For each mini-batch, it demands each positive
sample pair closer than all negative samples by at least a margin:

$\mathcal{L}_{\mathrm{trip}}=\sum_{i=1}^{C}[m+\mathrm{dist}(\boldsymbol{x}_{i},p(\boldsymbol{x}_{i}))-\mathrm{dist}(\boldsymbol{x}_{i},n(\boldsymbol{x}_{i}))]_{+},$
(2)

where $[z]_{+}=\max(z,0)$, $m$ denotes the margin. $p(\boldsymbol{x}_{i})$ is
the farthest sample belonging to the same person as $\boldsymbol{x}_{i}$, and
$n(\boldsymbol{x}_{i})$ is another identity’s image that is the nearest to
$\boldsymbol{x}_{i}$.

In video-based ReID tasks, each tracklet consists of a series of images, which
are usually generated by tracking techniques,  _i.e._ ,
$\boldsymbol{T}_{i}=\\{\boldsymbol{I}_{i}^{(1)},\boldsymbol{I}_{i}^{(2)},...,\boldsymbol{I}_{i}^{(n)}\\}$.
Compared to image-based ReID, video-based ReID tasks introduce a new temporal
dimension alongside existing spatial dimensions. Thus, the major challenges in
video-based ReID tasks are usually two-fold. First, a good feature embedding
for each image is required to encode appearance characteristics and spatial
information. Second, which is also the main difference compared to image-based
ReID tasks, video-based ReID tasks demand an ingenious module for modeling
temporal relations among all frames in each tracklet.

However, modeling temporal relations is extremely challenging in real-world
applications, especially for a non-ideal, unconstrained surveillance video.
Both the severe mismatch of human body parts and foreground position shifts
inside each frame might lead to significant performance decreases. To tackle
these issues, we disassemble the mismatch problem into two categories: spatial
mismatch in the image-level and temporal mismatch in the tracklet-level, and
attempts to adopt Transformer-based methods to address the above challenges
accordingly. Details of our method are presented in the next section.

###  3.2 Overall Framework

Benefited from the strong capability of “attention and aggregation”, we can
directly utilize Transformers for video-based ReID tasks, extracting and
aggregating the useful human information spatially and temporally while
neglecting disturbances such as occlusions and background areas.

The overall of our proposed framework is shown in Fig. 2. We first utilize a
truncated convolutional neural network, e.g., a ResNet-50 backbone with the
first three blocks, as the preliminary feature encoder. For each frame, the
CNN backbone encodes it into a 3-dimensional feature map. Next, each feature
map is split into small patches. We regard each patch as a “token” and feed
tokens from the same image into our Spatial Transformer (ST) module. In this
way, the ST module learns the spatial relations among all given tokens and
aggregates discriminative spatial information as well as appearance features
from them. Simultaneously, following the practice in [9, 34], we add another
trainable classification token for fusing the spatial information of each
input image. This token fuses the information of all patch-based tokens. Since
it further encodes the spatial relation between tokens, we name it the spatial
token. For each frame, this spatial token is regarded as the final output of
the ST module. After the ST module encodes each frame, the Temporal
Transformer (TT) module collects all spatial tokens from the same tracklet and
aggregates them into a temporal token. The final temporal token embeds the
discriminative identity information spatially and temporally. It is used as
the final representation for the entire tracklet.

The vanilla two-stage Transformer mentioned above processes both spatial and
temporal information at the same time, making it conceptually suitable for
video-based ReID tasks. However, when using it in video-based ReID tasks
directly, it suffers from severe over-fitting. As shown in Tab. 2 and Tab. 1,
compared to the CNN baseline, this framework reports much worse performance,
even though its preliminary feature encoder is the same as the first three
blocks of the CNN baseline. We attribute this performance decrease to three
reasons. First, the Transformer is heavily parameterized. Without any
restrictions, it could easily fall into the local minimal and generalize
poorly on the testing data. To address this issue, we propose a constrained
attention learning scheme especially to prevent the Transformer from over-
focusing on local regions. Second, the two-stage design separates the spatial
information and temporary information to avoid the over-fitting. However, with
this design, patches from one image cannot communicate with any other patches
of another image. Thus, we add a global attention learning branch that
associates patches across frames. Third, since the Transformer introduces much
more parameters, existing video-based ReID datasets are insufficient to tune
these parameters fully. To tackle this problem, we introduce pre-trained
video-based ReID models with synthesized images to achieve better
initialization. In the following section, we introduce our design principles
and the proposed pipeline in detail.

Figure 3: Illustration of the spatial constraint. The tokens of patches are
reshaped for horizontal average pooling.

###  3.3 Constrained Attention Learning

Owing to the two-stage architecture of the Spatiotemporal Transformer, which
contains the Spatial Transformer (ST) to process the patch tokens in image-
level and the Temporal Transformer (TT) to handle the image tokens in
tracklet-level, we set different constraints on ST and TT individually to
relieve the over-fitting problem.

####  3.3.1 Spatial Constraint

This Spatial Transformer module of STT processes patch tokens in each image.
To be more specific, each feature map extracted by the CNN backbone is split
into $H\times W$ patches and then flattened into $1$-dim tokens. Therefore,
with the extra classification token (or called spatial token in this paper),
the ST module receives $H\times W+1$ tokens in total. Aiming to learn
discriminative representation in spatial dimension, we add the cross entropy
loss,  _i.e._ , $\mathcal{L}_{\mathrm{spa\\_xent}}$, on the spatial token.
Therefore, the spatial token is forced to focus on the human information to
classify the input. Because of the small-scale data for video ReID, ST can
easily focus on limited regions but ignore detailed cues (  _e.g._ ,
backpack). To further avoid this, we add a spatial part cross entropy loss,
_i.e._ , $\mathcal{L}_{\mathrm{spa\\_part\\_xent}}$, to force each token to
learn useful recognition information as much as possible. The spatial part
loss is motivated by the recent part-based image ReID methods [38, 33].
Specifically, as shown in Fig. 3, we divide the $H\times W$ tokens into $P$
groups horizontally so that each group has $H/P\times W$ tokens. An average
pooling operation is conducted within every group. Finally, a cross entropy
classification loss is calculated for every averaged token, and the sum is
formulated as the spatial part cross entropy loss:

$\mathcal{L}_{\mathrm{spa\\_part\\_xent}}=\frac{1}{P}\sum_{p=1}^{P}\mathcal{L}_{\mathrm{spa\\_part\\_xent}}^{(p)},$
(3)

where $\mathcal{L}_{\mathrm{spa\\_part\\_xent}}^{(p)}$ represents the cross
entropy loss for the $p\mathrm{th}$ horizontal group. Obviously, this loss
term leads the model to discover person-related clues in every horizontal part
of the body, which makes sure each patch token learns enough human
information.

Therefore, the spatial constraint loss can be summarized as follows:

$\mathcal{L}_{\mathrm{SpaC}}=\mathcal{L}_{\mathrm{spa\\_part\\_xent}}+\mathcal{L}_{\mathrm{spa\\_xent}}.$
(4)

Figure 4: Illustration of the temporal constraint. Normalization layers and
skip connections are omitted in this figure for clear presentation. The first
row of the attention matrix (_i.e._ , weights of classification tokens) is
taken as the input of the temporal attention loss.

####  3.3.2 Temporal Constraint

As shown in Fig. 4, the Temporal Transformer (TT) module takes the spatial
tokens from the Spatial Transformer as input. One extra classification token
(or called the temporal token) is added as the final feature representation.
In a tracklet sequence, the pedestrian may be occluded in some images but
occur again in other images. With the cross entropy loss
$\mathcal{L}_{\mathrm{tem\\_xent}}$ that supervises the final output, we
expect the Temporal Transformer to exclude frames of lousy quality and utilize
more information from frames that contain complete human bodies. However, in
the meantime, the risk of over-fitting is raised because the learning
algorithm is not aware of the targeted human body. For instance, an image
containing a unique distractor person is also quite discriminative for the
model. Therefore, the Transformer module will attend to this image but ignore
the targeted human. More examples can be seen in Fig. 6. To solve this
problem, we expect the exploited representation should contain the common
information of all frames in the tracklet and other specific information on
each frame.

To achieve this, apart from the cross entropy loss that supervises the final
output, we add supervision for the temporal output tokens of each frame, named
$\mathcal{L}_{\mathrm{tem\\_trip}}$. As shown in Eq. 2, the triplet loss
shrinks the distances of positive pairs. Because frames within the same
tracklet are assigned with the same identity label, their distances are
naturally reduced so that the extracted information is shared by most of the
frames. Besides, we also encourage the Temporal Transformer to focus on as
many frames as possible to avoid biased attention to specific frames. We add a
temporal attention loss on the attention weights of the final classification
token:

$\mathcal{L}_{\mathrm{tem\\_attn}}=\sum_{i=1}^{N}[\exp(\sum_{k=1}^{L}\boldsymbol{a}_{i,k}\log(\boldsymbol{a}_{i,k}))-\alpha]_{+},$
(5)

where $L$ is the number of frames within a tracklet, $\boldsymbol{a}_{i,k}$ is
the attention weight of the $k\mathrm{th}$ frame in the $i\mathrm{th}$
tracklet, and $\alpha$ is a hyper-parameter that adjusts the upper bound. The
temporal attention loss increases the information entropy of the attention
weights in each tracklet. Also, it leaves much space for the Transformer to
decide which frame is more critical with the parameter $\alpha$.

Finally, the temporal constraint loss is as follows:

$\mathcal{L}_{\mathrm{TemC}}=\mathcal{L}_{\mathrm{tem\\_trip}}+\mathcal{L}_{\mathrm{tem\\_attn}}.$
(6)

###  3.4 Global Attention Learning

As described above, the Spatiotemporal Transformer learns image-level
attention and tracklet-level attention in succession. Therefore, the
relationships between patches of different frames are ignored. To address this
problem, we further introduce the global attention learning branch in our
framework. This part has a Global Transformer module, which takes the feature
map patches of all frames in a tracklet as the input. More specifically,
supposing we have $H\times W$ patches for every frame and $L$ frames in a
tracklet, we concatenate these tokens of every frame within the same tracklet,
resulting in an input, of which the length is $H\times W\times L$. These are
directly fed into the Global Transformer (GT) with an extra classification
token. Then, a cross entropy loss $\mathcal{L}_{\mathrm{global\\_xent}}$ is
adopted to supervise the learning of GT. When the global attention learning
branch is enabled, the final representation is generated by the concatenation
of STT outputs and GT outputs.

Models | Spatial Constraint | Temporal Constraint | Global Attention | Pre-training | MARS | Duke | LS-VID
---|---|---|---|---|---|---|---
|  |  |  |  | rank-1 | mAP | rank-1 | mAP | rank-1 | mAP
CNN baseline |  |  |  |  | 84.1 | 78.8 | 50.6 | 48.3 | 16.6 | 11.2
CNN+Transformer* |  |  |  |  | 72.3 | 63.1 | 24.5 | 24.5 | 4.5 | 3.9
CNN+Transformer | ✓ |  |  |  | 87.1 | 83.5 | 60.5 | 56.3 | 16.2 | 11.7
CNN+Transformer | ✓ | ✓ |  |  | 87.2 | 84.2 | 61.1 | 57.4 | 18.2 | 13.1
CNN+Transformer | ✓ | ✓ |  | ✓ | 88.6 | 83.9 | 62.4 | 61.0 | 18.7 | 13.1
CNN+Transformer | ✓ | ✓ | ✓ |  | 87.3 | 85.0 | 65.0 | 62.1 | 21.0 | 14.6
CNN+Transformer | ✓ | ✓ | ✓ | ✓ | 88.7 | 86.3 | 69.2 | 66.2 | 24.3 | 17.2
|  |  |  |  |  |  |  |  |  |

Table 1: Evaluation results when training on MARS. * denotes the vanilla
version of the Transformer.

Models | Spatial Constraint | Temporal Constraint | Global Attention | Pre-training | MARS | Duke | LS-VID
---|---|---|---|---|---|---|---
|  |  |  |  | rank-1 | mAP | rank-1 | mAP | rank-1 | mAP
CNN baseline |  |  |  |  | 34.1 | 18.8 | 91.2 | 88.0 | 20.8 | 13.7
CNN+Transformer* |  |  |  |  | 23.2 | 12.8 | 90.7 | 88.2 | 7.3 | 5.9
CNN+Transformer | ✓ |  |  |  | 45.0 | 28.2 | 96.2 | 96.0 | 32.0 | 22.0
CNN+Transformer | ✓ | ✓ |  |  | 46.4 | 28.9 | 96.6 | 96.0 | 32.9 | 22.6
CNN+Transformer | ✓ | ✓ |  | ✓ | 51.4 | 33.6 | 97.4 | 97.0 | 34.6 | 24.3
CNN+Transformer | ✓ | ✓ | ✓ |  | 48.6 | 30.8 | 96.7 | 96.6 | 34.7 | 24.6
CNN+Transformer | ✓ | ✓ | ✓ | ✓ | 56.5 | 39.1 | 97.6 | 97.4 | 41.3 | 29.4
|  |  |  |  |  |  |  |  |  |

Table 2: Evaluation results when training on Duke. * denotes the vanilla
version of the Transformer.

Models | Spatial Constraint | Temporal Constraint | Global Attention | Pre-training | MARS | Duke | LS-VID
---|---|---|---|---|---|---|---
|  |  |  |  | rank-1 | mAP | rank-1 | mAP | rank-1 | mAP
CNN baseline |  |  |  |  | 36.1 | 20.3 | 50.6 | 43.2 | 58.5 | 44.6
CNN+Transformer* |  |  |  |  | 12.0 | 7.7 | 16.8 | 17.9 | 31.9 | 28.3
CNN+Transformer | ✓ |  |  |  | 50.0 | 32.7 | 74.2 | 71.3 | 78.8 | 68.0
CNN+Transformer | ✓ | ✓ |  |  | 52.7 | 35.4 | 77.8 | 74.9 | 80.7 | 70.5
CNN+Transformer | ✓ | ✓ |  | ✓ | 54.0 | 37.8 | 83.1 | 80.5 | 84.6 | 74.7
CNN+Transformer | ✓ | ✓ | ✓ |  | 53.9 | 36.8 | 79.6 | 76.0 | 82.4 | 73.0
CNN+Transformer | ✓ | ✓ | ✓ | ✓ | 59.6 | 42.7 | 86.0 | 83.5 | 87.5 | 78.0
|  |  |  |  |  |  |  |  |  |

Table 3: Evaluation results when training on LS-VID. * denotes the vanilla
version of the Transformer.

###  3.5 Synthesized Video Pre-training

Previous works [9, 34] show that the data scale is one of the most important
keys for the Transformer and conclude that Transformers can work better than
pure CNN networks when the training data is sufficient. Limited by the
difficulty of annotations, it is nearly impossible to annotate a super large-
scale video ReID dataset. To further relieve the over-fitting problem of the
Transformer in the data aspect, we resort to the synthesized data and pre-
train our model on this dataset. Specifically, we adopt the UnrealPerson [47]
toolkit to generate videos in 3D virtual scenes. Four different environments
and $34$ cameras are used in our synthesized data. We implement two extra
modifications to UnrealPerson. First, we add large disturbance when cutting
the bounding boxes. In real scenes, the bounding boxes for video-based ReID
tasks are not ideally aligned. Therefore, persons in bounding boxes may not
appear in the middle. This helps the Transformer to concentrate on local
patches where the body parts exist. Second, to mimic the tracklets by tracking
algorithms, we also keep the severely occluded frames. Therefore, in a
tracklet, the quality of images may vary largely. This increases the
difficulty of the synthesized data and encourages the model to find images of
good quality in a tracklet. The synthesized dataset we adopt in the
experiments contains $90\mathrm{,}673$ tracklets of $2\mathrm{,}808$
identities.

##  4 Experiments

###  4.1 Datasets

We conduct experiments on two benchmark datasets, MARS [49], DukeMTMC-
VideoReID [40] and LS-VID [19]. MARS consists of $1\rm{,}261$ pedestrians
captured under $6$ different cameras on campus. The training set of MARS has
$8\rm{,}298$ tracklets of $625$ persons, and the testing set has $621$ persons
and $9\rm{,}330$ gallery images. There are $1\rm{,}840$ valid query tracklets
in total. In MARS, each tracklet has 59 images on average. DukeMTMC-VideoReID
(referred as Duke for short) has $2\rm{,}196$ tracklets of $702$ pedestrians
for training, and $2\rm{,}336$ tracklets of $1\rm{,}110$ persons for testing.
Each tracklet of Duke has $168$ frames on average. LS-VID is constructed with
videos from $15$ cameras. The training set contains $2\rm{,}831$ tracklets of
$842$ identities, and the testing set includes $11\rm{,}333$ tracklets of
$2\rm{,}730$ identities, among which $3\rm{,}504$ tracklets are used as
queries. Images and tracklets of MARS and LS-VID are generated automatically
by detection and tracking algorithms,  _i.e._ , DPM [10], GMMCP [6] and
Faster-RCNN [28], while Duke is manually labeled. In our experiments, we
follow the standard evaluation protocols [50] and report rank-1 accuracy and
mean average precision (mAP).

###  4.2 Implementation Details

Architecture. We adopt the first three residual blocks of ResNet-50 [15] that
is pre-trained on ImageNet [7] as the CNN backbone. The CNN baseline model has
the complete layers of all $4$ residual blocks of ResNet-50, while in the
proposed CNN+Transformer architecture, the $4\mathrm{th}$ residual block is
replaced by Transformer blocks. For our proposed Spatiotemporal Transformer,
we follow the implementation of ViT [9]. The output feature maps of the CNN
backbone go through a convolutional layer and are flattened to patch tokens.
The Spatial and Temporal Transformers share the same architecture design, with
$1$ layer and $6$ heads. The Global Transformer has $2$ layers and $6$ heads.
The embedding dimension of all Transformers is set to $768$. Positional
embeddings are only used in the Spatial Transformer. By discarding positional
embeddings in the Temporal Transformer and Global Transformer, our framework
is able to process tracklets of any length.

Input. Inspired by DeiT [34], we use more extreme data augmentation operations
in our experiments, including random erasing [51], random patch [53, 52],
random crop, random flip and AutoAugment [4]. The input size of images is set
as $256\times 128$. We sample $8$ images with the restricted sampling strategy
[22] from each tracklet when training. For evaluation, we sample $32$ images
from each tracklet. A balanced sampling strategy [17] is adopted to form the
mini-batches. Specifically, $15$ identities are selected first, and for each
identity, $4$ tracklets are chosen. Thus the mini-batch size is $60$. The size
of the output feature map in the CNN backbone is $1024\times 16\times 8$,
among which 1024 is the channel number. The feature map is further embedded
into $8\times 4$ patches by a convolution layer.

Training Details. For all cross entropy losses, we add label smoothing
regularization with $\epsilon=0.1$. The part number of the spatial part loss
is $4$. The margin of the temporal triplet loss is set as $0.3$. In the
temporal attention loss, the parameter $\alpha$ is set to $0.15$ empirically.
Besides loading the pre-trained weights for the CNN backbone, we also add an
extra cross entropy loss for it to stabilize training. In all our experiments,
we adopt AdamW [26, 18] optimizer with the weight decay of $5e-4$. The
learning rate is set to $3.5e-4$ initially and multiply by a factor $0.1$ at
the $80\mathrm{th}$ epoch and $160\mathrm{th}$ epoch, respectively. The
learning rate of all classifiers is set to a larger value, $1.75e-3$. During
the first $200$ iterations, the learning rate warm-up strategy is adopted. All
of the models are trained for $200$ epochs.

###  4.3 Ablation Study

In the ablation study, we evaluate the effectiveness of our proposed framework
and the extra constraints for Transformer modules. The results are shown in
Tab. 1, Tab. 2 and Tab. 3. The direct transfer evaluation is also conducted
among different datasets to validate the generalization ability of the trained
models.

Vanilla Transformer. Compared to the pure CNN baseline model, the Transformer
network without any additional constraints suffers performance drop,
especially when trained on MARS and LS-VID. The direct transfer performance
decays even more than self-domain evaluation, indicating serious over-fitting
exists in the trained model. The reason has been illustrated in Fig. 1(c),
_i.e._ , an unconstrained Transformer network produces biased attention
weights.

Constrained Attention Learning. When we add the spatial constraint losses in
the model, the performance surpasses the pure CNN baseline by a large margin,
especially on direct transfer results. For example, when we evaluate Duke with
the model trained on MARS, the rank-1 accuracy raises from $50.6\%$ to
$60.5\%$. The temporal constraint also improves all performances slightly.
These results demonstrate that our proposed loss terms are effective in
preventing over-fitting.

Global Attention Learning. The global attention learning empowers each patch
token to associate with patches from other frames. Experiment results show
that this global scope is an important complementary component for the
Spatiotemporal Transformer. For instance, in Tab. 1, when trained on MARS and
evaluated on Duke, the model with global attention learning improves the
rank-1 accuracy by $3.9\%$ compared to the model without global attention.

Synthesized Data Pre-training. The pre-training weights provide a good
initialization for the Transformer modules, avoiding over-fitting from early
training stages. Note that we have the pre-trained weights fine-tuned on the
downstream datasets for $200$ epochs, the same as training without synthesized
data pre-training. Therefore, there should leave very little information about
the synthesized dataset in the final model weights. Nevertheless, the direct
transfer performance is boosted significantly on all three datasets,
indicating that the Transformer benefits from better pre-trained weights and
reaches a more optimal convergence in the end.

###  4.4 Visualization Analysis

Figure 5: Spatial attention maps after adding spatial constraint losses.
Figure 6: Temporal attention weights after using constrained attention
learning. Figure 7: The attention maps of two images from the right figure of
Fig. 6. These two images are notable for the vanilla Transformer in the
tracklet because of distractors.

####  4.4.1 Comparison on Spatial Attention

As shown in Fig. 5, we draw the attention maps with the vanilla Transformer
models and compare them with attention maps after adding the spatial
constraints. The attention weights are extracted from the attention matrix in
the Spatial Transformer. The first row of the matrix,  _i.e._ ,
classification-token-related weights, is resized to reflect the importance of
local patches. The given examples are all successful detection bounding boxes,
and the human bodies are occupying most of the images. Despite this, the
vanilla Transformer learns biased attention and discards many useful regions,
indicating it suffers from serious over-fitting. In comparison, with our
proposed constrained spatial attention learning, we set up a more challenging
objective for models to learn. We encourage every image and every horizontal
part can be classified correctly, so more regions on human bodies are covered
by the attention maps.

####  4.4.2 Comparison on Temporal Attention

In Fig. 6, the left figure shows that in a tracklet where most of the images
are of good quality, the vanilla Transformer without extra constraints tends
to pay attention to very few frames. For example, only the $6\mathrm{th}$
frame has a high response in the attention matrix of the Temporal Transformer.
Our proposed temporal attention loss makes the weights more evenly distributed
across all frames via increasing the information entropy of attention weights.
We also set up an explicit upper bound of information entropy. Therefore,
there still exists space for the Transformer to distinguish less important
frames, like the $8\mathrm{th}$ frame in the left example of Fig. 6.

Another benefit the temporal constraint brings is that the final
representation of the tracklet should be shared by most frames. In the right
figure of Fig. 6, the vanilla Transformer gives high attention weight to the
$6\mathrm{th}$ frame. We observe the spatial attention map of this frame (as
shown in Fig. 7) and find that the person in the red box is considered as the
object of the tracklet. From the whole tracklet, we know that the red box
contains a distractor and should be ignored by the model. With the temporal
constraint, we require the output tokens of every frame within a tracklet to
be as close as possible. In this way, distractors are paid less attention
because they only appear in some frames. From Fig. 7, we see that with
constrained attention learning, the model mainly extracts features from the
target person.

###  4.5 Comparison with Other Methods

Methods | MARS | Duke
---|---|---
| rank-1 | mAP | rank-1 | mAP
TKP [14] | 84.0 | 73.3 | 94.0 | 91.7
STA [11] | 86.2 | 81.2 | 96.0 | 95.0
GLTR [19] | 87.0 | 78.5 | 96.3 | 93.7
MG-RAFA [48] | 88.8 | 85.9 | - | -
STE-NVAN [23] | 88.9 | 81.2 | 95.2 | 93.5
AGRL [39] | 89.5 | 81.9 | 97.0 | 95.4
NL-AP3D [13] | 90.7 | 85.6 | 97.2 | 96.1
STGCN [29] | 90.0 | 83.7 | 97.3 | 95.7
GRL [24] | 91.0 | 84.8 | - | -
Ours | 88.7 | 86.3 | 97.6 | 97.4
|  |  |  |
Table 4: Comparison with recent works on MARS and Duke datasets. Methods | rank-1 | rank-5 | rank-10 | mAP
---|---|---|---|---
Two-stream [31] | 48.2 | 68.7 | 75.1 | 32.1
LSTM [42] | 52.1 | 72.6 | 78.9 | 35.9
STMP [25] | 56.8 | 76.2 | 82.0 | 39.1
M3D [21] | 57.7 | 76.1 | 83.4 | 40.1
GLTR [19] | 63.1 | 77.2 | 83.8 | 44.3
Ours | 87.5 | 95.0 | 96.5 | 78.0
|  |  |  |
Table 5: Comparison with recent works on LS-VID.

In Tab. 4 and Tab. 5, we compare our method with recent video-based person
ReID methods on the three benchmark datasets. On both Duke and LS-VID, our
proposed framework outperforms state-of-the-art methods in rank-1 accuracy and
mAP. For example, we surpass the graph-based method STGCN [29] by $1.7\%$ in
mAP on Duke. This improvement is significant, considering the baseline
accuracy of Duke is rather high. Our method also outperforms GLTR [19] by
$24.4\%$ in rank-1 accuracy on LS-VID. On MARS dataset, our method achieves
state-of-the-art performance in mAP. From other methods, we see that there is
no absolutely positive correlation between rank-1 accuracy and mAP. For
instance, GRL [24] achieves the best rank-1 accuracy, but the mAP of GRL is
significantly lower than MG-RAFA [48]. In MARS, of which detection boxes and
tracking sequences are automatically generated by algorithms, the quality of
images and tracklets may vary largely. Therefore, different algorithms may
emphasize different aspects. Rank-1 accuracy reflects the ability to find the
most confident positive sample, while mAP is a more comprehensive indicator,
as it considers the ranking positions of all positive samples. Therefore, the
significant improvement in mAP on MARS validates the effectiveness of our
proposed method.

##  5 Conclusions

This paper presents a simple yet effective Transformer-based representation
learning framework for video-based person ReID tasks. In this framework, the
constrained attention learning scheme and global attention learning branch are
proposed to exploit spatial and temporal knowledge from ReID videos.
Furthermore, we introduce synthesized data pre-training for better
initializing our framework and reducing the over-fitting risk. Extensive
experiments have shown the effectiveness of our proposed approaches in this
paper. Importantly, to our best knowledge, it is the first work to evaluate
the ability of the Transformer in video-based ReID tasks, which paves a new
way for the application of the Transformer on more video-based tasks.

## References

  * [1] T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, T. Henighan, R. Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, E. Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, J. Clark, Christopher Berner, Sam McCandlish, A. Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [2] B. Bryan, Y. Gong, Y. Zhang, and C. Poellabauer.  Second-order non-local attention networks for person re-identification.  In ICCV, 2019.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  ArXiv, abs/2005.12872, 2020.
  * [4] E. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le.  Autoaugment: Learning augmentation strategies from data.  In CVPR, 2019.
  * [5] J. Dai, Pingping Zhang, D. Wang, H. Lu, and H. Wang.  Video person re-identification by temporal residual learning.  IEEE TIP, 28:1366–1377, 2019.
  * [6] Afshin Dehghan, S. M. Assari, and M. Shah.  Gmmcp tracker: Globally optimal generalized maximum multi clique problem for multiple object tracking.  In CVPR, 2015.
  * [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [8] J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL-HLT, 2019.
  * [9] A. Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. Dehghani, Matthias Minderer, Georg Heigold, S. Gelly, Jakob Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [10] Pedro F. Felzenszwalb, Ross B. Girshick, David A. McAllester, and D. Ramanan.  Object detection with discriminatively trained part based models.  IEEE TPAMI, 32:1627–1645, 2009.
  * [11] Y. Fu, Xiaoyang Wang, Yunchao Wei, and Thomas Huang.  Sta: Spatial-temporal attention for large-scale video-based person re-identification.  In AAAI, 2019.
  * [12] Kirill Gavrilyuk, R. Sanford, M. Javan, and Cees G. M. Snoek.  Actor-transformers for group activity recognition.  In CVPR, 2020.
  * [13] Xinqian Gu, H. Chang, Bingpeng Ma, Hongkai Zhang, and X. Chen.  Appearance-preserving 3d convolution for video-based person re-identification.  In ECCV, 2020.
  * [14] Xinqian Gu, Bingpeng Ma, H. Chang, S. Shan, and X. Chen.  Temporal knowledge propagation for image-to-video person re-identification.  In ICCV, 2019.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [16] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang.  Transreid: Transformer-based object re-identification, 2021.
  * [17] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv preprint arXiv:1703.07737, 2017.
  * [18] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [19] Jianing Li, J. Wang, Q. Tian, Wen Gao, and S. Zhang.  Global-local temporal representations for video person re-identification.  In ICCV, 2019.
  * [20] Jianing Li and Shiliang Zhang.  Joint visual and temporal consistency for unsupervised domain adaptive person re-identification.  In ECCV, 2020.
  * [21] Jianing Li, Shiliang Zhang, and Tiejun Huang.  Multi-scale 3d convolution network for video based person re-identification.  In AAAI, 2019.
  * [22] Shuang Li, Slawomir Bak, P. Carr, and Xiaogang Wang.  Diversity regularized spatiotemporal attention for video-based person re-identification.  In CVPR, 2018.
  * [23] C. Liu, Chih-Wei Wu, Yu-Chiang Frank Wang, and S. Chien.  Spatially and temporally efficient non-local attention network for video-based person re-identification.  In BMVC, 2019.
  * [24] Xuehu Liu, Pingping Zhang, Chenyang Yu, Huchuan Lu, and Xiaoyun Yang.  Watching you: Global-guided reciprocal learning for video-based person re-identification.  In CVPR, 2021.
  * [25] Yiheng Liu, Zhenxun Yuan, W. Zhou, and H. Li.  Spatial and temporal mutual promotion for video-based person re-identification.  In AAAI, 2019.
  * [26] I. Loshchilov and F. Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [27] N. McLaughlin, J. Rincón, and P. Miller.  Recurrent convolutional network for video-based person re-identification.  In CVPR, 2016.
  * [28] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE TPAMI, 39(6):1137–1149, 2017.
  * [29] Jin rui Yang, W. Zheng, Q. Yang, Y. Chen, and Q. Tian.  Spatial-temporal graph convolutional network for video-based person re-identification.  In CVPR, 2020.
  * [30] M Saquib Sarfraz, Arne Schumann, Andreas Eberle, and Rainer Stiefelhagen.  A pose-sensitive embedding for person re-identification with expanded cross neighborhood re-ranking.  In CVPR, 2018.
  * [31] K. Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In NeurIPS, 2014.
  * [32] Chi Su, Jianing Li, Shiliang Zhang, Junliang Xing, Wen Gao, and Qi Tian.  Pose-driven deep convolutional model for person re-identification.  In ICCV, 2017.
  * [33] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang.  Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline).  In ECCV, 2018.
  * [34] Hugo Touvron, M. Cord, M. Douze, Francisco Massa, Alexandre Sablayrolles, and H. Jégou.  Training data-efficient image transformers & distillation through attention.  ArXiv, abs/2012.12877, 2020.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [36] Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou.  Learning discriminative features with multiple granularities for person re-identification.  In ACMMM, 2018.
  * [37] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian.  Person transfer gan to bridge domain gap for person re-identification.  In CVPR, 2018.
  * [38] Longhui Wei, Shiliang Zhang, Hantao Yao, Wen Gao, and Qi Tian.  Glad: global-local-alignment descriptor for pedestrian retrieval.  In ACMMM, 2017.
  * [39] Yiming Wu, Omar El Farouk Bourahla, X. Li, F. Wu, and Q. Tian.  Adaptive graph representation learning for video person re-identification.  IEEE TIP, 29:8821–8830, 2020.
  * [40] Yu Wu, Yutian Lin, Xuanyi Dong, Yan Yan, Wanli Ouyang, and Yi Yang.  Exploit the unknown gradually: One-shot video-based person re-identification by stepwise learning.  In CVPR, 2018.
  * [41] Yichao Yan, B. Ni, Zhichao Song, Chao Ma, Yan Yan, and X. Yang.  Person re-identification via recurrent feature aggregation.  In ECCV, 2016.
  * [42] Yichao Yan, B. Ni, Zhichao Song, Chao Ma, Yan Yan, and X. Yang.  Person re-identification via recurrent feature aggregation.  In ECCV, 2016.
  * [43] Yichao Yan, J. Qin, Jiaxin Chen, Li Liu, F. Zhu, Ying Tai, and Ling Shao.  Learning multi-granular hypergraphs for video-based person re-identification.  In CVPR, 2020.
  * [44] Fuzhi Yang, Huan Yang, J. Fu, Hongtao Lu, and B. Guo.  Learning texture transformer network for image super-resolution.  In CVPR, 2020.
  * [45] Junbo Yin, J. Shen, Chenye Guan, Dingfu Zhou, and Ruigang Yang.  Lidar-based online 3d video object detection with graph-based message passing and spatiotemporal transformer attention.  In CVPR, 2020.
  * [46] Tianyu Zhang, Lingxi Xie, Longhui Wei, Yongfei Zhang, Bo Li, and Qi Tian.  Single camera training for person re-identification.  In AAAI, 2020.
  * [47] Tianyu Zhang, Lingxi Xie, Longhui Wei, Zijie Zhuang, Yongfei Zhang, Bo Li, and Qi Tian.  Unrealperson: An adaptive pipeline towards costless person re-identification.  In CVPR, 2021.
  * [48] Zhizheng Zhang, Cuiling Lan, Wenjun Zeng, and Zhibo Chen.  Multi-granularity reference-aided attentive feature aggregation for video-based person re-identification.  In CVPR, 2020.
  * [49] L. Zheng, Zhi Bie, Y. Sun, Jingdong Wang, Chi Su, S. Wang, and Q. Tian.  Mars: A video benchmark for large-scale person re-identification.  In ECCV, 2016.
  * [50] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In ICCV, 2015.
  * [51] Z. Zhong, L. Zheng, Guoliang Kang, Shaozi Li, and Y. Yang.  Random erasing data augmentation.  In AAAI, 2020.
  * [52] K. Zhou, Yongxin Yang, A. Cavallaro, and T. Xiang.  Learning generalisable omni-scale representations for person re-identification.  ArXiv, abs/1910.06827, 2019.
  * [53] K. Zhou, Yongxin Yang, A. Cavallaro, and T. Xiang.  Omni-scale feature learning for person re-identification.  In ICCV, 2019.
  * [54] Z. Zhou, Y. Huang, Wei Wang, Liang Wang, and T. Tan.  See the forest for the trees: Joint spatial and temporal recurrent neural networks for video-based person re-identification.  In CVPR, 2017.
  * [55] Zijie Zhuang, Longhui Wei, Lingxi Xie, Tianyu Zhang, Hengheng Zhang, Haozhe Wu, Haizhou Ai, and Qi Tian.  Rethinking the distribution gap of person re-identification with camera-based batch normalization.  In ECCV, 2020.