# Feature Completion Transformer for Occluded Person Re-identification

Tao Wang, Hong Liu†, Member, IEEE, Wenhao Li, Miaoju Ban, Tianyu Guo, Yidi Li

###### Abstract

Occluded person re-identification (Re-ID) is a challenging problem due to the
destruction of occluders. Most existing methods focus on visible human body
parts through some prior information (such as pose information, semantic
segmentation, human body parsing, etc.). However, when complementary
occlusions occur, features in occluded regions can interfere with matching,
which affects performance severely. In this paper, different from most
previous works that discard the occluded region, we propose a Feature
Completion Transformer (FCFormer) to implicitly complement the semantic
information of occluded parts in the feature space. Specifically, Occlusion
Instance Augmentation (OIA) is proposed to simulates real and diverse
occlusion situations on the holistic image. These augmented images not only
enrich the amount of occlusion samples in the training set, but also form
pairs with the holistic images. Subsequently, a dual-stream architecture with
a shared encoder is proposed to learn paired discriminative features from
pairs of inputs. Without additional semantic information, an occluded-holistic
feature sample-label pair can be automatically created. Then, Feature
Completion Decoder (FCD) is designed to complement the features of occluded
regions by using learnable tokens to aggregate possible information from self-
generated occluded features. Finally, we propose the Cross Hard Triplet (CHT)
loss to further bridge the gap between complementing features and extracting
features under the same ID. In addition, Feature Completion Consistency (FC2)
loss is introduced to help the generated completion feature distribution to be
closer to the real holistic feature distribution. Extensive experiments over
five challenging datasets demonstrate that the proposed FCFormer achieves
superior performance and outperforms the state-of-the-art methods by
significant margins on occluded datasets.

###### Index Terms:

Person Re-identification, Transformer, Occlusion, Feature Completion

##  I Introduction

Person Re-Identification (Re-ID) involves identifying a person-of-interest
across multiple non-overlapping cameras [1]. This task has a wide range of
application backgrounds in many fields, such as video surveillance, activity
analysis, sport understanding, and tracking.

In the past few years, most of the existing methods mainly focus on the
holistic person Re-ID problem, which assumes that pedestrians’ body is fully
visible. However, in real-world scenarios, such as stations, airports, and
shopping malls, person images from surveillance cameras can be easily occluded
by some obstacles, e.g. plants, umbrellas, cars, or other pedestrians, which
poses a challenge for holistic Re-ID to identify persons with incomplete and
invisible body parts. Therefore, the task of occluded person re-identification
[2] is of significant practical importance.

(a) Part Occlusion

(b) Complementary Occlusion

(c) Feature Completion

Figure 1: Illustration of part/complementary occlusion and our proposed
feature completion paradigm. FCFormer represents an occluded person image by
using the transformer decoder to implicitly recover the missing features in
occluded regions.

Occluded person Re-ID faces more challenging problems because of the following
three reasons: (1) The limited number of occlusion samples in the training
dataset [3] makes the model sensitive to diverse occlusions. (2) Occlusions
will introduce a lot of noise information, which will interfere with feature
extraction. (3) Occlusion will cause the loss of appearance information,
making the extracted features less discriminative and incorrect semantic
alignment for matching, as shown in Figure 1a and 1b.

To address the above problems, many occluded ReID methods are proposed. For
the first problem, some researches [4, 5] employ the occlusion augmentation
strategy to improve the robustness of the model. However, these occlusion
augmentation strategies still cannot simulate the real environment well. Most
of them focus on more occlusion samples for training, while do not improving
the diversity of occlusion. For the second challenge, a large proportion of
them exploits additional cues (e.g. pose estimation, semantic parsing or human
mask) to indicate non-occluded body parts. For example, PGFA [3] directly
utilizes pose information to indicate non-occluded body parts on the spatial
feature map. PVPM[6] and HoReID [7] use graph-based approaches to model
topology information by learning node-to-node or edge-to-edge correspondence
to further mine the visible parts. These methods use external models directly
at the inference stage to extract additional semantic information. However,
those approaches are sensitive and error-prone when facing complex backgrounds
or severe occlusions. For the third problem, some methods [8, 9] attempt to
utilize GANs to predict the occluded part at the image level to restore the
holistic pedestrian. However, the generated regions are still not convincing,
resulting in limited performance. A recent work [10] performs feature
completion on occluded regions by using occluded body part landmarks and
utilizing the region based encoder-decoder architecture, which efficiently
captures spatial information to recover invisible parts from neighboring
features. But this method still needs extra key-points information and pre-
designed the regions in the feature space, which is not flexible enough.

Currently, there is no effective approach to address all three of these issues
simultaneously. To this end, we propose an Feature Completion Transformer
(FCFormer) for occluded person ReID to mitigate the impact of the three
challenges. Specifically, the proposed method could adaptively complement the
occluded features, as shown in Figure 1. As Figure 3 shows, the encoder is
responsible for extracting global and local information. On this basis, four
designs are proposed to alleviate the above three problems. Firstly, to obtain
rich occlusion samples, we build an Occlusion Instances Library (OIL) that
contains 17 classes of occlusion samples obtained from the COCO [11] and
Occluded-duke [3] training sets. Then we propose an Occlusion Instance
Augmentation (OIA) strategy that produces more diverse occluded training
images pairs by pasting image patches from OIL with a specific strategy. As
Figure 2 shows, compared with the existing occlusion augmentation strategies,
such as random erase [12] and OAMN augmentation [4], the introduce of rich
occlusion samples can better simulate real-world occlusion scenarios.
Secondly, a dual-stream encoder paradigm with shared weights is proposed, and
the generated holistic-occluded image pairs are taken as input to obtain
aligned feature pairs. In general, holistic pedestrian features contain more
ID representations than occluded pedestrian features. Thirdly, based on the
above prior information, a self-supervised Feature Completion Decoder (FCD)
module is proposed to complement occluded pedestrian features. FCD has a
learnable completion embedding representation, which enables the model to
automatically complement missing features under the supervision of holistic
pedestrian features without pre-defining human body regions and additional
labels. Finally, to bridge the large feature gap between the occluded scene
and the holistic scene, we design a Cross Hard Triplet Loss (CHT) for metric
learning. In addition, we propose a Feature Completion Consistency Loss (FC2)
to explicitly narrow the distribution discrepancy between completion features
and holistic features in high-dimensional space, thereby ensuring that FCD
could conduct feature completion and train in a self-supervised manner.

Figure 2: Examples of occluded pedestrians and introducing Augmentation. (a)
shows the real-world occlusion scenarios. (b) and (c) show augmentation stragy
introduced by OAMN[4] and our proposed OIA respectively.

The main contributions can be summarized as follows:

  * (1)

We propose an Occlusion Instances Library and an Occlusion Instance
Augmentation (OIA) strategy for Occluded ReID task, which brings more
realistic image-level occlusion enhancement.

  * (2)

We propose a Feature Completion Decoder to exploit learnable tokens for
occluded features completion. Compared to pervious works, our method is more
flexible without any pre-designed regions.

  * (3)

We design a Cross Hard Triplet Loss and a Feature Completion Consistency Loss
to impel model’s perception ability and feature completion ability
respectively.

  * (4)

To prove the effectiveness of our method, we perform experiments on occluded
and holistic Re-ID datasets. The results validate the proposed method performs
favorably against state-of-the-art methods.

Figure 3: Overall architecture of feature completion transformer (FCFormer).
FCFormer consists of three parts (Occluded Instance Augmentation, Dual stream
architecture, and Feature Completion Stream) and two losses (Cross Hard
Triplet Loss and Feature Completion Consistency Loss). The holistic-occlusion
sample pairs generated from OIA are fed into dual stream architecture with
shared encoder. The non-shared parts of dual architecture are used to train
the specifical tasks. Then FCD takes the learnable tokens and occluded
features as input to recovery holistic features. We propose CHT Loss to allow
the model to better perform metric learning among three different modal
features (occlusion, holistic, and completion features). At last, FC2 Loss is
proposed to guide FCD to generate a completion feature similar enough to the
holistic feature. In the test stage, the features from three branches are
utilized for retrieving.

##  II Related Work

###  II-A Occluded Person Re-Identification

Existing methods can be roughly divided into three categories, including part-
to-part matching based methods, extra-clues based methods and feature recovery
based methods.

Part-to-part matching methods address the occlusion issue by evaluating the
similarity between the aligned local features. Sun et al. [13] present a
network called Part-based Convolution Baseline (PCB) that uniformly partitions
the feature map and learns local features directly. Zhang et al. [14] aligns
features by finding the shortest path among local features. sun et al. [15]
propose a Visibility-aware Part Model (VPM), which learns to perceive the
visibility of regions by self-supervised learning. Jia et al. [16] propose MoS
measures the similarity between person images by using the Jaccard similarity,
which formulates the occluded person re-ID as a set matching problem without
alignment. The second category is extra-clues based methods, which leverages
external cues to locate the human body part such as segmentation, pose
estimation or body parsing. Song et al.[17] propose a mask-guided attention
model to extract discriminative and robust features invariant to background
clutters. Miao et al. [3] introduce Pose-Guided Feature Alignment (PGFA) that
utilizes gaussian pose heatmap to mine discriminative parts without occlusion.
Gao et al. [6] propose a Pose-guided Visible Part Matching (PVPM) model to
learn discriminative local features with pose-guided attentions. Wang et al.
[7] propose HOReID that utilize GCN to embed the high-order relation and
human-topology information between vairous body joints. The last category is
feature recovery based methods, which mainly focus on how to recover the
features of occluded regions. Hou et al. [10] propose Spatial and Temporal
Region Feature Completion (RFC) to recover semantics of occluded regions in
feature space for image and video occluded person Re-ID respectively. Yu et
al. present that occluded person image features can be reconstructed by its
neighborhoods to tackle the problem of occluded person Re-ID. However, the
above recovery-based methods all seriously need the support of additional
semantic information. Different from the above methods, our method simulates a
variety of occlusions, and introduces learnable completion tokens to complete
the feature completion of occluded areas in a “self-supervised” paradigm,
which can be well adapted to various occlusion problems.

###  II-B Occlusion Augmentation

The existing person re-identification model is difficult to deal with the
occlusion problem, and the interference caused by occlusion limits their
robustness. One of the factors is the limited occluded samples in the training
set [18], making the model unable to learn the relationship between occlusions
and pedestrians. An effective way to solve this problem is the occlusion
augmentation strategy. Currently, occlusion augmentation strategies can be
divided into three categories: (1) Random erase. Zhong et al. [12] propose a
method to randomly erase pixel values directly on the image and replace them
with random values. This method is simple and helps to reduce the risk of
overfitting, but has low generalization. (2) Random cropping and random
pasting. Chen et al.[4] randomly cropped a rectangular patch in the training
images, and then scaled the cropped area and pasted it in the set four areas
randomly. (3) Occluded sample augmentation. Jia et al. [5] proposed a method
that crops different occlusions from train set and randomly synthesize
occlusions for each training batch. Compared with strategy (2), the generated
images can better simulate the real occlusion scene, which makes the model
could implicitly learn more robust features. However, the occlusions in the
Occluded-Duke training set are utilized to synthesize, where the categories
and number of occlusions are limited, and these methods do not take full
advantage of occlusion semantics and paired features brought by augmentation.

##  III Proposed Method

In this section, we introduce the proposed Feature Completion Transformer
(FCFormer) in detail. Firstly, in order to alleviate the model sensitivity
problem caused by the small number of occluded samples, we introduce an online
data augmentation module named Occlusion Instance Augmentation (OIA) that
produces image pairs and occlusion masks (see section III-A for details).
Then, a shared dual-stream encoder is proposed to extract pairwise aligned
features (see section III-B for details), which could better learn the
occlusion relationship from the paired images. A Feature Completion Decoder
(FCD) is further proposed to complement human body features in occluded
regions (see section III-C for details). The flowchart of our method is shown
in Figure 3 and the overall FCFormer approach is outlined in Algorithm 1.

###  III-A Occlusion Instance Augmentation

Most of the existing occlusion augmentation strategies use random cropping to
obtain occlusions from randomly sampled images, and stitch the randomly
cropped pictures to form an occlusion scene. However, the cropped occlusion
has no explicit semantic information and cannot well simulate the occlusion in
the real environment. To address above issues, we propose the diversity
occlusion instance augmentation strategy. The strategy contains Occlusion
Instance Library (OIL) and Occlusion Instance Augmentation strategy (OIA).

Occlusion Instance Library. In order to better utilize occlusion augmentation
to solve the occlusion problem, we propose a general occlusion dataset. By
utilizing this dataset, more diverse and realistic occlusion situations can be
simulated. We first merge the Occluded-Duke [3] training set and the COCO [11]
training set, then utilize the Mask R-CNN [19] to obtain the instance bounding
box. In order to avoid the interference of noise information, we erase the
pixels irrelevant to the instance. As shown in Figure 4, we manually selected
17 common classes (such as pedestrians, vehicles, bicycles, umbrella, etc.) as
occlusion samples with a total of 1000 images.

Figure 4: Some occlusion samples from Occlusion Instance Library. TABLE I: Classes in Occlusion Instance Library. OIL |  Strong prior set $O_{s}$ |  Weak prior set $O_{w}$
---|---|---
car | truck | umbrella | Backpack
bicycle | motorcycle | suitcase | road sign
fire hydrant | table | Kite | tennis racket
pedestrian | chair | suitcase | billboard
- | bench | - | -

Occlusion Instance Augmentation. Empirically, Some common occlusions have
position priors in detected person image (for example, as the fig 2(a) shows,
vehicles are generally in the lower half of the image and are unlikely to
appear in other areas of the image). So we determine the augmentation position
according to the category of occlusion. As Table I shows, the OIL is divided
into two sets: strong position priors set $O_{s}$ and weak position priors set
$O_{w}$. For strong position priors set, we align the bottom edge and place
them randomly in the horizontal directions. However, for the weak position
prior set, its location in the detection box can be relatively random.

Specifically, given an image batch $X$, for each $x_{i}\in X$ and
$x_{i}\in{\mathbb{R}^{H\times{W\times{C}}}}$, where $H,W,C$ denote the high,
width, and channel dimension respectively. Our augmentation scheme has
following few steps: (1) Randomly select an occlusion sample $x_{ob}$ from
$OIL$. (2) Randomly Scale the occlusion $x_{ob}$ to 10$\%$$\sim$70$\%$ of
image size $H\times W$. It can be described as

${\epsilon}=\frac{{\delta}(H{\times}W)}{h_{o}{\times}w_{o}},$ (1)

where $h_{o}\times w_{o}$ denotes the size of chosen occlusion sample
$x_{ob}$, $\epsilon$ is the ratio of $h_{o}$ and $w_{o}$ ,
$\delta\sim\mathcal{U}(0.1,0.7)$ is the scaling factor, $\mathcal{U}$ denotes
the uniform distribution. (3) Determine the augmentation area. If $x_{ob}\in
O_{s}$, we choose augmentation location $(h,w)$, where
$h\in\\{H-{\epsilon}h_{o},H\\}$ and $w\in\\{0,W\\}$. If $x_{ob}\in O_{w}$, we
randomly put the occlusion sample onto training image $x_{i}$. Finally, our
OIA could be formulated as:

$x_{op}=\psi(x_{ob},\epsilon)+M\odot\rho(x_{i}),$ (2)

where $x_{op}$ denotes the augmented image patches, $\psi(\cdot,\cdot)$ is the
resize operation, $M$ denotes the occlusion binary mask sampled from resized
occlusion $x_{ob}$, and $\rho(\cdot)$ means clipping operation. At last,
$x_{op}$ will be paste on the holistic image $x_{i}$. The overall process is
shown in Figure 5. Following above precess, we obtain an occluded copies of
each training image. The augmented image is denoted as $x_{o}$.

Figure 5: Schematic diagram of Occlusion Instance Augmentation.

###  III-B Dual Stream Architecture

Following method [20] and [21], we build our feature extractor based on ViT
[22]. Given a pair of holistic-occluded images $x_{h}\in\mathbb{R}^{H\times
W\times C}$ and $x_{o}\in\mathbb{R}^{H\times W\times C}$ from OIA, the shared
encoder will split the input image into $N$ non-overlapping patches by using a
2D convolution layer $p(\cdot)$, then patch embeddings
$E_{p}\in\mathbb{R}^{N\times d}$ can be obtained, where $d$ denotes the
embedding dimension. In order to alleviate the impact of camera perspective,
we follow the method in [20] and set a learnable parameter $E_{cm}$ to learn
camera perspective information. At the same time, a learnable global embedding
$E_{g}$ is prepended to the patch embeddings.

$E_{p}=p(x),$ (3) $E_{input}=Concat(E_{g},E_{p})+P_{E}+\lambda_{cm}E_{cm},$
(4)

where $\lambda_{cm}$ is the ratio of camera embeddings, $P_{E}$ is the
learnable position embedding. Then ViT will take $E_{input}$ as input. The
output feature for each stream is $f\in{\mathbb{R}^{(N+1)\times C}}$, where
$N+1$ denotes the image tokens and one global token, $c$ is the channel
dimension. The global tokens $f_{og}\in{\mathbb{R}^{1\times C}}$ and
$f_{hg}\in{\mathbb{R}^{1\times C}}$ are treat as global feature. Several
methods [13, 23] have proved that fine-grained part features are effective for
person re-identification tasks. Thus, the rest of image tokens
$f_{ot}\in{\mathbb{R}^{N\times C}}$ and $f_{ht}\in{\mathbb{R}^{N\times C}}$
will be fed into two parallel streams to generate local features. Following
the methods [20] and [21], we split image tokens into $M_{n}$ parts and
concatenate the corresponding global features for each part. After obtained
the $M_{n}$ parts features, we feed them into a non-shared transformer layer
and finally get local feature representations
$f_{hp}\in{\mathbb{R}^{M_{n}\times C}}$ and $f_{op}\in{\mathbb{R}^{M_{n}\times
C}}$. Normalized features $\hat{f_{hp}}$ and $\hat{f_{op}}$ can be obtained
through a BNNeck[24]. In short, dual stream architecture consists of a shared
ViT backbone and two non-shared transformer layers. The shared backbone can
extract general features, while the unshared transformer layer will learn
specific patterns for occluded and holistic Re-ID tasks.

Loss: To guarantee that the local features are related to the identity and
ensure their discriminability, we train it with following identity loss:

$\mathcal{L}_{id_{g}}=\frac{1}{B}\sum_{i}\log(p(\hat{f_{og}^{i}})\cdot
p(\hat{f_{hg}^{i}})),$ (5)
$\mathcal{L}_{id_{1}}=\frac{1}{B}\sum_{i}\log(p(\hat{f_{hp}^{i}})),$ (6)
$\mathcal{L}_{id_{2}}=\frac{1}{B}\sum_{i}\log(p(\hat{f_{op}^{i}})),$ (7)
$\mathcal{L}_{id}=\mathcal{L}_{id_{g}}+\mathcal{L}_{id_{1}}+\mathcal{L}_{id_{2}},$
(8)

where $B$ denotes the batch size, $p(\cdot)$ denotes the probability function.

###  III-C Feature Completion Stream

Figure 6: Illustration of the proposed feature completion decoder.

Even though the occlusion augmentation strategy and the training method of
shared dual-stream network can force the model to better focus on unoccluded
parts, the lack of human body part information caused by occlusion has not
received much attention. Therefore, we propose a feature completion stream to
recover the features of the occluded parts.

The feautre completion stream consist of a Feature Completion Decoder (FCD)
and a transformer layer, as shown in Figure 3. The FCD is illustrated in
Figure 6, which consists of a hybrid feature embedding, several transformer
layers, and training loss.

Hybrid Feature Embedding: we consider the completion pedestrian features to be
composed of occlusion features and recovery features. Inspired by MAE [25],
our approach reconstructs the holistic features from the latent occluded
features. For each occluded input features, learnable prototype completion
tokens $T_{c}\in{\mathbb{R}^{1\times K\times C}}$ are prepended to aggregate
missing feature information from $f_{ot}$. It is worth noting that the
position of MAE’s mask token is fixed within the batch, which means that each
instance can use the same set of mask tokens for feature completion. However,
unlike the MAE, the occlusion positions of samples generated by OIA are
randomly sampled, so we need to conduct completion on each instance. Thus, we
fuse the learnable prototype tokens $T_{c}$ with occlusion features to obtain
instance-level tokens. Formally,

$T_{b}=({W_{1}}^{T}f_{ot})T_{c},$ (9)

where $W_{1}\in{\mathbb{R}^{N\times K}}$ is linear projection,
$T_{b}\in{\mathbb{R}^{B\times K\times C}}$ is the instance completion tokens.
Here, $K=\alpha N$, and $\alpha$ is a hyper-parameter to control the number of
tokens. Then we map $f_{ot}$ into $L$ dimension, and concatenate with $T_{b}$.
Formally,

$f_{r}=Concat(f_{og},T_{b},({W_{2}}^{T}f_{ot})),$ (10)

where $Concat(\cdot)$ is operation of vector combination,
$W_{2}\in{\mathbb{R}^{N\times L}}$ is linear projection, and $L=(1-\alpha)N$.
$f_{og}$ is prepended to provide global information. Finally, reconstruct
feature $f_{r}\in{\mathbb{R}^{B\times(N+1)\times C}}$ is obtained. As
mentioned above, due to the randomness of the position of the occluded region,
the prepended features need to learn different regions for each instance.
However, the operation of prepending limits the position learning, so we use a
learnable position embedding to assist the model to encode the token position
of each instance. Formally,

$f_{r}=(W_{3}f_{r})P_{e}+f_{r},$ (11)

where $W_{3}$ is the parameter of convolution, $P_{e}$ is the positional
embedding.

Transformer Layers: the consolidated features are sent to the transformer
layers to complement unoccluded body parts. Following the Transformer [26],
the query, key, and value can be formulated as:

$Q=W_{q}f_{r},K=W_{k}f_{r},V=W_{v}f_{r},$ (12)

where $W_{q},W_{k},W_{v}$ are weights of linear projections. Through the
attention mechanism, the final completion feature can be expressed as:

$f_{m}=softmax(\frac{QK}{\sqrt{k}})V$ (13) $f_{cp}=Wf_{m}+b.$ (14)

Training Loss: Here we use the holistic feature $f_{ht}$ as the target, which
is taken from the complete branch of the dual-stream branch, thus forming a
self-supervised training method. The MSE loss function is utilized to drive
FCD for training. The training loss can be defined as:

$\mathcal{L}_{fcd}=||f_{cp}-f_{ht}||_{2}^{2}.$ (15)

###  III-D Overall Training Loss

We propose a Cross Hard Triplet Loss (CHT) to help the model to better perform
metric learning among three different modes of features. (occluded features,
holistic features, and complete features). Furthermore, In order to
successfully complete the occluded pedestrian features, we propose a Feature
Completion Consistency Loss (FC2). It is worth noting that this method does
not require any additional labels and can be trained in a self-supervised
manner.

Cross Hard Triplet Loss: Here, we set holistic feature $f_{hp}$ as anchor, and
we want ensure that $f_{hp}$ is closer to all positive samples than other
negative samples, which is the same as original triplet loss [27]. However,
most previous models only measure features in a single modality. We find the
hardest pair of positive samples and the hardest pair of negative samples
among holistic features, occluded features, and complementary features for
optimization. Formally,

$p_{1}=\mathop{\arg\max}\limits_{j}|f_{hp}^{i}-f_{op}^{j}|_{+},p_{2}=\mathop{\arg\max}\limits_{j}|f_{hp}^{i}-f_{c}^{j}|_{+},$
(16)
$n_{1}=\mathop{\arg\min}\limits_{j}|f_{hp}^{i}-f_{op}^{j}|_{-},n_{2}=\mathop{\arg\min}\limits_{j}|f_{hp}^{i}-f_{c}^{j}|_{-},$
(17)
$\begin{split}\mathcal{L}_{cht}=\sum_{i}^{B}\max([||f_{hp}^{i}-f_{op}^{p_{1}}||_{2}^{2}-||f_{hp}^{i}-f_{op}^{n_{1}}||_{2}^{2}+\alpha],0)\\\
+\sum_{i}^{B}\max([||f_{hp}^{i}-f_{c}^{p_{2}}||_{2}^{2}-||f_{hp}^{i}-f_{c}^{n_{2}}||_{2}^{2}+\alpha],0),\end{split}$
(18)

where $p_{1}$ is the index of the hardest positive sample in $f_{op}$, $p_{2}$
is the index of the hardest positive sample in $f_{c}$, $n_{1}$ is the index
of the hardest negative sample in $f_{op}$, and $n_{2}$ is the index of the
hardest negative sample in $f_{c}$. $\alpha$ is a margin hyperparameter.

Algorithm 1 Feature Completion Transformer

1: Require An Occluded training set $\mathcal{X}$, an occlusion Instance
library $\mathcal{O}$, Strong prior set $\mathcal{O}_{s}$, Weak prior set
$\mathcal{O}_{w}$, $\epsilon$ for Eq.(1);

2: $\%Training\enspace stage$

3: Initialize the ViT backbone;

4: Initialize the learnable completion token $T_{c}$;

5: for $epoch=1$ to $num\\_epoch$ do

6: for mini_batch $\mathcal{B}\in\mathcal{X}$ do

7: for each sample $x\in\mathcal{B}$ do

8: Randomly select occlusion sample $x_{ob}$ from $\mathcal{O}$;

9: Scale $x_{ob}$ according to $\epsilon$;

10: if $x_{ob}\in\mathcal{O}_{s}$ then

11: Randomly paste $x_{ob}$ onto $x$ at position $(h,w)$,
$h\in\\{H-{\epsilon}h_{o},H\\}$ and $w\in\\{0,W\\}$;

12: else {$x_{ob}\in\mathcal{O}_{w}$}

13: Randomly paste $x_{ob}$ onto $x$ at position $(h,w)$, $h\in\\{0,H\\}$ and
$w\in\\{0,W\\}$;

14: end if

15: end for

16: Extract holistic and occluded features $f_{hp}$ and $f_{op}$ through dual
stream architecture;

17: Generate complete feature $f_{c}$ using FCD by taking $T_{c}$ and $f_{op}$
as input;

18: Compute $\mathcal{L}_{id}$, $\mathcal{L}_{cht}$, $\mathcal{L}_{fc^{2}}$ by
Eqs.8, 18, 19;

19: Compute final loss in Eq.20;

20: Backward to update the Re-ID model;

21: end for

22: end for

Feature Completion Consistency Loss: it is almost impossible to complete the
ideal holistic feature distribution. So in order to make the distribution of
features obtained by completion consistent with the distribution of holistic
features, our goal then becomes to minimize the difference between the
completed feature distribution and the holistic feature distribution:

$\mathcal{L}_{fc^{2}}=\frac{1}{B}\sum_{i}[p(\hat{f_{c}^{i}})\log
p(\hat{f_{c}^{i}})-p(\hat{f_{c}^{i}})\log p(\hat{f_{hp}^{i}})]$ (19)

where $\hat{f_{c}}$ denotes the feature embedding after BNNeck, $p(\cdot)$
denotes the predicted probability of given features.

###  III-E Training and Inference

In the training stage, dual stream architecture and feature completion decoder
are trained together with the overall objective loss, which is formulated as
Eq.20.

$\mathcal{L}=\mathcal{L}_{id}+\mathcal{L}_{fcd}+\mathcal{L}_{cht}+\mathcal{L}_{fc^{2}},$
(20)

During inference stage, the model is relatively simple, a VIT encoder and a
feature completion decoder can perform the inference.

##  IV Experiments

TABLE II: Performance comparison with state-of-the-art methods on Occluded-
Duke and P-DukeMTMC. $\dagger$ means the encoder is with a small step sliding-
window setting. Hybird denotes ResNet50 + transformer encoder. $\bigstar$
means the results are reproduced by [28].

Methods | Backbone | Extra-clue | Occluded-Duke | P-DukeMTMC | Occluded-REID
---|---|---|---|---|---
Rank-1 | Rank-5 | Rank-10 | mAP | Rank-1 | mAP | Rank-1 | mAP
Part-Aligned (ICCV $17$)[29] | GoogLeNet | ✗ | 28.8 | 44.6 | 51.0 | 44.6 | - | - | - | -
PCB (ECCV $18$)[13] | ResNet50 | ✗ | 42.6 | 57.1 | 62.9 | 33.7 | 79.4 | 63.9 | 41.3 | 38.9
Part Bilinear (ECCV $18$)[30] | GoogLeNet | ✓ | 36.9 | - | - | - | - | - | - | -
FD-GAN (NIPS $18$)[31] | ResNet50 | ✓ | 40.8 | - | - | - | - | - | - | -
DSR (CVPR $18$)[32] | ResNet50 | ✗ | 40.8 | 58.2 | 65.2 | 30.4 | - | - | 72.8 | 62.8
SFR (Arxiv $18$)[33] | FCN | ✗ | 42.3 | 60.3 | 67.3 | 32.0 | - | - | - | -
Ad-Occluded (CVPR $18$)[34] | ResNet50 | ✗ | 44.5 | - | - | 32.2 | - | - | - | -
PGFA (ICCV $19$)[3] | ResNet50 | ✓ | 51.4 | 68.6 | 74.9 | 37.3 | 85.7 | 72.4 | 80.7 | 70.3
PVPM (CVPR $20$)[6] | ResNet50 | ✓ | 47.0 | - | - | 37.7 | 85.1 | 69.9 | 70.4 | 61.2
ISP (ECCV $20$)[35] | ResNet50 | ✗ | 62.8 | 78.1 | 82.9 | 52.3 | 89.0 | 74.7 | - | -
HOReID (CVPR $20$)[7] | ResNet50 | ✓ | 55.1 | - | - | 43.8 | - | - | 80.3 | 70.2
SORN (TCSVT $21$)[36] | ResNet50 | ✓ | 57.6 | 73.9 | 79.0 | 46.3 | - | - | - | -
MoS (AAAI $21$)[16] | ResNet50 | ✗ | 61.0 | 77.4 | 79.1 | 49.2 | - | - | - | -
OAMN (ICCV $21$)[4] | ResNet50 | ✗ | 62.6 | 77.5 | - | 46.1 | - | - | - | -
RFCnet (TPAMI $21$)[10] | ResNet50 | ✓ | 63.9 | 77.6 | 82.1 | 54.5 | - | - | - | -
Pirt (ACM MM $21$)[37] | ResNet50-ibn | ✓ | 60.0 | - | - | 50.9 | - | - | - | -
PGFL-KD (ACM MM $21$)[38] | ResNet50 | ✓ | 63.0 | - | - | 54.1 | - | - | 80.7 | 70.3
QPM (TMM $22$)[39] | ResNet50 | ✗ | 64.4 | 79.3 | 84.2 | 49.7 | 89.4 | 74.4 | - | -
TransReID (ICCV $21$)[20] | ViT-B | ✗ | 64.2 | - | - | 55.7 | - | - |  70.2★ |  67.3★
PAT (CVPR $21$)[23] | Hybrid | ✗ | 64.5 | - | - | 53.6 | - | - | 81.6 | 72.1
DRL (TMM $22$)[5] | Hybrid | ✗ | 65.8 | 80.4 | 85.2 | 53.9 | - | - | - | -
PFD (AAAI $22$)[21] | ViT-B | ✓ | 67.7 | 80.1 | 85.0 | 60.1 | - | - | 79.8 | 81.5
FED (CVPR $22$)[28] | ViT-B | ✗ | 68.1 | - | - | 56.4 | - | - | 86.3 | 79.3
FCFormer (Ours) | ViT-B | ✗ | 71.3 | 84.1 | 87.1 | 60.9 | 91.5 | 80.7 | 84.9 | 86.2
TransReID† (ICCV $21$)[20] | ViT-B | ✗ | 66.4 | - | - | 59.2 | - | - | - | -
PFD† (AAAI $22$)[21] | ViT-B | ✓ | 69.5 | - | - | 61.8 | - | - | 81.5 | 83.0
FCFormer† (Ours) | ViT-B | ✗ | 73.0 | 84.9 | 88.6 | 63.1 | 92.4 | 82.5 | 83.6 | 85.7
PGFA[3] \+ Re-ranking | ResNet50 | ✓ | 52.4 | 68.6 | 74.9 | 46.8 | - | - | - | -
HOReID[7] \+ Re-ranking | ResNet50 | ✓ | 58.3 | - | - | 49.2 | - | - | - | -
Pirt[37] \+ Re-ranking | ResNet50-ibn | ✓ | 62.1 | - | - | 59.3 | - | - | - | -
PFD[21] \+ Re-ranking | ViT-B | ✓ | 71.7 | 79.4 | 82.3 | 71.7 | - | - | \- - | -
FCFormer (Ours) \+ Re-ranking | ViT-B | ✗ | 76.8 | 85.7 | 88.4 | 75.8 | 89.1 | 85.3 | 84.6 | 86.9
FCFormer† (Ours) \+ Re-ranking | ViT-B | ✗ | 79.4 | 86.7 | 89.3 | 77.2 | 90.6 | 87.1 | - | -

###  IV-A Datasets and Evaluation Metrics

To illustrate the effectiveness of our method, we evaluate our method on five
Re-ID datasets for two tasks including occluded person Re-ID, and holistic
person Re-ID.

Occluded-Duke [3] consists of 15,618 training images of 702 persons, 2,210
occluded query images of 519 identities and 17,661 gallery images of 1110
persons. It is a subset of DukeMTMC-reID [40], which is currently the most
challenging dataset due to its complex scene.

P-DukeMTMC [2] is the subset of the DukeMTMC-reID, which consists of 12,927
training images with 665 identities, 2163 occluded images for query and 9053
images without occlusion for gallery.

Occluded-REID [2] is captured by the mobile phone, which consist of 2,000
images of 200 occluded persons. Each identity has five full-body person images
and five occluded person images with different types of severe occlusions.

Market-1501 [41] contains 1,501 identities observed from 6 camera viewpoints,
12,936 training images of 751 identities, 19,732 gallery images and 2,228
queries of 750 persons.

DukeMTMC-reID [40] contains 36,411 images of 1,404 identities captured from 8
camera viewpoints. It contains 16,522 training images, 17,661 gallery images
and 2,228 queries.

Evaluation Metrics. We adopt Cumulative Matching Characteristic (CMC) curves
and mean average precision (mAP) to evaluate different Re-ID models.

###  IV-B Implementation Details

Our encoder is followed TransReID [20], which consists of 12 transformer
layers. The initial weights of encoder are pre-trained on ImageNet-21K and
then finetuned on ImageNet-1K. During the training and testing time, input
images are both resized to 256 $\times$ 128\. The training images are
augmented with random horizontal flipping, and padding. The number of the
split tokens $M_{N}$ is set to 4 for Occluded-Duke dataset (Details are
described in Sec. IV-E ). The number of decoder layer is set to 2. And the
number of token $\alpha$ in FCD is 0.7. The hidden dimension $D$ is set to
768. The transformer decoder is same with [26]. The batch size is set to 64
with 4 images per ID. The learing rate is initialized at 0.008 with cosine
learning rate decay. We conduct all experiments on one RTX 3090 GPU.

###  IV-C Comparison with the State-of-the-Art Models

We compare our method with the state-of-the-art methods on seven benchmarks
including occluded person ReID and holistic person ReID.

Results on Occluded datasets. Table II shows the results on three occluded
datasets, i.e., Occluded-Duke and P-DukeMTMC. As table shows, two classes of
methods will be compared: CNN based ReID methods [29, 13, 30, 31, 34, 42, 43,
33, 3, 38, 6, 44, 7, 35, 16, 36, 4, 10, 37, 39] and Transformer based ReID
methods [20, 23, 5, 21, 28]. The results show that the proposed FCFormer
consistently achieves the competitive performance on Occluded datasets. It is
worth noting that some previous methods adopt additional models, such as PFD
[21] and HOReID [7], both utilize skeleton topology information to cluster
unoccluded features, thus achieving competitive results. In comparison,
FCFormer achieves the best result with Rank-1 accuracy of 71.3% and mAP of
60.9% on the challenging Occluded-Duke dataset without any external clues,
which outperforms the previous state-of-the-art methods by a large margin (at
least +3.2%Rank-1/+0.8%mAP). On the P-DukeMC dataset, our FCFormer achieves
91.5% Rank-1 accuracy and 80.7% mAP, surpassing the state-of-the-art model QPM
[39] by 2.1% and 6.3% in terms of Rank-1 accuracy and mAP respectively. In
addition, FCFormer is flexible and scalable. We change the step of sliding-
window, and also introduce the Re-ranking technology. As the table shows,
FCFormer† achieves 73.0%/63.1% on Rank-1/mAP on Occluded-Duke dataset by
simply reducing the step size, surpassing others by at least 4.9%/3% in terms
of Rank-1 and mAP respectively. Further, with the help of re-ranking, our
model FCFormer† \+ Re-ranking achieves the highest results by far, reaching
79.4% Rank-1 and 77.2% mAP.

TABLE III: Performance comparison with state-of-the-art models on Market-1501
and DukeMTMC-reID datasets.

Methods | Market-1501 | DukeMTMC
---|---|---
Rank-1 | mAP | Rank-1 | mAP
PCB (ECCV $18$)[13] | 92.3 | 77.4 | 81.8 | 66.1
DSR (CVPR $18$)[32] | 83.6 | 64.3 | - | -
BOT (CVPRW $19$)[45] | 94.1 | 85.7 | 86.4 | 76.4
VPM (CVPR $19$)[15] | 93.0 | 80.8 | 83.6 | 72.6
MVPM (ICCV $19$)[46] | 91.4 | 80.5 | 83.4 | 70.0
SFT (ICCV $19$)[47] | 93.4 | 82.7 | 86.9 | 73.2
CAMA (CVPR $19$)[48] | 94.7 | 84.5 | 85.8 | 72.9
IANet (CVPR $19$)[49] | 94.4 | 83.1 | 87.1 | 73.4
Circle (CVPR $20$)[50] | 94.2 | 84.9 | - | -
SPReID (CVPR $18$)[51] | 92.5 | 81.3 | 84.4 | 70.1
P2Net (ICCV $19$)[52] | 95.2 | 85.6 | 86.5 | 73.1
PGFA (CVPR $19$)[3] | 91.2 | 76.8 | 82.6 | 65.5
AANet (CVPR $19$)[53] | 93.9 | 82.5 | 86.4 | 72.6
HOReID (CVPR $20$)[7] | 94.2 | 84.9 | 86.9 | 75.6
Pirt (TMM $22$)[37] | 94.1 | 86.3 | 88.9 | 77.6
TransReID (ICCV $21$)[20] | 95.0 | 88.2 | 89.6 | 80.6
PAT (CVPR $21$)[23] | 95.4 | 88.0 | 88.8 | 78.2
DRL-Net (TMM $22$)[5] | 94.7 | 86.9 | 88.1 | 76.6
PFD (AAAI $22$)[21] | 95.5 | 89.6 | 90.6 | 82.2
FED (CVPR $22$)[28] | 95.0 | 86.3 | 89.4 | 78.0
FCFormer ($Ours$) | 95.0 | 86.8 | 89.7 | 78.8

Results on Holistic ReID datasets. We conduct experiments on two holistic ReID
datasets including Market-1501, and DukeMTMC-reID. Table III shows the results
on Market-1501 and DukeMTMC-reID datasets. Specifically, our method FCFormer
achieves comparable performance (95.0%/86.8% Rank-1 accuracy and 89.7%/78.8%
mAP, respectively) on Market-1501 and DukeMTMC-reID datasets. As shown in the
results, PFD achieves better results than ours on the holistic datasets. This
is mainly because PFD incorporates skeleton information during encoding and
decoding process. Moreover, PFD uses more parts for feature retrieval, and the
feature dimension is larger, so the features are more fine-grained. In
addition, FCFormer applies a shared backbone, the occlusion branch in the Dual
Stream Architecture will affect the backpropagation of the complete feature.
Our method FCFormer, although not designed for holistic person re-
identification, still outperforms CNN-based methods without the help of
external models and has a small gap with current state-of-the-art models,
illustrating the robustness of FCFormer.

###  IV-D Performance under Transfer setting

In order to further vertify the effectiveness of the method, we follow the
methods [7, 6, 39] that adopts Market-1501 or MSMT17 as training set, then
directly evaluate on occluded dataset (Occluded-REID and P-DukeMTMC). A number
of methods [54, 55, 56, 57, 13, 35, 3, 6, 39] are involved in the comparison.
Table II and IV show that FCFormer achieves the competitive Rank-1 accuracy
and mAP on Occluded-REID dataset and P-DukeMTMC dataset respectively. On the
P-DukeMTMC dataset, our FCFormer outperforms QPM by 1.0% and 8.3% in terms of
Rank-1 accuracy and mAP respectively. On the Occluded-REID dataset, FCFormer
produces the comparable results with FED [28], achieving 84.9% Rank-1. We fail
to achieve the highest Rank-1 performance on Occluded-REID for the following
reasons: First, the Transformer has poor cross-domain generalization ability
on small datasets. Secondly, we use the same backbone as TransReID, which
contains some dataset-specific tokens, further limiting the generalization
ability. However, it is worth noting that our method achieves 86.2% mAP on
Occluded-REID and 39.4% mAP on P-DukeMTMC, surpassing previous model by at
least 4.7% and 8.3% on Occluded-REID and P-DukeMTMC respectively. The reason
for this is that the completion feature participates in the feature distance
calculation, which helps the average accuracy.

TABLE IV: Performance comparisons on P-DukeMTMC under transfer setting.

Methods | Rank-1 | Rank-5 | Rank-10 | mAP
---|---|---|---|---
IDE (CVPR $17$)[56] | 36.0 | 49.3 | 55.2 | 19.7
HACNN (CVPR $18$)[54] | 30.4 | 42.1 | 49.0 | 17.0
PCB (ECCV $18$)[13] | 43.6 | 57.1 | 63.3 | 24.7
OSNet (ICCV $19$)[55] | 33.7 | 46.5 | 54.0 | 20.1
Part Bilinear (ECCV $18$)[30] | 39.2 | 50.6 | 56.4 | 25.4
ISP* (ECCV $20$)[35] | 46.3 | 56.9 | 60.8 | 26.4
PGFA+ (TNNLS $21$)[3] | 48.2 | 59.6 | 65.8 | 26.8
PVPM (CVPR $20$)[6] | 51.5 | 64.4 | 69.6 | 29.2
QPM (TMM $22$)[39] | 57.3 | 69.9 | 75.5 | 31.1
FCFormer (Market $\to$ P-Duke) | 58.3 | 69.7 | 74.4 | 39.4
FCFormer (MSMT17 $\to$ P-Duke) | 58.2 | 69.8 | 74.6 | 41.4

Index | Baseline | Dual-structure | OIA | FCD | CHT loss |  FC2 loss | R-1 | R-5 | R-10 | mAP
---|---|---|---|---|---|---|---|---|---|---
1 | ✓ |  |  |  |  |  | 59.3 | 76.5 | 82.2 | 50.0
2 | ✓ |  | ✓ |  |  |  |  63.4(+4.1) | 77.1 | 82.6 |  53.2(+3.2)
3 | ✓ | ✓ | ✓ |  |  |  |  66.5(+7.2) | 78.6 | 83.6 |  55.7(+5.7)
4 | ✓ |  | ✓ | ✓ |  |  |  65.1(+5.8) | 77.9 | 83.1 |  54.8(+4.8)
5 | ✓ | ✓ | ✓ | ✓ |  |  |  67.3(+8.0) | 79.5 | 84.1 |  56.9(+6.9)
6 | ✓ | ✓ | ✓ |  | ✓ |  |  68.4(+9.1) | 81.7 | 85.7 |  58.5(+8.5)
7 | ✓ | ✓ | ✓ | ✓ |  | ✓ |  67.8(+8.5) | 80.9 | 85.0 |  57.8(+7.8)
8 | ✓ | ✓ | ✓ | ✓ | ✓ |  |  70.3(+11.0) | 82.6 | 86.8 |  60.2(+10.2)
9 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |  71.3(+12.0) | 84.1 | 87.1 |  60.9(+10.9)
TABLE V: Ablation study over Occluded-Duke.

###  IV-E Ablation Study

In this part, we perform ablation studies on Occluded-Duke dataset to futher
analyze the effectiveness of each component.

Effectiveness of proposed Modules. we present the ablation studies of Dual-
structure, Occlusion Instance Augmentation (OIA) and Feature Completion
Decoder(FCD). Table V shows the experimental results. Index-1 denotes the
baseline transformer structure [20]. (1) From index-1 and index-2, we can see
that the OIA module can improve 4.1% Rank-1 accuracy and 3.2% mAP compared
with baseline model. OIA successfully incorporates a wider occlusion ratio and
a variety of occlusion samples into the training set. (2) From index-2 and
index-3, when dual-structure is performed, the dual-structure improves
performance by 3.1% Rank-1 accuracy and 2.5% mAP. This demonstrates that dual-
stream structure can effectively enable the shared part to learn a general
occlusion and non-occlusion pattern, while the non-shared part completes the
training of specific tasks. (3) The performance gain brought by FCD can be
demonstrated by index-6 and index-8. After adding FCD to index-6, the
performance increased by 1.9% and 1.7% on rank-1 and mAP respectively. By
comparing index-3 and index-5, FCD could increase the performance by 0.8%
Rank-1 and 1.2% mAP. This shows that the features obtained by the holistic
branch are more robust with the help of CHT loss, making the supervision data
of FCD more precise and the completion features produced by FCD more
discriminative.

Effectiveness of proposed Losses. we also present the ablation studies of
Cross Hard Triplet Loss (CHT loss) and Feature Completion Consistency Loss
(FC2 loss). (1) Index-3 and index-6 show that the proposed CHT loss can
contribute to network, improving the performance by 1.9% Rank-1 and 2.8% mAP.
The CHT loss allows it to find the most difficult features that have different
modalities than the anchors in the fused features. This is advantageous for
attracting features with the same identity but distinct modalities in the
feature space. (2) Both (Index-5, Index-7) and (Index-8, index-9) demonstrate
that FC2 loss can guide FCD to bring the generated pedestrian feature
distribution closer to the holistic feature distribution, enabling FCD to
complete pedestrian features while inputting occlusion features.

Figure 7: Parameter analysis for the number of local tokens.
Figure 8: Parameter analysis for the scaling ratio $\delta$ in OIA.

Analysis of the number of part tokens $M_{n}$. The number of part tokens
determines the granularity of output features. Figure 7 shows the performance
of the REID influenced by the number of part tokens $M_{n}$. As we can see,
FCFormer achieves the best Rank-1/mAP when $M_{n}=4$. The initial performance
improves as $M_{n}$ increases, which shows adequate part tokens can
effectively help the model learn more robust occlusion features. However,
increasing its value further contributes to performance decline. We conclude
from Figure 7 that redundant tokens increase the dimensionality of features,
and the model pays attention to more details, which weakens the anti-noise
ability.

Analysis of scaling ratio $\delta$ in OIA. Figure 8 depicts the experimental
findings of FCFormer at various fixed occlusion ratios. We note that the
scaling ratio of 0.4 allows the FCFormer to perform at its best. Figure 9
shows that OIA may generate some undesired augmented images. For example,
large portions of the human body’s information are preserved in the augmented
images produced by tiny occlusion ratios (Figure 9a). As a result, the network
is unable to learn the relationship between person and occluder effectively.
Another scenario about the small scaling ratio is that the occlusion sample
does not hit the target person, resulting in an invalid augmentation (Figure
9b). On the contrary, a high occlusion ratio ($\delta>0.8$) (Figure 9c)
results in a significant loss of pedestrian information, preventing the model
from learning ID-related properties. Therefore, to imitate the real occlusion
situation and prevent people from being entirely obscured, we take a uniform
distribution centered around $\delta=0.4$ ($\delta\sim\mathcal{U}(0.1,0.7)$)
instead of a fixed scaling ratio to scale the size of occlusion samples in
III-A.

(a)

(b)

(c)

Figure 9: The augmented image of (a) small scaling ratio. (b) failure case. (c) large scaling ratio. TABLE VI: Different augmentation methods within a batch. Augmentation | R-1 | R-5 | R-10 | mAP
---|---|---|---|---
Fixed | 67.8 | 83.2 | 86.1 | 57.4
Random | 71.3 | 84.1 | 87.1 | 60.9

Figure 10: Retrieval results of RFCnet and our proposed FCFormer on Occluded-
DukeMTMC dataset.

Analysis of augmentation locations in OIA. Table VI shows that OIA uses
different position enhancement methods. The first involves pasting images for
each instance in a batch at a “Fixed” location, and the second involves
pasting images for each instance in a batch at “Random” locations. As
mentioned in III-C, our FCD adapts MAE’s notion [25] of restoring entire
features using implicit unoccluded features. According to the results, the
”random” enhancement approach can perform better than the ”fixed” method. This
is because the ”fixed” method only selects a random position at the beginning,
and then every instance in the batch follows this position, which reduces the
diversity of occlusion relationships.

TABLE VII: Comparison of different backbones. $*$ denotes that camera perspective information is introduced. Backbone | Param. | R-1 | R-5 | R-10 | mAP
---|---|---|---|---|---
ResNet50§ | - | 42.6 | 57.1 | 62.9 | 33.7
ResNet50 | 25.6M | 47.6 | 64.3 | 71.1 | 35.6
ResNet101 | 44.7M | 47.7 | 62.5 | 68.7 | 35.6
ViT-B§ | - | 59.3 | 76.5 | 82.2 | 50.0
ViT-S* | 22M | 63.7 | 77.9 | 82.9 | 53.3
ViT-B* | 86M | 71.3 | 84.1 | 87.1 | 60.9
ViT-L* | 307M | 68.8 | 79.9 | 83.9 | 57.9
DeiT-S* | 22M | - | - | - | -
DeiT-B* | 86M | 69.5 | 82.9 | 86.1 | 59.7
DeiT-L* | 307M | - | - | - | -

Analysis of different backbones. In this section, we conduct experiments to
show how the encoder Influences the performance. CNN-based and Transformer-
based backbones are compared in Table VII. $\S$ means only the backbone is
used for training. From the table, we can observer that the ResNet series and
the Transformer series have a huge performance gap in occlusion scenarios.
Here ResNet-50$\S$ is similar to the PCB [13] that directly learns local
features. The second line shows that the FCFormer method takes ResNet-50 as
backbone, surpassing ResNet$\S$ by 5% Rank-1 accuracy and 1.9% mAP. It
demonstrates that our proposed paradigm is effective. Unlike ResNet,
Transformer has a high performance starting point when dealing with occlusion
problems. ViT-Small has better performance and less parameters compared to
ResNet-50. DeiT-B achieves the comparable results compared with ViT-B, the
reason is that ViT series are pre-trained on ImageNet-21K and then finetuned
on ImageNet-1K, but DeiT series only pre-trained on ImageNet-1K. The
performance of ViT-L is lower than ViT-B, which shows that training large
models on small dataset can easily lead to overfitting.

Figure 11: Probability scores of part tokens

###  IV-F Qualitative Analysis

In this section, we present qualitative experimental results and demonstrate
the superiority of our proposed FCFormer.

Probability scores of part tokens. We present the probability scores after the
non-shared transformer layers for some occluded pedestrian images. As shown in
Figure 11, occlusion heavily affects the probability scores of tokens in the
baseline model. However, the recovered feature tokens obtained from FCD have
significantly improved scores in occluded regions, which indicates that the
feature completion decoder uses unoccluded information to compensate occluded
region features.

Visualization of the Feature Completion Transformer. In Figure 10, we present
the retrieval results of RFCnet [10] and our FCFormer. The retrieval results
show that RFCnet is prone to mix the information of the target person and
obstacles, resulting in retrieving a wrong person with similar obstacle or
treating the front person as target. This is because pre-designing the regions
still does not tackle the extreme occlusion problem well. Unlike RFCnet’s
completion approach, we implicitly learn the completion features from the
occlusion features without encoding and decoding on the feature space map.

Visualization of attention map of feature completion decoder. We visualize the
attention heatmaps for feature completion decoder in Figure 12 by using Grad-
CAM [58]. The model concentrates on body parts with significant information
when the pedestrian is not occluded. On the contrary, if the target person is
occluded, FCD will pay attention to the position of the human body on the
occluder’s edge, as this helps to complement missing features with nearby
features.

##  V Conclusion

In this paper, we propose a Feature Completion Transformer (FCFormer)
framework to alleviate occluded person re-identification problem. The core
idea is to construct a variety of occlusion sample pairs, and train a decoder
that can complement incomplete pedestrian features through known occlusion
relationships without relying on extra semantic model. Specifically, OIA
provides a new occlusion augmentation scheme with generality, providing a
variety of holistic-occlusion sample pairs. Subsequently, a dual stream
structure can use the shared encoder to train two different branches for
holistic and occluded images respectively by using holistic-occlusion sample
pairs. Then we propose a feature completion decoder (FCD) to recovery holistic
features from occluded features. Extensive experiments on occluded and
holistic datasets demonstrate the effectiveness of our proposed method.

In the future, we intend to extend the proposed occlusion augmentaiton and
feature completion paradigm to various computer vision tasks in order to
achieve a unified solution to the occlusion problem.

Figure 12: Visualization of attention heatmaps by Grad-CAM [58] in feature
completion decoder.

## References

  * [1] L. Zheng, Y. Yang, and A. G. Hauptmann, “Person re-identification: Past, present and future,” _arXiv preprint arXiv:1610.02984_ , 2016.
  * [2] J. Zhuo, Z. Chen, J. Lai, and G. Wang, “Occluded person re-identification,” in _2018 IEEE International Conference on Multimedia and Expo (ICME)_ , 2018, pp. 1–6.
  * [3] J. Miao, Y. Wu, P. Liu, Y. Ding, and Y. Yang, “Pose-guided feature alignment for occluded person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 542–551.
  * [4] P. Chen, W. Liu, P. Dai, J. Liu, Q. Ye, M. Xu, Q. Chen, and R. Ji, “Occlude them all: Occlusion-aware attention network for occluded person re-id,” in _Proceedings of the IEEE/CVF international conference on computer vision (CVPR)_ , 2021, pp. 11 833–11 842.
  * [5] M. Jia, X. Cheng, S. Lu, and J. Zhang, “Learning disentangled representation implicitly via transformer for occluded person re-identification,” _IEEE Transactions on Multimedia (TMM)_ , 2022.
  * [6] S. Gao, J. Wang, H. Lu, and Z. Liu, “Pose-guided visible part matching for occluded person reid,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 11 744–11 752.
  * [7] G. Wang, S. Yang, H. Liu, Z. Wang, Y. Yang, S. Wang, G. Yu, E. Zhou, and J. Sun, “High-order information matters: Learning relation and topology for occluded person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 6449–6458.
  * [8] S. Iodice and K. Mikolajczyk, “Partial person re-identification with alignment and hallucination,” in _Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision (ACCV)_. Springer, 2019, pp. 101–116.
  * [9] X. Jin, C. Lan, W. Zeng, G. Wei, and Z. Chen, “Semantics-aligned representation learning for person re-identification,” in _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 34, no. 07, 2020, pp. 11 173–11 180.
  * [10] R. Hou, B. Ma, H. Chang, X. Gu, S. Shan, and X. Chen, “Feature completion for occluded person re-identification,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , vol. 44, no. 9, pp. 4894–4912, 2021\.
  * [11] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European Conference on Computer Vision (ECCV)_ , 2014, pp. 740–755.
  * [12] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, “Random erasing data augmentation,” _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 34, no. 7, 2017.
  * [13] Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, “Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline),” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 480–496.
  * [14] X. Zhang, H. Luo, X. Fan, W. Xiang, Y. Sun, Q. Xiao, W. Jiang, C. Zhang, and J. Sun, “Alignedreid: Surpassing human-level performance in person re-identification,” _arXiv preprint arXiv:1711.08184_ , 2017.
  * [15] Y. Sun, Q. Xu, Y. Li, C. Zhang, Y. Li, S. Wang, and J. Sun, “Perceive where to focus: Learning visibility-aware part-level features for partial person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 393–402.
  * [16] M. Jia, X. Cheng, Y. Zhai, S. Lu, S. Ma, Y. Tian, and J. Zhang, “Matching on sets: Conquer occluded person re-identification without alignment,” in _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 35, 2021, pp. 1673–1681.
  * [17] C. Song, Y. Huang, W. Ouyang, and L. Wang, “Mask-guided contrastive attention model for person re-identification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 1179–1188.
  * [18] H. Huang, A. Zheng, C. Li, R. He _et al._ , “Parallel augmentation and dual enhancement for occluded person re-identification,” _arXiv preprint arXiv:2210.05438_ , 2022.
  * [19] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision (ICCV)_ , 2017, pp. 2961–2969.
  * [20] S. He, H. Luo, P. Wang, F. Wang, H. Li, and W. Jiang, “Transreid: Transformer-based object re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [21] T. Wang, H. Liu, P. Song, T. Guo, and W. Shi, “Pose-guided feature disentangling for occluded person re-identification based on transformer,” in _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , vol. 36, no. 3, 2022, pp. 2540–2549.
  * [22] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [23] Y. Li, J. He, T. Zhang, X. Liu, Y. Zhang, and F. Wu, “Diverse part discovery: Occluded person re-identification with part-aware transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 2898–2907.
  * [24] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. pmlr, 2015, pp. 448–456.
  * [25] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2022, pp. 16 000–16 009.
  * [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017, pp. 5998–6008.
  * [27] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2015, pp. 815–823.
  * [28] Z. Wang, F. Zhu, S. Tang, R. Zhao, L. He, and J. Song, “Feature erasing and diffusion network for occluded person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR))_ , 2022, pp. 4754–4763.
  * [29] L. Zhao, X. Li, Y. Zhuang, and J. Wang, “Deeply-learned part-aligned representations for person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2017, pp. 3219–3228.
  * [30] Y. Suh, J. Wang, S. Tang, T. Mei, and K. M. Lee, “Part-aligned bilinear representations for person re-identification,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 402–419.
  * [31] Y. Ge, Z. Li, H. Zhao, G. Yin, S. Yi, X. Wang, and H. Li, “Fd-gan: Pose-guided feature distilling gan for robust person re-identification,” _arXiv preprint arXiv:1810.02936_ , 2018.
  * [32] L. He, J. Liang, H. Li, and Z. Sun, “Deep spatial feature reconstruction for partial person re-identification: Alignment-free approach,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 7073–7082.
  * [33] L. He, Z. Sun, Y. Zhu, and Y. Wang, “Recognizing partial biometric patterns,” _arXiv preprint arXiv:1810.07399_ , 2018.
  * [34] H. Huang, D. Li, Z. Zhang, X. Chen, and K. Huang, “Adversarially occluded samples for person re-identification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 5098–5107.
  * [35] K. Zhu, H. Guo, Z. Liu, M. Tang, and J. Wang, “Identity-guided human semantic parsing for person re-identification,” in _European Conference on Computer Vision (ECCV)_ , 2020, pp. 346–363.
  * [36] X. Zhang, Y. Yan, J.-H. Xue, Y. Hua, and H. Wang, “Semantic-aware occlusion-robust network for occluded person re-identification,” _IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)_ , vol. 31, no. 7, pp. 2764–2778, 2021.
  * [37] Z. Ma, Y. Zhao, and J. Li, “Pose-guided inter-and intra-part relational transformer for occluded person re-identification,” in _Proceedings of the 29th ACM International Conference on Multimedia (ACM MM)_ , 2021, pp. 1487–1496.
  * [38] K. Zheng, C. Lan, W. Zeng, J. Liu, Z. Zhang, and Z.-J. Zha, “Pose-guided feature learning with knowledge distillation for occluded person re-identification,” in _Proceedings of the 29th ACM International Conference on Multimedia (ACM MM)_ , 2021, pp. 4537–4545.
  * [39] P. Wang, C. Ding, Z. Shao, Z. Hong, S. Zhang, and D. Tao, “Quality-aware part models for occluded person re-identification,” _IEEE Transactions on Multimedia (TMM)_ , pp. 1–1, 2022.
  * [40] Z. Zheng, L. Zheng, and Y. Yang, “Unlabeled samples generated by gan improve the person re-identification baseline in vitro,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2017, pp. 3754–3762.
  * [41] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian, “Scalable person re-identification: A benchmark,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2015, pp. 1116–1124.
  * [42] L. He, Y. Wang, W. Liu, H. Zhao, Z. Sun, and J. Feng, “Foreground-aware pyramid reconstruction for alignment-free occluded person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 8450–8459.
  * [43] L. He, J. Liang, H. Li, and Z. Sun, “Deep spatial feature reconstruction for partial person re-identification: Alignment-free approach,” in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2018, pp. 7073–7082.
  * [44] L. He and W. Liu, “Guided saliency feature learning for person re-identification in crowded scenes,” in _European Conference on Computer Vision (ECCV)_ , 2020, pp. 357–373.
  * [45] H. Luo, Y. Gu, X. Liao, S. Lai, and W. Jiang, “Bag of tricks and a strong baseline for deep person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2019.
  * [46] H. Sun, Z. Chen, S. Yan, and L. Xu, “Mvp matching: A maximum-value perfect matching for mining hard samples, with application to person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 6737–6747.
  * [47] C. Luo, Y. Chen, N. Wang, and Z. Zhang, “Spectral feature transformation for person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 4976–4985.
  * [48] W. Yang, H. Huang, Z. Zhang, X. Chen, K. Huang, and S. Zhang, “Towards rich feature discovery with class activation maps augmentation for person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1389–1398.
  * [49] R. Hou, B. Ma, H. Chang, X. Gu, S. Shan, and X. Chen, “Interaction-and-aggregation network for person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 9317–9326.
  * [50] Y. Sun, C. Cheng, Y. Zhang, C. Zhang, L. Zheng, Z. Wang, and Y. Wei, “Circle loss: A unified perspective of pair similarity optimization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 6398–6407.
  * [51] M. M. Kalayeh, E. Basaran, M. Gökmen, M. E. Kamasak, and M. Shah, “Human semantic parsing for person re-identification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 1062–1071.
  * [52] J. Guo, Y. Yuan, L. Huang, C. Zhang, J.-G. Yao, and K. Han, “Beyond human parts: Dual part-aligned representations for person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 3642–3651.
  * [53] C.-P. Tay, S. Roy, and K.-H. Yap, “Aanet: Attribute attention network for person re-identifications,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 7134–7143.
  * [54] W. Li, X. Zhu, and S. Gong, “Harmonious attention network for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2018, pp. 2285–2294.
  * [55] K. Zhou, Y. Yang, A. Cavallaro, and T. Xiang, “Omni-scale feature learning for person re-identification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 3702–3712.
  * [56] L. Zheng, H. Zhang, S. Sun, M. Chandraker, Y. Yang, and Q. Tian, “Person re-identification in the wild,” in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2017, pp. 1367–1376.
  * [57] Y. Suh, J. Wang, S. Tang, T. Mei, and K. M. Lee, “Part-aligned bilinear representations for person re-identification,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 402–419.
  * [58] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision (ICCV)_ , 2017, pp. 618–626.