∎

11institutetext: Y. Yan 22institutetext: MoE Key Lab of Artificial
Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China.
22email<EMAIL_ADDRESS>33institutetext: J. Li 44institutetext:
Inception Institute of Artificial Intelligence, Abu Dhabi, UAE.
44email<EMAIL_ADDRESS>55institutetext: J. Qin 66institutetext: College of
Computer Science and Technology, Nanjing University of Aeronautics and
Astronautics
66email<EMAIL_ADDRESS>77institutetext: P. Zheng 88institutetext:
Inception Institute of Artificial Intelligence, Abu Dhabi, UAE.
88email<EMAIL_ADDRESS>99institutetext: S. Liao 1010institutetext:
Inception Institute of Artificial Intelligence, Abu Dhabi, UAE.
1010email<EMAIL_ADDRESS>1111institutetext: X. Yang 1212institutetext: MoE
Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong
University, Shanghai, China.
1212email<EMAIL_ADDRESS>
# Efficient Person Search: An Anchor-Free Approach

Yichao Yan  Jinpeng Li  Jie Qin  Peng Zheng  Shengcai Liao  Xiaokang Yang

(Received: date / Accepted: date)

###### Abstract

Person search aims to simultaneously localize and identify a query person from
uncropped images. To achieve this goal, state-of-the-art models typically add
a re-id branch upon two-stage detectors like Faster R-CNN. Owing to the ROI-
Align operation, this pipeline yields promising accuracy as re-id features are
explicitly aligned with the corresponding object regions, but in the meantime,
it introduces high computational overhead due to dense object anchors. In this
work, we present an anchor-free approach to efficiently tackling this
challenging task, by introducing the following dedicated designs. First, we
select an anchor-free detector (i.e., FCOS) as the prototype of our framework.
Due to the lack of dense object anchors, it exhibits significantly higher
efficiency compared with existing person search models. Second, when directly
accommodating this anchor-free detector for person search, there exist several
misalignment issues in different levels (i.e., scale, region, and task). To
address these issues, we propose an aligned feature aggregation module to
generate more discriminative and robust feature embeddings. Accordingly, we
name our framework as Feature-Aligned Person Search Network (AlignPS). Third,
by investigating the advantages of both anchor-based and anchor-free models,
we further augment AlignPS with an ROI-Align head, which significantly
improves the robustness of re-id features while still keeping our model highly
efficient. Our framework not only achieves state-of-the-art or competitive
performance on two challenging person search benchmarks, but can be also
extended to other challenging searching tasks such as animal and object
search. All the source codes, data, and trained models are available at:
https://github.com/daodaofr/alignps.

###### Keywords:

Person search anchor-free model efficient learning feature alignment.

††journal: IJCV

##  1 Introduction

Person search DBLP:conf/cvpr/ZhengZSCYT17; DBLP:conf/cvpr/XiaoLWLW17 aims to
localize and identify a target person from a gallery of realistic, uncropped
scene images, and it has recently emerged as a practical task with real-world
applications, e.g., video surveillance. Two fundamental computer vision tasks,
i.e., pedestrian detection DBLP:conf/iccv/OuyangW13; DBLP:conf/cvpr/ZhangBS17
and person re-identification (re-id) DBLP:conf/cvpr/FarenzenaBPMC10;
DBLP:conf/cvpr/AhmedJM15, need to be addressed to tackle this task. Both
detection and re-id are very challenging tasks and have received tremendous
attention in the past decade. In person search, we need to not only address
the challenges (e.g., occlusions, pose/viewpoint variations, and background
clutter) of the two individual tasks, but also pursue a unified and optimized
framework to simultaneously perform detection and re-id.

(a) Two-step person search framework (b) One-step two-stage person search
framework (c) AlignPS: the proposed one-step anchor-free framework (d) ROI-
AlignPS: AlignPS augmented with an ROI-Align head

Figure 1: Comparison of four person search frameworks. (a) The two-step
framework addresses detection and re-id as two separate tasks. (b) The one-
step model enables end-to-end training of detection and re-id with an ROI-
Align operation based on a two-stage detector; however, re-id is considered as
a secondary task after detection. (c) The proposed framework enables single-
stage inference for both detection and re-id, while making re-id the primary
task. (d) By appending an ROI-Align head to the bounding box output of
AlignPS, the re-id features are further improved with explicit region
alignment.

Previous person search frameworks can be generally divided into two
categories. The first line of works DBLP:conf/cvpr/ZhengZSCYT17;
DBLP:conf/eccv/ChenZOYT18; DBLP:conf/eccv/LanZG18 can be summarized as _two-
step_ approaches, which attempt to deal with detection and re-id separately.
As shown in Fig. 1a, multiple persons are first localized with off-the-shelf
detection models, and then cropped out and fed to re-id networks to extract
feature representations. Although two-step models can obtain satisfactory
results, the disentangled treatment of the two tasks is time- and resource-
consuming. In contrast, the second category DBLP:conf/cvpr/XiaoLWLW17;
DBLP:conf/iccv/LiuFJKZQJY17; DBLP:conf/eccv/ChangHSLYH18;
DBLP:conf/cvpr/MunjalATG19; DBLP:conf/cvpr/ChenZYS20 provides a _one-step_
solution that unifies detection and re-id in an end-to-end manner. As shown in
Fig. 1b, one-step models first apply an ROI-Align layer to aggregate features
in the detected bounding boxes. The features are then shared by detection and
re-id; with an additional re-id loss, the simultaneous optimization of the two
tasks becomes feasible. Since these models adopt two-stage detectors like
Faster R-CNN DBLP:journals/pami/RenHG017, we refer to them as _one-step two-
stage_ models. However, these methods inevitably inherit the limitations of
two-stage detectors, e.g., high computational complexity caused by dense
anchors, and high sensitivity to the hyperparameters including the size,
aspect ratio and number of anchor boxes, etc.

Compared with two-stage detectors, anchor-free models exhibit unique
advantages (e.g., simpler structure and higher speed), and have been actively
studied in recent years DBLP:conf/cvpr/RedmonDGF16; DBLP:conf/eccv/LawD18;
DBLP:conf/cvpr/LiuLRHY19; DBLP:conf/iccv/DuanBXQH019. Inspired by this, an
open question is naturally thrown at us - _Is it possible to develop an
anchor-free framework for person search?_ Our answer is yes. However, this is
a non-trivial task due to the following three misalignment issues. 1) Many
anchor-free models learn multi-scale features using feature pyramid networks
(FPNs) DBLP:conf/cvpr/LinDGHHB17 to achieve scale invariance for object
detection. However, this introduces the misalignment issue for re-id (i.e.,
scale misalignment), as a query person needs to be compared with all the
people of various scales in the gallery set, while the re-id features would be
inconsistently taken from different FPN levels. 2) In the absence of
operations like ROI-Align, anchor-free models cannot align the features for
re-id and detection according to a specific region. Therefore, re-id
embeddings must be directly learned from feature maps without explicit region
alignment, which brings additional challenges as re-id features are sensitive
to the foreground regions DBLP:conf/cvpr/DongZST20a;
DBLP:conf/iccv/HanYZTZGS19. 3) Person search can be intuitively formulated as
a multi-task learning framework with detection and re-id as its sub-tasks.
However, there exist conflicts between the objectives of these two tasks,
i.e., pedestrian detection tries to learn features that are commonly shared by
all the people, while re-id tries to extract unique features for individual
persons. Hence, we need to find a better tradeoff/alignment between the two
tasks.

In this work, we present an anchor-free framework for efficient person search,
which we name the Feature-Aligned Person Search Network (AlignPS). Our model
employs the typical architecture of an anchor-free detection model (i.e., FCOS
DBLP:conf/iccv/TianSCH19), which allows our framework to be more efficient
than prior person search models. To address the above-mentioned challenges, we
design an aligned feature aggregation (AFA) module to make our model focus
more on the re-id subtask. Specifically, AFA reshapes some building blocks of
FPN to overcome the issues of region and scale misalignment in re-id feature
learning. For example, we exploit deformable convolution to make the re-id
embeddings adaptively aligned with the foreground regions. In the meantime, we
design a feature fusion scheme to better aggregate features from different FPN
levels, which makes the re-id features more robust to scale variations. We
also optimize the training procedures of re-id and detection to place more
emphasis on generating robust re-id embeddings (as shown in Fig. 1c). These
simple yet effective designs successfully transform a classic anchor-free
detector into a powerful and efficient person search framework, and allow the
proposed model to outperform its anchor-based competitors. Moreover, we find
that although the proposed AlignPS framework implicitly aligns re-id features
with the corresponding regions, there inevitably exists a gap between the
adapted regions and the foreground bounding boxes. As observed in previous
works DBLP:conf/cvpr/DongZST20a, re-id features are sensitive to the context
outside the foreground regions. Inspired by the learning scheme in two-stage
models (as shown in Fig. 1b), we further augment AlignPS with an ROI-Align
head, to explicitly extract more robust re-id features corresponding to the
foreground regions. We name this variant as ROI-AlignPS, as shown in Fig. 1d.
These explicitly aligned re-id features can be viewed as complements to the
implicitly aligned features from AlignPS, and these two kinds of features are
fused to yield even better re-id representations. More importantly, the ROI-
Align head only receives the output bounding boxes from AlignPS during
inference, avoiding computing dense region proposals. Therefore, ROI-AlignPS
still inherits the high efficiency from AlignPS.

In summary, our main contributions include:

  * •

We propose the first _anchor-free_ framework (AlignPS) for efficient person
search, which will significantly foster future research in this direction.

  * •

We design an AFA module that simultaneously addresses the issues of scale,
region, and task misalignment to successfully accommodate an anchor-free
detector for the task of person search.

  * •

We further propose a novel model variant (ROI-AlignPS) by augmenting AlignPS
with an ROI-Align head, which complements the implicitly aligned re-id
features. With mutual learning strategy, this variant further improves the
performance while remaining highly efficient.

  * •

As an anchor-free framework, our model surprisingly achieves state-of-the-art
or competitive performance on two challenging person search benchmarks, while
running at a higher speed.

  * •

Our framework can be naturally extended to handle other object search tasks,
such as animal and object search. We evaluate our framework on a Chinese
dolphin and a tattoo search benchmarks, and the results demonstrate that our
framework consistently outperforms prior arts.

Part of this work has been published in Yan_2021_CVPR. In this paper, we make
the following major extensions: 1) We augment AlignPS with an ROI-Align head,
which explicitly aligns the foreground regions with their re-id features. This
architecture variant notably improves the performance, while keeping the
framework efficient. 2) We investigate a mutual feature learning strategy,
which allows the explicitly and implicitly aligned re-id features to promote
each other during training. We find these strategies yield more robust re-id
representations. 3) We provide more thorough ablation studies and component
analysis on both CUHK-SYSU DBLP:conf/cvpr/XiaoLWLW17 and PRW
DBLP:conf/cvpr/ZhengZSCYT17, and present more qualitative results. These
analyses further illustrate the effectiveness of the proposed framework. 4) We
further extend our frameowrk from person search to animal and object search
tasks, and provide comparative results with prior arts.

##  2 Related Work

Pedestrian Detection. Pedestrian or object detection can be considered as a
preliminary task of person search. Current deep learning-based detectors are
generally categorized into one-stage and two-stage models, according to
whether they employ a region proposal network (RPN) to generate object
proposals. Alternatively, object detectors can also be categorized into
anchor-based and anchor-free detectors, depending on whether they utilize
anchor boxes to associate objects. One of the most representative two-stage
anchor-based detectors is Faster R-CNN DBLP:journals/pami/RenHG017, which has
been extended into numerous variants DBLP:conf/iccv/DaiQXLZHW17;
DBLP:conf/cvpr/CaiV18; DBLP:conf/cvpr/PangCSFOL19; DBLP:conf/cvpr/SongLW20;
DBLP:journals/ijcv/ZhangCYS21. Notably, some one-stage detectors
DBLP:conf/eccv/LiuAESRFB16; DBLP:conf/iccv/LinGGHD17;
DBLP:conf/cvpr/RedmonF17; DBLP:conf/cvpr/ZhangWBLL18 also work with anchor
boxes. Compared with the above models, one-stage anchor-free detectors
DBLP:conf/cvpr/RedmonDGF16; DBLP:conf/eccv/LawD18; DBLP:conf/cvpr/LiuLRHY19;
DBLP:journals/corr/abs-1904-07850; DBLP:conf/iccv/YangLHWL19;
DBLP:conf/iccv/TianSCH19; DBLP:conf/mm/LiLJ020 have been attracting more and
more attention recently due to their simple structures and efficient
implementations. In this work, we develop our person search framework based on
an one-stage anchor-free detector, thus making the whole framework simpler and
faster.

Feature Alignment. How to align image features with objects is a fundamental
problem in object detection. Two-stage detectors DBLP:journals/pami/RenHG017
typically perform dense proposals and employ ROI-Pooling or ROI-Align to
explicitly align features, or further apply deformable convolution
DBLP:conf/iccv/DaiQXLZHW17; DBLP:conf/cvpr/WangCYLL19 to better adapt the
features with anchor geometry. In one-stage and anchor-free detectors,
however, it requires implicitly aligning image features with objects. For
example, FCOS DBLP:conf/iccv/TianSCH19 considers all the features within the
ground-truth bounding boxes as positive samples, which are employed to predict
location. FoveaBox DBLP:journals/tip/KongSLJLS20 finds that center points are
more likely to produce high-quality bounding boxes, and therefore proposes a
center sampling strategy. DAFS DBLP:conf/iccv/LiYH0Z19 proposes an anchor
refinement module and a dynamic feature selection strategy to match the anchor
with more accurate features. RefineDet DBLP:conf/cvpr/ZhangWBLL18 proposes an
anchor refinement module to improve the location and size of the anchors.
AlignDet DBLP:journals/corr/abs-1908-01570 first predicts the offsets on the
feature maps and then performs deformable conv to align the features. Our
framework also employs deformable convolution for adaptive learning, but it
jointly addresses the detection and re-id tasks, while previous works mainly
focus on detection. Moreover, we make the alignment on the FPN module, to
yield a joint solution for the scale, region, and task misalignment issues.

Person Re-identification. Person re-id is also closely related to person
search, aiming to learn identity embeddings from cropped person images.
Traditional methods employed various handcrafted features
DBLP:journals/ijcv/Lowe04; DBLP:conf/cvpr/FarenzenaBPMC10;
DBLP:conf/eccv/GrayT08 before the renaissance of deep learning. However, to
pursue better performance, current re-id models are mostly based on deep
learning. Some models employ structure/part information in the human body or
add additionl constraint to learn more robust representations
DBLP:conf/iccv/SuLZX0T17; DBLP:conf/eccv/SunZYTW18; DBLP:conf/iccv/MiaoWLD019;
9233968; DBLP:journals/ijcv/ZhengHJWZGQL22; DBLP:journals/ijcv/ZhuZLMMG21,
while others focus on learning better distance metrics
DBLP:conf/cvpr/AhmedJM15; DBLP:journals/corr/HermansBL17;
DBLP:conf/cvpr/ChenCZH17; DBLP:journals/pami/ChenZZL18;
DBLP:journals/pami/WangGZW16; DBLP:journals/ijcv/WangZGX18;
DBLP:journals/ijcv/LiSFSHZQ20. As person re-id usually lacks large-scale
training data, data augmentation DBLP:conf/nips/GeLZYYWL18;
DBLP:conf/cvpr/LiuNYZCH18; DBLP:conf/cvpr/WeiZ0018 also becomes popular for
tackling this task. Compared with detection which aims to learn common
features of pedestrians, re-id needs to focus more on fine-grained details and
unique features of each identity. Therefore, we propose to follow the “re-id
first” principle to raise the priority of the re-id task, resulting in more
discriminative identity embeddings.

Person Search. Existing person search frameworks can be divided into two-step
and one-step models. Two-step models first perform pedestrian detection and
subsequently crop the detected people for re-id. Zheng et al.
DBLP:conf/cvpr/ZhengZSCYT17 introduced the pioneering two-step framework for
person search and evaluated the combinations of different detectors and re-id
models. Since then, several models DBLP:conf/eccv/ChenZOYT18;
DBLP:conf/eccv/LanZG18; DBLP:conf/iccv/HanYZTZGS19; DBLP:conf/cvpr/WangMCSC20
have followed this pipeline. In DBLP:conf/cvpr/XiaoLWLW17, Xiao et al.
proposed the first one-step person search framework based on Faster R-CNN.
Specifically, a joint framework enabling end-to-end training of detection and
re-id was proposed by stacking a re-id embedding layer after the detection
features and proposing the Online Instance Matching (OIM) loss. So far, a
number of improvements DBLP:conf/iccv/LiuFJKZQJY17;
DBLP:journals/pr/XiaoXTHWF19; DBLP:conf/eccv/ChangHSLYH18;
DBLP:conf/cvpr/YanZNZXY19; DBLP:conf/cvpr/MunjalATG19;
DBLP:conf/cvpr/DongZST20a; DBLP:conf/cvpr/ChenZYS20 have been made based on
this framework. In general, two-step models may achieve better performance,
while one-step models have the advantages of simplicity and efficiency.
However, there is still room for improving one-step methods due to the
aforementioned shortcomings of the two-stage anchor-based detectors they
usually adopt. In this work, we introduce an anchor-free model to further
improve the simplicity and efficiency of one-step models, without any
sacrifice in accuracy.

Mutual Learning. Hinton et al. DBLP:journals/corr/HintonVD15 firstly
introduced knowledge distillation in neural networks in 2015. Since then, it
has been widely employed in various computer vision tasks to improve the
capability of neural networks, e.g., image recognition
DBLP:journals/pami/LiH18a; DBLP:conf/eccv/WangFLWLM20;
DBLP:conf/iccv/PengLZLQT19, object detection DBLP:conf/cvpr/LiJY17;
DBLP:conf/nips/ChenCYHC17; DBLP:conf/cvpr/ZhuHLD19, image segmentation
DBLP:conf/iccv/MullapudiCZRF19; DBLP:conf/cvpr/LiuCLQLW19;
DBLP:conf/cvpr/Hou0LHL20, etc. In person search, several recent work have also
adopted this approach. For example, QEEPS DBLP:conf/cvpr/MunjalATG19 and IGPN
DBLP:conf/cvpr/DongZST20 employ the query person to find the most similar
proposals in the RPN, which reduces the number of proposals and enhances the
efficiency. In BiNet DBLP:conf/cvpr/DongZST20a and DKD
DBLP:conf/aaai/ZhangWBSY21, the features from cropped pedestrians are utilized
to distill the re-id features in person search network, such that the
robustness of the re-id features are improved. In this work, we design two
feature learning branches in ROI-AlignPS. Instead of employing the query
person or the cropped image, we investigate a mutual feature promotion
strategy to improve the discriminative capability of the aggregated features.

##  3 Feature-Aligned Person Search Networks

In this section, we introduce the proposed anchor-free frameworks (i.e.,
AlignPS and ROI-AlignPS) for person search. First, we give an overview of the
network architecture. Second, the proposed AFA module is elaborated with the
aim of mitigating different levels of misalignment issues when transforming an
anchor-free detector into a superior person search framework. Then, we present
the designed loss function to obtain more discriminative features for person
search. Finally, we present ROI-AlignPS, a model variant which further
improves the performance of AlignPS.

Figure 2: Architecture of the proposed AlignPS framework, which shares the
basic structure of FCOS DBLP:conf/iccv/TianSCH19. The components in yellow are
newly designed to accommodate FCOS for the task of person search. “Dconv”
means deformable convolution.

###  3.1 Framework Overview

The basic framework of the proposed AlignPS is based on FCOS
DBLP:conf/iccv/TianSCH19, one of the most popular one-stage anchor-free object
detectors. Differently, we adhere to the “re-id first” principle to put
emphasis on learning robust feature embeddings for the re-id subtask, which is
crucial for enhancing the overall performance of person search.

As illustrated in Fig. 2, our model simultaneously localizes multiple people
in the image and learns re-id embeddings for them. Specifically, an AFA module
is developed to aggregate features from multi-level feature maps in the
backbone network. To learn re-id embeddings, which is the key of our method,
we directly take the flattened features from the output feature maps of AFA as
the final embeddings, without any extra embedding layers. For detection, we
employ the detection head from FCOS which is good enough for the detection
subtask. The detection head consists of two branches, both of which contain
four 3$\times$3 _conv_ layers. In the meantime, the first branch predicts
regression offsets and centerness scores, while the second makes
foreground/background classification. Finally, each location on the output
feature map of AFA will be associated with a bounding box with classification
and centerness scores, as well as a re-id feature embedding.

###  3.2 Aligned Feature Aggregation

Following FPN DBLP:conf/cvpr/LinDGHHB17, we make use of different levels of
feature maps to learn detection and re-id features. As the key of our
framework, the proposed AFA performs three levels of alignment, beyond the
original FPN, to make the output re-id features more discriminative.

Scale Alignment. As shown in Fig. 3a, the original FCOS model employs
different levels of features to detect objects of different sizes. This
significantly improves the detection performance since the overlapped
ambiguous samples will be assigned to different layers. For the re-id task,
however, the multi-level prediction could cause feature misalignment between
different scales. In other words, when matching a person of different scales,
re-id features are inconsistently taken from different levels of FPN, thus
preventing the re-id features from being robust to scale variations.
Furthermore, the people in the gallery set are of various scales, which could
eventually make the multi-level model fail to find correct matches for the
query person. Therefore, in our framework, we only make predictions based on a
single layer of AFA, which explicitly addresses the feature misalignment
caused by scale variations. Specifically, we employ the
$\\{C_{3},C_{4},C_{5}\\}$ feature maps from the ResNet-50 backbone, and AFA
sequentially outputs $\\{P_{5},P_{4},P_{3}\\}$, with strides of 32, 16, and 8,
respectively. As shown in Fig. 3b, our framework only generates features from
$\\{P_{3}\\}$, which is the largest output feature map, for both the detection
and re-id subtasks, and $\\{P_{6},P_{7}\\}$ are no longer generated as in the
original FPN. Although this design may slightly influence the detection
performance, we will show in Sec. 4.3 that it achieves a good trade-off
between the detection and re-id subtasks, which adapts well to the person
search task.

(a) The prediction structure of FPN (b) The prediction structure of AlignPS

Figure 3: Illustration of scale misalignment. (a) FPN divides persons of
different sizes to different feature maps, i.e., $\\{P_{3},P_{4},P_{5}\\}$,
which causes scale misalignment for re-id. (b) AlignPS addresses this issue by
only making predictions on a single feature map, i.e., $\\{P_{3}\\}$.

Region Alignment. On the output feature map of AFA, each location perceives
the information from the whole input image based on a large receptive field.
Due to the lack of the ROI-Align operation as in Faster R-CNN, it is difficult
for our anchor-free framework to learn more accurate features within the
pedestrian bounding boxes, and thus leading to the issue of region
misalignment. The re-id subtask is even more sensitive to this issue as
background features could greatly impact the discriminative capability of the
learned features. In AlignPS, we address this issue from three perspectives.
_First_ , we replace the 1$\times$1 _conv_ layers in the lateral connections
with 3$\times$3 _deformable conv_ layers. As the original lateral connections
are designed to reduce the channels of feature maps, a 1$\times$1 _conv_ is
enough. In our design, moreover, the 3$\times$3 _deformable conv_ enables the
network to adaptively adjust the receptive field on the input feature maps,
thus implicitly fulfilling region alignment. _Second_ , we replace the “sum”
operation in the top-down pathway with a “concatenation” operation, which can
better aggregate multi-level features. _Third_ , we again replace the
3$\times$3 _conv_ with a 3$\times$3 _deformable conv_ for the output layer of
FPN, which further aligns the multi-level features to finally generate a more
accurate feature map. The above three designs work seamlessly to address the
region misalignment issue, and we notice that these simple designs are
extremely effective when accommodating the basic anchor-free model for our
person search task.

Task Alignment. Existing person search frameworks typically treat pedestrian
detection as the primary task, i.e., re-id embeddings are just generated by
stacking an additional layer after the detection features. A recent work
DBLP:journals/corr/abs-2004-01888 investigated a parallel structure by
employing independent heads for the two tasks to achieve robust multiple
object tracking results. In our task of person search, we find the inferior
re-id features largely hinder the overall performance. Therefore, we opt for a
different principle to align these two tasks by treating re-id as our primary
task. Specifically, the output features of AFA are directly supervised with a
re-id loss (which will be introduced in the following subsection), and then
fed to the detection head. This “re-id first” design is based on two
considerations. _First_ , the detection subtask has been relatively well
addressed by existing person search frameworks, which directly inherit the
advantages from existing powerful detection frameworks. Therefore, learning
discriminative re-id embeddings is our primary concern. As we discussed, re-id
performance is more sensitive to region misalignment in an anchor-free
framework. Therefore, it is desirable for the person search framework to be
inclined towards the re-id subtask. We also show in our experiments that this
design significantly improves the discriminative capability of the re-id
embeddings, while having negligible impact on detection. _Second_ , compared
with “detection first” and parallel structures, the proposed “re-id first”
structure does not require an extra layer to generate re-id embeddings, and is
thus more efficient.

###  3.3 Network Training

Online Instance Matching Loss. Existing person search frameworks typically
employ the OIM loss DBLP:conf/cvpr/XiaoLWLW17 to supervise the training of re-
id. The objective of OIM is to minimize the distance between the features of
the same identity, while maximizing the distance between different
pedestrians. Meanwhile, person search datasets (e.g., CUHK-SYSU) include
unlabeled identities, i.e., some pedestrians are only annotated with bounding
box labels (but without identity labels). Employing the unlabeled identities
can make the learned features more discriminative, as the identities in the
labeled and unlabeled sets do not overlap. To make full use of all the
training samples, OIM maintains two external memories. Specifically, it stores
the feature centers of all labeled identities in a lookup table (LUT),
$V\in\mathbb{R}^{D\times L}=\\{v_{1},...,v_{L}\\}$, which contains $L$ feature
vectors with $D$ dimensions. Meanwhile, a circular queue containing the
features of $Q$ unlabeled identities is maintained, $U\in\mathbb{R}^{D\times
Q}=\\{u_{1},...,u_{Q}\\}$. At each iteration, given an input feature $x$ with
label $i$, OIM computes the similarity between $x$ and all the features in the
LUT and circular queue by $V^{T}x$ and $Q^{T}x$, respectively. The probability
of $x$ belonging to the identity $i$ is calculated as:

$p_{i}=\frac{{\rm exp}(v_{i}^{T}x)/\tau}{\sum_{j=1}^{L}{\rm
exp}(v_{j}^{T}x)/\tau+\sum_{k=1}^{Q}{\rm exp}(u_{k}^{T}x)/\tau},$ (1)

where $\tau=0.1$ is a hyperparameter that controls the softness of the
probability distribution. The numerator ${\rm exp}(v_{i}^{T}x)/\tau$ denotes
the normalized similarity between $x$ and the $i$-th centroid, and the
denominator denotes the similarity between $x$ and all the centroids in LUT as
well as the unlabeled identities in the circular queue. Therefore, the
unlabeled identities only serve as the negative classes for all the labeled
identities. The objective of OIM is to minimize the expected negative log-
likelihood:

$\mathcal{L}_{\text{OIM}}=-{\rm E}_{x}[{\rm log}~{}p_{t}],\ t=1,2,...,L.$ (2)

During the backward pass, the centroids in LUT are gradually updated:

$v_{i}\leftarrow\gamma v_{i}+(1-\gamma)x,$ (3)

where $\gamma\in[0,1]$ is a weighting term. In the meantime, the unlabeled
identities in the circular queue are also updated by pushing the new features
in the queue and removing the old features, while keeping the size of the
queue fixed.

Optional Triplet-aided OIM Loss. Although OIM effectively employs both labeled
and unlabeled identities, we still observe two limitations. First, the
distances are only computed between the input features and the features stored
in the lookup table and circular queue, while no comparisons are made between
the input features. Second, the log-likelihood loss term does not give an
explicit distance metric between feature pairs.

To improve OIM, we propose a specifically designed triplet loss. For each
person in the input images, we employ the center sampling strategy as in
DBLP:journals/tip/KongSLJLS20. As shown in Fig. 4, for each person, a set of
features located around the person center are considered as positive samples.
The objective is to pull the feature vectors from the same person close, and
push the vectors from different people away. Meanwhile, the features from the
labeled persons should be close to the corresponding features stored in the
LUT, and away from the other features in the LUT.

Figure 4: Illustration of the Triplet-aided Online Instance Matching loss,
where both the features from the input image and the lookup table are sampled
to form the triplet.

(a) Training pipeline of ROI-AlignPS (b) Test pipeline of ROI-AlignPS

Figure 5: The architecture of ROI-AlignPS, which can be regarded as AlignPS
augmented with a Faster R-CNN branch.

More specifically, suppose we sample $S$ vectors from one person; we get
$X_{m}=\\{x_{m,1},...,x_{m,S},v_{m}\\}$ and
$X_{n}=\\{x_{n,1},...,x_{n,S},v_{n}\\}$ as the candidate feature sets for the
persons with identity labels $m$ and $n$, respectively, where $x_{i,j}$
denotes the $j$-th feature of person $i$, and $v_{i}$ is the $i$-th feature in
the LUT. Given $X_{m}$ and $X_{n}$, positive pairs can be sampled within each
set, while negative pairs are sampled between the two sets. The triplet loss
can be calculated as:

$\mathcal{L}_{\text{tri}}=\sum_{\text{pos,~{}neg}}[M+D_{\text{pos}}-D_{\text{neg}}],$
(4)

where $M$ denotes the distance margin, and $D_{\text{pos}}$ and
$D_{\text{neg}}$ denote the Euclidean distances between the positive pair and
the negative pair, respectively. Finally, the Triplet-aided OIM (TOIM) loss is
the summation of these two terms:

$\mathcal{L}_{\text{TOIM}}=\mathcal{L}_{\text{tri}}+\mathcal{L}_{\text{OIM}}.$
(5)

As the triplet loss is only employed based on the generated features, it
therefore basically does not bring in much additional computation or memory
issue. Here, we treat the triplet loss as an optional loss term in our
framework.

###  3.4 AlignPS Augmented with Explicit Region Alignment

ROI-AlignPS. Although the proposed AFA module implicitly addresses the region
misalignment issue with deformable convolution, it is still desirable to
investigate the explicitly aligned learning schemes, e.g., cropping
pedestrians for re-id learning DBLP:conf/cvpr/ZhengZSCYT17;
DBLP:conf/eccv/ChenZOYT18 and the ROI-Align operation in Faster R-CNN
DBLP:conf/cvpr/XiaoLWLW17; DBLP:conf/cvpr/ChenZYS20. However, learning based
on the cropped pedestrians inevitably transfers the whole framework into a
two-step model, which significantly downgrades the efficiency. In the
meantime, the dense region proposals in Faster R-CNN also make it inefficient
during inference. To address these issues, we propose a novel variant, which
we name ROI-AlignPS, by augmenting our efficient anchor-free framework with an
explicit region alignment module.

In the training phase, two parallel branches are simultaneously trained with
implicit and explicit region alignment, respectively. As shown in Fig. 5a, the
bottom branch is the original AlignPS model, while the top branch follows the
architecture of NAE DBLP:conf/cvpr/ChenZYS20, which includes a RPN and ROI-
Align operation to output the re-id features of a certain region. Both
branches share the ResNet-50 backbone from res1 to res4 layers, while having
separate res5 layers. In the test phase, as illustrated in Fig. 5b, the bottom
AlignPS branch first outputs bounding boxes and re-id features. Then, the
processed bounding boxes are fed to the top branch to extract explicitly
aligned re-id features corresponding to each person. Finally, the re-id
features from these two branches are combined to generate a more robust
representation.

Although ROI-AlignPS contains an anchor-free branch and an anchor-based
branch, the dense anchors in Faster R-CNN are only involved in the training
stage. During inference, RPN is discarded and only a small number of detected
bounding boxes are processed with ROI-Align. In this way, ROI-AlignPS can
still remain highly efficient during the inference stage. Moreover, the
explicit and implicit region alignment strategies are complementary to each
other. Specifically, the ROI-Align branch extracts re-id features based on
bounding boxes, while AlignPS dynamically adapts to certain regions.
Meanwhile, the performance of both methods may be influenced by occlusions and
backgrounds. Therefore, combining these two kinds of features intuitively
reduces such impacts, and the effectiveness of this design is validated in
Sec. 4.4.

Branch-Level Mutual Learning. Prior works DBLP:journals/pami/LiH18a;
DBLP:conf/iccv/PengLZLQT19; Hong_2021_CVPR; Dai_2021_CVPR found that feature
interaction and knowledge distillation can benefit feature learning in neural
networks. In our case, it would be desirable if the prediction of the two
branches can reach a better consensus. To fullfile this, we investige a mutual
information maximization strategy.

Mutual information is defined to measure the dependency between two random
variables $X$ and $Y$:

$\mathcal{I}(X,Y)=\int_{X}\int_{Y}p(x,y){\rm log}\frac{p(x,y)}{p(x)p(y)}dxdy,$
(6)

where $p(x,y)$ is the joint probability density function, $p(x)$ and $p(y)$
are the marginal probability density functions of $X$ and $Y$, respectively.
As demonstrated in prior works Kinney3354; DBLP:conf/iclr/HjelmFLGBTB19,
mutual information is able to depict the mutual dependence between $X$ and
$Y$, no matter how nonlinear the dependence is. Thus, mutual information is
more flexible than the measurement of correlation. In this work, random
variables $X$ and $Y$ represent the re-id features from the two branches in
ROI-AlignPS, respectively. We aim to maximize the mutual information between
$X$ and $Y$, such that the overall representation will focus on the identity
information. As it is non-trivial to directly calculate the mutual
information, we employ a neural estimator DBLP:conf/icml/BelghaziBROBHC18 to
maximize the lower bound of the mutual information. Specifically,

$\mathcal{I}_{\Theta}(X,Y)=\sup_{\theta\in\Theta}\mathbb{E}_{\mathbb{P}_{{X}{Y}}}[T_{\theta}]-\log(\mathbb{E}_{\mathbb{P}_{{X}\otimes{Y}}}[e^{T_{\theta}}]),$
(7)

where $\mathbb{P}_{{X}{Y}}$ is the joint distribution and
$\mathbb{P}_{{X}\otimes{Y}}$ represents the marginal distribution,
$T_{\theta}$ is a neural network parameterized by $\theta\in\Theta$. To
maximize $\mathcal{I}_{\Theta}(X,Y)$, we minimize its negative value:

$\mathcal{L}_{mi}=-\mathcal{I}_{\Theta}(X,Y).$ (8)

Discussions. A recent work, SeqNet DBLP:conf/aaai/LiM21, proposes a sequential
architecture based on Faster R-CNN, which first predicts the detection
results, and then extracts re-id features with an additional ROI-Align branch.
Although ROI-AlignPS shares certain spirits with SeqNet, there exist several
significant differences. (1) The motivation of SeqNet is to improve the
quality of proposals, such that the re-id features could be better learned. In
ROI-AlignPS, rather than considering the detection issue, the ROI-Align branch
serves as an explicitly aligned re-id feature extractor, to complement the
features from the AlignPS branch. (2) Despite the strong performance, SeqNet
relies on a post graph matching technique. In contrast, the re-id features in
ROI-AlignPS could be mutually promoted and the ensemble results are highly
discriminative without post-processing. (3) In the inference stage, ROI-
AlignPS still runs in an anchor-free way. Consequently, our framework is more
efficient than SeqNet. The comparative results between ROI-AlignPS and SeqNet
can be found in Sec. 4.5.

##  4 Experiments

###  4.1 Datasets and Settings

CUHK-SYSU DBLP:conf/cvpr/XiaoLWLW17 is a large-scale person search dataset
which contains 18,184 images, with 8,432 different identities and 96,143
annotated bounding boxes. The images come from two kinds of data sources
(i.e., real street snaps and movies/TV), covering diverse scenes and including
variations of viewpoints, lighting, resolutions, and occlusions. We utilize
the standard training/test split, where the training set contains 5,532
identities and 11,206 images, and the test set contains 2,900 query persons
and 6,978 images. This dataset also defines a set of protocols with gallery
sizes ranging from 50 to 4,000. We report the results using the default
gallery size of 100 unless otherwise specified.

PRW DBLP:conf/cvpr/ZhengZSCYT17 was captured using six static cameras in a
university campus. All the images are extracted from the surveillance videos,
which consist of 11,816 video frames in total. Person identities and bounding
boxes are manually annotated, resulting in 932 labeled persons with 43,110
bounding boxes. The dataset is split into a training set of 5,704 images with
482 different identities, and a test set of 2,057 query persons and 6,112
images. Results are reported based on this split.

PICWD111https://github.com/PICWD/PICWD is a large-scale dataset collected for
Chinese white dolphin detection and re-id. The images are collected from
Qinzhou and Shantou, covering different environment conditions, including
illumination, occlusion, reflection of sea surface, etc. We select 48,273
images, with 158 identities and 52,570 bounding boxes localizing the dorsal
fins for white-dolphin search. Specifically, the whole dataset is divided into
train/test parts, where the training set contains 100 identities and 36,825
images, and the test set contains 58 identities and 11,448 images.

(a) Comparative results on CUHK-SYSU (b) Comparative results on PRW

Figure 6: Comparative results on CUHK-SYSU and PRW with different alignment
strategies, i.e., scale alignment (SA), region alignment (RA), and task
alignment (TA).

Webtattoo DBLP:journals/pami/HanLJSC19 is dataset collected for tattoo
detection and search. In this work, we focus on the tattoo search task and
follow the setting in DC-I-Net DBLP:journals/pami/ZhangHYWG22, where the
training set contains 1428 images of 400 tatoo classes, and the test set
includes 200 query images and 355 gallery images.

Evaluation Metric. We employ the mean average precision (mAP) and top-1
accuracy to evaluate the performance for person and animal search. We also
employ recall and average precision (AP) to measure the detection performance.

###  4.2 Implementation Details

We employ ResNet-50 DBLP:conf/cvpr/HeZRS16 pretrained on ImageNet
DBLP:conf/cvpr/DengDSLL009 as the backbone. We set the batch size to 4, and
adopt the stochastic gradient descent (SGD) optimizer with weight decay of
0.0005. The initial learning rate is set to 0.001 and is reduced by a factor
of 10 at epoch 16 and 22, with a total of 24 epochs. We use a warmup strategy
for 300 steps. We employ a multi-scale training strategy, where the longer
side of the image is randomly resized from 667 to 2000 during training, while
zero padding is utilized to fit the images with different resolutions. For
inference, we rescale the test images to a fixed size of 1500$\times$900\.
Following DBLP:conf/aaai/ChenZO0S20, we add a focal loss
DBLP:conf/iccv/LinGGHD17 to the original OIM loss. The center sampling radius
in the detector is set to 1.5 as its default value. In ROI-AlignPS, the raw
output bounding boxes from AlignPS are processed with non-maximum supression
(NMS) and then fed to the ROI-Align branch. We employ TOIM as re-id loss for
AlignPS and OIM for ROI-Align. All the experiments are implemented based on
PyTorch DBLP:conf/nips/PaszkeGMLBCKLGA19 and MMDetection
DBLP:journals/corr/abs-1906-07155, with an NVIDIA Tesla V100 GPU. It takes
around 38 and 24 hours for ROI-AlignPS to finish training on CUHK-SYSU and
PRW, respectively.

###  4.3 Analysis of AlignPS

Baseline. Our baseline model is built upon an anchor-free detector, i.e.,
FCOS. To enable FCOS for the task of person search, we add a re-id head at
each FPN level, in parallel with each detection head, such that the model can
simultaneously output the bounding box predictions and the re-id features. It
includes three baseline modules: 1) Multi-level prediction in FPN. To reduce
the overlapping ambiguity, different objects are assigned to different FPN
layers according to their spatial scales. However, this raises the scale
misalignment issue for the re-id task. 2) The feature aggregation module in
FPN. The original FPN only employs $1\times 1$ and $3\times 3$ convolutional
layers to perform feature learning, incapable of capturing the region-
sensitive foreground features. 3) Parallel prediction for detection and re-id,
which does not seriously consider the relationship between the two subtasks.
To address these issues, we design scale, region, and task alignment upon
these baseline modules. As shown in Fig. 6, each of the alignment strategies
brings notable improvements to the baseline, and combining all of them yields
23% and 26.1% improvements in mAP on CUHK-SYSU and PRW, respectively.

(a) Deformable conv at lateral $C_{3}$ layer in AFA (b) Deformable conv at
lateral $C_{4}$ layer in AFA

Figure 7: Each image shows the sampling locations of two levels of 3$\times$3 ($9^{2}=81$ points at each location) deformable filters: (a) Lateral deformable conv $C_{3}$ \+ Output deformable conv; (b) Lateral deformable conv $C_{4}$ \+ Output deformable conv. We illustrate different locations with different colors, while center locations of people are marked in green. Please zoom in for better visualization. | Detection | Re-id
---|---|---
CUKH-SYSU |  Recall |  AP |  mAP |  top-1
$P_{3}$ |  90.3 |  81.2 |  93.1 |  93.4
$P_{4}$ |  87.5 |  78.7 |  92.7 |  93.1
$P_{5}$ |  79.0 |  71.7 |  89.3 |  89.5
$P_{3}$, $P_{4}$ |  90.4 |  80.5 |  91.1 |  91.6
$P_{3}$, $P_{4}$, $P_{5}$ |  90.9 |  80.4 |  90.0 |  90.5
| Detection | Re-id
PRW |  Recall |  AP |  mAP |  top-1
$P_{3}$ |  94.8 |  92.9 |  45.9 |  81.9
$P_{4}$ |  93.4 |  91.8 |  41.4 |  77.5
$P_{5}$ |  85.7 |  83.7 |  25.3 |  57.4
$P_{3}$, $P_{4}$ |  94.7 |  92.9 |  40.8 |  77.1
$P_{3}$, $P_{4}$, $P_{5}$ |  95.4 |  93.5 |  39.5 |  74.3
Table 1: Comparative results on CUHK-SYSU and PRW by employing different
levels of features. $P_{3}$, $P_{4}$, and $P_{5}$ are the feature maps with
strides of 8, 16, and 32, respectively.

Scale Alignment. To evaluate the effects of scale alignment, we employ feature
maps from different levels of AFA and report the results in Table 1.
Specifically, we evaluate the features from $P_{3}$, $P_{4}$, and $P_{5}$ with
strides of 8, 16, and 32, respectively. As can be observed, features from the
largest scale $P_{3}$ yield the best performance on both CUHK-SYSU and PRW,
due to the fact that they absorb different levels of features from AFA,
providing richer information for detection and re-id. Similar to FCOS, we also
evaluate the performance by assigning people of different scales to different
feature levels. We set the size ranges for {$P_{3},P_{4}\\}$ as [0, 128] and
[128, $\infty$], while the prediction ranges for {$P_{3},P_{4},P_{5}\\}$ are
[0, 128], [128, 256], and [256, $\infty$], respectively. We can see that these
dividing strategies only achieve slightly better detection results w.r.t. the
recall rate. As nearby people typically have similar scales, i.e., the person
scale changes are not that significant in the same scene, it is expected that
dividing them to different FPN layers does not significantly improve
detection. However, the dividing strategy brings back the scale misalignment
issue to person re-id, resulting in worse performance compared with single-
scale features. Also note that this issue is not well addressed with the
multi-scale training strategy. All these results demonstrate the necessity and
effectiveness of the proposed scale alignment strategy.

Lateral |  Output |  Feature | CUHK-SYSU | PRW
---|---|---|---|---
dconv |  dconv |  concat |  mAP | top-1 | mAP | top-1
|  |  |  83.4 |  83.7 |  27.4 |  62.3
$\checkmark$ |  |  |  90.6 |  90.8 |  39.4 |  74.7
|  $\checkmark$ |  |  91.4 |  91.9 |  40.6 |  77.0
|  |  $\checkmark$ |  84.0 |  84.1 |  29.1 |  62.4
$\checkmark$ |  $\checkmark$ |  |  91.8 |  92.2 |  42.8 |  80.1
$\checkmark$ |  |  $\checkmark$ |  90.7 |  91.0 |  39.6 |  75.3
|  $\checkmark$ |  $\checkmark$ |  92.0 |  92.5 |  40.7 |  77.5
$\checkmark$ |  $\checkmark$ |  $\checkmark$ |  93.1 |  93.4 |  45.9 |  81.9
Table 2: Comparative results on CUHK-SYSU and PRW by employing different components in AFA for region alignment. “dconv” stands for deformable convolution.  Dconv |  FLOPS |  Time | CUHK-SYSU | PRW
---|---|---|---|---
|  (G) |  (ms) |  mAP | top-1 | mAP | top-1
w/o |  298 |  57 |  84.0 |  84.1 |  29.1 |  62.4
w/ |  308 |  61 |  93.1 |  93.4 |  45.9 |  81.9
Table 3: Comparasion of runtime, FLOPS, and performance with or without
deformable convolution (dconv).

Region Alignment. We conduct experiments with different combinations of
lateral deformable conv, output deformable conv and feature concatenation, and
analyze how different region alignment components influence the overall
performance. The results are reported in Table 2. Without all these modules,
the framework only achieves 83.4% and 27.4% in mAP on CUHK-SYSU, and PRW,
respectively, which is 9.7% and 18.5% lower than the full model. The
individual components of lateral deformable conv and output deformable conv
improve the model by $\sim$7% and $\sim$8% on CUHK-SYSU, respectively. Feature
concatenation also brings $\sim$1% improvements. By combining two of the three
components, we observe consistent improvements. Finally, employing all the
three modules yields 93.1% in mAP and 93.4% in top-1 accuracy on CUHK-SYSU,
significantly boosting the performance. On PRW, we also observe consistent
improvement by introducing these strategies. These ablation studies thoroughly
demonstrate the effectiveness of region alignment.

Deformable convolutions inevitably bring in additional computation. To
investigate how it affects the efficiency, we compare the runtime and FLOPS in
Table 3. As can be observed, the deformable convolution only requires limited
additional computation (10 GFLOPS) and runs slightly slower (4ms), but it
improves the mAP by 9.1% and 16.8% on CUHK-SYSU and PRW, respectively.

To further illustrate how the deformable convolutions work in our framework,
we visualize the learned offsets of the deformable filters in Fig. 7. We
observe that the proposed framework is capable of learning adaptive receptive
field according to the layout of the human body, and is robust to occlusion,
crowding, and scale/illumination variations. Specifically, in the last column,
when there are a crowd of people with severe occlusion, our framework can
still accurately locate most of the people. We also observe that the lateral
deformable conv in $C_{3}$ learns tighter offsets around the body center,
while the offsets in the $C_{4}$ layer cover larger regions, which makes the
two layers complementary to each other.

(a) $T_{1}$ (b) $T_{2}$

(c) $T_{3}$ (d) AlignPS

Figure 8: Illustration of different structures for training the detection and re-id tasks. | Detection | Re-id
---|---|---
CUHK-SYSU |  Recall |  AP |  mAP |  top-1
$T_{1}$ |  87.5 |  79.0 |  80.3 |  79.2
$T_{2}$ |  89.1 |  78.6 |  77.1 |  75.9
$T_{3}$ |  90.1 |  81.4 |  80.7 |  80.2
AlignPS |  90.3 |  81.2 |  93.1 |  93.4
| Detection | Re-id
PRW |  Recall |  AP |  mAP |  top-1
$T_{1}$ |  94.3 |  92.5 |  35.5 |  67.4
$T_{2}$ |  96.9 |  92.8 |  31.8 |  64.3
$T_{3}$ |  94.5 |  92.6 |  35.8 |  69.1
AlignPS |  94.8 |  92.9 |  45.9 |  81.9
Table 4: Comparative results on CUHK-SYSU and PRW with different training
structures.

Task Alignment. Since person search aims to simultaneously address detection
and re-id subtasks in a single framework, it is important to understand how
different configurations of the two subtasks influence the overall task and
which subtask should be paid more attention to. To this end, we design several
structures to compare different training options (as shown in Fig. 8), the
performance of which is summarized in Table 4. As can be observed, the
structures of $T_{1}$ and $T_{2}$, where re-id features are shared with the
regression and classification heads, respectively, yield significantly lower
performance in re-id compared with our design. This indicates that the
detection task takes advantage of the shared heads. As for $T_{3}$ where re-id
and detection have independent feature heads, it achieves slightly better
performance compared with $T_{1}$ and $T_{2}$, but still remarkably
underperforms our design. These results indicate that our “re-id first”
structure achieves the best task alignment among all these designs.

| CUHK-SYSU | PRW
---|---|---
Center Radius |  mAP |  top-1 |  mAP |  top-1
1 |  93.0 |  93.5 |  45.2 |  80.3
1.5 (default) |  93.1 |  93.4 |  45.9 |  81.9
2 |  92.8 |  93.3 |  45.3 |  80.4
3 |  92.7 |  93.1 |  45.6 |  81.4
5 |  92.1 |  92.9 |  45.3 |  81.0
8 |  92.0 |  92.6 |  45.1 |  80.9
box size |  91.3 |  92.0 |  42.5 |  79.4
Table 5: Comparative results on CUHK-SYSU and PRW with different center
sampling radiuses. Center sampling benifits the feature learning, and a
relatively small radius (1.5) achieves the best performance.

Center Sampling. In the detection branch, we employ the center sampling
strategy, i.e., the locations around the bounding box center are treated as
positive samples, which will generate slightly different but relatively
accurate bounding box predictions. These bounding box variations play similar
roles as data augmentation for re-id learning, making the re-id branch more
robust. However, if the center radius is too large, there would be higher
chance to produce inaccurate bounding box predictions, thus introducing noisy
samples for re-id learning. As can be observed in Table 5, the performance
consistently decreases as the center sampling radius increases, and all the
setting with center sampling performs better than that without center
sampling.

CUHK-SYSU |  mAP |  top-1 |  $\Delta$ mAP |  $\Delta$ top-1
---|---|---|---|---
OIM |  92.4 |  92.9 |  - |  -
TOIM w/o LUT |  92.8 |  93.2 |  +0.4 |  +0.3
TOIM w/ LUT |  93.1 |  93.4 |  +0.7 |  +0.5
PRW |  mAP |  top-1 |  $\Delta$ mAP |  $\Delta$ top-1
OIM |  45.7 |  81.8 |  - |  -
TOIM w/o LUT |  45.8 |  81.8 |  +0.1 |  +0
TOIM w/ LUT |  45.9 |  81.9 |  +0.2 |  +0.1
Table 6: Comparative results on CUHK-SYSU and PRW with different loss functions. ‘w/o LUT’ indicates that the features in the LUT are not calculated in the triplet loss. ‘w/ LUT’ denotes the full TOIM loss. CUHK-SYSU |  Deformable conv |  mAP |  top-1
---|---|---|---
ResNet-50 |  none |  93.1 |  93.4
ResNet-50 |  res3 |  93.5 |  93.9
ResNet-50 |  res3 & res4 |  93.5 |  94.0
ResNet-50 |  res3 & res4 & res5 |  94.0 |  94.5
PRW |  Deformable conv |  mAP |  top-1
ResNet-50 |  none |  45.9 |  81.9
ResNet-50 |  res3 |  45.8 |  81.9
ResNet-50 |  res3 & res4 |  45.9 |  82.1
ResNet-50 |  res3 & res4 & res5 |  46.1 |  82.1
Table 7: Comparative results on CUHK-SYSU and PRW with different deformable
conv layers in the backbone model. Deformable conv in the backbone layers
slightly improve the performance.

TOIM Loss. We evaluate the performance of our framework when adopting
different loss functions and report the results in Table 6. We find that
directly employing a triplet loss brings slight improvement. When employing
the items in the LUT, the TOIM improves the mAP and top-1 accuracy on CUHK-
SYSU by 0.7% and 0.5%, respectively. These two terms also slightly improve the
performance on PRW. This indicates that it is beneficial to consider the
relations between the input features and the features stored in the LUT. As
its impacts are limited, we can treat the triplet loss in TOIM as an optional
term in our framework.

Deformable Conv in the Backbone. As shown in Table 7, inserting deformable
convolutions into the backbone network has positive effects on our framework.
However, the contribution of the deformable conv layers in the backbone
network is less significant than the deformable conv layers in our AFA module,
e.g., with all the res3 & res4 & res5 deformable conv layers, only $\sim$1%
and 0.2% improvements are observed on CUHK-SYSU and PRW, respectively. These
results indicate that the proposed AFA works as the key module for successful
feature alignment.

###  4.4 Analysis of ROI-AlignPS

We analyze the effectiveness of ROI-AlignPS by answering the following
questions.

Separate | CUHK-SYSU | PRW | Inference
---|---|---|---
Training |  mAP |  top-1 |  mAP |  top-1 |  Time
ROI-Align |  92.0 |  92.3 |  43.1 |  80.7 |  83
AlignPS |  93.1 |  93.4 |  45.9 |  81.9 |  61
Concatenation |  93.0 |  93.6 |  44.5 |  81.3 |  144
Joint Training | CUHK-SYSU | PRW |  Inference
w/o RPN |  mAP |  top-1 |  mAP |  top-1 |  Time
ROI-Align |  90.4 |  91.3 |  43.9 |  81.5 |  75
AlignPS |  93.2 |  92.8 |  44.9 |  80.0 |  61
Concatenation |  94.3 |  94.8 |  48.6 |  83.2 |  75
ROI-Align∗ |  93.4 |  93.8 |  47.5 |  84.5 |  75
Joint Training | CUHK-SYSU | PRW |  Inference
w/ RPN |  mAP |  top-1 |  mAP |  top-1 |  Time
ROI-Align |  93.3 |  93.9 |  47.8 |  83.3 |  75
AlignPS |  94.0 |  94.1 |  46.4 |  81.2 |  61
Concatenation |  95.0 |  95.3 |  50.3 |  84.3 |  75
Table 8: Comparative results with different structures. ROI-Align∗ denotes the
model trained without re-id feature in the AlignPS branch, i.e., AlignPS
concentrates on the detection task while ROI-Align focuses on re-id. This also
yields an efficient anchor-free framework with fair performance. The training
pipeline of ‘Joint Training w/ RPN’ follows Fig. 5a, while the training
pipeline of ‘Joint Training w/o RPN’ follows Fig. 5b. Both these two settings
share the same inference structure shown in Fig. 5b. Employing RPN during
training improves the performance.

Is the ROI-Align branch necessary? In ROI-AlignPS, the ROI-Align branch
extracts the explicitly aligned re-id features, complementary to the
implicitly aligned features in the AlignPS branch. To evaluate the
effectiveness of this design, we test each branch under different settings.
(1) Separate Training/Test, i.e., two person search models are separately
trained and evaluated. During inference, we also concatenate the features from
the ROI-Align model (with RPN) and the AlignPS model. (2) Joint Training w/o
RPN, where the ROI-Align branch and the AlignPS branch are jointly trained,
and both the training and test pipelines follow the structure in Fig. 5b,
i.e., no RPN is employed during training and inference. (3) Joint Training w/
RPN, which denotes the proposed ROI-AlignPS framework, where the training
pipeline follows Fig. 5a, and the test pipeline follows Fig. 5b. In this case,
RPN is only employed in the ROI-Align branch during training. As shown in
Table 8, although the separately trained models both obtain promising results,
directly concatenating their re-id features does not yield better performance,
which may be because the detection results of the two models are not perfectly
aligned. In our joint training setting, in contrast, the branch-level features
are well aligned. When the ROI-Align branch is trained without RPN, although
the performance of each branch is similar to the separately trained models,
the concatenated re-id features are notably improved by 1.3% and 4.1% w.r.t.
mAP on CUHK-SYSU and PRW, respectively. Moreover, when the ROI-Align branch is
trained with RPN, the performance of this branch is further improved, which in
turn yields enhanced performance by concatenating the re-id features. These
results not only indicate that the jointly training pipeline can better
regularize features in the shared layers (i.e., res1-4), but also validate the
effectiveness and necessity of the ROI-Align branch, which is complementary to
AlignPS.

How does the ROI-Align branch influence the efficiency? In ROI-AlignPS, the
ROI-Align branch inevitably brings additional computational overhead. As shown
in Table 8, in the inference stage, it takes 75 milliseconds (ms) to process
an image with both AlignPS and ROI-Align, in comparison to 61 ms for AlignPS
only. However, this is still less than the Faster R-CNN based ROI-Align model
(83 ms), because our model does not require dense region proposals during
inference. Therefore, ROI-AlignPS achieves significant performance improvement
and remains highly efficient.

| CUHK-SYSU | PRW
---|---|---
Mutual Learning |  mAP |  top-1 |  mAP |  top-1
None |  95.0 |  95.3 |  50.3 |  84.3
$\mathcal{L}_{mi}$ |  95.4 |  96.0 |  51.6 |  84.4
Table 9: Comparative results on CUHK-SYSU and PRW with mutual learning loss
function.

(a) w/o Mutual Learning (b) w/ Mutual Information

Figure 9: The distribution of re-id features from 15 random identities. ‘*’ denotes the re-id features from the ROI-Align branch, and ‘$\cdot$’ denotes the features from AlignPS branch. Different colors represent different identities.  Figure 10:  Qualitative results of the ROI-Align branch, AlignPS branch, and ROI-AlignPS. The yellow bounding boxes denote the queries, while the green and red bounding boxes denote correct and incorrect top-1 matches, respectively. Zoom in for best visualization.  | CUHK-SYSU | PRW
---|---|---
Methods | mAP | top-1 | mAP | top-1
one-step |  OIM DBLP:conf/cvpr/XiaoLWLW17 |  75.5 |  78.7 |  21.3 |  49.4
IAN DBLP:journals/pr/XiaoXTHWF19 |  76.3 |  80.1 |  23.0 |  61.9
NPSM DBLP:conf/iccv/LiuFJKZQJY17 |  77.9 |  81.2 |  24.2 |  53.1
RCAA DBLP:conf/eccv/ChangHSLYH18 |  79.3 |  81.3 |  - |  -
CTXG DBLP:conf/cvpr/YanZNZXY19 |  84.1 |  86.5 |  33.4 |  73.6
QEEPS DBLP:conf/cvpr/MunjalATG19 |  88.9 |  89.1 |  37.1 |  76.7
HOIM DBLP:conf/aaai/ChenZO0S20 |  89.7 |  90.8 |  39.8 |  80.4
BINet DBLP:conf/cvpr/DongZST20a |  90.0 |  90.7 |  45.3 |  81.7
NAE DBLP:conf/cvpr/ChenZYS20 |  91.5 |  92.4 |  43.3 |  80.9
NAE+ DBLP:conf/cvpr/ChenZYS20 |  92.1 |  92.9 |  44.0 |  81.1
PGA Kim_2021_CVPR |  92.3 |  94.7 |  44.2 |  85.2
DKD DBLP:conf/aaai/ZhangWBSY21 |  93.1 |  94.2 |  50.5 |  87.1
SeqNet DBLP:conf/aaai/LiM21 |  93.8 |  94.6 |  46.7 |  83.4
SeqNet+CBGMDBLP:conf/aaai/LiM21 |  94.8 |  95.7 |  47.6 |  87.6
AlignPS |  93.1 |  93.4 |  45.9 |  81.9
|  ROI-AlignPS${{\color[rgb]{0,0,0}{(128)}}}$ |  94.5 |  95.2 |  50.7 |  84.0
|  ROI-AlignPS${(256)}$ |  95.4 |  96.0 |  51.6 |  84.4
two-step |  DPM+IDE DBLP:conf/cvpr/ZhengZSCYT17 |  - |  - |  20.5 |  48.3
CNN+MGTS DBLP:conf/eccv/ChenZOYT18 |  83.3 |  83.9 |  32.8 |  72.1
CNN+CLSA DBLP:conf/eccv/LanZG18 |  87.2 |  88.5 |  38.7 |  65.0
FPN+RDLR DBLP:conf/iccv/HanYZTZGS19 |  93.0 |  94.2 |  42.9 |  70.2
IGPN DBLP:conf/cvpr/DongZST20 |  90.3 |  91.4 |  47.2 |  87.0
OR DBLP:journals/tip/YaoX21 |  92.3 |  93.8 |  52.3 |  71.5
|  TCTS DBLP:conf/cvpr/WangMCSC20 |  93.9 |  95.1 |  46.8 |  87.5
Table 10: Comparison with state-of-the-art methods. The upper block lists the
results of one-step models, while the lower block shows the results of two-
step methods. Figure 11: Visualization of some difficult cases in person
search datasets. The yellow bounding boxes denote the queries, while the green
and red bounding boxes denote correct and incorrect top-1 matches,
respectively.

Are the mutual learning strategies effective? By applying the mutual learning
strategy, we observe 0.4% and 1.3% performance gains in mAP on the two
datasets, as shown in Table 9. Furthermore, we visualize the distribution of
re-id features based on t-SNE. As shown in Fig. 9, although the features
generated from the two branches are generally separable before mutual
learning, there exist several non-separable classes. After mutual learning,
the non-separable cases are reduced, and the features are visually better
distributed.

To further investigate how the two branches reinforces each other, we show
some qualitative results in Fig. 10. We observe that the AlignPS is more
sensitive in extreme cases where the query and targets have a large scale
variation, while the ROI-Align branch can be confused when the query and
target share a similar appearance. As the sensitivities of these two branches
are different, they are complementary to each other in certain difficult
cases. As can be observed, when one of the branches or both branches give
inaccurate prediction, the complementary property yields more robust final
results.

Figure 12: Comparative results on CUHK-SYSU with different gallery sizes. Our
models (AlignPS and ROI-AlignPS) are compared with both one-step models and
two-step models.

###  4.5 Comparison to State-of-the-Art Methods

We compare our framework with state-of-the-art one-step models
DBLP:conf/cvpr/XiaoLWLW17; DBLP:journals/pr/XiaoXTHWF19;
DBLP:conf/iccv/LiuFJKZQJY17; DBLP:conf/eccv/ChangHSLYH18;
DBLP:conf/aaai/ChenZO0S20; DBLP:conf/cvpr/MunjalATG19;
DBLP:conf/cvpr/DongZST20a; DBLP:conf/cvpr/ChenZYS20; Kim_2021_CVPR;
DBLP:conf/aaai/ZhangWBSY21; DBLP:conf/aaai/LiM21 and two-step models
DBLP:conf/eccv/ChenZOYT18; DBLP:conf/eccv/LanZG18; DBLP:conf/iccv/HanYZTZGS19;
DBLP:conf/cvpr/DongZST20; DBLP:conf/cvpr/WangMCSC20.

| PICWD | Webtattoo
---|---|---
Methods |  mAP |  top-1 |  mAP |  top-1
DC-I-Net DBLP:journals/pami/ZhangHYWG22 |  - |  - |  48.5 |  51.0
NAE DBLP:conf/cvpr/ChenZYS20 |  41.3 |  65.1 |  45.3 |  47.6
NAE+ DBLP:conf/cvpr/ChenZYS20 |  42.6 |  66.5 |  46.9 |  48.7
SeqNet DBLP:conf/aaai/LiM21 |  44.7 |  68.2 |  51.2 |  53.4
AlignPS |  47.3 |  70.1 |  51.8 |  53.7
ROI-AlignPS(128) |  48.8 |  70.5 |  54.0 |  56.6
ROI-AlignPS(256) |  49.5 |  71.3 |  54.6 |  57.3
Table 11: Comparison with state-of-the-art methods on PICWD and Webtattoo
dataset.

Results on CUHK-SYSU. As ROI-AlignPS concatenate features from two branches,
we evaluate it with both 256- and 128-dimensional features for a fair
comparision. The results are shown in Table 10. We observe that both ROI-
AlignPS(256) and ROI-AlignPS(128) outperform most previous SOTA models,
including TCTS, SeqNet, DKD, and NAE+. These results demonstrate that the
improvements are achieved mainly due to the two-branch structure, rather than
the increase in feature dimension. In the following, we make ROI-AlignPS(256)
as our default setting. Note that SeqNet requires a graph matching strategy as
post-processing to achieve its best performance, while ROI-AlignPS does not
need such a process. We also observe from the table that our model outperforms
all the two-step models, even though they employ two separate models for
detection and re-id. In contrast, our model allows joint inference with a very
simple structure, whilst running at a higher speed.

Figure 13: Visualization of some difficult cases in PICWD and Webtattoo
datasets. The yellow bounding boxes denote the queries, while the green and
red bounding boxes denote correct and incorrect top-1 matches, respectively.

We visualize the results of AlignPS and ROI-AlignPS w.r.t. mAP with various
gallery sizes and compare our model with both one-step and two-step models.
Fig. 12 illustrates the detailed results, where ROI-AlignPS outperforms all
the models by notable margins, in terms of all the gallery sizes.

Results on PRW. Compared to CUHK-SYSU, PRW contains fewer training samples and
has a larger gallery size, thus it is more challenging. Nevertheless, as can
be observed from Table 10, ROI-AlignPS still outperforms all the one-step
models in terms of mAP. In the meantime, we notice that SeqNet
DBLP:conf/aaai/LiM21, DKD DBLP:conf/aaai/ZhangWBSY21, PGA Kim_2021_CVPR, IGPN
DBLP:conf/cvpr/DongZST20, and TCTS DBLP:conf/cvpr/WangMCSC20 achieve higher
top-1 accuracy on PRW. A high top-1 accuracy only requires one instance of the
same class to be top-ranked, while a good mAP requires all instances of the
same class to have high similarities with the query. Therefore, top-1 accuracy
only cares about the easiest sample of each class, while mAP depends on all
the samples, including the hard ones. In this sense, mAP is a more general
metric than top-1 in the retrieval task.

Moreover, it is worth noting that SeqNet achieves strong top-1 accuracy with a
graph matching strategy (CBGM) as its post-processing strategy; otherwise, its
top-1 accuracy is 83.4%. This strategy improves the top-1 by 4.2%, while only
improving the mAP by 0.9%, which demonstrates that the post-processing
strategy cares more about achieving top-1 for the easiest samples of each
class. The other methods also seek to address the feature misalignment issue,
but they either resort to feature distillation from the cropped instances
DBLP:conf/aaai/ZhangWBSY21; DBLP:conf/cvpr/DongZST20;
DBLP:conf/cvpr/WangMCSC20, or apply attention mechanisms Kim_2021_CVPR.
Differently, our model efficiently addresses this issue with the proposed AFA
module, without post-ranking or additional supervision. The alignment
strategies in ROI-AlignPS make it more robust to difficult examples, thus
achieving strong mAP while with compromised top-1 accuracy under limited
training samples. It is also noteworthy that the strategies in
DBLP:conf/aaai/ZhangWBSY21; DBLP:conf/cvpr/DongZST20;
DBLP:conf/cvpr/WangMCSC20; Kim_2021_CVPR are compatible with our framework. We
will try to improve the top-1 accuracy on PRW with these strategies in our
future work.

Results on PICWD and Webtattoo. As PICWD and Webtatto only annotates the small
dorsal fin and the tattoo area, it is more challenging to locate and identify
the Chinese dolphins and tattoos. By comparing our framework with DC-I-Net
DBLP:journals/pami/ZhangHYWG22, NAE DBLP:conf/cvpr/ChenZYS20 and SeqNet
DBLP:conf/aaai/LiM21 in Table 11, we obeserve that both AlignPS and ROI-
AlignPS consistently achieve higher performance, which makes it a more general
framework for animal and object search.

Methods |  Train Anchor |  Test Anchor |  GPU |  Time (ms)
---|---|---|---|---
PGA Kim_2021_CVPR |  $\checkmark$ |  $\checkmark$ |  Titan X |  356
DKD DBLP:conf/aaai/ZhangWBSY21 |  $\checkmark$ |  $\checkmark$ |  1080 Ti |  124
OIM DBLP:conf/cvpr/XiaoLWLW17 |  $\checkmark$ |  $\checkmark$ |  V100 |  118
NAE+ DBLP:conf/cvpr/ChenZYS20 |  $\checkmark$ |  $\checkmark$ |  V100 |  98
NAE DBLP:conf/cvpr/ChenZYS20 |  $\checkmark$ |  $\checkmark$ |  V100 |  83
SeqNet DBLP:conf/aaai/LiM21 |  $\checkmark$ |  $\checkmark$ |  V100 |  86
AlignPS |  $\times$ |  $\times$ |  V100 |  61
ROI-AlignPS |  $\checkmark$ |  $\times$ |  V100 |  75
Table 12: Runtime comparison of different models. Both AlignPS and ROI-AlignPS are anchor-free during inference. Query Number |  IGPN DBLP:conf/cvpr/DongZST20 |  QEEPS DBLP:conf/cvpr/MunjalATG19 |  AlignPS |  ROI-AlignPS
---|---|---|---|---
1 |  6s |  30s |  7s |  8s
10 |  60s |  300s |  8s |  9s
100 |  600s |  3000s |  14s |  17s
1000 |  6000s |  30000s |  77s |  93s
Table 13: Runtime comparison with query-guided models.

Efficiency Comparison. Since different methods are evaluated with different
GPUs, it is difficult to conduct a fair comparison of the efficiency among all
the models. Here, we compare our method with OIM222We test the PyTorch
implementation at https://github.com/serend1p1ty/person_search.
DBLP:conf/cvpr/XiaoLWLW17, NAE/NAE+ DBLP:conf/cvpr/ChenZYS20 and SeqNet
DBLP:conf/aaai/LiM21 on the same Tesla V100 GPU. All the test images are
resized to 1500$\times$900 before being fed to the networks. As shown in Table
12, our anchor-free AlignPS only takes 61 ms to process an image, which is 27%
and 38% faster than NAE and NAE+, respectively. Meanwhile, ROI-AlignPS also
runs faster than prior state-of-the-art models DBLP:conf/cvpr/ChenZYS20;
DBLP:conf/aaai/LiM21. This is because our models are anchor-free during
inference, validating the advantage of the proposed framework. For query-
guided frameworks, e.g., IGPN DBLP:conf/cvpr/DongZST20 and QEEPS
DBLP:conf/cvpr/MunjalATG19, they need to re-compute all the gallery features
given each query. As our framework only computes the gallery features once,
the total computation of these models can be much slower compared to AlignPS
when there are a large number of queries, as shown in Table 13. In practice,
IGPN is more applicable for a small set of queries, while AlignPS and ROI-
AlignPS are efficient for both small and large number of queries.

Qualitative Results. Some qualitative results on person search are illustrated
in Fig. 11, while the animal and tattoo search results are illustrated in Fig.
13. We observe that AlignPS and ROI-AlignPS are more robust in handling
occlusions, scale and viewpoint variations, where OIM, NAE, and SeqNet fail.
Some failure cases can be observed in distinguishing very tiny objects or the
objects that share similar appearances, we will further work towards this
direction in our future work.

##  5 Conclusion and Future Work

In this paper, we propose an anchor-free approach to efficiently tackling the
task of person search, by developing a framework based on an anchor-free
detector. We design the aligned feature aggregation module to effectively
address the scale, region, and task misalignment issues when accommodating the
detector for the person search task. Furthermore, we propose to augment our
anchor-free model with an ROI-Align branch, which additionally takes advantage
of the anchor-based models. Extensive experiments on person search, animal
search, and object search demonstrate that the proposed framework not only
outperforms existing person search methods but also runs at a higher speed.

There are several interesting directions to be explored in our future work.
First, in this work, we only explore FCOS, a special detection model. It would
be interesting to investigate a more general framework that is compatible with
other detectors for person search. Second, although our two-branch framework
is effective, it lacks elegance. We will try to design a unified way to
perform explicit and implicit feature alignment. Third, transformer-based
frameworks have demonstrated promising results in detection and recognition,
we will explore it for person search. Finally, the top-1 accuracy of our
framework is slightly inferior to several recent models on PRW. We will try to
improve it with more advanced strategies under limited training samples.

###### Acknowledgements.

This paper was supported by NSFC (62201342, U19B2035), Shanghai Municipal
Science and Technology Major Project (2021SHZDZX0102).