11institutetext:  Department of Computer Science, UNC Charlotte
22institutetext: Shanghai Key Laboratory of Intelligent Information Processing
33institutetext: School of Computer Science, Fudan University
33email<EMAIL_ADDRESS>33email<EMAIL_ADDRESS>
# Cluster-level Feature Alignment
for Person Re-identification

Qiuyu Chen 11 Wei Zhang 2233 Jianping Fan 11

###### Abstract

Instance-level alignment is widely exploited for person re-identification,
_e.g_. spatial alignment, latent semantic alignment and triplet alignment.
This paper probes another feature alignment modality, namely cluster-level
feature alignment across whole dataset, where the model can see not only the
sampled images in local mini-batch but the global feature distribution of the
whole dataset from distilled anchors. Towards this aim, we propose anchor loss
and investigate many variants of cluster-level feature alignment, which
consists of iterative aggregation and alignment from the overview of dataset.
Our extensive experiments have demonstrated that our methods can provide
consistent and significant performance improvement with small training efforts
after the saturation of traditional training. In both theoretical and
experimental aspects, our proposed methods can result in more stable and
guided optimization towards better representation and generalization for well-
aligned embedding.

###### Keywords:

Person Re-Identification, Feature Alignment, Metric Learning

##  1 Introduction

Person re-identification (ReID) is an essential component of intelligent
computer vision systems. It has drawn increasing interest in many
applications, such as surveillance, activity analysis and long-term tracking.
Given an image of a person-of-interest captured by one camera, the goal is to
re-identify this person from images captured by multiple cameras without
overlapping viewpoints. As an instance-level recognition problem, the ReID
task is inherently challenging. First, intra-class variations are typically
huge due to significant changes of visual appearances caused by camera viewing
conditions, human pose variations, occlusions, _et al_. Second, the inter-
class variations can be quite small because people may wear similar clothes.

Figure 1: From the view of alignment modality: instance-level alignment (left)
and cluster-level alignment (right).

To address these challenges of intra-class diversity and inter-class
inseparability, a lot of efforts have been devoted to deep learning for its
strong capability on discriminative feature extraction. Most of existing
methods put the training process of person ReID under classification
framework, where intermediate features are extracted to compute the similarity
between query and gallery images during test. Some tailor-made neural networks
are proposed to incorporate localization/attention or disentanglement for
feature alignment. The former one aligns features in 2D spatial dimension and
the latter one targets latent semantic alignment, but the essence of those
approaches is instance-level alignment (_c.f_. Fig. 1 left). In addition, a
large variety of loss functions have been proposed for metric learning in
person ReID. For example, the two most prevalent loss functions are
classification loss, _e.g_. cross entropy loss, and metric-learning based
loss, _e.g_. hard-negative mining triplet loss [9]. For those advanced
networks driven by such popular loss definitions, although successful, we
argue that they can still be categorized as instance-level alignment (_c.f_.
Fig. 1 left). The intrinsic reason of such constraint is attributed to the
adoption of general classification framework, where the interaction it builds
can only dwell within the sampled mini-batch but cannot see more neighbors in
the distribution of the whole dataset. As a result, it inhibits the growth of
intra-class compactness and inter-class separability.

Aiming to break through the aforementioned limitations and step beyond the
instance-level feature alignment, we propose a succinct and efficient method
to enable cluster-level interaction in feature space, targeting the alignment
from an overview of latent feature distribution (_c.f_. Fig. 1 right). ReID is
in essential a metric learning problem. When projected to the learned feature
space, feature points are expected to gather into compact clusters respecting
their labels and such cluster-level interaction may inhabit better formulation
of the clusters. We define the center of each feature cluster as the anchor.
In a computational efficient manner, the anchors generated from aggregation
serve as a supervision from the distribution of whole dataset and enable the
model to see other training images in the dataset indirectly. In practice,
after the saturation of traditional training, we manipulate two iterative
steps to further intensify the cluster compactness: (a) Aggregate cluster
features across dataset, stepping beyond the limitation of a classification
framework; (b) Align features under the guidance from aggregated anchors. We
claim such cluster-level feature alignment is much more promising for
identity-related representation.

Figure 2: From the view of optimization: Anchor loss provides more stability
and consistency for optimization. (a) Classification loss pulls feature sample
$f_{i}$ towards the corresponding classifier vector $p_{j}$ in fully connected
layer; (b) Triplet loss probes the interaction within sampled mini-batch
(denoted as solid color); (c-d) Anchor loss enables the sampled mini-batch to
see the anchors aggregated from all the siblings through iterative
aggregation(c) and alignment(d).

Besides the view of feature alignment modality (_c.f_. Fig. 1), the proposed
method, called anchor loss, provides consistent optimization for metric
learning which benefits training as well as generalization process. The
classification loss tries to align the features in orders according to the
classification labels. Specifically, the inner product
$\textbf{f}^{\top}_{i}\textbf{p}_{j}$ between classifier $p_{j}$ in the full-
connected layer and the feature vector $f_{i}$ is increased as $p_{j}$ and
$f_{i}$ pulling towards each other (_c.f_. Fig. 2(a)). It has promising
convergence but unnecessary penalize the intra-class variance if classifier
diversifies channel-wise focus on decision. Also, it handles training mini-
batch samples by simply averaging individual losses, thus can only build
sample connection from the identity implicitly. Triplet loss [9] tries to
align features in more explicitly way. It optimizes the intra-class and inter-
class distance by mini-batch interaction with proper sampling. From Fig. 2(b),
we can see that the optimization direction of triplet loss is highly dependent
on the mini-batch sampling and inevitably introduces uncertainty and
inconsistency. On the other hand, anchor loss enables the sampled mini-batch
to see the anchors $\textbf{a}_{j}$ aggregated from all the siblings, bearing
more consistency (_c.f_. Fig. 2(c-d)). Anchors generated from aggregation
provide strong guidance and propagate the global information from the
distribution of dataset to local mini-batch training. Based on extensive
experiments, we demonstrate that anchor loss can consistently boost the
generalization.

Overall, in this proposed paper, we learn a metric to overcome intra-class
diversity and inter-class confusion for person ReID based on anchor-based min-
batch training. Although each mini-batch of samples is a small subset of the
dataset, we can successfully capture the global information during training
through anchors which play key roles in the proposed cluster-level feature
alignment. A small number of representative anchors propagate rich knowledge
from the distribution of dataset into local training batch in a computational
efficient manner.

##  2 Related Work

Person Re-Identification A large group of person re-identification network
focuses on feature alignment. In general, there are two kinds of feature
alignment: (a) spatial feature alignment by attention and localization (b)
latent feature alignment by disentanglement.

In spatial feature alignment, it can be categorized as self-supervised and
extra-supervised methods. We consider hand-crafted splitting as one
representation of self-supervision. Sun _et al_. [26] propose PCB to split the
intermediate features horizontally in order to align the feature in local
spatial parts and is widely used by [6, 7, 19, 25, 40]. Quite some works [11,
5, 22, 31, 12] proposed similar and effective part-aligning CNN networks for
locating context regions and then extract these regional features for ReID.
Extra-supervision leverages human part detector [8], human pose [15] or human
body parsing [13] to provide more accurate localization. For example, [17, 21,
23, 35] incorporate external pose attention maps to align the feature in
deformable spatial space of human body. SPReID [13] utilizes a parsing model
to generate five different predefined human part masks to compute more
reliable part representations, which achieves promising results on various
person ReID benchmarks. Dense semantic alignment [38] went one step further,
it addressed the body misalignment by leveraging the estimation of the dense
semantics of a person image, and constructed a set of densely semantically
aligned part images for re-identification. Other methods for spatial alignment
include the attention from attribute [27], forground mask [28], _et al_.

For latent feature alignment, DG-net [43] proposed a disentanglement solution
by GAN and Autoencoder to decouple input into appearance code and structure
code, and extract pose-invariant features. Whereas, there has been a growing
interest in using generative models to augment training data and enhance the
invariance to input changes [39, 34, 36].

Metric Learning: Center Loss and Triplet Loss In end-to-end learning process,
several methods propose to explore iteration within mini-batch for feature
alignment . Zheng _et al_. [44] propose a verification loss to align the
pairwise features. Hermans _et al_. [9] target the triplet samples and point
out the triplet loss on hard examples mining is superior to batch-all triplet
loss. In face recognition, parametric center loss [32] is proposed to align
the intra-class distance and it is used by [16] for person reID, which looks
similar to our proposed anchors. Whereas, our motivation is essentially
different. Center loss treats the parametric center as an auxiliary decision
factor similar to the classifier $p_{j}$ in Fig. 2(a), which is jointly
optimized under classification framework and only builds the connection from
identity label implicitly. On the contrary, our proposed anchor loss complies
the embedded feature distribution and distills the knowledge from sibling
samples (_c.f_. Fig. 2(c)) to enable the interaction in cluster level
explicitly. More recently, Wen _et al_. [33] revisit center loss and propose
to use the classifier layer as the center for each class, which further
validates our motivation difference. Moreover, it suffers from large
instability due to random initialization for parametric center. On the other
hand, our method provides constant improvement because the aggregation
distills knowledge from dataset distribution.

In summary, those methods never push towards the constraint of classification
framework in mini-batch training and scrutinize the modality in cluster-level
feature alignment. Probably due to the concern about training efficiency,
cross-dataset aggregation is not fully investigated in deep CNN methods.
However, we conduct a comprehensive study on different variants of the
cluster-level interaction, which has demonstrated our method could be trained
effectively and efficiently with small training efforts.

##  3 Proposed Method

Person ReID aims to establish the identity correspondences between each query
image and gallery images across different cameras. We use Convolutional neural
network (CNN) to extract image features due to its strong representation
power. To learn discriminative representation that is robust against intra-
class variation and interclass confusion, we take advantage of three different
loss functions to train the model: (1) cross-entropy classification loss
$L_{cls}$ (Fig. 2(a)); (2) triplet loss $L_{trip}$ (Fig. 2(b)); (3) anchor
loss $L_{anchor}$ (Fig. 2(d)). $L_{cls}$ and $L_{trip}$ [9] are widely used
for person ReID, and they are both instance-level optimization as illustrated
in Fig. 2(a-b). To propagate rich knowledge from outside samples into each
mini-batch, the designed anchor loss $L_{anchor}$ targets cluster-level
supervision which has two options:

(a) Anchor Loss for Intra-Class Compactness: $L_{anchor}$ pulls the feature
vector towards the anchor where its label belongs:

$L_{anchor}=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\sum_{j=1}^{C}\delta(y_{i}=j)D(\textbf{f}_{i},\textbf{a}_{j})$
(1)

where $|\mathcal{B}|$ denotes the number of samples in the mini-batch
$\mathcal{B}$, and $C$ is the number of classes. $\textbf{f}_{i}$ and $y_{i}$
are the feature vector and the label of the sample $i$ in $\mathcal{B}$,
respectively. $\textbf{a}_{j}$ is the anchor for the $j-$th class.
$D(\textbf{f}_{i},\textbf{a}_{j})$ is the distance between the sample
$\textbf{f}_{i}$ and the anchor $\textbf{a}_{j}$. $\delta(y_{i}=j)=1$ if the
condition is satisfied, _i.e_. the label of sample $i$ in the mini-batch
equals $j$; otherwise, $\delta(y_{i}=j)=0$.

(b) Triplet Anchor Loss for Intra-Class Compactness $\&$ Inter-Class
Separability: Motivated by hard sample mining [9], we add extra inter-class
penalty to target the hard/confused anchor mining:

$L_{TripletAnchor}=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\sum_{j=1}^{C}\delta(y_{i}=j)[D(\textbf{f}_{i},\textbf{a}_{j})-\underset{k\neq
j}{min}D(\textbf{f}_{i},\textbf{a}_{k})+margin]$ (2)

Triplet anchor loss not only pulls the samples to the anchor in the same class
close, but also pushes the negative samples further away than the distance
between anchor and positive samples. It could be more discriminative by
simultaneously taking into account intra-class compactness and inter-class
separability. On the other hand, it may import inconsistency during the
optimization in a similar way as triplet loss (_c.f_. Fig. 2(b)).

###  3.1 Two-Staged Training

Figure 3: Two training stages of the proposed metric learning framework. Stage
I (left): instance-level alignment with $L_{cls}+L_{trip}$. Stage II (right):
feature aggregation respecting class label to generate anchors, and cluster-
level alignment with $L_{cls}+L_{anchor}$.

As shown in Fig. 2, anchors generated from aggregation contribute more
consistent optimization and less noisy guidance during training process.
However, it is based on the assumption that distribution of embedded features
is approximately cluster-formed. During the early training phase when the
feature distribution is still random and stochastic, such aggregated anchors
may contain misleading information and impair the training process. The
cluster-level supervision could be more effective after the saturation of
traditional training stage, and therefore our training consists of two stages:

Stage I: Train the model in the traditional manner with loss function
$L=L_{cls}+L_{trip}$, cultivating the initial formulation of clusters (_c.f_.
Fig. 3(left));

Stage II: Train the model under the cluster level supervision with loss
function $L=L_{cls}+L_{anchor}$, capturing distribution of whole dataset in
embedded feature space (_c.f_. Fig. 3(right)).

###  3.2 Generation and Update of Anchors

During the learning process, we take the anchors as the global supervision
from data distribution to align feature towards a better representative
embedding in local mini-batch training, _i.e_. pushing towards the target
anchor (Eq. 1) and pulling samples away from the confusion anchor (Eq. 2).
Aggregation and alignment are iteratively performed to further reduce the
intra-class variance and inter-class entanglement.

####  3.2.1 Generation

When the latent features are extracted over training dataset respecting their
labels, we consider two approaches to estimate optimal anchors during
aggregation:

(a) Average aggregation: When the embedded features are approximately cluster-
formed, aggregate the embedded features $f_{i}$ for each class in training
dataset $\mathcal{T}$:

$\textbf{a}_{j}=\frac{\sum_{i\in\mathcal{T}}\delta(y_{i}=j)\textbf{f}_{i}}{\sum_{i\in\mathcal{T}}\delta(y_{i}=j)}$
(3)

(b) Voting by confidence: Taking the prediction probability $P(j|i)$ for class
$j$ as the contribution, aggregate the embedded features by a weighted mean:

$\textbf{a}_{j}=\frac{\sum_{i\in\mathcal{T}}\delta(y_{i}=j)P(j|i)\textbf{f}_{i}}{\sum_{i\in\mathcal{T}}\delta(y_{i}=j)P(j|i)}$
(4)

Eq. 3 treats each sample’s contribution to anchor equally and could be an
effective estimation to eliminate the variance of latent feature distribution
caused by pose, camera view condition, background, _et al_. It may work well
supposing the training samples are equally distributed in terms of variance.
Eq. 4 takes the classifier confidence as the contribution and help to
revealing the early portrait of anchors. Intuitively, when the feature cluster
distribution is still stochastic in the early training stage, easy samples,
which may contains less noise and variance thus converges faster, could be
close to optimal anchors and guide the hard samples moving towards the
estimated optimal centers.

for  _$i\leftarrow 1$ to StartEpoch $E_{start}$ _ do

Update model parameter $W$ by $L=L_{trip}+L_{cls}$;

Update $\textbf{c}_{j}$ for each class $j$;

for  _$i\leftarrow E_{start}$ to EndEpoch $E_{end}$ _ do

for  _each iteration_ do

Update model parameter $W$ by $L=L_{anchor}+L_{cls}$;

Algorithm 1 Fix Anchors

for  _$i\leftarrow 1$ to StartEpoch $E_{start}$ _ do

Update model parameter $W$ by $L=L_{trip}+L_{cls}$;

for  _$i\leftarrow E_{start}$ to EndEpoch $E_{end}$ _ do

for  _each iteration_ do

Update model parameter $W$ by $L=L_{anchor}+L_{cls}$;

Update $\textbf{c}_{j}$ for each class $j$;

Algorithm 2 Update Anchors Each Epoch

for  _$i\leftarrow 1$ to StartEpoch $E_{start}$ _ do

Update model parameter $W$ by $L=L_{trip}+L_{cls}$;

for  _$i\leftarrow E_{start}$ to EndEpoch $E_{end}$ _ do

for  _each iteration_ do

Update model parameter $W$ by $L=L_{anchor}+L_{cls}$;

Update $\textbf{c}_{j}$ for each class $j$;

Algorithm 3 Update Anchors Each Iteration

####  3.2.2 Update Frequency

Ideally, we should calculate anchors by either Eq. 3 or Eq. 4 after each
forward and backward process when the training parameters are updated.
However, such a process is unrealistic in terms of training efficiency and
thus we consider three options for the update frequency:

(a) Constant (Algorithm 1): When the model is trained until initial
convergence, the cluster-level feature aggregation is calculated and serves as
the fixed anchors during the following fine-tuning process;

(b) Each Epoch (Algorithm 2): When the model is trained until some epoch,
$E_{start}$, the anchors $\textbf{c}_{j}$ are updated after each following
training epoch by either Eq. 3 or Eq. 4;

(c) Each Iteration (Algorithm 3): In each iteration trained with anchor loss,
the anchor for class $j$ is updated as:

$\textbf{a}_{j}^{t+1}=[1-\eta\cdot\sum_{i\in\mathcal{B}}\delta(y_{i}=j)]\cdot\textbf{a}_{j}^{t}+\eta\cdot\sum_{i\in\mathcal{B}}\delta(y_{i}=j)\textbf{f}_{i}$
(5)

where $\textbf{a}_{j}^{t}$ and $\textbf{a}_{j}^{t+1}$ are the anchors for
class $j$ at the $t-$th and $t+1-$th iterations respectively. To approximate
the anchors calculated per iteration, we design the weight as
$\eta=\frac{1}{\sum_{i\in\mathcal{T}}\delta(y_{i}=j)}$ where $\mathcal{T}$ is
the total training dataset.

Option (a) take the anchors calculated from the initial convergence as the
optimal one and only one aggregation process is performed. It is based on the
observation that the clusters are almost formed after the convergence of Stage
I training and thus may provide stable optimization. Option (b) adaptively
updates the anchors after each training epoch in a manner similar to EM
optimization: Estimate the anchor location according to the current feature
cluster distribution in aggregation step and maximize the cluster compactness
in alignment step. Option (c) is a trade-off option between training
efficiency and adaptive estimation for anchors, which can be viewed as an
approximate approach to option (b). From our experiments demonstrated later,
we show this method could have comparable performance with option (b) and thus
provides an alternative when the size of training samples are large or in the
context of online learning.

##  4 Experiments and Analysis

####  4.0.1 Experiment Setup

We adopt the bag of tricks proposed by [16], _i.e_. warm-up learning rate
scheduler, random erasing augmentation [45], label smoothing, no stride down-
sampling in last bottleneck of ResNet50 and bnneck (one additional batch
normalization layer after classifier). We use $L_{2}$ distance for the anchor
loss and its variants, which benefits stable training. We experiment our
methods on three datasets, Market1501 [41], DukeMTMC-ReID [20] and CUHK03
[14]. Market-1501 have 12,936 training images with 751 different identities.
Gallery and query sets have 19,732 and 3,368 images respectively with another
750 identities. DukeMTMC-ReID includes 16,522 training images of 702
identities, 2,228 query and 17,661 gallery images of another 702 identities.
CUHK03-NP is a new training-testing split protocol for CUHK03, it contains two
subsets which provide labeled and detected (from a person detector) person
images. The detected CUHK03 set includes 7,365 training images, 1,400 query
images and 5,332 gallery images. The labeled set contains 7,368 training,
1,400 query and 5,328 gallery images respectively. The new protocol splits the
training and testing sets into 767 and 700 identities. We note that our method
is only used during the training stage and the evaluation methods stay the
same with previous approaches.

Firstly, we present the experimental comparison without triplet loss to
analyze three factors: starting time, aggregation methods and anchor loss
functions. Secondly, we delve into the effects of aggregation anchors from a
reconstruction experiments. Thirdly, the comparison, when triplet loss is
incorporated in the Stage I training until convergence, will be further
analysed. Lastly, we demonstrate the advantages of our method comparing to the
parametric center loss [32].

###  4.1 Ablation Study for Three Factors

From the experimental results in Table 1, we conclude the impacts of three
factors:

(a) When to start Attributed to better cluster distribution, stating
aggregation and alignment after the convergence of initial training results in
better generalization. From Fig. 4, one can see the anchors change rapidly
during the early training phase. The transition becomes steady as the training
process towards saturation. It is in line with our analysis that anchor loss
may impose unexpected prior and abet densely distributed clusters when applied
early during training, impairing the generalization consequently.

$E_{start}$ | Rank@1 | mAP
---|---|---
$L_{Anchor}(f,y,a_{avg})$ | $L_{Anchor}(f,y,a_{weighted})$ | $L_{TripletAnchor}(f,y,a_{avg})$ | $L_{Anchor}(f,y,a_{avg})$ | $L_{Anchor}(f,y,a_{weighted})$ | $L_{TripletAnchor}(f,y,a_{avg})$
- | 93.79% | 93.79% | 93.79% | 84.69% | 84.69% | 84.69%
0 | 93.85% | 94.06% | 83.86% | 83.93% | 84.95% | 67.48%
10 | 93.32% | 93.29% | 93.29% | 83.77% | 83.59% | 83.65%
40 | 93.97% | 93.91% | 94.09% | 85.49% | 85.45% | 85.43%
70 | 94.09% | 94.09% | 94.15% | 85.75% | 85.89% | 85.81%
120 | 94.18% | 94.03% | 94.09% | 85.98% | 85.96% | 85.90%

Table 1: Ablation study for three factors on Market1501 dataset: starting
epoch $E_{start}$, aggregation methods (Eq. 3&Eq. 4) and anchor loss
choices(Eq. 1&Eq. 2). $f$ and $y$ denote the extracted feature and its label.
Before $E_{start}$ $L=L_{cls}$ is used in the first stage training.
Afterwards, either $L=L_{cls}+L_{Anchor}$ or $L=L_{cls}+L_{TripletAnchor}$ is
applied. We updates the anchors each epoch as in Algorithm 2. Figure 4: The
change of sampled anchors (id = 0,1,…,7) along training epochs (_i.e_. 0, 10,
20, … 120): Each image corresponds to an ID (0,1,…,7). Each row in image
represents the anchor feature vector (2048 dimension) in the sampled epoch.
Total 13 epochs from 0 to 120 with step of 10 is sampled. Zoom in to see the
details. The sampled anchors are calculated from the checkpoints of training
without anchor loss. Figure 5: Reconstruction results of anchors in the
training dataset (ID=0,1,2,3…,19).

(b) How to calculate anchors When the feature alignment is still stochastic,
_i.e_. early training phase, calculating anchors with probability contribution
(Eq. 4) produces better results. Easy samples, which converge earlier, may
contain less noises and variance, revealing the approximation of optimal
anchors, _e.g_. the anchors generated from reconstruction in Fig. 5. After the
initial training approaches convergence, the benefits of voting by confidence
become less significant.

Figure 6: Reconstruction pipeline of anchors in image space: Encoder is
transfered and frozen during the training of decoder. During the inference,
the decoder reconstructs the anchor into image space.

(c) How to look at anchors Triplet anchor loss and intra-class anchor loss
achieve comparable results when starting the aggregation and alignment in the
intermediate stage of training convergence. After the initial training
saturation, intra-class anchor loss performs slightly better, implying the
consistency weights more in training Stage II. In experiments, we also find
triplet anchor loss needs a proper tuning for the margin hyper-parameter.
Training is unstable when set to a high margin while easily saturated when set
to a low one. The best results are found when $margin=0$, where however
training converges slower comparing to intra-class anchor loss. It validates
our initial assumption that, when the anchors are well aggregated after Stage
I training convergence, optimization with consistency provides more benefits
for generalization.

###  4.2 What Would the Anchors Look Like

In order to further validate our assumption about the benefits of aggregation,
we train a decoder network to reconstruct the images from the feature maps
before GAP (global average pooling), where encoder is the well-trained model
without anchor loss. A decoder structure slightly modified from DG-Net [43] is
used. Then we generate anchors in image space by feeding the average
aggregated feature maps before GAP into trained decoder (_c.f_. Fig. 6). We
note that GAP applied anchor feature map produce the same as anchor feature
vector and 2D maps are used to preserve the spatial information for better
reconstruction. This anchor feature vector can be taken as the initial anchor
during the start of training Stage II. As the results in Fig. 5, the anchors
could be a feature vector which dissects the view-variance, pose-variance as
well as background-variance. Comparing to the sample images from same class
(_c.f_. Fig. 6), anchors generated from aggregation acts like a implicit
regularization to remove noise and variance. It complies that the average
aggregation distills the constitutional id-related features over the sampling
distribution.

###  4.3 Further Discussion

Figure 7: Test result of anchor loss from checkpoints: Without bells and
whistles, anchor losses boost the performance significantly after the initial
training is saturated. Figure 8: T-SNE visualization of the samples
(ID=0,1,…99) on training dataset: Training result of Stage I with $L_{trip}$
(left) v.s. Stage II with $L_{anchor}$ (right). Zoom in to see details.

Triplet Loss From the results in Table 3, our methods consistently improve the
results over the original model no matter which variant is chosen. As in Fig.
7, our methods boost the generalization after the training Stage I converges,
where the triplet loss reaches saddle point and there is still room to further
intensify the cluster compactness (Fig. 8 left). After the Stage II training,
the intra-class distance is further reduced (Fig. 8 right) and boost the
generalization in terms of both rank@1 accuracy and mAP (Table 3). As
illustrated in Fig. 2 and the experiments in Section 4.1, our methods perform
more effectively after the training Stage I is converged, where a stable
feature distribution is provided for aggregation. On the other hand, triplet
loss may provide beneficial effects during the initial stochastic training
process. Triplet loss can inhabit more compacted feature embedding for each
class in euclidean space than cross-entropy loss, which has been discussed in
BNNeck [16]. Consequently, the improvement that the proposed anchor loss
brings, is more significant for the model trained with $L_{trip}$ than the one
trained without $L_{trip}$ (_c.f_. Table 3&Fig. 7). In summary, anchor loss
stimulates stable and effective optimization to find better local optimal when
triplet loss suffers from stochastic saddle point (_c.f_. Fig. 8).

Frequency | Market1501 | DukeMTMC reID
---|---|---
| rank@1 | mAP | rank@1 | mAP
constant | 95.34% | 87.91% | 88.3% | 78.9%
epoch | 95.37% | 87.99% | 88.3% | 79.1%
iteration | 95.25% | 88.11% | 88.5% | 79.1%

Table 2: Ablation study for frequency to update anchors

Stage I | Stage II | aggregation method | Rank@1 | mAP
---|---|---|---|---
$L_{cls}$ | - | - | 93.79% | 84.69%
$L_{cls}+L_{Anchor}$ | avg | 94.18% | 85.98%
$L_{cls}+L_{triplet}$ | - | - | 94.42% | 86.18%
$L_{cls}+L_{Anchor}$ | avg | 95.37% | 87.99%
$L_{cls}+L_{Anchor}$ | weighted | 95.04% | 87.95%
$L_{cls}+L_{TripletAnchor}$ | avg | 95.25% | 87.84%
$L_{cls}+L_{TripletAnchor}$ | weighted | 95.13% | 87.87%
$L_{cls}+L_{Anchor}+L_{triplet}$ | avg | 95.16% | 87.87%
$L_{cls}+L_{TripletAnchor}+L_{triplet}$ | avg | 95.04% | 87.94%
$L_{cls}+L_{TtripletAnchor}+L_{triplet}$ | weighted | 95.28% | 88.07%
Table 3: Ablation study of applying anchor loss after the initial training
stage, _i.e_. epoch 120.

Applicability In terms of the comparison about update frequency for anchors,
three methods (Algorithm 1,Algorithm 2,Algorithm 3) derive comparable results
as shown in Table 3. Considering the cluster formulation, the anchors
generated at the end of traditional training are already well-complied with
data distribution regarding their identity labels (_c.f_. Fig. 8 left). After
Stage II fine-tuning, the aggregated anchors stay close to the initial one
since all the samples are pulled towards their anchors in the optimization
(_c.f_. Fig. 8 right). From another perspective, those three methods in Table
3 are alternatives concerning computational cost and training availability
while provide comparable result. For example, when training with large
training dataset or in the context of online learning, Algorithm 1 and
Algorithm 3 would be preferred with little sacrifice of performance. Hence,
our proposed method could be tremendously flexible and widely applicable in
terms of training efficiency.

Methods | Market1501 $\xrightarrow{}$ DukeMTMC | DukeMTMC $\xrightarrow{}$ Market1501
---|---|---
Rank@1 | mAP | Rank@1 | mAP
Resnet50 | 27.9%(24.3%) | 15.5%(13.0%) | 47.7%(47.2%) | 21.7%(21.1%)
Resnet50(ours) | 35.3%(33.3%) | 20.9%(18.6%) | 49.6%(47.3%) | 23.0%(21.8%)
Resnet50-ibn-a | 40.7%(37.9%) | 25.9%(23.2%) | 56.0%(50.6%) | 27.8%(24.5%)
Resnet50-ibn-a(ours) | 46.1%(46.0%) | 30.3%(29.1%) | 55.3%(52.0%) | 28.2%(25.2%)

Table 4: The performance of different models is evaluated on cross-domain
datasets. Market1501 $\xrightarrow{}$ DukeMTMC means that we train the model
on Market1501 and evaluate it on DukeMTMC-reID. () denotes the models trained
and tested with input size $384\times 192$.

Robustness A natural question about the Stage II fine-tuning is that whether
the further cluster-level alignment tends to be domain dependent and overfit
the domain distribution. From the cross-domain testing experiments in Table 4,
our methods invariably outperform the baseline model. It verifies the proposed
methods are an effective and robust approach to embed images into identity-
related space for metric learning.

###  4.4 Non-Parametric Anchor vs Parametric Center

Although anchors in our work looks like the centers proposed in [32], they are
intrinsically not the same: the former is non-parametric while the latter is
parametric. In fact there is no essential difference between the center loss
$\min_{\textbf{c}_{j}}||\textbf{f}_{i}-\textbf{c}_{j}||^{2}$ and the
classification loss
$\max_{\textbf{p}_{j}}(\textbf{f}^{\top}_{i}\textbf{p}_{j})$, both of which
are distance metrics, _i.e_. $L_{2}$ distance and inner product. The role
played by centers $\textbf{c}_{j}$ [32] corresponds to the role of the
hyperplanes $\textbf{p}_{j}$ in traditional classification. As a result,
parametric center loss still conforms to instance-level alignment similar to
classification loss under the local mini-batch training framework, and cannot
build the global connection in cluster level. On the other hand, in the
proposed anchor loss (Eq. 1&Eq. 2), anchors $\textbf{a}_{j}$ are not
optimization variables but calculated from cluster distribution instead. The
anchors are iteratively updated from the aggregation of dataset features,
which enables them to have the global view of feature distribution during the
local mini-batch training.

Figure 9: Test result comparison between our method and parametric center
loss: We train 12 models independently for each method on Market1501 dataset.

We train 12 models independently using the the proposed anchor loss in
comparison with another 12 independently trained models using the center loss
[32] on Market1501 dataset. Fig. 9 illustrates the model performance histogram
in terms of mAP (left) and rank1 (right). As can be observed, our proposed
anchor loss consistently outperform those center loss [32]. It validates that
anchor loss distills the knowledge in latent feature space from the images
belonging to the same identity and aggregate them into anchors to guide the
training towards well-aligned embedding. Such embedding complies intrinsic
feature distribution and thus helps the both training and generalization.
Furthermore, the result variance of parametric center loss is much higher than
anchor loss, implying the dependency to random initialization for parametric
center which may impose some stochastic prior to the cluster formulation. On
the contrary, our methods consistently outperform center loss with small
variance.

Methods | Labeled | Detected
---|---|---
Rank@1 | mAP | Rank@1 | mAP
MLFN(CVPR2018) [1] | 49.2% | 54.7% | 47.8% | 52.8%
PCB(ECCV2018) [26] | - | - | 57.5% | 63.7%
Mancs(ECCV2018) [29] | 69.0% | 53.9% | 65.5% | 60.5%
MGN [30] | 67.4% | 68.0% | 66.0% | 68.0%
Mltb [37](CVPR2019) | 66.5% | 70.1% | 64.2% | 66.6%
CASN+PCB [42] (CVPR2019) | 73.7% | 68.0% | 71.5% | 64.4%
MHN-4 (PCB) [2](ICCV2019) | 75.1% | 70.6% | 71.6% | 66.1%
Resnet50 [16]* | 63.36% | 61.60% | 60.07% | 51.79%
Resnet50(ours) | 76.36% | 74.50% | 72.36% | 70.32%

Table 5: Comparison with SOTA on CUHK03: CUHK03 evaluation with the setting of
767/700 training/test split on both the labeled and detected images. * denotes
our implementation.

Methods | alignment method | Market-1501 | DukeMTMC-reID
---|---|---|---
Rank@1 | mAP | Rank@1 | mAP
Mancs (ECCV2018) | - | 93.1% | 82.3% | 84.9% | 71.8%
PCB+RPP [26] (ECCV2018) | vertical partition | 93.8% | 81.6% | 83.3% | 69.2%
VPM [25] (CVPR2019) | soft vertical partition | 93.8% | 80.8% | 83.6% | 72.6%
AANet152 [27] (CVPR2019) | attribute attention | 93.9% | 83.4% | 87.7% | 74.3%
IANet [10](CVPR2019) | spatial semantic | 94.4% | 83.1% | 87.1% | 73.4%
MltB [37] (CVPR2019) | CAM | 94.7% | 84.5% | 85.8% | 72.9 %
DG-Net [43] (CVPR2019) | latent code | 94.8% | 86.0% | 86.6% | 74.8 %
MVP Loss [24] (ICCV2019) | - | 91.4% | 80.5% | 83.4% | 70.0%
OSNet [46] (ICCV2019) | channel attention | 94.8% | 84.9% | 88.6% | 73.5%
MHN-6 [2] (ICCV2019) | high-order attention | 95.1% | 85.0% | 89.1% | 77.2%
$P^{2}$-Net [7] (ICCV2019) | part | 95.2% | 85.6% | 86.5% | 73.1%
BDB+Cut [4] (ICCV2019) | partition | 95.3% | 86.7% | 89.0% | 76.0%
ABD-Net [3] (ICCV2019) | diverse attention | 95.6% | 88.3% | 89.0% | 78.6%
Resnet50 [16] | - | 94.1%(93.6%) | 85.7%(85.8%) | 86.2%(86.9%) | 75.9%(76.8%)
Resnet50 [16] | parametric center | 94.5% | 85.9% | 86.4% | 76.4%
Resnet50(ours) | cluster anchor | 95.4%(94.9%) | 88.0% | 88.3%(89.1%) | 79.1%(79.6%)
Resnet50-ibn-a [16] | - | 95.2%(95.5%) | 87.2%(88.2%) | 89.0%(89.7%) | 79.4%(80.0%)
Resnet50-ibn-a [16] | parametric center | 95.0% | 87.2% | 89.4% | 78.8%
Resnet50-ibn-a(ours) | cluster anchor | 95.7%(95.8%) | 88.9%(89.7%) | 90.2%(91.0%) | 80.6%(81.8%)

Table 6: Comparison of SOTA on Market1501 dataset and DukeMTMC-reID dataset.
() denotes the results with a larger input size $384\times 192$.

##  5 Comparison with the State-of-the-Art Methods

We compare our method with the recent state-of-the-art methods in Table 5 and
Table 6. Comparing to spatial alignment by either attention or localization
[26, 27, 10, 37, 7, 4], our method is much succinct without extra modules or
classifier heads to handle subspace alignment. We have made several attempts
to incorporate spatial alignment in our baseline model, _e.g_. PCB [26], only
to find slightly worse results. Based on our observation from reconstruction
experiment (_c.f_. Fig. 6), we notice that decoder can be trained well to
reconstruct images in both training and test dataset, implying spatially
diversified features are preserved in feature maps before GAP (global average
pooling). However, the feature vectors aggregated after GAP is aligned,
meaning GAP could effectively eliminate the spatial variance while preserving
the channel-wise diversity when a unified strong model is well trained, _e.g_.
the strong baseline model[16]. Hence, we infer that the benefits of spatial
alignment is marginal in our context. Comparing to channel attention and
several variants [46, 2, 3] which aim to generate diverse and uncorrelated
feature embedding with the efforts on convolution filters in self-supervised
manner, our methods achieve the cluster encoding with focus on feature space
by an explicit supervision of aggregated anchors from other images. DG-Net
[43] endeavors to disentangle appearance code and structure code by GAN and
auto-encoder, our method accomplishes similar effect that dissects the
variance and preserves the identity-related features in the direction towards
aggregated anchors (_c.f_. Fig. 5). IBN-Net [18] is proposed to reduce
appearance variance and keep discriminative feature extraction by unifying
both instance batch normalization and batch normalization. Luo _et al_. [16]
apply it as the backbone network for person re-identification and we report
our implementation result in Table 6. We note that resnet50-ibn-a network has
the same parameter size and computational cost with original resnet50. Without
bells and whistles, our method consistently boosts the performance in terms of
both Rank@1 and mAP comparing to corresponding baseline (resnet50 and
resnet50-ibn-a [16]), achieving the state-of-the-art results on Market-1501,
DukeMTMC-reID(Table 6) and CUHK03(Table 5) datasets.. Specially, due to
further reduce of intra-class variance towards a compact cluster in latent
feature space, our method improves mAP significantly and benefits the
robustness for the application of person re-identification. In summary,
comparing to the recent state-of-the-art methods, our methods visit another
modality of alignment, cluster-level alignment, validating that exploration of
interaction of clusters observed from dataset feature distribution improves
both training and generalization.

##  6 Conclusion

In this paper, we investigate the person re-identification from the view of
alignment and find an interesting and effective approach to delve in another
scale of feature alignment, cluster level. By performing aggregation and
alignment iteratively, our proposed anchor loss is enabled to interact with
more images indirectly from aggregated anchors, which pass the distilled
knowledge from the feature distribution and provide a consistent optimization
to further boost the performance significantly after traditional training
convergence. It shows the cluster-level alignment guided by the aggregation of
dataset distribution, which steps beyond the general classification framework,
is essential and beneficial for identity-related embedding.

## References

  * [1] Chang, X., Hospedales, T.M., Xiang, T.: Multi-level factorisation net for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2109–2118 (2018)
  * [2] Chen, B., Deng, W., Hu, J.: Mixed high-order attention network for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 371–381 (2019)
  * [3] Chen, T., Ding, S., Xie, J., Yuan, Y., Chen, W., Yang, Y., Ren, Z., Wang, Z.: Abd-net: Attentive but diverse person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 8351–8361 (2019)
  * [4] Dai, Z., Chen, M., Gu, X., Zhu, S., Tan, P.: Batch dropblock network for person re-identification and beyond. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3691–3701 (2019)
  * [5] Dangwei, L., Xiaotang, C., Zhang, Z., Kaiqi, H.: Learning deep context-aware features over body and latent parts for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 384–393 (2017)
  * [6] Fu, Y., Wei, Y., Zhou, Y., Shi, H., Huang, G., Wang, X., Yao, Z., Huang, T.: Horizontal pyramid matching for person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 8295–8302 (2019)
  * [7] Guo, J., Yuan, Y., Huang, L., Zhang, C., Yao, J.G., Han, K.: Beyond human parts: Dual part-aligned representations for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3642–3651 (2019)
  * [8] Haiyu, Z., Maoqing, T., Shuyang, S., Jing, S., Junjie, Y., Shuai, Y., Xiaogang, W., Xiaoou, T.: Spindle net: Person re-identification with human body region guided feature decomposition and fusion. In: Proceedings of the IEEE International Conference on Computer Vision (2017)
  * [9] Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737 (2017)
  * [10] Hou, R., Ma, B., Chang, H., Gu, X., Shan, S., Chen, X.: Interaction-and-aggregation network for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 9317–9326 (2019)
  * [11] Jie, H., Li, S., Gang, S.: Squeeze-and-excitation networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 7132–7141 (2018)
  * [12] Jing, X., Rui, Z., Feng, Z., Huaming, W., Wanli, O.: Attention-aware compositional network for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2119–2128 (2018)
  * [13] Kalayeh, M.M., Basaran, E., Gökmen, M., Kamasak, M.E., Shah, M.: Human semantic parsing for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1062–1071 (2018)
  * [14] Li, W., Zhao, R., Xiao, T., Wang, X.: Deepreid: Deep filter pairing neural network for person re-identification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 152–159 (2014)
  * [15] Liu, J., Ni, B., Yan, Y., Zhou, P., Cheng, S., Hu, J.: Pose transferrable person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4099–4108 (2018)
  * [16] Luo, H., Jiang, W., Gu, Y., Liu, F., Liao, X., Lai, S., Gu, J.: A strong baseline and batch normalization neck for deep person re-identification. IEEE Transactions on Multimedia (2019)
  * [17] Miao, J., Wu, Y., Liu, P., Ding, Y., Yang, Y.: Pose-guided feature alignment for occluded person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 542–551 (2019)
  * [18] Pan, X., Luo, P., Shi, J., Tang, X.: Two at once: Enhancing learning and generalization capacities via ibn-net. In: The European Conference on Computer Vision (ECCV) (September 2018)
  * [19] Quan, R., Dong, X., Wu, Y., Zhu, L., Yang, Y.: Auto-reid: Searching for a part-aware convnet for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3750–3759 (2019)
  * [20] Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C.: Performance measures and a data set for multi-target, multi-camera tracking. In: European Conference on Computer Vision. pp. 17–35. Springer (2016)
  * [21] Saquib Sarfraz, M., Schumann, A., Eberle, A., Stiefelhagen, R.: A pose-sensitive embedding for person re-identification with expanded cross neighborhood re-ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 420–429 (2018)
  * [22] Shuang, L., Slawomir, B., Peter, C., Xiaogang, W.: Diversity regularized spatiotemporal attention for videobased person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 369–378 (2018)
  * [23] Subramaniam, A., Nambiar, A., Mittal, A.: Co-segmentation inspired attention networks for video-based person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 562–572 (2019)
  * [24] Sun, H., Chen, Z., Yan, S., Xu, L.: Mvp matching: A maximum-value perfect matching for mining hard samples, with application to person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 6737–6747 (2019)
  * [25] Sun, Y., Xu, Q., Li, Y., Zhang, C., Li, Y., Wang, S., Sun, J.: Perceive where to focus: Learning visibility-aware part-level features for partial person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 393–402 (2019)
  * [26] Sun, Y., Zheng, L., Yang, Y., Tian, Q., Wang, S.: Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 480–496 (2018)
  * [27] Tay, C.P., Roy, S., Yap, K.H.: Aanet: Attribute attention network for person re-identifications. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7134–7143 (2019)
  * [28] Tian, M., Yi, S., Li, H., Li, S., Zhang, X., Shi, J., Yan, J., Wang, X.: Eliminating background-bias for robust person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5794–5803 (2018)
  * [29] Wang, C., Zhang, Q., Huang, C., Liu, W., Wang, X.: Mancs: A multi-task attentional network with curriculum sampling for person re-identification. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 365–381 (2018)
  * [30] Wang, G., Yuan, Y., Chen, X., Li, J., Zhou, X.: Learning discriminative features with multiple granularities for person re-identification. In: Proceedings of the 26th ACM international conference on Multimedia. pp. 274–282 (2018)
  * [31] Wei, L., Xiatian, Z., Shaogang, G.: Harmonious attention network for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2285–2294 (2018)
  * [32] Wen, Y., Zhang, K., Li, Z., Qiao, Y.: A discriminative feature learning approach for deep face recognition. In: European conference on computer vision. pp. 499–515. Springer (2016)
  * [33] Wen, Y., Zhang, K., Li, Z., Qiao, Y.: A comprehensive study on center loss for deep face recognition. International Journal of Computer Vision 127(6-7), 668–683 (2019)
  * [34] Xiang, L., Ancong, W., Wei-Shi, Z.: Adversarial open-world person re-identification. In: ECCV (2018)
  * [35] Xu, J., Zhao, R., Zhu, F., Wang, H., Ouyang, W.: Attention-aware compositional network for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2119–2128 (2018)
  * [36] Yan, H., Jinsong, X., Qiang, W., Zhedong, Z., Zhaoxiang, Z., Jian, Z.: Multi-pseudo regularized label for generated samples in person re-identification. TIP (2018)
  * [37] Yang, W., Huang, H., Zhang, Z., Chen, X., Huang, K., Zhang, S.: Towards rich feature discovery with class activation maps augmentation for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1389–1398 (2019)
  * [38] Zhang, Z., Lan, C., Zeng, W., Chen, Z.: Densely semantically aligned person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 667–676 (2019)
  * [39] Zhedong, Z., Liang, Z., Yi, Y.: Unlabeled samples generated by gan improve the person re-identification baseline in video. In: ICCV (2017)
  * [40] Zheng, F., Deng, C., Sun, X., Jiang, X., Guo, X., Yu, Z., Huang, F., Ji, R.: Pyramidal person re-identification via multi-loss dynamic training. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8514–8522 (2019)
  * [41] Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., Tian, Q.: Scalable person re-identification: A benchmark. In: Computer Vision, IEEE International Conference on (2015)
  * [42] Zheng, M., Karanam, S., Wu, Z., Radke, R.J.: Re-identification with consistent attentive siamese networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5735–5744 (2019)
  * [43] Zheng, Z., Yang, X., Yu, Z., Zheng, L., Yang, Y., Kautz, J.: Joint discriminative and generative learning for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2138–2147 (2019)
  * [44] Zheng, Z., Zheng, L., Yang, Y.: A discriminatively learned cnn embedding for person reidentification. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 14(1), 1–20 (2017)
  * [45] Zhong, Z., Zheng, L., Kang, G., Li, S., Yang, Y.: Random erasing data augmentation. arXiv preprint arXiv:1708.04896 (2017)
  * [46] Zhou, K., Yang, Y., Cavallaro, A., Xiang, T.: Omni-scale feature learning for person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3702–3712 (2019)