# ReFace: Improving Clothes-Changing Re-Identification With Face Features

Daniel Arkushin∗1 Bar Cohen††footnotemark: ∗2 Shmeul Peleg1 Ohad Fried2
1The Hebrew University of Jerusalem 2Reichman University
http://www.vision.huji.ac.il/reface/
{daniel.arkushin<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
These authors contributed equally

###### Abstract

Person re-identification (ReID) has been an active research field for many
years. Despite that, models addressing this problem tend to perform poorly
when the task is to re-identify the same people over a prolonged time, due to
appearance changes such as different clothes and hairstyles. In this work, we
introduce a new method that takes full advantage of the ability of existing
ReID models to extract appearance-related features and combines it with a face
feature extraction model to achieve new state-of-the-art results, both on
image-based and video-based benchmarks. Moreover, we show how our method could
be used for an application in which multiple people of interest, under
clothes-changing settings, should be re-identified given an unseen video and a
limited amount of labeled data. We claim that current ReID benchmarks do not
represent such real-world scenarios, and publish a new dataset, 42Street,
based on a theater play as an example of such an application. We show that our
proposed method outperforms existing models also on this dataset while using
only pre-trained modules and without any further training.

##  1 Introduction

Person re-identification (ReID) aims to match the same people appearing at
different times and locations. As explained in Section 2.1, in a typical ReID
setting, the predicted label of each query sample is defined based on a given
gallery set. Most commonly used ReID benchmarks [45, 30, 35, 19], capture
people within a single day, where clothes are rarely changed [12]. These
benchmarks were used by models that rely on clothes and not necessarily on
unique person characteristics [14, 36, 11]. To enable ReID research under
clothes-changing settings, benchmarks including clothes changes, such as LTCC
[29], PRCC [41] and CCVID [13] have been released. To solve this problem,
several models try to extract clothes-independent, global features [12, 41, 1,
29, 40].

In spite of the efforts invested in solving the clothes-changing problem, ReID
models still perform better in the same-clothes setting [12]. Current ReID
models do not use explicit face features [9] — although, when available, face
features distinguish between people with much higher accuracy, across
different times and environments. Therefore, we suggest a method that relies
on both clothes features and face features to automatically enrich a given
labeled gallery in a weakly supervised manner (Section 3.1). Our model
combines face feature extraction and ReID modules (Section 3.2) and achieves
new state-of-the-art results. We note that every module of our method could be
replaced by new ReID or face models having better capabilities.

In existing clothes-changing ReID benchmarks, data samples are clear crops of
a single person to be re-identified. Moreover, these benchmarks address the
closed-set setting only, and therefore, do not represent real-world scenarios
which include occlusions, crowded scenes, and open-set settings (see Section
2.1 for more details). We show how our method, when using a tracking module,
can address such complex scenarios and release a new dataset, 42Street
(Section 4.2) as an example for a real-world application. We show that without
any further training our method achieves strong results on this dataset while
outperforming the existing ReID models.

Our contribution in this work is twofold:

  1. 1.

We suggest a method that combines ReID and face modules, and enriches a given
gallery using face features, to achieve new state-of-the-art results on
existing clothes-changing ReID benchmarks.

  2. 2.

We publish a new dataset, 42Street, as an example for an application of person
ReID on a complex real-world scenario, and show that our method achieves
strong results on it while outperforming existing models.

##  2 Background and Related Work

###  2.1 Person Re-Identification

The inference process of person ReID can be seen as an instance retrieval
problem. Given labeled data samples and unlabeled query samples, the goal is
to predict the correct label for every query sample. In a standard ReID
pipeline, first, a gallery is created by applying a feature-extraction model
(the ReID model) on the labeled samples to represent them using feature
vectors. Next, given an unseen query sample, the distances between its feature
vector and the gallery feature vectors are computed. Finally, the predicted
label of the query sample is defined as the label of the gallery feature
vector which is closest to the query. A typical ReID dataset provides train
and test sets that do not share common identities.
To address the ReID challenge, in addition to using RGB images, several models
suggest using different input modalities such as depth images [37], text
descriptions [18], infrared images [38], and more.

##### Image-based and Video-based ReID

In image-based ReID datasets, every data sample is a single-person crop. Over
the years, multiple image-based models have been developed [36, 14, 23, 24],
and they achieve impressive results on image-based benchmarks. In video-based
ReID datasets, every data sample is a sequence of crops and the ReID model
produces a single feature vector to represent the entire sequence by using the
spatial-temporal information in the sequence [27, 12, 10, 15].

##### Clothes-changing ReID

In this setting, a person in the query set might wear different clothes than
in the gallery set. Some models try to extract clothes-independent features by
modeling body shape and gait using skeletons [29], silhouettes [1] and contour
sketches [40]. Another model, CAL [12], proposes a Clothes-based Adversarial
Loss to mine clothes-independent features, and uses the video input to extract
spatio-temporal patterns. Training these models requires ground-truth clothes
annotations.

##### Open-Set and Closed-Set

A closed-set setting assumes that every identity in the query set has at least
one corresponding sample in the gallery set. In an open-set setting, this
might not be the case, as some identities in the query are not present in the
gallery. Open-set ReID is usually formulated as a person verification problem
in which a model is required to discriminate whether two person images belong
to the same identity[34, 46]. Models that address this problem [22, 39, 21],
typically learn to discriminate between a given query and gallery according to
their similarity[17].

##  3 Method

In order to address the clothes-changing ReID problem, our method enriches the
labeled gallery (Section 3.1) and combines standard ReID approaches together
with face feature extraction (Section 3.2). Both elements work in conjunction
and define our method. Our method can be applied to image-based, video-based,
and raw-video input.

###  3.1 Creating an Enriched Gallery

The objective of most ReID models is to produce a feature extraction function
that generates a feature vector for each data sample. Given two different data
samples of the same person, the feature extraction function is expected to
generate feature vectors with a lower cosine distance compared to two samples
from different persons. The richer the gallery is with samples that are
similar to the query set, the more likely it is to find a correct gallery
match. Hence, we propose an algorithm to enrich a given labeled gallery using
face features, when available, from the unlabeled query.
Building an enriched gallery is done using the following weakly supervised
algorithm and summarized in Fig. 1. Given labeled and unlabeled samples:

  1. 1.

For every labeled data sample in which a face was detected, we extract a face
feature vector and label it using the label of the original sample, to create
a face gallery.

  2. 2.

For every unlabeled sample in which a face was detected and verified (Section
3.1.1), we predict a label by computing its face feature vector and comparing
it to the labeled face gallery from step 1.

  3. 3.

(optional) Under the open-set setting, we label out-of-gallery queries as
”Unknown” (Section 3.3).

  4. 4.

We create an enriched gallery by extracting ReID feature vectors from the
originally labeled data and the predicted samples from step 2.

Figure 1: The Gallery Enrichment Process. For every labeled sample in which a
face was detected, a face feature vector is extracted to create a labeled face
gallery. Then, for every unlabeled sample in which a face was detected, a
label is predicted by computing cosine similarities between its face feature
vector and the labeled face gallery. Finally, an enriched gallery is created
by extracting ReID features from both the original labeled samples and the
newly labeled samples.

We note that the above process assumes that the query set is given beforehand,
however, this process could also be made online by adding one query sample at
a time.

####  3.1.1 Matching Face to Pose Estimation

A person crop is a part of an image cropped according to the posture of a
specific person. Often, in person crops from crowded scenes, multiple people
appear in the background of a person’s crop. In such cases, a face detected in
an image might belong to a person in the background and not to the main person
according to whom the crop was created. Therefore, it is necessary to verify
whether the detected face belongs to the main person in the crop. To do this,
we use a separate pose estimation of the main person in the crop and verify
that the keypoints of the eyes and nose, as given by the pose estimator, are
within the bounding box given by the face detector. Examples are shown in Fig.
2.

|  |
---|---|---
|  |
Figure 2: Matching Face Detection and Pose Estimation. Left: A single detected
face that matches the main person. Middle: Multiple detected faces, the main
person’s pose matches one of the detected faces. Right: A single detected face
that does not match the pose estimation of the main person.

###  3.2 Combining ReID and Face Modules

In our method we propose to use a face feature extraction module and a ReID
module to compute face and ReID score vectors respectively. These score
vectors represent the confidence of each module that the given track belongs
to each of the possible identities. Our method combines these two score
vectors into a final score vector which is used to label the identity of the
person in a given data sample. Next, we describe in detail our method for the
video-based setting, where every data sample is a person track. The prediction
process for the image-based setting is treated as a special case of the video-
based setting, in which a data sample, i.e. a single image, is a track of
length 1.

Figure 3: Track Identity Prediction Overview. Given an input track, using all
track images, the ReID and face modules produce a track score vector
representing the confidence of every module that the track belongs to each of
the possible identities. Then, these score vectors are combined into a final
score vector and the track is labeled according to the identity with the
highest score (green circle).

##### Predicting an Identity of a Track

Given labeled data of a set of identities ${I}$, we first build an enriched
gallery, $G_{enriched}$, as described in Section 3.1, and a labeled face
gallery denoted $G_{face}$, by extracting face feature vectors from the
labeled data. Then, given a person track, we apply the face feature extraction
and the ReID modules on it to create a score vector of size $|{I}|$,
representing the confidence of our model that the given track belongs to each
identity in $I$. This process is illustrated in Fig. 3 and detailed next.

##### Face and ReID Score Vectors

Given a person track with $N$ consecutive image crops of the same person,
denote $v_{ReID}\in\mathbb{R}^{|I|}$ to be a vector representing the ReID
score for each identity, initialized with zeros.
For every image $q$ in the track, we compute the feature vector representing
this image using the ReID module. Then, for each identity $i\in I$ the
confidence that image $q$ belongs to identity $i$ is defined as the maximum
cosine-similarity between the feature vector of $q$ and the feature vectors of
all gallery samples of identity $i$ in $G_{enriched}$. The final score for
identity $i$, i.e. $v_{ReID}[i]$, is the mean confidence for identity $i$ for
all the images in the track.

Denote $v_{face}\in\mathbb{R}^{|I|}$ to be a vector representing the face
score for each identity, initialized with zeros. To compute $v_{face}$ we
follow the same procedure as above while changing the input images and
gallery. In this case, the input images for the procedure are $M$ detected
face images from the original person track. Those are created by applying a
face-detector on every image in the original track, and verifying that the
pose matches the main person in the image (Section 3.1.1). The gallery used
for similarity comparison is the labeled face gallery $G_{face}$.

##### Combining Score Vectors

To compute the final score vector $v_{pred}\in\mathbb{R}^{|I|}$, similarly to
Wan et al. [33], we take a linear combination of the output of the two score
vectors with a hyper-parameter $\alpha$. The resulting score vector is:

$v_{pred}=\alpha\cdot v_{ReID}+(1-\alpha)\cdot v_{face}$ (1)

The final label prediction for the entire track is given by taking the
identity with the highest score in $v_{pred}$.

###  3.3 Addressing the Open-set Challenge

In the open-set setting, we approach the task of labeling an out-of-gallery
query sample by utilizing the gallery enrichment process (Section 3.1). During
this process we label a given query sample as “Unknown” and add it to the
enriched gallery if it fulfills the following criteria:

  1. 1.

A face was detected with confidence above a certain threshold, and verified
(Section 3.1.1).

  2. 2.

The cosine similarity between the feature vector of the detected face and the
labeled face gallery feature vectors is below a certain threshold.

  3. 3.

The difference between the rank-1 and rank-2 predictions in the face score
vector is below a certain threshold.

###  3.4 Raw-Video ReID: A Real-World Application

In order to apply our method to a given raw-video, we use a tracking module
without applying any changes to it. The tracker module detects and tracks
people across multiple frames and outputs a sequence of per person image crops
(tracks). For each track, we predict an identity as described in Section 3.2.
Finally, the tracks and their predictions are displayed on the input video to
create an annotated output video. This process is illustrated in Fig. 4.

##  4 Evaluation

In this section, we list the models we compare our results against and provide
details for the evaluation of existing clothes-changing ReID benchmarks
(Section 4.1). Then, we describe the 42Street dataset curation (Section 4.2)
and detail the protocol to evaluate the performance of various models on the
clothes-changing ReID scenarios in it.

###  4.1 Evaluation on Existing Benchmarks

We evaluate the performance of our model and compare it with CAL[12], CTL[36],
and InsightFace [6, 8, 7, 5]. CAL is a clothes-changing ReID model, CTL is a
model that focuses on the same-clothes setting, and InsightFace is a face
analysis project. All models are evaluated on four widely used clothes-
changing ReID benchmarks: three image-based benchmarks, LTCC [29], PRCC [41],
and LAST [31], and one video-based benchmark, CCVID [13]. For consistency with
the results reported by CAL, the current SoTA model on these benchmarks, Top-1
accuracy and mAP are used as evaluation metrics and three kinds of test
settings are defined as follows:

  * •

General: both clothes-changing and same-clothes ground truth samples are used
to calculate accuracy.

  * •

Clothes-Changing (CC): only clothes-changing ground truth samples are used to
calculate accuracy.

  * •

Same-Clothes (SC): only same-clothes ground truth samples are used to
calculate accuracy.

Figure 4: Raw-Video Annotation overview. Given an unlabeled video, person
tracks are generated using a tracking module. Next, the identity of each track
is predicted in the Track Identity Prediction process (Fig. 3). Finally, all
tracks and their predicted identities, are displayed on the input video to
create an annotated video.

###  4.2 The 42Street Dataset

As described in Section 1, current clothes-changing ReID benchmarks do not
represent complex real-world scenarios with open-set settings. For this
reason, we create and publish the 42Street dataset, as an evaluation dataset,
i.e. we do not provide training data with a different set of identities,
compared to the test set.
The dataset is created using a public recording of the 42Street theatre play
[26]. The play is $\sim$1.5 hours long and was split into 5 equally long parts
of $\sim$20 minutes each, with various clothes changes between the different
parts. From Part 1 we label samples that are used by our method and other
models we compare against. Parts 2–3 and 4–5 are used for validation and test,
respectively, and from them, we randomly extract 5 validation videos and 10
test videos, each 17 seconds long. We define a set of people-of-interest $I$,
which consists of 13 people that appear in all parts of the play in various
clothes, and define a single class, “Unknown”, for all people that are not
part of set $I$.
In Appendix D, we describe a semi-supervised process for generating labeled
samples quickly. As this process requires relatively low effort, our method
can be easily applied to new unseen data.

##### Annotating Evaluation Videos

To annotate an evaluation video with ground truth labels, a tracker, ByteTrack
[43], is applied to automatically extract tracks of detected people, followed
by a manual annotation of every track. These ground-truth labels, as well as
person bounding boxes, track ids, video names, and more are saved in a
designated database published alongside the dataset. In Appendix E, we provide
more details about the database and how it can be used for further research
and to reproduce our results.

##### Evaluation on the 42Street Dataset

Given the labeled samples created from part 1 of the play, we evaluate the
performance of our method on the evaluation videos (Section 4.2) of the
42Street dataset and compare it with the same models as described in Section
4.1. In addition, we also compare the results of the video-based version of
CAL for which we provide labeled video samples. As the 42Street dataset is an
evaluation dataset only, all models are evaluated without any fine-tuning.

##### Evaluation Metrics

The 42Street dataset represents a real-world scenario in which, given a raw-
video, a model should produce a single prediction for each person crop
detected in the video. Therefore, we measure a model’s performance using
classification metrics: Per-Image and Per-Track accuracy. We note that these
accuracies are equivalent to the top-1 metric, typically used in ReID works.

  * •

Per-Image Accuracy: the number of correctly identified person crops in a
video, divided by the total number of person crops in the video, across all
evaluation videos.

  * •

Per-Track Accuracy: the number of correctly classified tracks, i.e. whether
the model’s single prediction on the entire track is correct, divided by the
total number of tracks in the video, across all evaluation videos.

We note that the image-based models we evaluate generate a per-image
prediction, and do not provide a single prediction for an entire track. In
order to address this limitation, while calculating their Per-Track accuracy,
we set a single prediction for the entire track based on the most confidently
predicted identity, across all the track images, similarly to Section 3.2.
Furthermore, as this work does not focus on improving tracking capabilities,
while calculating this metric any track with less than 10 frames is ignored,
as those are likely to be tracking mistakes.

##### Open-Set and Closed-Set ReID Settings

ReID models we compare our results against assume that the input datasets are
of a closed-set setting. Hence, in order to correctly evaluate against these
models we compare the results in both the open-set and closed-set settings.
The latter is done by ignoring out-of-gallery queries during evaluation.

|
---|---
Figure 5: Performance Visualization on the 42Street Dataset. Results
visualization of the different compared models under the closed-set setting
(left) and open-set setting (right). For each bounding box, the abbreviation
’OTAF’ refers to the different models — O (Ours), T (CTL), A (CAL), F
(InsightFace). The green and red colors correspond to a correct and incorrect
prediction of each model on the bounding-box. In the open-set frame (right),
all people other than the one in the center are out-of-gallery and should be
labeled as ”Unknown”.

##  5 Experiments

###  5.1 Setting Hyper-Parameters

A typical face-detection model produces a confidence score that a face exists
in a given image. Additionally, cosine distances between the face feature
vector of the input image and a face gallery are calculated. These distances
can be used as the confidence that the given input image belongs to a certain
identity. To achieve the best results, we set thresholds for both confidence
scores. Given a dataset, we apply our gallery enrichment method (Section 3.1)
on the training set to find the best combination of these two thresholds.
Ideally, we would like to find the optimal combination that achieves the
highest prediction accuracy based on face features, whilst maintaining a high
detection accuracy per person. Meaning, we would like the model to predict at
least one image of each identity in order to create an enriched gallery with
examples of all identities. However, there exists a trade-off between the per-
identity detection accuracy and the prediction accuracy: for a higher
detection threshold, the number of unique predicted identities decreases. At
the same time, since the detected faces are of better quality, the prediction
accuracy increases. Vice versa, a lower detection threshold, leads to a larger
number of predicted identities, while achieving lower prediction accuracy.
Fig. 6 demonstrates this trade-off for the PRCC benchmark.

Figure 6: Prediction Accuracy vs. Number of Unique People. Example of the thresholds trade-off on the PRCC benchmark. Each data tuple represents the values of the detection threshold (left) and similarity threshold (right). We look for the combination that achieves the highest prediction accuracy while predicting the highest number of unique people (bold). Method | CCVID | LTCC | PRCC | LAST
---|---|---|---|---
General | General | CC | SC | CC | General
top-1 | mAP | top-1 | mAP | top-1 | mAP | top-1 | mAP | top-1 | mAP | top-1 | mAP
CTL | 71.8 | 64.8 | 36.7 | 11.1 | 11.7 | 4.8 | 78.6 | 68.8 | 24.0 | 21.0 | 20.1 | 3.2
CAL | 82.9 | 81.3 | 74.4 | 41.2 | 40.1 | 19.0 | 100 | 99.8 | 55.7 | 56.3 | 73.7 | 28.8
InsightFace | 95.3 | N/A | 27.4 | 9.2 | 24.5 | 8.3 | 95.6 | 70.0 | 77.9 | 54.7 | 57.8 | 31.1
Ours | 89.2 | N/A | 75.5 | 41.2 | 48.2 | 19.8 | 99.7 | 95.6 | 83.7 | 66.7 | 78.0 | 38.1
Table 1: Models comparison on existing clothes-changing benchmarks. Ours
refers to our method using CAL as the ReID module, CAL was fine-tuned on each
benchmark and CTL was pre-trained on DukeMTMC. General considers all gallery
samples for evaluation, SC considers only same-clothes gallery samples, and CC
considers only clothes-changing gallery samples. Notice that our model
outperforms other models on most benchmarks, and especially in the clothes-
changing setting. We note that although the face model achieves the best
results on CCVID, the results on other benchmarks suggest that it is
insufficient by itself.

###  5.2 Implementation Details

For our face module, we use InsightFace [6, 8, 7, 5] for face detection,
alignment, and feature extraction. For the ReID module, we examine both
CTL[36], pre-trained on DukeMTMC [30] and CAL[12]. The hyper-parameter
$\alpha$ used for combining the face score vector and the ReID score vector in
Eq. 1 is set to 0.75, giving more weight to the ReID module.
The detection and similarity thresholds used by our face module (Section 5.1)
are determined according to the training set for the existing benchmarks, and
the validation set for the 42Street dataset. In Appendix F, we detail all the
thresholds used for the different datasets.
For 42Street, the entire raw videos of test-set parts are used as query input
for the gallery enrichment process. In Appendix C, we analyze the impact of
using more raw-data for the gallery enrichment process. To apply our method on
the real-world application described in Section 3.4, we use ByteTrack [43] as
our tracking module as well as the MMCV [3] and MMTRACK [4] frameworks. To
deal with some of the tracking limitations, as the original play includes many
scene cuts, i.e. abrupt changes of the camera angle or zoom, we apply a scene
cut detection algorithm, and split tracks accordingly if necessary.

###  5.3 Results

Next, we present the results of the compared models on existing clothes-
changing ReID benchmarks as well as on the 42Street dataset and show the
significant improvement that can be achieved by using our method. In Appendix
B, we provide an ablation study of the various components in our method and
their significance.

####  5.3.1 Existing Benchmarks Results

Table 1 compares the results of our method versus the evaluated models on the
existing clothes-changing ReID benchmarks. From the table, we notice that our
method significantly outperforms the other evaluated models on all benchmarks,
except for the SC setting on PRCC where it achieves comparable results to the
current SoTA model. In addition, we note that although the face model achieved
superior results compared to other models on CCVID, the results on other
benchmarks suggest that it is not sufficient by itself. This is due to the
nature of the CCVID benchmark, which was curated from a gait-recognition
dataset (FVG [44]), recording people as they walk toward the camera with
clearly visible faces.
We note that as explained in Section 3.2, given an input track, our model
produces a score vector representing the confidence that the track belongs to
each of the possible identities. Therefore, as our model does not rank all
gallery tracks, we cannot calculate mAP for the video-based benchmark CCVID.

####  5.3.2 Cross-Dataset Results

Next, we analyze the generalization ability of different models. In this
experiment, we evaluate the performance of the compared models on PRCC and
LTCC without fine-tuning on the evaluated benchmark. We compare the
performance of CAL, pre-trained on LTCC when evaluated on PRCC and vice versa,
CTL, pre-trained on DukeMTMC, and our model, both with CAL and CTL as the ReID
module. Table 2 summarizes the results of this experiment and shows that our
model has better generalization abilities to unseen data than existing ReID
models.

Method PRCC $\rightarrow$ LTCC LTCC $\rightarrow$ PRCC General CC SC CC top-1
mAP top-1 mAP top-1 mAP top-1 mAP CAL 21.9 6.1 7.4 3.3 99.5 99.7 39.3 36.5 CTL
36.7 11.1 11.7 4.8 78.6 68.8 24.0 21.0 Our-CAL 27.8 6.9 16.6 3.9 99.7 100 81.5
53.0 Our-CTL 40.8 11.8 21.4 5.5 91.7 58.3 78.3 32.9

Table 2: Cross-Dataset Generalization without Fine-Tuning. Results comparison
using a pre-trained ReID model, without fine-tuning on the evaluated dataset.
$X\rightarrow Y$ means that CAL was pre-trained on dataset X and evaluated on
dataset Y. CTL was pre-trained on DukeMTMC. In Ours(X), X refers to the ReID
model our method used.

####  5.3.3 42Street Dataset Results

Table 3 compares the results of our method versus the evaluated models on the
42Street dataset.
Under the closed-set setting, similar to the results of the cross-dataset
experiment (Section 5.3.2), it is clear from the table that both CTL and CAL
models have limited generalization ability on the 42Street dataset as well.
Furthermore, the face-recognition model itself achieves only mediocre results.
This is due to the fact that in the 42Street dataset the face detector
detected faces only in 74% of the total person crops, some of which are not of
sufficient quality to be recognized correctly. However, our method, whilst not
requiring any further training, achieves strong results and significantly
outperforms the other models.
Under the open-set setting, all other evaluated models achieve poor results,
as they are not designed to address this setting. However, by using simple
adaptations as described in Section 3.3, our method is able to handle the
open-set challenge and achieve solid results. Fig. 5 visualizes the
performance of the various models under both settings.

As image-based models are not capable of generating a single prediction for a
track, per-track accuracy cannot be directly calculated for them. In Table 5,
we provide an extended table with the results of such models when adapted to
generate predictions over tracks.

We note that the accuracy results on the 42Street dataset concern only the
detected person crops. The person detector of ByteTrack [43] which we used,
achieved an IDF1 score of 80.5 for the MOT17[25] challenge. As such, results
for all evaluated models could potentially be discounted by a 19.5% margin to
bridge the possible gap of misidentified person crops. On the other hand,
additional tracker limitations, as described in Section 5.4, may affect the
ability of our method to correctly classify a track. These limitations do not
impact the image-based models we compare against, as their input is a single
crop and not a track.

Method | Closed-Set | Open-Set
---|---|---
Per Image | Per Track | Per Image | Per Track
CTL | 0.31 | N/A | 0.20 | N/A
CAL-Image | 0.20 | N/A | 0.13 | N/A
CAL-Video | 0.21 | 0.16 | 0.26 | 0.17
InsightFace | 0.55 | N/A | 0.36 | N/A
Ours-CTL | 0.92 | 0.81 | 0.79 | 0.65
Table 3: Models comparison on the 42Street dataset. All models are evaluated
under both closed-set and open-set settings. The models are pre-trained on
other datasets and are not fine-tuned on this dataset: CTL on DukeMTMC, CAL-
Image on LTCC, CAL-Video on CCVID. Ours(X) refers to using our model with X as
the ReID module. Note, as CTL, CAL-Image and InsightFace are image-based
models, per track accuracy is not applicable for them.

###  5.4 Limitations

Our method heavily relies on the assumption that every clothes-set of each
identity has at least one high quality face-image within the given labeled
samples and query sets. For datasets that do not hold this assumption due to
limited number of detected faces or low quality of the detected faces, our
method would only show slight improvement over other models, as only a limited
amount of query samples will be added to the original gallery in the gallery
enrichment process. That said, we believe our assumption holds for many real-
world scenarios and as such can introduce significant improvement compared to
existing models on such scenarios.

To apply our method to real-world applications and annotate a raw-video we use
a tracking module to extract person tracks. Using a tracker introduces several
limitations:

  * •

Identity Switch - often, when dealing with person occlusions in crowded
scenes, the tracker begins with tracking one person and mistakenly switches to
a different person. In this case, track-based predictions which provide a
single prediction for the entire track, incorrectly label the ’switched’ part
of the track.

  * •

Missed Detected Persons - people that were not identified by the tracker’s
person detector module.

  * •

Incorrectly Detected Persons - occurs when the person detector of the tracker
”detects” a non-existing person.

Currently, our method uses a tracking module without applying any changes to
it, and does not deal with these potential tracking mistakes.

##  6 Ethical Considerations

The release of new person ReID and tracking datasets often raises privacy
concerns as individuals that appear in the dataset are not always asked for
their consent. To comply with the regulations on data collection and usage,
such as European General Data Protection Regulation (GDPR) and California
Consumer Privacy Act (CCPA), some papers suggest data anonymization by
removing identifying information and blurring faces [32, 28]. Since our work
depends on the existence of faces in the dataset, blurring is not an option,
however, we address this concern by using publicly available videos only.
Furthermore, while our method utilizes face features to improve ReID
performance it is only used for image retrieval and distance measurement and
does not require any further identity matching or recognition. Another ethical
consideration is the usage of ReID and tracking systems. While these systems
have many positive applications for public safety, improving city planing,
etc., they could also be used maliciously, to spy on and suppress people. Our
method and published dataset should only be used for academic purposes.

##  7 Discussion & Future Work

In this work, we show a novel approach to address the clothes-changing ReID
challenge by creating an enriched gallery from the given labeled and unlabeled
samples, based on face features. By combining the enriched gallery with face
feature extraction and ReID modules, our proposed method achieves state-of-
the-art results on clothes-changing ReID benchmarks and on the real-world
clothes-changing dataset we publish, 42Street. Furthermore, our method shows
that with minimal labeling efforts, solid results can be achieved on raw-
videos, without any further training. This is in contrast to current models
that lack the ability to generalize to unseen data.

Future work ideas to improve our method include:

  * •

Use the fact that two people that appear in the same frame, cannot receive the
same label.

  * •

Use complete tracks instead of a single crop in the gallery enrichment
process. This could make the gallery both richer and more accurate.

  * •

Support real-time gallery enrichment, by adding extracted face and ReID
features to the galleries in inference time.

##  Appendix A Extended Results

In Table 4 we compare the results of additional clothes-changing ReID models.
Since not all papers have released their code, the reported results are taken
from the original papers. From the table we learn that our method outperforms
these models as well.

Table 5 shows the results of different models on the 42Street dataset. In
addition to the image-based versions of these models presented in the paper,
the track-based version is shown in order to support evaluation according to
the Per Track metric. This version is created in a similar way to the track
inference process (Section 3.2).

In Table 6 we evaluate the performance under two additional settings compared
to the results shown in the paper: Clothes-Changing (CC) for the CCVID
benchmark, and General for the PRCC benchmark.

The General setting for the PRCC benchmark is calculated using a weighted
average of the Same-Clothes and Different-Clothes settings. Even though it is
not provided by previous works, we include this setting as it is important to
understand the overall performance of the model. From the table we can see
that under this setting as well, our method significantly outperforms the
other models.

Method | LTCC (CC) | PRCC (CC)
---|---|---
top-1 | mAP | top-1 | mAP
CAL [12] | 40.1 | 19.0 | 55.7 | 56.3
CACC [20] | 40.3 | 18.4 | 57.4 | –
LCVN [42] | – | – | 64.4 | 62.6
3DSL [2] | 31.2 | 14.8 | 51.3 | –
FSAM [16] | 38.5 | 16.2 | 54.5 | –
Our | 48.2 | 19.8 | 83.7 | 66.7
Table 4: Additional models comparison on LTCC and PRCC. We evaluate the perfomrance under the clothes-changing setting only (CC) as this is the only setting included in the original papers. Method | Closed-Set | Open-Set
---|---|---
Per Image | Per Track | Per Image | Per Track
CTL | 0.31 | N/A | 0.20 | N/A
CTL (Track) | 0.35 | 0.27 | 0.21 | 0.15
CAL-Image | 0.20 | N/A | 0.13 | N/A
CAL-Image (Track) | 0.22 | 0.22 | 0.12 | 0.11
CAL-Video | 0.21 | 0.16 | 0.26 | 0.17
InsightFace | 0.55 | N/A | 0.36 | N/A
InsightFace (Track) | 0.69 | 0.62 | 0.45 | 0.36
Ours-CTL | 0.92 | 0.81 | 0.79 | 0.65
Table 5: Extended models comparison on the 42Street dataset. Image-based models predict an identity per-image, and therefore are not evaluated on the Per-Track accuracy metric. To address this, a (Track-Based) prediction is calculated using a track score vector, generating a single prediction for the entire track. All evaluated models are pre-trained on other datasets: CTL on DukeMTMC; CAL-Image on LTCC; CAL-Video on CCVID. Ours refers to using our model with CTL as the ReID module.  Method | CCVID | PRCC
---|---|---
General | CC | General | SC | CC
top-1 | mAP | top-1 | mAP | top-1 | mAP | top-1 | mAP | top-1 | mAP
CTL | 71.8 | 64.8 | 69.3 | 61.2 | 52.5 | 46.0 | 78.6 | 68.8 | 24.0 | 21.0
CAL | 82.9 | 81.3 | 81.9 | 79.6 | 78.8 | 79.0 | 100 | 99.8 | 55.7 | 56.3
InsightFace | 95.3 | N/A | 73.5 | N/A | 87.1 | 62.7 | 95.6 | 70.0 | 77.9 | 54.7
Ours-CAL | 89.2 | N/A | 90.5 | N/A | 92.1 | 81.8 | 99.7 | 95.6 | 83.7 | 66.7
Table 6: Extended models comparison on existing clothes-changing benchmarks.
In this table, in addition to the results shown in the paper for CCVID and
PRCC, we present the clothes-changing setting for CCVID and the general
setting for PRCC. Ours refers to our method using CAL as the ReID module, CAL
was fine-tuned on each benchmark and CTL was pre-trained on DukeMTMC. General
considers all gallery samples for evaluation, SC considers only same-clothes
gallery samples, and CC considers only clothes-changing gallery samples.
Additionally to the results shown in the paper, we note that under the CC
setting in CCVID, our model also outperforms Insightface, as the number of
clothes-changing gallery samples is significantly lower, which leads to the
lower performance of this model. For PRCC, the general setting is computed as
the weighted sum between the SC and CC settings, i.e. computed according to
the relative sizes of the CC and SC query samples, and was added to provide a
comparison of the overall performance of each model.

Figure 7: Accuracy as a function of the percentage of additional raw data used
for gallery enrichment in the 42Street dataset. This figure shows the impact
of gallery enrichment with a ReID module(CTL), while using additional raw-data
on both closed-set (left) and open-set (right) settings. The X-axis shows the
percentage of raw-data that is used in the gallery enrichment process. The red
point represents the results of the image-based ReID module, without using any
enrichment. Query Only refers to using only the data in the evaluation videos
for the gallery enrichment process. Each step represents the percentage of
additional data from the raw-video of the test part that was used in the
gallery enrichment process. The results suggest that enriching the gallery
using the Query Only data, significantly improves the same ReID module
compared to no enrichment. Moreover, the total accuracy of all compared
modules increases when using more raw-data.

##  Appendix B Ablation Study

In this section we evaluate the impact of each component of our method on the
overall performance of our method, both on the existing benchmarks and the
42Street dataset. From Table 7 and Table 8 we conclude the following:

  * •

The Gallery Enrichment Process introduces a significant improvement compared
to using only a ReID module, without enrichment. In Appendix C, we further
discuss the influence of additional raw data on the performance of our method.

  * •

Combining ReID and face modules (even without gallery enrichment),
significantly improves the results of current ReID models.

  * •

The face module by itself achieves superior results compared to the original
ReID module. Although this shows the strength of face features and their
importance for ReID tasks, it is not a sufficient model as not all images
include a clearly visible face.

  * •

Examining the Per Image Acc in Table 8 shows that the track-based version of
the different modules improves the results compared to their image-based
versions.

  * •

The open-set setting leads to weaker results of all components, as it
introduces a more difficult challenge of verification, i.e. labeling an out-
of-gallery query sample as “Unknown”.

  * •

Our method, combining ReID and face modules with an enriched gallery achieves
the best results on the evaluated datasets.

Method | General | Same-Clothes | Changing-Clothes
---|---|---|---
top-1 | mAP | top-1 | mAP | top-1 | mAP
ReID Module | 78.8 | 79.0 | 100 | 99.8 | 55.7 | 56.3
\+ Enriched Gallery | 91.3 | 77.3 | 99.7 | 93.1 | 82.2 | 60.0
Face Module | 87.1 | 62.7 | 95.6 | 70.0 | 77.9 | 54.7
ReID + Face Modules | 91.2 | 73.2 | 99.8 | 83.3 | 81.9 | 62.2
Ours-CAL | 92.1 | 81.8 | 99.7 | 95.6 | 83.7 | 66.7
Table 7: Ablation study of our method on the PRCC benchmark. ReID+Face Modules is the combination of both face and ReID modules without gallery enrichment. Ours is the complete proposed model that uses both face and ReID modules, and an enriched gallery. The ReID module used is CAL.  Method | Closed-Set | Open-Set
---|---|---
Per Image Acc | Per Track Acc | Per Image Acc | Per Track Acc
ReID Module (Image-based) | 0.31 | N/A | 0.20 | N/A
\+ Enriched Gallery | 0.83 | N/A | 0.71 | N/A
ReID Module (Track-based) | 0.31 | 0.27 | 0.21 | 0.15
\+ Enriched Gallery | 0.88 | 0.76 | 0.77 | 0.61
Face Module (Image-based) | 0.55 | N/A | 0.36 | N/A
Face Module (Track-based) | 0.69 | 0.62 | 0.45 | 0.36
ReID + Face Modules | 0.81 | 0.67 | 0.50 | 0.38
Ours-CTL | 0.92 | 0.81 | 0.79 | 0.65
Table 8: Ablation study of our method on the 42Street dataset. Image/Track-
based refers to whether the labeling decision of the model is done per a
single image or on the entire track, respectively. ReID + Face Modules is the
combination of both face and ReID modules without gallery enrichment. Ours is
the complete proposed model that uses both face and ReID modules, and an
enriched gallery. The ReID module used is CTL. Note, that Per Track Acc is not
applicable for image-based models.

##  Appendix C Using Raw-Data For Gallery Enrichment

As explained in the 42Street dataset creation (Section 4.2), the test data of
this dataset consists of two parts from the play, each around 20 minutes long.
Out of these parts, we extract short videos, 17-seconds each, for evaluation.
In this section, in addition to enriching the gallery with the query data
(from the evaluation videos), we analyze the impact of using more raw-data
(from the test part) for the gallery enrichment process. We start by using
only the evaluation videos for gallery enrichment, and gradually add more
randomly sampled raw data from the test parts. Fig. 7 shows the accuracy of
the compared models as a function of the amount of raw-data used for the
gallery enrichment process. Notice that the initial gallery enrichment with
query data only, already introduces a significant improvement compared to not
using enrichment at all, and that the accuracy increases as we add more raw-
data. This is true for the Image-based ReID model, Track-based ReID model and
our method, on both the closed and open set settings, with the best results
achieved by our method. Even though raw-data is not available in the standard
ReID task, we argue that such data is common in real-world scenarios like the
application we presented, and show how it can be used to boost the performance
of our method.

##  Appendix D Creation of Labeled Galleries

In this section, we explain in detail how to create labeled face and full-body
crop data samples, in a generic, semi-supervised process. We provide the
details for the 42Street dataset as an example, but this process can be
applied on every raw-data video.

This process is illustrated in Fig. 8 and works as follows:

  1. 1.

Given the entire video of the first part (Part1) of the 42Street play, which
is allocated for gallery creation, apply a person detector on every frame in
the video.

  2. 2.

For each detected person-crop, use face detection and feature extraction
modules to extract face features.

  3. 3.

Apply K-means clustering over all face feature vectors with a high number of
clusters compared to $|I|$ ($K>>|I|$), where $I$ is the set of people of
interest.

  4. 4.

Manually iterate over each cluster, remove images that are not of the main
person in the cluster and label the cluster as one of the identities in $I$.

  5. 5.

For each newly labeled face image, automatically label the matching full-body
crop image with the same label.

  6. 6.

Save both labeled full-body crops and face images to generate labelled
galleries.

We note that the manual traversal and labeling of clusters took approximately
2 hours, to curate $\sim$500 images per identity, with a total of $\sim$8K
labeled images.

##  Appendix E The 42Street Database

We save each labeled crop from the 42Street dataset in a designated database,
which includes spatial and temporal information for each crop. Every entry in
the database includes the following information:

  * •

label.

  * •

im_name: unique entry.

  * •

frame_num: the number of the frame in the video from which the crop was taken.

  * •

x1,y1,x2,y2: top-left and bottom-right coordinates of the bounding box.

  * •

conf: the confidence of the person detector that a person exists in the crop.

  * •

vid_name: the name of the video from which the crop was taken

  * •

track_id: the track number given by the tracker.

  * •

crop_id: the number of the crop within the track.

  * •

invalid: a boolean that is set to true for crops that do not present a clear
person.

Figure 8: Process of Labeled Gallery Creation The above process describes how
a raw video is proceeded into labeled full-body and face galleries. For the
42Street dataset, manual traversal and labeling of clusters took approximately
2 hours, to curate $\sim{500}$ images per identity, with a total $\sim{8K}$
labeled images.

##  Appendix F Threshold Details

In Table 9 we detail the detection and similarity thresholds used by our
method for the existing benchmarks. These thresholds are determined according
to the training set for the existing benchmarks, and the validation set for
the 42Street dataset. Moreover, for the 42Street dataset, we used the
evaluation data to extract the following thresholds:

  * •

Detection Threshold: for the gallery enrichment process we used a threshold of
0.8 and during inference we used 0.7. The reason we use a higher threshold for
the gallery enrichment process is that we are interested in creating a highly
accurate gallery. On the other hand, during inference we want to use more
images with faces, and we are more tolerate for mistakes.

  * •

Similarity Threshold: during the gallery enrichment process we use a threshold
of 0.4 for the cosine similarity between an unlabeled sample and the labeled
gallery. That is, if the maximal cosine similarity between the feature vector
of the unlabeled sample and the labeled gallery is below this threshold, this
sample will not be added to the enriched gallery.

  * •

Rank Difference: in addition to detecting only clearly visible faces with a
high similarity to the labeled gallery, we want to use only samples for which
the face model was confident about their identity. We measure this confidence
as the difference between the similarity score of the top-1 and top-2
predictions of the model. In the 42Street dataset, if the difference is below
0.1, we do not add the sample to the enriched gallery.

Under the open-set setting, to label people which are not in the people-of-
interest set as “Unknown”, we set another threshold. This threshold asserts
that a person will be labeled as “Unknown” if the similarity to all people-of-
interest is below 0.3. Meaning that whilst having high detection confidence,
the model is not confident of the identity.

Threshold | Detection | Similarity
---|---|---
CCVID | 0.5 | 0.75
LTCC | 0.8 | 0.5
PRCC | 0.7 | 0.65
LaST | 0.7 | 0.45
Table 9: Threshold values for existing benchmarks. The different detection and
similarity thresholds used for the gallery enrichment process in each of the
existing benchmarks.

## References

  * [1] Hanqing Chao, Yiwei He, Junping Zhang, and Jianfeng Feng.  Gaitset: Regarding gait as a set for cross-view gait recognition.  CoRR, abs/1811.06186, 2018.
  * [2] Jiaxing Chen, Xinyang Jiang, Fudong Wang, Jun Zhang, Feng Zheng, Xing Sun, and Wei-Shi Zheng.  Learning 3d shape feature for texture-insensitive person re-identification.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8142–8151, 2021.
  * [3] MMCV Contributors.  MMCV: OpenMMLab computer vision foundation.  https://github.com/open-mmlab/mmcv, 2018.
  * [4] MMTracking Contributors.  MMTracking: OpenMMLab video perception toolbox and benchmark.  https://github.com/open-mmlab/mmtracking, 2020.
  * [5] Jiankang Deng, Jia Guo, Tongliang Liu, Mingming Gong, and Stefanos Zafeiriou.  Sub-center arcface: Boosting face recognition by large-scale noisy web faces.  In Proceedings of the IEEE Conference on European Conference on Computer Vision, 2020.
  * [6] Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In CVPR, 2019.
  * [7] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou.  Retinaface: Single-shot multi-level face localisation in the wild.  In CVPR, 2020.
  * [8] Jiankang Deng, Anastasios Roussos, Grigorios Chrysos, Evangelos Ververas, Irene Kotsia, Jie Shen, and Stefanos Zafeiriou.  The menpo benchmark for multi-pose 2d and 3d facial landmark localisation and tracking.  IJCV, 2018.
  * [9] Julia Dietlmeier, Joseph Antony, Kevin McGuinness, and Noel E. O’Connor.  How important are faces for person re-identification?  CoRR, abs/2010.06307, 2020.
  * [10] Chanho Eom, Geon Lee, Junghyup Lee, and Bumsub Ham.  Video-based person re-identification with spatial and temporal memory networks, 2021.
  * [11] Dengpan Fu, Dongdong Chen, Jianmin Bao, Hao Yang, Lu Yuan, Lei Zhang, Houqiang Li, and Dong Chen.  Unsupervised pre-training for person re-identification.  Proceedings of the IEEE conference on computer vision and pattern recognition, 2021.
  * [12] Xinqian Gu, Hong Chang, Bingpeng Ma, Shutao Bai, Shiguang Shan, and Xilin Chen.  Clothes-changing person re-identification with rgb modality only.  In CVPR, 2022.
  * [13] Xinqian Gu, Hong Chang, Bingpeng Ma, Shutao Bai, Shiguang Shan, and Xilin Chen.  Clothes-changing person re-identification with rgb modality only.  In CVPR, 2022.
  * [14] Lingxiao He, Xingyu Liao, Wu Liu, Xinchen Liu, Peng Cheng, and Tao Mei.  Fastreid: A pytorch toolbox for general instance re-identification, 2020\.
  * [15] Tianyu He, Xin Jin, Xu Shen, Jianqiang Huang, Zhibo Chen, and Xian-Sheng Hua.  Dense interaction learning for video-based person re-identification, 2021\.
  * [16] Peixian Hong, Tao Wu, Ancong Wu, Xintong Han, and Wei-Shi Zheng.  Fine-grained shape-appearance mutual learning for cloth-changing person re-identification.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10508–10517, 2021.
  * [17] Qingming Leng, Mang Ye, and Qi Tian.  A survey of open-world person re-identification.  IEEE Transactions on Circuits and Systems for Video Technology, 30(4):1092–1108, 2020.
  * [18] Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang.  Person search with natural language description.  CoRR, abs/1702.05729, 2017.
  * [19] Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang.  Deepreid: Deep filter pairing neural network for person re-identification.  2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 152–159, 2014.
  * [20] Xulin Li, Bin Liu, Yan Lu, Qi Chu, and Nenghai Yu.  Cloth-aware center cluster loss for cloth-changing person re-identification.  In Shiqi Yu, Zhaoxiang Zhang, Pong C. Yuen, Junwei Han, Tieniu Tan, Yike Guo, Jianhuang Lai, and Jianguo Zhang, editors, Pattern Recognition and Computer Vision, pages 527–539, Cham, 2022. Springer International Publishing.
  * [21] Xiang Li, Ancong Wu, and Wei-Shi Zheng.  Adversarial open-world person re-identification, 2018.
  * [22] Giuseppe Lisanti, Niki Martinel, Alberto Del Bimbo, and Gian Luca Foresti.  Group re-identification via unsupervised transfer of sparse features encoding.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2468–2477, 2017.
  * [23] Hao Luo, Wei Jiang, Youzhi Gu, Fuxu Liu, Xingyu Liao, Shenqi Lai, and Jianyang Gu.  A strong baseline and batch normalization neck for deep person re-identification.  CoRR, abs/1906.08332, 2019.
  * [24] Niki Martinel, Gian Luca Foresti, and Christian Micheloni.  Aggregating deep pyramidal representations for person re-identification.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1544–1554, 2019.
  * [25] A. Milan, L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler.  MOT16: A benchmark for multi-object tracking.  arXiv:1603.00831 [cs], Mar. 2016.  arXiv: 1603.00831.
  * [26] SOTA music.  42 street - the complete musical, 2013.
  * [27] Athira Nambiar, Alexandre Bernardino, and Jacinto C. Nascimento.  Gait-based person re-identification: A survey.  ACM Comput. Surv., 52(2), apr 2019.
  * [28] E.M. Newton, L. Sweeney, and B. Malin.  Preserving privacy by de-identifying face images.  IEEE Transactions on Knowledge and Data Engineering, 17(2):232–243, 2005.
  * [29] Xuelin Qian, Wenxuan Wang, Li Zhang, Fangrui Zhu, Yanwei Fu, Tao Xiang, Yu-Gang Jiang, and Xiangyang Xue.  Long-term cloth-changing person re-identification.  arXiv preprint arXiv:2005.12633, 2020.
  * [30] Ergys Ristani, Francesco Solera, Roger S. Zou, R. Cucchiara, and Carlo Tomasi.  Performance measures and a data set for multi-target, multi-camera tracking.  In ECCV Workshops, 2016.
  * [31] Xiujun Shu, Xiao Wang, Xianghao Zang, Shiliang Zhang, Yuanqi Chen, Ge Li, and Qi Tian.  Large-scale spatio-temporal person re-identification: Algorithms and benchmark, 2021.
  * [32] Latanya Sweeney.  K-anonymity: A model for protecting privacy.  Int. J. Uncertain. Fuzziness Knowl.-Based Syst., 10(5):557–570, oct 2002.
  * [33] Fangbin Wan, Yang Wu, Xuelin Qian, and Yanwei Fu.  When person re-identification meets changing clothes.  CoRR, abs/2003.04070, 2020.
  * [34] Hanxiao Wang, Xiatian Zhu, Tao Xiang, and Shaogang Gong.  Towards unsupervised open-set person re-identification.  In 2016 IEEE International Conference on Image Processing (ICIP), pages 769–773, 2016.
  * [35] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian.  Person transfer GAN to bridge domain gap for person re-identification.  CoRR, abs/1711.08565, 2017.
  * [36] Mikolaj Wieczorek, Barbara Rychalska, and Jacek Dabrowski.  On the unreasonable effectiveness of centroids in image retrieval, 2021\.
  * [37] Ancong Wu, Wei-Shi Zheng, and Jianhuang Lai.  Robust depth-based person re-identification.  CoRR, abs/1703.09474, 2017.
  * [38] Ancong Wu, Wei-Shi Zheng, Hong-Xing Yu, Shaogang Gong, and Jianhuang Lai.  Rgb-infrared cross-modality person re-identification.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5390–5399, 2017.
  * [39] Hao Xiao, Weiyao Lin, Bin Sheng, Ke Lu, Junchi Yan, Jingdong Wang, Errui Ding, Yihao Zhang, and Hongkai Xiong.  Group re-identification: Leveraging and integrating multi-grain information.  In 2018 ACM Multimedia Conference on Multimedia Conference, pages 192–200. ACM, 2018.
  * [40] Qize Yang, Ancong Wu, and Wei-Shi Zheng.  Person re-identification by contour sketch under moderate clothing change.  CoRR, abs/2002.02295, 2020.
  * [41] Qize Yang, Ancong Wu, and Wei-Shi Zheng.  Person re-identification by contour sketch under moderate clothing change.  IEEE Transactions on Pattern Analysis and Machine Intelligence (DOI 10.1109/TPAMI.2019.2960509), 2020.
  * [42] Renjie Zhang, Yu Fang, Huaxin Song, Fangbin Wan, Yanwei Fu, Hirokazu Kato, and Yang Wu.  Specialized re-ranking: A novel retrieval-verification framework for cloth changing person re-identification.  Pattern Recognition, 134:109070, 2023.
  * [43] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang.  Bytetrack: Multi-object tracking by associating every detection box, 2021\.
  * [44] Ziyuan Zhang, Luan Tran, Xi Yin, Yousef Atoum, Xiaoming Liu, Jian Wan, and Nanxin Wang.  Gait recognition via disentangled representation learning.  CoRR, abs/1904.04925, 2019.
  * [45] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In Computer Vision, IEEE International Conference on, 2015.
  * [46] Xiatian Zhu, Botong Wu, Dongcheng Huang, and Wei-Shi Zheng.  Fast open-world person re-identification.  IEEE Transactions on Image Processing, 27(5):2286–2300, 2018.


# Interstellar Dust and Gas in the Heliosphere

Jonathan Slavin  Center for Astrophysics $\mid$ Harvard & Smithsonian, 60
Garden Street, Cambridge, MA 02138<EMAIL_ADDRESS>
###### Abstract

Interstellar dust and gas that enter the heliosphere provide us with important
clues about both the heliosphere and the local interstellar medium (LISM). The
picture we have of the Local Interstellar Cloud (LIC) from both _in situ_
detections and absorption line data presents questions that have important
implications for the origins and evolution of the cloud. New detections of
60Fe on Earth in deep sea crusts and Antarctic snow cores provide evidence for
the role of supernovae in shaping the LISM. We discuss our models for the
evolution of the LIC inside the Local Bubble and possible explanations for the
source of the supernova produced dust.

##  1 Introduction

The interstellar dust and gas that flow into the heliosphere are coming from
the interstellar cloud that surrounds the Solar System known as the Local
Interstellar Cloud (LIC). To understand both the LIC and the heliosphere we
need information on the state of the LISM. In turn, a knowledge of the current
state of the LISM can provide important information on the history and
evolution of the LISM, in particular the LIC and other nearby clouds (the
Complex of Interstellar Clouds or CLIC) and the Local Bubble that surrounds
the clouds. The detailed observations that we have of the LISM can then help
us to understand interstellar processes and evolution more broadly.

To focus our discussion of the nature of the nearby ISM, I have chosen to
examine four outstanding questions regarding the LISM. The first two are
longstanding questions while the second two are relatively new:

  * •

How did the low density clouds of the CLIC get into the Local Bubble?

  * •

Why is the grain size distribution of interstellar dust in the heliosphere so
strange?

  * •

What is the source of the 60Fe peaks seen in deep sea crusts?

  * •

What does the 60Fe in Antarctic snow cores tell us?

These questions have some interesting interconnections and could hold
important clues to how the interstellar dust and gas evolve.

##  2 The origins of the CLIC

To understand why the existence of the CLIC within the Local Bubble is
mysterious, one needs to be aware of the way the thermal phases in the ISM are
created and evolve, as inferred from both observations and theoretical
modeling. It has been found that the interstellar plasma tends to roughly
segregate into different temperature-density-ionization phases. These are cold
($T\sim 100$ K, $n\sim 50$ cm-3), warm ($T\sim 6000-9000$ K, $n\sim 0.2$ cm-3)
and hot ($T\sim 10^{6}$ K, $n\sim 5\times 10^{-3}$ cm-3). The warm and cold
phases can be in approximate thermal equilibrium given known heating and
cooling processes whereas the hot phase is associated with shock heating and
is not in equilibrium but cools slowly. The warm phase is seen to be widely
and fairly smoothly distributed [see e.g., H$\alpha$ maps, 1], whereas the
cold phase gas tends to be confined to clouds. Such structures are also seen
in Galaxy scale simulations with realistic heating and cooling rates in which
supernovae drive the medium [e.g., 2].

The Local Bubble, which surrounds the CLIC, is filled with hot gas [3, 4, 5]
and must therefore have been created by fast shocks. In fact because of the
size and thermal pressure of the bubble it must have bee created by multiple
supernovae [e.g., 6]. It is well known [7, 8] that supernova remnant (SNR)
shocks propagating through a cloudy medium will sweep around and leave behind
dense clouds that they encounter. However, that implies that the CLIC must
have been denser than the surrounding medium. That would seem to contradict
the fact that the clouds now have densities ($n\sim 0.2$ cm-3) that are
characteristic of the lower density, warm inter-cloud medium.

The solution to this problem is that the clouds _were_ dense and cold at the
time they were encountered in the ISM by the first shock that carved out the
Local Bubble. We must then ask how (and if) the clouds could have evolved from
that cold, dense state to the warm, low density state that we see them in now.
This is a complex question since it involves a number of processes including
the heating and cooling rates caused by photoionization heating and radiative
cooling as well as the effects of the shock(s) that hit the clouds and shear
flow as the surrounding gas sweeps past the clouds.

To assess the possibility that the clouds could be heated in this way to
achieve their current state, we have carried out a series of numerical
simulations. The heating and cooling rates used were realistic and allow for
two stable temperature equilibria to exist, one for cold clouds and one for
warm intercloud gas. A magnetic field is used with a strength typical of that
derived for the ISM (5 $\mu$G) and a supernova explosion is set off near a
complex of cold clouds. We have experimented with the parameters of the
simulations, setting off a second explosion and sometimes a third at intervals
of $\sim 5\times 10^{5}$ yr. A general finding is that two or more explosions
are necessary. We find that indeed, after being compressed by the first
shocks, the clouds do rebound and at late times as the bubble pressure drops
near the currently inferred values of $P/k_{B}\sim 7000$ cm${}^{-3}\,$K, the
clouds are mostly warm with temperatures near those observed for the CLIC (see
Figure 1). (More details of the simulations and results will be presented in a
forthcoming paper.) Thus we conclude that the LIC was originally cold and
dense and has been subject to at least two shocks.

Figure 1: Temperature for a complex of clouds embedded in the hot bubble in
our magnetohydrodynamical model for the LISM. The model has two supernova
explosions that occur at an interval of $5\times 10^{5}$ yr. The clouds, which
were initially cold and dense have rebounded after having been shocked and
have warmed to temperatures similar to those derived for the CLIC.

##  3 The interstellar grain size distribution in the heliosphere

Another longstanding problem is the grain size distribution detected by
several spacecraft in the heliosphere, especially Ulysses [9, 10]. As pointed
out by Frisch et al. [9], Weingartner & Draine [11], and Draine [12] the
detected grain size distribution is far from that inferred for the ISM and is
in fact not consistent with that distribution (see Figure 2). Since the
charged grains in the LIC will be filtered at the heliopause, the fact that
the distribution lacks small grains is not surprising. However, the large
excess of large grains is. Such large grains, if they were prevalent in the
ISM, would produce essentially wavelength independent (gray) extinction to an
extent that would be inconsistent with the wavelength dependence of the many,
many extinction curves observed for Galactic sources. So what has caused this
odd size distribution? Since shocks tend to erode large grains rather than
enhance them, that part of our proposed history for the LIC only exacerbates
the problem. We discuss a possible solution to this mystery below (§7).

Figure 2: Comparison of grain size distribution for interstellar grains
observed in the heliosphere with size distribution models derived from
observations [from 11, used with permission]. The heliospheric distribution
(triangles) is clearly extremely discrepant.

##  4 60Fe in deep sea crusts

A different piece of evidence that may bear on the history of the LISM is the
discovery of a peak in the 60Fe isotope fraction in deep sea ferromanganese
crusts [13, 14]. As can be seen in Figure 3, there was a peak at about 2.8 Myr
ago. 60Fe is not produced in abundance by cosmic rays, but is produced in core
collapse supernova (SN) explosions. Because Fe is ionized essentially
everywhere in the ISM except inside of very dense clouds, in order for the
60Fe, to be transported to the Earth’s surface through the heliosphere, it had
to have been in the form of dust. There is abundant evidence that core
collapse SNe produce dust and so the 60Fe spike can be interpreted as evidence
of a nearby SN that exploded roughly $2\times 10^{6}$ yr ago and created dust
that was transported to the Earth. Because the half life of 60Fe is 2.6 Myr,
the SN could not have exploded much earlier than that.

Figure 3: 60Fe/Fe ratios versus the age of the crust layer [From 13, used with
permission]. The peak at $\sim 2.8$ Myr is evidence for an influx of
supernova-produced dust reaching the Earth at that time.

We note that the Earth was not in the LIC at that time so the connection to
the state of the LIC in particular is not clear. However, it is quite likely
that the explosion that created the dust was one of the explosions that
created the Local Bubble, as has been modeled by [6]. The LIC was also likely
to have been affected by the shock generated by that explosion.

##  5 60Fe in Antarctic snow cores

More directly relevant to the LIC is the detection of 60Fe in Antarctic snow
cores as reported by [15]. Those authors eliminated terrestrial and cosmogenic
sources via the 60Fe/53Mn ratio and conclude that the 60Fe is of interstellar
origin. These cores include snow that has precipitated over the past 20 yr, so
this is evidence that SN created dust is present in the LIC now. However, we
do not know when (and how) the dust got into the LIC. It seems likely that
that dust was also generated in a SN that helped to form the Local Bubble,
however how the dust was transported and captured by the cloud is uncertain.

##  6 Source and transport of 60Fe dust

Aiming to understand the source of the 60Fe in deep sea crusts, Schulreich et
al. [16] modeled the dust transport from a SN that contributed to the heating
of the Local Bubble. For their model of Local Bubble evolution, they use 16
SNe going off over time. They tied their model to the locations and history of
nearby OB associations which could plausibly (based on their stellar content)
have produced SNe during the relevant time period to produce the dust. They
assume that the dust moves with the gas (using passive gas tracers) and are
able to find a good match between the times of the detected 60Fe peaks and the
inferred peak fluxes in their model.

A limitation of the Schulreich et al. [16] model is that they do not consider
sputtering of the grains along the way from the SN explosion to the Earth.
They also ignore the magnetic field in the bubble. In our simulations, we find
that the field morphology becomes quite complex and, although weak on average
compared to its magnitude in the ambient medium, it has regions where it is
strong enough to substantially deflect the grains. Most importantly, the
assumption of close coupling of the grains to the gas is not well founded.

In a project aimed at examining the ability of dust grains formed in SNe to
escape into the ISM, we have carried out simulations that relax the assumption
of close coupling (Slavin et al. 2020, ApJ submitted). Though there is
evidence of efficient production of dust in SN ejecta, a longstanding question
has been how much of that dust survives the passage of the reverse shock to
escape into the ISM.

Our simulations start with grains embedded in dense knots such as those
observed in the Cas A SNR. Both the ejecta knots and the surrounding smooth
ejecta are rapidly expanding into the surrounding medium. The expansion
creates a forward shock and a reverse shock, which propagates back into the
ejecta. The grains are subject drag and sputtering from the surrounding gas
but are otherwise free to move independently of the gas. In this context the
magnetic field is unlikely to be important and we do not include it.

Figure 4: Dust and gas for simulation of dust evolution in a young supernova
remnant. The image shows the density (magnitude in cm-3 is shown in the color
bar) while the blue dots represent dust grains. It can be seen that most of
the dust has propagated ahead of the expanding shock. Here the dust is assumed
to be silicate grains that have an initial radius of $0.25$ $\mu$m. This is
for a time 3000 yr after the explosion.

As can be seen in Figure 4 the grains fully decouple from the ejecta knots and
can escape ahead of the forward shock. Here many of the dust grains (blue
dots) have already escaped into the ISM ahead of the forward shock. However
all have lost some fraction of their initial mass via sputtering and will lose
more mass as they are slowed in the gas. We estimate that only roughly 10% of
the mass in these grains will remain after the grains have slowed below 20 km
s-1. However, larger grains can retain more of their initial mass. We
speculate that grains formed in SNe that explode within pre-existing cavities
may suffer less destruction, though such calculations have yet to be carried
out.

##  7 Capture of dust in the CLIC

The ability of SN created dust to escape the remnant raises interesting
possibilities related to dust in the LIC. If large SN-created grains could
have escaped from a nearby explosion and become embedded in the LIC, it could
explain the surplus of large grains in the LIC – grains added to the cloud
without any addition of gas – as well as the 60Fe dust seen in the Antarctic
snow cores.

Figure 5: Gyroradius of silicate grains propagating through the a low density
warm cloud such as the LIC with $B=3\,\mu$G and $T=7500\,$K.

However there are several stumbling blocks to this explanation of SN dust in
the LIC. First, many models for SN dust formation find that most of the grains
produced are small, $\lesssim 0.1\,\mu$m, though there are substantial
uncertainties in these calculations. Second, the dust needs to be stopped
without being destroyed. We discuss this further below. Finally, the cloud
magnetic field could simply reflect the grain back into the intercloud medium.

Figure 6: The magnetic field near a cloud from MHD simulations. The color
corresponds to the magnitude of the field, while the arrows indicate the field
direction. The red (density) contours show the cloud shapes.

Dust grains are charged in the ISM in general, by impact and sticking of ions
and electrons and by photoelectric ejection of electrons. Their charge and
mass lead to gyroradii in many situations that can be large compared even to
relevant interstellar distance scales, e.g. the size of clouds,

$R_{\mathrm{gyr}}=\frac{8.5\,a(\mu m)^{3}\,v(\mathrm{km/s})}{|Z|\,B(\mu
G)}\;\mathrm{pc},$ (1)

where $a$ is the grain radius, $v$ is the speed of the grain in the direction
perpendicular to the field direction, $Z$ is the number of charges on the
grain and $B$ is the the field strength. To find $Z$, we have carried out
calculations of the charging following [17] and [11] for conditions like those
in the LIC. These lead to gyroradii as shown in Figure 5. Clearly for a fast
moving grain, the gyroradius will exceed the size of the CLIC. More
importantly, the magnetic field only diverts the path of the grains and does
not slow them.

Our MHD calculations of the LISM that we mention in §2 show that the magnetic
field in the Local Bubble is likely disordered at least if not fully
turbulent, with the plasma $\beta\gg 1$ on average, though with small regions
of larger field strength. We find that the field strength does not correlate
simply with density as shown in Figure 6. Such a disordered field could act to
retain grains by deflecting them between clouds enough to allow them to slow.

For the cloud to actually capture the grains they need to be slowed
sufficiently that their velocity does not differ substantially from the gas
they are embedded in. That can only be done by drag. The rule of thumb for
grain slowing is that a grain needs to sweep up a mass of gas roughly equal to
the grain mass. This can be expressed as requiring a column density of

$N\approx\frac{4}{3}a_{gr}\rho_{gr}/m\approx 1.7\times 10^{20}a_{gr}(\mu
m)\;\mathrm{cm}^{-2},$ (2)

where $\rho_{gr}$ is the solid material density of a grain and $m$ is the mean
mass per particle of the gas and the numerical value assumes silicate grains.
The LIC has a column density presently of only $\sim 10^{18}$ cm-2 and even
the entire CLIC has a total column density only a few times that.

Figure 7: Fraction of initial mass remaining in a grain that is slowed by drag
vs. the initial speed of the grain. This plot is for gas abundances like those
in the LIC. Here the assumption is that grain is slowed to a speed below which
it is no longer sputtered, which is $\sim 20$ km s-1.

Another issue is that the same collisions with gas particles that slow grains
sputter them as well. In fact it can be shown [18] that the fraction of mass
left in a grain as it is slowed (in the limit that only inertial sputtering is
important, $T\lesssim 10^{5}$ K) is a only a function of the initial grain
speed. This dependence is illustrated in Figure 7. The assumed gas phase
abundances are those for the LIC from [19] and the grains are assumed to have
slowed below the threshold for sputtering, which for both silicate and
carbonaceous grains is $\sim 20$ km s-1. Thus to have slowed the largest
interstellar grains observed in the heliosphere, $a\sim 1\,\mu$m, requires
that they were initially larger than that by a substantial amount.

Given these pitfalls it would seem unlikely that the grains in the LIC were
injected into the cloud from a nearby SN. However, there is one way out. In
our simulations of the LISM, the clouds of the CLIC start out as cold, dense
($n=28$ cm-3) and smaller in volume as compared their current state. Their
initial column density though, is substantially larger than that of the
current LIC. We do not have any information about when the 60Fe dust got into
the LIC, so it could have been a couple Myr ago and the cloud could have had a
much larger column density at the time – large enough to stop 1 $\mu$m grains.

##  8 Summary

  * •

The existence and characteristics of the Complex of Local Interstellar Clouds
can be explained if they started as cold, neutral clouds embedded in warm ISM
and were overrun by SN shocks that also heated the Local Bubble.

  * •

The 60Fe in deep sea crusts is likely from dust created in one of the
supernovae that created the Local Bubble, and reached Earth before it was in
the LIC.

  * •

The 60Fe in Antarctic snow can be explained if SN created dust was captured by
the LIC, which may also explain the excess of large interstellar grains
detected in the heliosphere.

  * •

Explaining the 60Fe dust both in deep sea crusts and in Antarctic snow
requires more careful examination of dust transport within the Local Bubble.

We thank the organizers for the invitation to present this work and for
putting on an excellent conference under trying circumstances. This research
was supported by NASA Astrophysics Theory Program grant no. NNX17AH80G and a
Smithsonian Institution Scholarly Studies grant. We thank Priscilla Frisch for
helpful conversations. We are grateful to Gunther Korschinek, Thomas
Faestermann and Joe Weingartner for permission to use their figures.

## References

  * [1] Haffner L M, Reynolds R J, Tufte S L, Madsen G J, Jaehnig K P and Percival J W 2003 ApJS 149 405
  * [2] Hill A S, Mac Low M M, Gatto A and Ibáñez-Mejía J C 2018 ApJ 862 55
  * [3] McCammon D, Burrows D N, Sanders W T and Kraushaar W L 1983 ApJ 269 107
  * [4] Galeazzi M, Chiao M, Collier M R, Cravens T, Koutroumpa D, Kuntz K D, Lallement R, Lepri S T, McCammon D, Morgan K, Porter F S, Robertson I P, Snowden S L, Thomas N E, Uprety Y, Ursino E and Walsh B M 2014 Nature 512 171
  * [5] Liu W, Chiao M, Collier M R, Cravens T, Galeazzi M, Koutroumpa D, Kuntz K D, Lallement R, Lepri S T, McCammon D, Morgan K, Porter F S, Snowden S L, Thomas N E, Uprety Y, Ursino E and Walsh B M 2017 ApJ 834 33
  * [6] Breitschwerdt D, Feige J, Schulreich M M, Avillez M A D, Dettbarn C and Fuchs B 2016 Nature 532 73
  * [7] McKee C F and Ostriker J P 1977 ApJ 218 148
  * [8] Slavin J D, Smith R K, Foster A, Winter H D, Raymond J C, Slane P O and Yamaguchi H 2017 ApJ 846 77
  * [9] Frisch P C, Dorschner J M, Geiss J, Greenberg J M, Grün E, Landgraf M, Hoppe P, Jones A P, Krätschmer W, Linde T J, Morfill G E, Reach W, Slavin J D, Svestka J, Witt A N and Zank G P 1999 ApJ 525 492
  * [10] Krüger H, Strub P, Grün E and Sterken V J 2015 ApJ 812 139
  * [11] Weingartner J C and Draine B T 2001 ApJ 548 296
  * [12] Draine B T 2009 Space Sci. Rev. 143 333–345
  * [13] Knie K, Korschinek G, Faestermann T, Dorfi E A, Rugel G and Wallner A 2004 Phys. Rev. Lett. 93 171103
  * [14] Fitoussi C, Raisbeck G M, Knie K, Korschinek G, Faestermann T, Goriely S, Lunney D, Poutivtsev M, Rugel G, Waelbroeck C and Wallner A 2008 Phys. Rev. Lett. 101 121101
  * [15] Koll D, Korschinek G, Faestermann T, Gómez-Guzmán J M, Kipfstuhl S, Merchel S and Welch J M 2019 Phys. Rev. Lett. 123 072701
  * [16] Schulreich M M, Breitschwerdt D, Feige J and Dettbarn C 2017 A&A 604 A81
  * [17] McKee C F, Hollenbach D J, Seab G C and Tielens A G G M 1987 ApJ 318 674
  * [18] Micelotta E R, Dwek E and Slavin J D 2016 A&A 590 A65
  * [19] Slavin J D and Frisch P C 2008 A&A 491 53