# WiC = TSV = WSD:
On the Equivalence of Three Semantic Tasks

Bradley Hauer, Grzegorz Kondrak
Alberta Machine Intelligence Institute, Department of Computing Science
University of Alberta, Edmonton, Canada
<EMAIL_ADDRESS>
###### Abstract

The WiC task has attracted considerable attention in the NLP community, as
demonstrated by the popularity of the recent MCL-WiC SemEval task. WSD systems
and lexical resources have been used for the WiC task, as well as for WiC
dataset construction. TSV is another task related to both WiC and WSD. We aim
to establish the exact relationship between WiC, TSV, and WSD. We demonstrate
that these semantic classification problems can be pairwise reduced to each
other, and so they are theoretically equivalent. We analyze the existing WiC
datasets to validate this equivalence hypothesis. We conclude that our
understanding of semantic tasks can be increased through the applications of
tools from theoretical computer science. Our findings also suggests that more
efficient and simpler methods for one of these tasks could be successfully
applied in the other two.

##  1 Introduction

The word-in-context (WiC) task is to decide whether a word has the same
meaning in two different contexts Pilehvar and Camacho-Collados (2019), which
is closely related to word sense disambiguation (WSD), the task of identifying
the sense of a word in context Navigli (2009). Despite its relevance to
semantics research, the exact relationship between WiC and WSD is still an
open question. Is one of the tasks easier than the other? The goal of this
paper is to shed light on the exact nature of their relationship via
theoretical analysis.

Answering this open question is important as the WiC task has already
attracted considerable attention from the research community. A number of
proposed systems can be applied to both WSD and WiC. The original motivation
of Pilehvar and Camacho-Collados (2019) included “evaluating contextualized
word and sense representations and WSD.” Raganato et al. (2020) present XL-
WiC, an extension of the original WiC dataset to twelve other languages. Breit
et al. (2021) propose the target sense verification (TSV) task, which can be
viewed as intermediate between WiC and WSD: decide whether a single example of
a word in context has a given sense. Most recently, the first SemEval Shared
Task on Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-
WiC) reports 48 actively participating teams, most of which focus on the
English dataset Martelli et al. (2021).

In this short paper, we formalize the WiC, TSV, and WSD problems using a
common notation, and demonstrate that they are theoretically equivalent. Our
argument is based on the observation that the soundness of all three tasks
hinges on the consistency of judgments on the sameness of word meaning. We
show that the tasks are reducible to one another; that is, a solution to one
of them can be used to solve the other two. This finding suggests that the
development of more resource-efficient, less complex methods for one of these
three tasks could yield efficiency improvements in the other two. While we
focus on English in this paper, we postulate that our results will generalize
to other languages.

In order to empirically validate the equivalence hypothesis, we conduct an
analysis of a sample of instances in the MCL-WiC dataset. The results of this
manual annotation experiment indicate that our hypothesis holds with high
probability. This is a surprising finding because top-performing WiC systems
tend to be binary classifiers independent of any predefined sense inventory
Martelli et al. (2021).

This paper has the following structure: In Section 2, we present our
theoretical argument for the equivalence of the three problems. In Section 3,
we provide empirical evidence that the problem reductions work in practice. In
Section 4, we conclude with some general observations on lexical semantics in
light of our work.

##  2 The Equivalence Hypothesis

In this section, we formally define the three problems, present a theoretical
argument for their equivalence, and specify the reductions.

###  2.1 Problem Definitions

  * •

WSD$(C,w)$: Given a context $C$ which contains a single target word $w$,
return the sense $s$ of $w$ in $C$.

  * •

TSV$(C,e,s)$: Given a context $C$ which contains a single target word $w$, and
a sense $s$, return True if $s$ is the sense of $w$ in $C$, and False
otherwise.

  * •

WiC$(C_{1},C_{2},w)$: Given two contexts $C_{1}$ and $C_{2}$ which contain the
same target word $w$, return True if $w$ has the same meaning in both $C_{1}$
and $C_{2}$, and False otherwise.

###  2.2 Problem Equivalence

The theoretical argument for the equivalence of the three problems is based on
the assumption that the relation of sameness of word meaning is shared between
the three problems. This is supported by the lack of distinction between
meanings and senses in the original WiC task proposal.111“The proposed
dataset, WiC, is based on lexicographic examples, which constitute a reliable
basis to […] discern different meanings of words.” Pilehvar and Camacho-
Collados (2019). On the other hand, WordNet exhibits a strict one-to-one
correspondence between distinct meanings, synsets, and concepts, with each
word sense corresponding to a specific synset. This implies that senses are
ultimately grounded in sameness of meaning as well.222“[Each] synonym set
represents one underlying lexical concept. […] Word meaning [refers] to the
lexicalized concept that a [word] form can be used to express.” Miller (1995).
Therefore, every word meaning distinction should correspond to a pair of
distinct senses. Contrariwise, if two tokens of the same word express
different concepts, their meaning must be different. This equivalence also
includes the TSV problem, provided that each sense corresponds to a single
WordNet synset.

###  2.3 Problem Reductions

We now present the three problem reductions.

To reduce WiC to WSD, solve the WSD problem for the target word tokens in both
WiC sentences; return True if both returned senses of the target word are
identical, and False otherwise. Formally:

$\mbox{WiC}(C_{1},C_{2},w)\Leftrightarrow\mbox{WSD}(C_{1},w)=\mbox{WSD}(C_{2},w)$

To reduce WSD to TSV, for each possible sense $s$ of $w$, construct a separate
TSV instance that pairs the original context $C$ with the sense $s$; return
the sense $s$ for which TSV$(C,e,s)$ is True.333The uniqueness of $s$ follows
from the assumption that every content word in a meaningful utterance is used
in exactly one sense. This is empirically supported by the fact that 99.7% of
annotated tokens in SemCor are assigned a single sense. Formally:

$\mbox{WSD}(C,w)=s\Leftrightarrow\mbox{TSV}(C,e,s)$

To reduce TSV to WiC, construct a WiC instance that pairs the original context
$C$ with an example context $C_{s}$ that contains $w$ used in the sense $s$;
return the value of WiC$(C,C_{s},w)$. which is either True or False. Formally:

$\mbox{TSV}(C,e,s)\Leftrightarrow\mbox{WiC}(C,C_{s},w)$

##  3 Datasets

In this section, we discuss and analyze the existing WiC datasets with the aim
of validating the equivalence hypothesis. A genuine exception to the
hypothesis would be an instance that contradicts the WiC-to-WSD reduction
equivalence in Section 2.3. Since natural language is not pure logic,
falsifying the hypothesis would require finding that true exceptions
constitute a a substantial fraction of instances, excluding errors and
omissions in lexical resources.

###  3.1 WiC

The original WiC dataset Pilehvar and Camacho-Collados (2019) consists of 7466
pairs of sentences from WordNet Fellbaum (1998), VerbNet Schuler (2005), and
Wiktionary, divided into training, development, and test sets. Each word sense
in the latter two resources was mapped to a WordNet sense via BabelNet Navigli
and Ponzetto (2010, 2012). The authors report that 68% of the initial
instances were drawn from WordNet, but the source of each sentence is not
indicated in the data. The instances were further filtered to remove fine-
grained sense distinctions. The reported inter-annotator agreement was 80% for
the final pruned set, and only 57% for the pruned-out instances, which is
attributed to the lack of reference sense inventories.

Since, regardless of the source, all instances were annotated automatically by
checking the sense identity in WordNet, the WiC dataset cannot, by
construction, contain any exceptions to the equivalence hypothesis.
Nevertheless, we conducted a partial check by attempting to match each
sentence in the dataset to one of the example sentences in WordNet 3.0. The
fraction of instances in which both sentences are identical to a WordNet
example sentence, allowing us to identify both senses of the target word, was
0.479, 0.534, and 0.522, for the training, development and test sets,
respectively.

Based on these findings, we can implement the WiC-to-WSD reduction defined in
Section 2 by checking whether both input sentences are among the WordNet usage
examples for the target word. Combined with a random back-off, this approach
would be expected to obtain the accuracy of $0.479+0.521/2=0.740$ on the
training set, and 0.767 and 0.761 on the development and test sets,
respectively. We note that this latter figure exceeds the current state-of-
the-art of $0.721$ Levine et al. (2020), which also uses WordNet.

###  3.2 WiC-TSV

Breit et al. (2021) propose _Target Sense Verification_ (TSV), a task of
deciding whether a given word in context is used in a given sense. TSV is
similar to WiC in that it is also a binary classification task, but only one
context per instance is provided. TSV is also similar to WSD in that there is
an explicit representation of senses, but only one sense per instance is
provided. Three TSV sub-tasks are defined depending on the method of defining
a sense: (a) definition, (b) hypernyms, and (c) both definition and hypernyms.
The definition-based subtask closely matches the formulation of TSV that we
formalized in Section 2.1.

The majority of the WiC-TSV datasets of Breit et al. (2021) are derived
directly from the original WiC dataset. The training, development, and one of
the four test sets contain 2137, 389, and 717 instances, respectively. The
remaining three test sets are devoted to cocktail, medical, and computer
terms, and contain 216, 205, and 168 instances, respectively. Since their
focus is on named entities, these three test sets are arguably more relevant
to named entity resolution than to WSD.

The instances that are derived from the original WiC are ultimately based on
WordNet senses. Specifically, at most one sentence from each WiC instance was
taken, and the sense of the word was recovered by reversing the process by
which the WiC instances were created; that is, by finding the source of that
example sentence, and identifying the corresponding WordNet synset (similar to
the procedure we applied to the WiC dataset in Section 3.1). Because of this
dataset construction method, no exceptions to the equivalence hypothesis can
be found in WiC-TSV, just as in WiC.

###  3.3 MCL-WiC

Martelli et al. (2021) introduce the Multilingual and Cross-lingual Word-in-
Context dataset. The English portion of the dataset consists of 10k WiC
instances, divided into a training set (8k instances), as well as development
and test sets (1k instances each). The monolingual task is exactly the same as
the original WiC task, and matches our WiC problem formalization in Section
2.1.

In contrast with the original WiC dataset, which was mostly derived from
WordNet, the sentence pairs in MCL-WiC were manually selected and annotated by
human annotators. Annotators were instructed to consult “multiple reputable
dictionaries” to minimize the subjectivity of the annotation process, and
provide support for their decisions. The inter-annotator agreement is reported
as $\kappa=0.968$, which is much higher than on the original WiC dataset. The
best-performing shared task system Gupta et al. (2021) on the English dataset
achieved 93.3% accuracy, which is also much higher than the results on the
original WiC dataset.

####  3.3.1 Annotation Experiment

The MCL-WiC dataset provides a unique test for the equivalence of WiC and WSD
because its annotation method is independent of WordNet. We analyzed 60
randomly selected instances from the English training set by manually
identifying the WordNet sense of each target word. We found that 55 out of 60
instances conform to the equivalence hypothesis. The remaining five instances
can be divided into three categories: (1) tokenization errors in MCL-WiC; (2)
missing senses in WordNet; (3) possible annotation errors in MCL-WiC. We
discuss these three types of errors below.

In two instances, word tokenization errors interfere with the MCL-WiC
annotations: (1) together in “the final coming together” is annotated as an
adverb instead of a particle of a phrasal verb, and (2) shiner in “shoes
shiners met the inspector” is annotated as a stand-alone noun instead of a
part of a compound noun. These tokenization errors prevent the proper
assignment of WordNet senses.

In two instances, one of the senses of the target word is missing in WordNet:
(1) breach referring to an animal breaking through the surface of the water,
and (2) spotter referring to a member of a motor racing team who communicates
by radio with the driver. Neither of these senses is subsumed by another sense
in WordNet, and both of them are present in the Oxford English
dictionary444https://www.lexico.com, which was consulted by the MCL-WiC
annotators.

In the final problematic instance, MCL-WiC classifies the noun campaign as
having the same meaning in the contexts “during the election campaign” and
“the campaign had a positive impact on behavior.” The distinction between
these two senses of campaign is found in Oxford and WordNet, but not in
Merriam-Webster or Wiktionary. Since the exact algorithm for resolving
differences between dictionaries is not specified by Martelli et al. (2021),
we are unable to decide whether this is an actual exception to the hypothesis,
or simply an annotation error in MCL-WiC.

In summary, a careful analysis of the sample of 60 instances reveals no clear
exceptions to the equivalence hypothesis. If we conservatively assume that the
instance involving the word campaign does represent an actual exception, the
binomial confidence interval for the probability of the WiC-to-WSD reduction
being successful is $0.983\pm 0.032$, which is at the level of inter-annotator
agreement. We therefore conclude that the results of the annotation experiment
strongly support the hypothesis.

##  4 Discussion

Having presented a theoretical argument for the equivalence of WiC, WSD, and
TSV in Section 2, and empirical evidence for the correctness of one of the key
reductions in Section 3, we now discuss our inferences, making the case for
WordNet as the underlying sense inventory in WiC.

Most English WiC and TSV datasets are based on predefined sense inventories
that come either directly or indirectly from WordNet. If no sense inventory is
used for grounding decisions about meaning, the inter-annotator agreement is
reported to be about 80% Pilehvar and Camacho-Collados (2019); Breit et al.
(2021). For the MCL-WiC dataset, however, annotators consulted other
dictionaries, and obtained “almost perfect agreement" Martelli et al. (2021).
This suggests that sense inventories, and semantic resources in general, are
crucial to reliable annotation for semantic tasks. However, because the exact
MCL-WiC procedure for resolving differences between dictionaries is
unspecified, and because such dictionaries vary in their availability, the
correctness of the annotations cannot be verified (c.f. Section 3.3).

The adoption of WordNet as the standard sense inventory for WiC has several
practical benefits: (1) it has been adopted as the standard inventory for WSD,
and so would simplify multi-task evaluation; (2) it allows seamless
application of systems across datasets; (3) it facilitates rapid creation of
new WiC datasets based on existing sense-annotated corpora; (4) it is freely
available; (5) it can be modified to correct errors and omissions; and finally
(6) it can be extended to facilitate work with other languages, as in the XL-
WiC dataset.

In addition, WordNet has strong theoretical advantages. Its fine-granularity
is a consequence of the underlying principle of distinguishing lexical
concepts. Therefore, the sense distinctions found in dictionaries either
already correspond to different concepts, or should result in adding new
concepts to WordNet. Furthermore, unlike in dictionaries, sense inventories of
different words are inter-connected via semantic relations such as synonymy
and hypernymy, which facilitate an objective assignment of every word usage to
a single WordNet concept. This has allowed the WSD methods based on contextual
and sense embeddings to surpass the inter-annotator agreement ceiling of
around 70% Navigli (2006).

##  5 Conclusion

We proposed a novel hypothesis on the equivalence of three semantic tasks, and
demonstrated how they can be reduced to one another. We analyzed the existing
datasets to verify that the reductions work in practice. We further
corroborated our conclusions by performing an annotation experiment on a
dataset constructed independently from WordNet. The hypothesis provides
theoretical support for the application of the WiC-to-WSD reduction. This
finding is based on the properties of the WordNet sense inventory, which we
have argued are highly desirable in semantics research. Future work could
include applying techniques from WiC research to WSD, or joint model
optimization for both tasks.

## References

  * Breit et al. (2021) Anna Breit, Artem Revenko, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2021.  WiC-TSV: An evaluation benchmark for target sense verification of words in context.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1635–1645.
  * Fellbaum (1998) Christiane Fellbaum, editor. 1998.  _WordNet: An Electronic Lexical Database_.  MIT Press.
  * Gupta et al. (2021) Rohan Gupta, Jay Mundra, Deepak Mahajan, and Ashutosh Modi. 2021.  Mcl@ iitk at semeval-2021 task 2: Multilingual and cross-lingual word-in-context disambiguation using augmented data, signals, and transformers.  In _Proceedings of the Fifteenth Workshop on Semantic Evaluation_.
  * Levine et al. (2020) Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020.  SenseBERT: Driving some sense into BERT.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4656–4667.
  * Martelli et al. (2021) Federico Martelli, Najla Kalach, Gabriele Tola, and Roberto Navigli. 2021.  SemEval-2021 task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC).  In _Proceedings of the Fifteenth Workshop on Semantic Evaluation (SemEval-2021)_.
  * Miller (1995) George A Miller. 1995.  WordNet: A lexical database for English.  _Communications of the ACM_ , 38(11):39–41.
  * Navigli (2006) Roberto Navigli. 2006.  Meaningful clustering of senses helps boost word sense disambiguation performance.  In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_ , pages 105–112.
  * Navigli (2009) Roberto Navigli. 2009.  Word sense disambiguation: A survey.  _ACM Computing Surveys (CSUR)_ , 41(2):10.
  * Navigli and Ponzetto (2010) Roberto Navigli and Simone Paolo Ponzetto. 2010.  BabelNet: Building a very large multilingual semantic network.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 216–225.
  * Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012.  BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network.  _Artificial Intelligence_ , 193:217–250.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  Wic: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of NAACL-HLT_ , pages 1267–1273, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Raganato et al. (2020) Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2020.  XL-WiC: A multilingual benchmark for evaluating semantic contextualization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7193–7206.
  * Schuler (2005) Karin Kipper Schuler. 2005.  _VerbNet: A broad-coverage, comprehensive verb lexicon_.  University of Pennsylvania.


# Learning Transferable Pedestrian Representation from Multimodal
Information Supervision

Liping Bao1 Longhui Wei1 Xiaoyu Qiu1 Wengang Zhou1
Houqiang Li1 Qi Tian2 1University of Science and Technology of China 2Huawei
Cloud
{baoliping<EMAIL_ADDRESS><EMAIL_ADDRESS>
{zhwg<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent researches on unsupervised person re-identification (reID) have
demonstrated that pre-training on unlabeled person images achieves superior
performance on downstream reID tasks than pre-training on ImageNet. However,
those pre-trained methods are specifically designed for reID and suffer
flexible adaption to other pedestrian analysis tasks. In this paper, we
propose VAL-PAT, a novel framework that learns transferable representations to
enhance various pedestrian analysis tasks with multimodal information. To
train our framework, we introduce three learning objectives, _i.e.,_ self-
supervised contrastive learning, image-text contrastive learning and multi-
attribute classification. The self-supervised contrastive learning facilitates
the learning of the intrinsic pedestrian properties, while the image-text
contrastive learning guides the model to focus on the appearance information
of pedestrians. Meanwhile, multi-attribute classification encourages the model
to recognize attributes to excavate fine-grained pedestrian information. We
first perform pre-training on LUPerson-TA dataset, where each image contains
text and attribute annotations, and then transfer the learned representations
to various downstream tasks, including person reID, person attribute
recognition and text-based person search. Extensive experiments demonstrate
that our framework facilitates the learning of general pedestrian
representations and thus leads to promising results on various pedestrian
analysis tasks.

Figure 1: VAL-PAT achieves the best performance on a series of downstream
pedestrian analysis tasks compared with other pre-trained models. “INsup”
refers to the supervised pre-trained model on ImageNet. “LUP” refers to MoCo
v2 pre-trained on LUPerson. “UP-ReID” represents the existing best pre-trained
model designed for person reID.

##  1 Introduction

As an emerging research topic in computer vision, pedestrian analysis has
attracted substantial attention thanks to its significant potential in video
surveillance applications. Generally, pedestrian analysis involves many tasks,
such as person re-identification (reID) [83, 84, 63, 77], pedestrian attribute
recognition [65, 33, 32], and text-based person search [38, 56, 60, 64]. With
the rise of deep learning, the state-of-the-art methods on pedestrian analysis
heavily rely on deep neural networks trained with large amounts of annotated
data [72, 37, 43, 38], which is a non-trivial issue, as it involves annotating
fine-grained pedestrian attributes or associating person images captured by
different cameras. With limited labeled data, most of current methods [63, 30,
77, 12] are firstly pre-trained on ImageNet and then fine-tuned on downstream
pedestrian datasets. Nonetheless, those solutions suffer a domain gap between
the ImageNet [14] and the pedestrian datasets, since the former consists of
general images and focuses on category-level classification while the latter
only contains pedestrain images and focuses on instance-level classification.

Recently, self-supervised learning [26, 8, 25, 2] has shown remarkable success
in visual pre-training tasks. Inspired by these advances, Fu et al. [19] make
the pioneering attempt at large-scale unsupervised pre-training for person
reID and establish a large-scale unlabeled person reID dataset, LUPerson.
However, this method directly utilizes MoCo v2 [9], a self-supervised method
designed for the ImageNet, without taking into account the unique
characteristics of the person reID tasks. To address this issue, UP-ReID [76]
is further developed to consider the global image-level and local patch-level
features simultaneously, allowing for the extraction of fine-grained clues
from person images. It is worth noting that these self-supervised methods are
designed specifically for person reID and may suffer flexible adaption to
other pedestrian analysis tasks such as person attribute recognition and text-
based person search. As shown in Figure 1, LUP and UP-ReID only achieve
marginal performance improvements over INsup on person attribute recognition
and text-based person search tasks.

To learn general pedestrian representations that transfer well to various
pedestrian analysis tasks, we introduce a novel multimodal learning framework
trained with Vision, Attribute and Language supervisions, named as VAL-PAT.
Unfortunately, none of the existing large-scale pedestrian datasets contain
both text and attribute annotations. Therefore, we construct a multimodal
pedestrian dataset LUPerosn-TA, where each image is paired with a
corresponding textual description and attribute annotations. Instead of
crawling pedestrian image-text pairs from Web, we utilize a model on image
captioning [75] to generate textual descriptions for the images in LUPerson,
and then extract common nouns and adjectives from each textual description as
attributes for its corresponding image.

Based on the aforementioned multimodal pedestrian dataset, we introduce three
optimization objectives to train our proposed framework. We first employ self-
supervised contrastive loss (SSL) to minimize the distance between different
views of the same image while maximizing the distance between different
images, with the aim of extracting the inherent properties of the pedestrian
images. We also use image-text contrastive loss (ITC) to pull matched image-
text pairs closer while pushing away non-matched ones. Since texts provide
comprehensive and detailed descriptions of the pedestrians’ appearance, ITC
encourages the model to prioritize information about the pedestrians’
appearance, while disregarding unnecessary information like background.
Additionally, a multi-attribute classification loss (MAC) is introduced to
guide the model to recognize the attributes presented in the image-text pairs,
thereby facilitating the extraction of fine-grained information. Incorporating
these three objectives into a single multimodal learning framework can enable
the candidate pre-trained model to learn more general and robust
representations that can easily transfer to various pedestrian-related tasks.

In summary, our major contributions are three-fold as follows:

  * •

We introduce a simple but effective framework, VAL-PAT, optimized with three
objectives, _i.e.,_ image self-supervised contrastive loss, image-text
contrastive loss and multi-attribute classification loss. To our best
knowledge, this is the first attempt to learn general representations that
transfer well to various pedestrian analysis tasks.

  * •

To train the framework, we construct a large-scale multimodal pedestrian
dataset, LUPerson-TA, where each image corresponds to a textual description
and attribute annotations.

  * •

Extensive experiments on various pedestrian analysis tasks reveal that our
VAL-PAT achieves superior transfer performances on most of pedestrian
datasets, which validates the effectiveness of our proposed framework.

##  2 Related Works

###  2.1 Visual Representation Learning

In the past two decades, supervised learning [35, 28] has played a dominant
role in visual representation learning. Pre-trained weights on ImageNet have
been widely used in various downstream vision tasks, such as object detection
[53, 52, 42], semantic segmentation [27, 44, 6], person reID [63, 30, 12, 82],
_etc._ , to alleviate the problem of insufficient labeled data. Recently, the
prevailing trend has been disrupted by self-supervised learning based on
contrastive learning [26, 8, 70] and masked image modeling [2, 25, 74], which
learns better transferable representations with only unlabeled images.
Contrastive learning takes the discriminative learning paradigm by maximizing
the similarity between different views of the same image while minimizing the
similarity between different images. Differently, masked image modeling works
in a generative way by reconstructing the masked part of the original input
from the given visible parts.

Instead of image self-supervision, some researchers have explored language as
supervision to learn meaningful visual representations. For example, VirTex
[15] has demonstrated that natural language can provide more data-efficient
supervision for learning transferable visual representations. Recently, CLIP
[51] and ALIGN [31] have achieved impressive zero-shot classification
performance by leveraging image-text contrastive learning on 400M/1B image-
text pairs. Furthermore, combining image self-supervision and language
supervision, SLIP [49] significantly improves performances in the zero-shot
transfer, linear classification and fine-tuning settings compared to CLIP and
SIMCLR [8], showing language can effectively aid image self-supervised
learning to learn better representations.

Figure 2: Illustration of our pre-training framework and LUPerson-TA dataset.
(a) is the overview of our framework, optimized with image self-supervised
contrastive loss $\mathcal{L}_{I}$, image-text contrastive loss
$\mathcal{L}_{ITC}$ and multi-attribute classification loss $\mathcal{L}_{A}$.
We leverage the logits computed over $k^{\prime}_{T_{i}}$ and
$k^{\prime}_{I_{i}}$ as soft labels to alleviate the impact of noise
attributes while computing $\mathcal{L}_{A}$. (b) shows several image-text
pairs in the LUPerson-TA, where the sentences can describe pedestrians’
appearance in details. (c) is a histogram of the top-15 attributes. The “M”
represents million.

###  2.2 Unsupervised Pretraining for Person Re-identification

Limited by the size of labeled reID datasets, previous works [12, 63, 30]
usually load weights pre-trained on ImageNet before training on reID datasets.
However, it may be unreasonable that there exists a large domain gap between
ImageNet and reID datasets. Recently, Fu et al. [19] first establish a large-
scale unlabeled person reID dataset, LUperson, and verify that pre-training on
LUPerson significantly improves performances on downstream reID tasks compared
to pre-training on ImageNet. Luo et al. [47] extensively investigate the
influence of different model architectures and pre-trained methods and
demonstrate that the transformer-based DINO [4] can achieve better
performance. However, these works directly utilize self-supervised learning
methods designed for ImageNet classification without specialized adaption for
reID tasks. Yang et al. [76] propose a reID-specific pre-training framework
UP-ReID and introduce global consistency and intrinsic contrastive constraints
to learn augmentation-invariant and fine-grained representations.
Concurrently, Zhu et al. [88] propose PASS based on DINO, which automatically
extracts part-level features to offer fine-grained information. Nevertheless,
these pre-training models are tailored for reID tasks and may not be
transferred to other pedestrian-related tasks, such as pedestrian attribute
recognition, text-based person search, _etc_. In this paper, we propose a
novel multimodal learning framework that utilizes multimodal supervision to
guide model to learn general pedestrian representations beneficial for a
series of downstream pedestrian analysis tasks.

##  3 VAL-PAT Framework

In this section, we present our proposed multimodal learning framework VAL-PAT
in details. As shown in Figure 2(a), VAL-PAT is optimized using image self-
supervised contrastive loss, image-text contrastive loss and multi-attribute
classification loss. To this end, text and attribute annotations are required
for each pedestrian image. Unfortunately, none of the existing pedestrian
datasets fully satisfy these requirements. Therefore, we first introduce how
to construct a multimodal pedestrian dataset, where each image is paired with
a corresponding textual description and attribute annotations. Then, we
elaborate on each optimization objective and present the full pre-training
objective.

###  3.1 LUPeron-TA Dataset

The typical approaches[51, 31, 58] to collect numerous image-text pairs are
through crawling from Web. However, crawling image-text pairs featuring
pedestrian is more challenging due of the limitation of image category and the
requirements for high-quality text that can describe the pedestrians’
appearance in details. To tackle this issue, we develop a simple method based
on LUPerson dataset to construct a pedestrian dataset with text and attribute
annotations and name this dataset as LUPerson-TA. We first train an image
captioning model [75] from scratch on the CUHK-PEDES [38] and ICFG-PEDES [17]
datasets, which are commonly used in text-based person search tasks and where
each image has one or two textual descriptions. Next, we utilize the trained
model to generate textual descriptions for each image in the LUPerson dataset.
We provide some examples of image-text pairs in Figure 2(b), where these texts
describe the characteristics of pedestrians in details. To obtain attribute
annotations, we extract all nouns and adjectives from these descriptions using
the NLTK tagger [3] and retain only the most frequently occurring ones,
resulting in a list of 1359 attributes. The histogram of the top-15 attributes
is demonstrated in Figure 2(c).

###  3.2 Self-supervised Contrastive Learning

One of the representative paradigms in visual self-supervised learning is
contrastive learning [26, 8, 73], which can be considered as an instance
classification task where each image is treated as a separate class. Instead
of traditional classifier, contrastive learning leverages the “noise
contrastive estimator (NCE) [24]” to maximize the similarity between different
augmented views of the same image and minimize the similarity between
different images. Specifically, given an image $I_{i}$ and two separate data
augmentation operators $t$ and $t^{\prime}$ sampled from the same family of
augmentations, we can obtain two augmented views $t\left(I_{i}\right)$ and
$t^{\prime}\left(I_{i}\right)$ of the image $I_{i}$, one encoded by image
encoder $E_{q}^{I}$ as $q_{I_{i}}$ and the other encoded by image encoder
$E_{k}^{I}$ as $k_{I_{i}}$, where $E_{k}^{I}$ is the momentum updating version
of $E_{q}^{I}$. Moreover, we maintain a queue of negative samples
$\left\\{k_{I_{1}}^{-},k_{I_{2}}^{-},\cdots,k_{I_{K}}^{-}\right\\}$, which are
encoded by $E_{k}^{I}$ over augmented views from different images. Formally,
the image self-supervised contrastive loss is defined as:

$\mathcal{L}_{I}=-\text{log}\frac{\text{exp}(q_{I_{i}}\cdot
k_{I_{i}}/\tau)}{\text{exp}(q_{I_{i}}\cdot
k_{I_{i}}/\tau)+\sum_{j=1}^{K}\text{exp}(q_{I_{i}}\cdot k_{I_{j}}^{-}/\tau)},$
(1)

where $\tau$ is a temperature hyper-parameter, and $K$ is the number of
negative samples.

###  3.3 Image-text Contrastive Learning

CLIP [51] and ALIGN [31] have demonstrated that utilizing contrastive loss to
maximize the similarity between the matched image-text pairs, while minimizing
the similarity between the non-matched pairs, encourages the model to learn
rich semantic visual representations. To be specific, given an image-text pair
$\\{I_{i},T_{i}\\}$, we adopt two modality-specific encoders $E_{q}^{I}$ and
$E_{q}^{T}$ to extract the image query $q^{\prime}_{I_{i}}$ and text query
$q^{\prime}_{T_{i}}$, respectively. Simultaneously, image key
$k^{\prime}_{I_{i}}$ and text key $k^{\prime}_{T_{i}}$ are obtained by using
$E_{k}^{I}$ and $E_{k}^{T}$, which are momentum updating versions of
$E_{q}^{I}$ and $E_{q}^{T}$, respectively. Similar to images self-supervised
contrastive loss, we also maintain two negative queues, the image negative
queue
$\left\\{k_{I_{1}}^{\prime-},k_{I_{2}}^{\prime-},\cdots,k_{I_{K}}^{\prime-}\right\\}$
and the text negative queue
$\left\\{k_{T_{1}}^{\prime-},k_{T_{2}}^{\prime-},\cdots,k_{T_{K}}^{\prime-}\right\\}$,
to expand the number of negative samples. The contrastive loss, in the
scenario of image-text pairs, is given as:

$\displaystyle\mathcal{L}_{I2T}=-\text{log}\frac{\text{exp}(q^{\prime}_{I_{i}}\cdot
k^{\prime}_{T_{i}}/\tau^{\prime})}{\text{exp}(q^{\prime}_{I_{i}}\cdot
k^{\prime}_{T_{i}}/\tau^{\prime})+\sum_{j=1}^{K}\text{exp}(q^{\prime}_{I_{i}}\cdot
k^{\prime-}_{T_{j}}/\tau^{\prime})},$
$\displaystyle\mathcal{L}_{T2I}=-\text{log}\frac{\text{exp}(q^{\prime}_{T_{i}}\cdot
k^{\prime}_{I_{i}}/\tau^{\prime})}{\text{exp}(q^{\prime}_{T_{i}}\cdot
k^{\prime}_{I_{i}}/\tau^{\prime})+\sum_{j=1}^{K}\text{exp}(q^{\prime}_{T_{i}}\cdot
k^{\prime-}_{I_{j}}/\tau^{\prime})},$ (2)

where $\tau^{\prime}$ is the temperature to scale the logits, which is learned
together with all other parameters.

###  3.4 Multi-Attribute Classification

While self-supervised contrastive learning and image-text contrastive learning
can result in powerful representations, they focus solely on exploring the
global features of images and texts, potentially neglecting significant fine-
grained information. Motivated by this, we introduce a multi-attribute
classification objective, which is a multi-label classification task. By
training model to recognize attributes (_e.g._ , backpack, tie, watch) in both
images and texts, the model can mine fine-grained local clues and learn more
discriminative representations. Moreover, as illustrated in Figure 2(a),
applying multi-attribute classification to image and text encoders encourages
both encoders to capture the same fine-grained attributes, facilitating fine-
grained alignment between image and text modalities.

Formally, given an attribute label $\bm{y}_{i}\in\left\\{0,1\right\\}^{M}$ and
an attribute classifier $W_{q}$ with bias $\bm{z}_{q}$, where $M$ denotes the
number of attributes and the zeros and ones indicate the absence and presence
of the corresponding attribute, we adopt the widely used binary cross-entropy
loss in pedestrian attribute recognition [32] as our optimization target:

$\begin{split}\mathcal{L}_{A}^{H}=\sum_{j=1}^{N}\Big{(}&w_{j}\left(y_{i,j}\log
p_{i,j}^{I}+\left(1-y_{i,j}\right)\log\left(1-p_{i,j}^{I}\right)\right)\\\
+&w_{j}\left(y_{i,j}\log
p_{i,j}^{T}+\left(1-y_{i,j}\right)\log\left(1-p_{i,j}^{T}\right)\right)\Big{)},\end{split}$
$p_{i}^{I}=\sigma\left(W_{q}q^{\prime}_{I_{i}}+\bm{z}_{q}\right),p_{i}^{T}=\sigma\left(W_{q}q^{\prime}_{T_{i}}+\bm{z}_{q}\right),\vspace{1mm}$
(3)

where $\bm{p}_{i,j}^{I}$ and $\bm{p}_{i,j}^{T}$ is the prediction probability
of $j$-th attribute, $\sigma\left(\cdot\right)$ is the sigmoid function and
$w_{j}$ is the attribute weight of $j$-th attribute to alleviate the
distribution imbalance between attributes.

However, since the mined attribute labels are noisy, using the hard binary
attribute labels $y_{i}$ as supervision causes the model to overfit to noise,
which significantly harms to transferability of the model. To tackle this, we
adopt the prediction probability $\hat{y}_{i}^{I}$ and $\hat{y}_{i}^{T}$
computed over $k^{\prime}_{I_{i}}$ and $k^{\prime}_{T_{i}}$ as soft attribute
labels. To further enhance the fine-grained alignment between image and text
modalities, we use $\hat{y}_{i}^{I}$ and $\hat{y}_{i}^{T}$ as the soft label
for $p_{i}^{T}$ and $p_{i}^{I}$, respectively, based on the assumption that
matched image-text pairs should predict the same attributes. The binary cross-
entropy loss is organized as follows:

$\begin{split}\mathcal{L}_{A}^{S}=\sum_{j=1}^{N}\Big{(}&w_{j}\left(\hat{y}^{T}_{i,j}\log
p_{i,j}^{I}+\left(1-\hat{y}^{T}_{i,j}\right)\log\left(1-p_{i,j}^{I}\right)\right)\\\
+&w_{j}\left(\hat{y}^{I}_{i,j}\log
p_{i,j}^{T}+\left(1-\hat{y}^{I}_{i,j}\right)\log\left(1-p_{i,j}^{T}\right)\right)\Big{)},\end{split}$
$\hat{y}_{i}^{I}=\sigma\left(W_{k}k^{\prime}_{I_{i}}+\bm{z}_{k}\right),\hat{y}_{i}^{T}=\sigma\left(W_{k}k^{\prime}_{T_{i}}+\bm{z}_{k}\right),$
(4)

where $W_{k}$ and $\bm{z}_{k}$ is the momentum updating versions of $W_{q}$
and $\bm{z}_{q}$ .

Finally, the full pre-trained objective can be expressed as:

$\displaystyle\mathcal{L}=\mathcal{L}_{I}+\beta\left(\mathcal{L}_{I2T}+\mathcal{L}_{T2I}\right)+\gamma\left(\left(1-\alpha\right)\mathcal{L}_{A}^{H}+\alpha\mathcal{L}_{A}^{S}\right),$
(5)

where $\alpha$, $\beta$ and $\gamma$ are hyper-parameters balancing the
importance of different objectives.

##  4 Experiments

###  4.1 Pretraing Implementation

To ensure a fair comparison with previous approaches, we use a standard
ResNet50 [28] as our image encoder, while our text encoder is a 12-layer,
8-head transformer [66] with a hidden size of 512, following CLIP. All
experiments are conducted using the PyTorch framework [50] on 8 NVIDIA V100
GPUs. The model is trained from scratch with randomly initialized weights,
using an AdamW optimizer [45] with a weight decay of 0.1. We train the model
for 140 epochs, with a batch size of 2048. The learning rate is initialized at
$1e^{-6}$ and gradually increased to $1e^{-3}$ after 7 epochs, before being
decreased to $1e^{-6}$ using the linear decay strategy. The trainable hyper-
parameter $\tau^{\prime}$ is initialized as 0.07. For other parameters, unless
otherwise stated, we set $K=65536$, $\tau=0.07$, $\alpha=0.2$, $\beta=0.5$,
$\gamma=0.01$, and $M=1359$. We adopt two data augmentation strategies for
different objectives. For image self-supervised contrastive loss, we use the
same data augmentation strategy as [19]. For image-text contrastive loss and
multi-attribute classification loss, we use simple data augmentation
techniques, including random horizontal flip and image normalization.

###  4.2 Various Downstream Tasks

We evaluate the capabilities of VAL-PAT on multiple pedestrian analysis tasks,
including fully-supervised reID, unsupervised reID, occluded reID, person
attribute recognition and text-based person search.

pre-train | PCB [63] | BOT [46] | MGN [69]
---|---|---|---
INsup | 74.6/86.7 | 76.4/86.4 | 79.4/89.0
LUP | 78.2/87.3 | 79.2/88.5 | 82.1/91.0
UP-ReID |  79.0/88.5 |  80.0/89.1 | 82.1/90.9
VAL-PAT | 78.6/89.5 |  80.0/88.6 |  83.0/91.5
(a) DukeMTMC

pre-train | PCB [63] | BOT [46] | MGN [69]
---|---|---|---
INsup | 80.9/92.7 | 86.7/94.8 | 85.3/94.3
LUP | 80.7/92.9 | 86.5/94.6 | 85.8/94.2
UP-ReID | 81.7/93.2 | 88.0/95.3 |  89.7/96.1
VAL-PAT | 86.0/94.9 |  88.7/96.1 | 89.6/96.3
(b) PersonX

Table 1: Comparison of different pre-trained models on three representative
fully-supervised reID methods. The first number is mAP and the second is
Rank-1.

pre-train | SPCL [22] | RLCC [79] | C-Contrast [13]
---|---|---|---
INsup | 65.3/81.2 | 69.2/83.2 | 72.6/84.9
LUP | 67.1/81.6 | 69.3/82.6 | 72.7/84.7
UP-ReID | 63.0/79.8 | 71.3/83.3 | 73.4/85.3
VAL-PAT | 69.8/83.5 |  73.0/85.3 | 74.9/86.1
(a) DukeMTMC

pre-train | SPCL [22] | RLCC [79] | C-Contrast [13]
---|---|---|---
INsup | 19.1/42.3 | 27.9/56.5 | 33.0/62.0
LUP | 11.3/25.6 | 21.4/45.1 | 22.9/48.5
UP-ReID | 10.6/23.9 | 20.2/42.9 | 24.2/49.7
VAL-PAT |  29.5/56.6 |  38.9/67.5 | 36.1/64.9
(b) MSMT17

Table 2: Comparison of different pre-trained models on three representative
unsupervised reID methods. The first number is mAP and the second is Rank-1.

Fully-supervised person reID aims to associate images of a specified
pedestrian from a large pool of images. We conduct experiments on two reID
datasets, DukeMTMC-reID [54] and PersonX [62]111We only test on DukeMTMC-reID
and PersonX, because CUHK-PEDES and ICFG-PEDES, which are used to train image
captioning model, contain images from Market1501 and MSMT17, respectively..
DukeMTMC-reID contains 16,522 training images of 702 identities, 2,228 query
images of the other 702 identities and 17,661 gallery images. PersonX is a
synthetic reID dataset that consists of 45,792 images of 1,266 identities from
6 cameras, with 9,840 images of 410 identities for training, 5,136 images of
856 identities for query and the other images for gallery. We evaluate the
performance using mean Average Precision (mAP) and cumulative matching
characteristic (CMC), as done in [19]. To validate the transferability of our
VAL-PAT, we utilize three representative reID baselines: PCB [63], BOT [46],
and MGN [69].

Unsupervised Person reID aims to learn discriminative pedestrian
representations without annotations. It can be categorized into unsupervised
domain adaptation and unsupervised learning, depending on the availability of
labeled source data during training. In this task, we focus on the setting
where no labeled source data are available, and evaluate the performance on
two widely used datasets: DukeMTMC-reID and MSMT17 [72]222To test on MSMT17,
we train image captioning model with only CUHK-PEDES and keep the other
setting unchanged to avoid information leakage.. MSMT17 is the largest labeled
reID dataset, containing 126,441 images of 4,101 identities. We use the same
metrics as in fully-supervised reID and select three state-of-the-art
unsupervised reID methods, SpCL [22], RLCC [79] and C-Contrast [13], to
evaluate the effectiveness of our pre-trained model.

Occluded Person reID aims identify the same person with a full-body appearance
in a disjoint camera, using an occluded query image. We conduct experiments on
Occluded-Duke [48], which is selected from DukeMTMC-reID by keeping occluded
images and filtering out some overlapping images. It contains 15,618 training
images, 17,661 gallery images, and 2,210 occluded query images. We use the
same metric as in fully-supervised reID and test the performance of different
pre-trained models using HOReID [68].

Person Attribute Recognition is a multi-label classification task, aiming at
recognizing attributes presented in pedestrian images, such as the clothes
style, clothes color, _etc._ To evaluate the performance of our VAL-PAT, we
adopt the strong baseline RePAR [33] (ResNet50 followed by an attribute
classifier) and implement experiments on two popular datasets: RAP [37] and
PA-100K [43]. The RAP dataset contains 41,585 images, where each image is
annotated with 72 attributes. Following [37], we split the PAR dataset into
33,268 training images and 8,317 testing images. PA-100K contains 80,000
training images, 10,000 validation images and 10,000 test images, with each
images annotated with 26 commonly used attributes. Following commonly used
setting, the mean accuracy (mA), accuracy, precision, recall and F1 score are
used for evaluation.

Text-based Person Search utilizes textual descriptions as queries to retrieve
relevant pedestrian images from a large gallery set. We implement experiments
on CUHK-PEDES dataset [38]333To ensure a fair comparison with other methods in
this task, we train image captioning model with only CUHK-PEDES and keep other
setting unchanged. , which is comprised of 40,206 pedestrian images of 13,003
identities with 80,412 textual descriptions. The total dataset is split into
34,054 images of 11,003 identities for training, 3,078 images of 1,000
identities for validation and 3,074 images of 1,000 identities for testing. We
adopt the top-$K$ accuracy metric ($K$=1, 5, 10) for evaluation, following the
standard setting. Our baseline method for this task is NAFS [20], and we
reimplement it with a better data augmentation strategy [61], resulting in
improved performance compared to the original paper.

###  4.3 Comparing with Pre-trained Methods

In this section, we compare VAL-PAT with other pre-trained methods based on
ResNet50 backbone, including INsup (a supervised model pre-trained on
ImageNet), LUP (pre-training MoCo v2 on LUPerson) and UP-reID (a pre-trained
model elaborately designed for reID task). To ensure a fair comparison, we
search for optimal hyper-parameters for each pre-trained model to present the
best performance during fine-tuning on downstream tasks.

Fully-supervised person reID. Table 1 presents the improvements using the
three representative fully-supervised reID methods on the two popular person
reID datasets. In contrast to our baseline LUP, the MGN with VAL-PAT obtain
0.9% and 3.8% improvements in terms of mAP and 0.5% and 2.1% improvements in
terms of Rank-1 on DukeMTMC and PersonX, respectively. Although UP-ReID is a
sophisticated pre-trained model for reID tasks, our VAL-PAT still achieves
competitive performance on DukeMTMC and PersonX. Notably, the UP-ReID
framework can be used to replace the vanilla image self-supervised branch in
VAL-PAT, which has the potential to further improve the performance on
downstream tasks.

Unsupervised person reID. Table 2 demonstrates that our pre-trained model is
more effective in unsupervised reID settings than in fully-supervised reID.
Generally, our VAL-PAT consistently outperforms other pre-trained models by a
considerable margin with different baselines on DukeMTMC and MSMT17.
Specifically, the C-Contrast with our VAL-PAT surpasses the one with UP-reID
by 1.5%, 11.9% in terms of mAP and 0.8%, 15.2% in terms of Rank-1 on DukeMTMC
and MSMT17, respectively. Moreover, it is observed that the performance of LUP
and UP-ReID is far below INsup on MSMT17, but our VAL-PAT significantly
exceeds INsup. This indicates that our VAL-PAT trained with multimodal
supervision can learn more discriminative representations.

Occluded person reID. As shown in Table 3, our VAL-PAT can also benefit
occluded reID tasks. Specifically, compared to initializing with LUP, HOReID
with VAL-PAT achieves 2.4% and 2.6% improvements in terms of mAP and Rank-1.
Additionally, VAL-PAT also outperforms UP-ReID by 1.6% on mAP and 2.4% on
Rank-1, showing the effectiveness of our pre-training framework.

pre-train |  Occluded-Duke
---|---
mAP |  Rank-1
INsup |  43.8 |  55.1
LUP |  51.5 |  60.3
UP-ReID |  52.3 |  60.5
VAL-PAT |  53.9 |  62.9
Table 3: Comparison of different pre-trained models on occluded reID. HOReID
[68] is adopted as baseline method.

Pedestrian Attribute Recognition. As illustrated in Table 4, our VAL-PAT
consistently outperforms other pre-trained models in terms of mA and F1 on
both RAP and PA-100K datasets. Compared with our VAL-PAT, the LUP and UP-reID
only achieve slight performance improvements over INsup on both datasets,
indicating that these pre-trained models specifically designed for reID tasks
lack the ability to generalize well to other pedestrian analysis tasks.

pre-train | RAP | PA-100K
---|---|---
mA | F1 | mA | F1
INsup | 80.2 | 80.5 | 80.9 | 87.4
LUP | 80.6 | 80.7 | 81.1 | 88.1
UP-ReID | 80.5 | 80.5 | 81.1 | 88.1
VAL-PAT | 80.8 | 81.0 | 82.3 | 88.5
Table 4: Comparison of different pre-trained models on pedestrian attribute
recognition. RePAR [33] is adopted as baseline method.

Text-based person search. We compare the VAL-PAT with other pre-trained models
on CUHK-PEDES and present the results in Table 5. Since text-based person
search is a multimodal retrieval task, the text encoder is required to extract
text representations. For INsup, LUP and UP-ReID, we adopt BERT [16] as their
text encoder. In contrast, our VAL-PAT directly utilizes the text encoder from
pre-training phase. We can observe that our VAL-PAT outperforms LUP and UP-
ReID by 2.7% and 2.3% on top-1 accuracy, respectively. The superiority of our
pre-trained model can be attributed to the fact that it has established a
strong relationship between the image and text modalities through the use of
image-text contrastive loss and multi-attribute classification loss during
pre-training, making it easily adaptable to this task.

pre-train | CUHK-PEDES
---|---
top-1 | top-1 | top-10
INsup | 61.2 | 80.6 | 87.2
LUP | 62.0 | 81.2 | 87.7
UP-ReID | 62.4 | 81.4 | 88.0
VAL-PAT | 64.7 | 83.3 | 88.9
Table 5: Comparison of different pre-trained models on text-based person search. NAFS [20] is used as our baseline method. Methods | DukeMTMC | Occluded-Duke
---|---|---
mAP | Rank-1 | mAP | Rank-1
PCB [63] (ECCV’18) | 69.2 | 83.3 | 33.7 | 42.6
MGN [69] (MM’18) | 78.4 | 88.7 | - | -
MGN* | 79.4 | 89.0 | - | -
BOT [46] (CVPRW’19) | 76.4 | 86.4 | - | -
BDB [12] (ICCV’19) | 76.0 | 89.0 | - | -
PGFA [48] (ICCV’19) | 65.5 | 82.6 | 37.3 | 51.4
SAN [34] (AAAI’20) | 75.5 | 87.9 | - | -
ISP [87] (ECCV’20) | 80.0 | 89.6 | 52.3 | 62.8
GASM [29] (ECCV’20) | 74.4 | 88.3 | - | -
PVPM [21] (CVPR’20) | - | - | 37.7 | 47.0
HOReID [68] (CVPR’20) | 75.6 | 86.9 | 43.8 | 55.1
AMD [10] (ICCV’21) | 75.5 | 88.2 | - | -
OAMN [7] (ICCV’21) | 72.6 | 86.3 | 46.1 | 62.6
PAT [40] (CVPR’21) | 78.2 | 88.8 | 53.6 | 64.5
VAL-PAT (Ours) | 83.0 | 91.5 | 53.9 | 62.9

Table 6: Performance (%) comparisons with state-of-the-art fully-supervised reID approaches on DukeMTMC-reID and occluded reID approaches on Occluded-Duke. * refers to the re-implementation based on official released code. The best performance is marked as bold and our baseline is highlighted with underline. Methods | DukeMTMC | MSMT17
---|---|---
mAP | Rank-1 | mAP | Rank-1
MMCL [67] (CVPR’20) | 40.2 | 65.2 | 11.2 | 35.4
HCT [78] (CVPR’20) | 50.7 | 69.6 | - | -
MMT [18] (ICLR’20) | 60.3 | 75.6 | - | -
SpCL [22] (NeurIPS’20) | 65.3 | 81.2 | 19.1 | 42.3
GCL [5] (CVPR’21) | 62.8 | 82.9 | 21.3 | 45.7
RLCC [79] (CVPR’21) | 69.2 | 83.2 | 27.9 | 56.5
C-Contrast [13] (ACCV’22) | 72.6 | 84.9 | 33.0 | 62.0
PPLR [11] (CVPR’22) | - | - | 31.4 | 61.1
ISE [80] (CVPR’22) | - | - | 37.0 | 67.6
VAL-PAT (Ours) | 74.9 | 86.1 | 38.9 | 67.5

Table 7: Performance (%) comparisons with state-of-the-art unsupervised reID
approaches on DukeMTMC and MSMT17.

###  4.4 Comparing with State-of-the-Art Methods

In this section, we compare our results with the state-of-the-art methods in
aforementioned downstream pedestrian-related tasks and report these results in
Table 6, Table 7, Table 9 and Table 10. As shown in Table 6, using our pre-
trained weights, the MGN achieves the best performance in terms of mAP and
Rank-1 on DukeMTMC-reID for fully-supervised reID tasks. Additionally, the
HOReID with our VAL-PAT surpasses the state-of-the-art PAT by 0.3% in terms of
mAP on Occluded-Duke. In Table 7, for the unsupervised reID, we achieve a new
state-of-the-art performance on DukeMTMC and MSMT17 by simply applying our
pre-training ResNet50 model on C-Contrant and RLCC, respectively. In Table 9,
the simplest baseline method RePAR (ResNet50 followed by an attribute
classifier) with our VAL-PAT has exceeded all sophisticated methods except for
Recall metric on PA-100K.

Moreover, as presented in Table 10, the NAFS with our VAL-PAT outperforms the
state-of-the-art LGUR by 0.5% in terms of top-1 accuracy on CUHK-PEDES, thanks
to our multimodal learning framework. In summary, the consistent improvements
on these tasks prove that our pre-trained model can learn transferable
representations that boost most of pedestrian-related tasks.

pre-train | DukeMTMC | MSMT17 | PA-100K | CUHK-PEDES
---|---|---|---|---
mAP | Rank-1 | mAP | Rank-1 | mA | Accu | Prec | Recall | F1 | top-1 | top-10
INsup | 76.4 | 86.4 | 27.9 | 56.5 | 80.9 | 79.2 | 87.7 | 87.2 | 87.4 | 61.2 | 87.2
SSL | 77.9 | 87.3 | 18.7 | 38.4 | 81.2 | 80.4 | 88.2 | 88.3 | 88.3 | 61.5 | 87.2
ITC | 78.6 | 87.8 | 35.5 | 64.1 | 81.6 | 79.6 | 87.8 | 87.6 | 87.7 | 62.9 | 88.2
SSL+ITC | 79.6 | 88.9 | 37.7 | 66.3 | 82.1 | 80.7 | 88.5 | 88.3 | 88.4 | 63.5 | 88.6
SSL+ITC+MAC w/o soft | 78.1 | 88.1 | 34.0 | 62.2 | 81.9 | 80.5 | 88.3 | 88.1 | 88.2 | 62.5 | 87.8
SSL+ITC+MAC | 80.0 | 88.6 | 38.9 | 67.5 | 82.3 | 80.9 | 88.5 | 88.5 | 88.5 | 64.7 | 88.9

Table 8: Ablating the different components of VAL-PAT pre-training w.r.t downstream task performances.  Methods | mA | Accu | Pre | Recall | F1
---|---|---|---|---|---
VeSPA [57] (BMVC’17) | 76.3 | 73.0 | 85.0 | 81.5 | 83.2
LGNet [41] (BVMC’18) | 77.0 | 75.6 | 87.0 | 83.2 | 85.0
PGDM [36] (ICME’18) | 75.0 | 73.0 | 84.4 | 82.2 | 83.3
MsVAA [55] (ECCV’18) | 80.1 | 77.0 | 86.2 | 85.6 | 85.5
VAC [23] (CVPR’19) | 79.0 | 79.0 | 88.4 | 86.1 | 86.8
ALM [65] (ICCV’19) | 79.3 | 78.6 | 87.3 | 86.7 | 86.6
RePAR [33] (ArXiv’21) | 80.2 | 79.2 | 87.8 | 87.0 | 87.4
RePAR* | 80.9 | 79.2 | 87.7 | 87.2 | 87.4
SSC [32] (ICCV’21) | 81.9 | 78.9 | 86.0 | 89.1 | 86.9
Label2Label [39] (ECCV’22) | 82.2 | 79.2 | 86.4 | 88.6 | 87.1
VAL-PAT (Ours) | 82.3 | 80.9 | 88.5 | 88.5 | 88.5

Table 9: Performance (%) comparisons with state-of-the-art pedestrian
attribute recognition approaches on PA-100K.

###  4.5 Ablation Study

We analyze the effectiveness of different components in our VAL-PAT through
ablation studies on four downstream tasks: fully-supervised reID, unsupervised
reID, pedestrian attribute recognition and text-based person search. For these
tasks, we adopt BOT, RLCC, RePAR and NAFS as the baseline methods,
respectively. The complete results are listed in Table 8, where the SSL refers
to image self-supervised contrastive loss, the ITC denotes image-text
contrastive loss and the MAC stands for multi-attribute classification loss.
We first validate the effectiveness of pre-training objectives SSL and ITC by
testing their standalone performance. As shown in Tabel 8, with the exception
of SSL in unsupervised setting, both per-training objectives contribute to
downstream tasks compared to the INsup.

In contrast to using SSL alone, we observe significant improvements on all
downstream tasks after adding ITC (SSL+ITC), demonstrating that language
supervision can effectively guide visual representation learning by providing
detailed and accurate information about pedestrians’ appearance. We further
investigate the impact of adding MAC to SSL+ITC, but only using mined noisy
labels as attribute supervision (SSL+ITC+MAC w/o soft). As expected, the
performances on downstream tasks are even worse than SSL+ITC, supporting our
hypothesis that enforcing the model to fit these noisy labels impairs the
generalization ability of the representations. However, with the incorporation
of soft labels as a complement to the mined noisy attributes (SSL+ITC+MAC), we
observe notable improvements in the performance across various downstream
tasks, particularly in pedestrian attribute recognition and text-based person
search, as compared to SSL+ITC. This proves that MAC not only facilitates the
learning of fine-grained visual representations but also helps bridge the
modality gap between image and text by encouraging the model to recognize
common attributes that are presented in matched image-text pairs. In
conclusion, the best results in SSL+ITC+MAC illustrate these three objectives,
SSL, ITC and MAC, are highly synergistic and guide the model to learn more
discriminative visual representations.

Methods | top-1 | top-5 | top-10
---|---|---|---
TIMAM [56] (ICCV’19) | 54.5 | 77.6 | 84.8
ViTAA [71] (ECCV’20) | 56.0 | 75.8 | 83.5
CMAAM [1] (WACV’20) | 56.7 | 77.2 | 84.9
HGAN [85] (MM’20) | 59.0 | 79.5 | 86.6
NAFS [20] (ArXiv’21) | 59.9 | 79.9 | 86.7
NAFS* | 61.2 | 80.6 | 87.2
DSSL [86] (MM’21) | 60.0 | 80.4 | 87.6
SSAN [17] (ArXiv’21) | 61.4 | 80.2 | 86.7
SRCF [64] (ECCV’22) | 64.0 | 83.0 | 88.8
LGUR [60] (MM’22) | 64.2 | 81.9 | 87.9
VAL-PAT (Ours) | 64.7 | 83.3 | 88.9

Table 10: Performance (%) comparisons with state-of-the-art text-based person
search approaches on CUHK-PEDES.

##  5 Conclusion

In this paper, we present a novel multimodal learning framework VAL-PAT that
learns transferable pedestrian representation from multimodal information
supervision. We first introduce a multimodal pedestrian dataset, LUPerson-TA,
with text and attribute annotations, and then train the framework on this
dataset using image self-supervised contrastive learning, image-text
contrastive learning and multi-attribute classification. Extensive experiments
show that the representation learned by VAL-PAT can transfer well to
downstream pedestrian analysis tasks, including person reID, pedestrian
attribute recognition, and text-based person search. Compared to previous pre-
trained models, our VAL-PAT achieves superior transfer performance. Moreover,
our proposed framework even outperforms previous state-of-the-art methods on
these tasks.

## References

  * [1] Surbhi Aggarwal, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty.  Text-based person search via attribute-aided matching.  In WACV, pages 2617–2625, 2020.
  * [2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * [3] Steven Bird, Ewan Klein, and Edward Loper.  Natural language processing with python: analyzing text with the natural language toolkit.  https://www.nltk.org, 2009.
  * [4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, pages 9650–9660, 2021.
  * [5] Hao Chen, Yaohui Wang, Benoit Lagadec, Antitza Dantcheva, and Francois Bremond.  Joint generative and contrastive learning for unsupervised person re-identification.  In CVPR, pages 2004–2013, 2021.
  * [6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE TPAMI, pages 834–848, 2017.
  * [7] Peixian Chen, Wenfeng Liu, Pingyang Dai, Jianzhuang Liu, Qixiang Ye, Mingliang Xu, Qi’an Chen, and Rongrong Ji.  Occlude them all: Occlusion-aware attention network for occluded person re-id.  In ICCV, pages 11833–11842, 2021.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, pages 1597–1607, 2020.
  * [9] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [10] Xiaodong Chen, Xinchen Liu, Wu Liu, Xiao-Ping Zhang, Yongdong Zhang, and Tao Mei.  Explainable person re-identification with attribute-guided metric distillation.  In ICCV, pages 11813–11822, 2021.
  * [11] Yoonki Cho, Woo Jae Kim, Seunghoon Hong, and Sung-Eui Yoon.  Part-based pseudo label refinement for unsupervised person re-identification.  In CVPR, pages 7308–7318, 2022.
  * [12] Zuozhuo Dai, Mingqiang Chen, Xiaodong Gu, Siyu Zhu, and Ping Tan.  Batch dropblock network for person re-identification and beyond.  In ICCV, pages 3691–3701, 2019.
  * [13] Zuozhuo Dai, Guangyuan Wang, Weihao Yuan, Siyu Zhu, and Ping Tan.  Cluster contrast for unsupervised person re-identification.  In ACCV, pages 1142–1160, 2022.
  * [14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, pages 248–255, 2009.
  * [15] Karan Desai and Justin Johnson.  Virtex: Learning visual representations from textual annotations.  In CVPR, pages 11162–11173, 2021.
  * [16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [17] Zefeng Ding, Changxing Ding, Zhiyin Shao, and Dacheng Tao.  Semantically self-aligned network for text-to-image part-aware person re-identification.  arXiv preprint arXiv:2107.12666, 2021.
  * [18] Fabian Dubourvieux, Romaric Audigier, Angelique Loesch, Samia Ainouz, and Stephane Canu.  Unsupervised domain adaptation for person re-identification through source-guided pseudo-labeling.  In ICPR, pages 4957–4964, 2021.
  * [19] Dengpan Fu, Dongdong Chen, Jianmin Bao, Hao Yang, Lu Yuan, Lei Zhang, Houqiang Li, and Dong Chen.  Unsupervised pre-training for person re-identification.  In CVPR, pages 14750–14759, 2021.
  * [20] Chenyang Gao, Guanyu Cai, Xinyang Jiang, Feng Zheng, Jun Zhang, Yifei Gong, Pai Peng, Xiaowei Guo, and Xing Sun.  Contextual non-local alignment over full-scale representation for text-based person search.  arXiv preprint arXiv:2101.03036, 2021.
  * [21] Shang Gao, Jingya Wang, Huchuan Lu, and Zimo Liu.  Pose-guided visible part matching for occluded person reid.  In CVPR, pages 11744–11752, 2020.
  * [22] Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, et al.  Self-paced contrastive learning with hybrid memory for domain adaptive object re-id.  NeurIPS, pages 11309–11321, 2020.
  * [23] Hao Guo, Kang Zheng, Xiaochuan Fan, Hongkai Yu, and Song Wang.  Visual attention consistency under image transforms for multi-label image classification.  In CVPR, pages 729–739, 2019.
  * [24] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In AISTATS, pages 297–304, 2010.
  * [25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In CVPR, pages 16000–16009, 2022.
  * [26] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, pages 9729–9738, 2020.
  * [27] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, pages 2961–2969, 2017.
  * [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [29] Lingxiao He and Wu Liu.  Guided saliency feature learning for person re-identification in crowded scenes.  In ECCV, pages 357–373, 2020.
  * [30] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang.  Transreid: Transformer-based object re-identification.  In ICCV, pages 15013–15022, 2021.
  * [31] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, pages 4904–4916, 2021.
  * [32] Jian Jia, Xiaotang Chen, and Kaiqi Huang.  Spatial and semantic consistency regularizations for pedestrian attribute recognition.  In ICCV, pages 962–971, 2021.
  * [33] Jian Jia, Houjing Huang, Xiaotang Chen, and Kaiqi Huang.  Rethinking of pedestrian attribute recognition: A reliable evaluation under zero-shot pedestrian identity setting.  arXiv preprint arXiv:2107.03576, 2021.
  * [34] Xin Jin, Cuiling Lan, Wenjun Zeng, Guoqiang Wei, and Zhibo Chen.  Semantics-aligned representation learning for person re-identification.  In AAAI, pages 11173–11180, 2020.
  * [35] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, 2012.
  * [36] Dangwei Li, Xiaotang Chen, Zhang Zhang, and Kaiqi Huang.  Pose guided deep model for pedestrian attribute recognition in surveillance scenarios.  In ICME, pages 1–6, 2018.
  * [37] Dangwei Li, Zhang Zhang, Xiaotang Chen, Haibin Ling, and Kaiqi Huang.  A richly annotated dataset for pedestrian attribute recognition.  arXiv preprint arXiv:1603.07054, 2016.
  * [38] Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang.  Person search with natural language description.  In CVPR, pages 1970–1979, 2017.
  * [39] Wanhua Li, Zhexuan Cao, Jianjiang Feng, Jie Zhou, and Jiwen Lu.  Label2label: A language modeling framework for multi-attribute learning.  In ECCV, pages 562–579, 2022.
  * [40] Yulin Li, Jianfeng He, Tianzhu Zhang, Xiang Liu, Yongdong Zhang, and Feng Wu.  Diverse part discovery: Occluded person re-identification with part-aware transformer.  In CVPR, pages 2898–2907, 2021.
  * [41] Pengze Liu, Xihui Liu, Junjie Yan, and Jing Shao.  Localization guided learning for pedestrian attribute recognition.  arXiv preprint arXiv:1808.09102, 2018.
  * [42] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In ECCV, pages 21–37, 2016.
  * [43] Xihui Liu, Haiyu Zhao, Maoqing Tian, Lu Sheng, Jing Shao, Shuai Yi, Junjie Yan, and Xiaogang Wang.  Hydraplus-net: Attentive deep features for pedestrian analysis.  In ICCV, pages 350–359, 2017.
  * [44] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In CVPR, pages 3431–3440, 2015.
  * [45] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [46] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang.  Bag of tricks and a strong baseline for deep person re-identification.  In CVPR workshops, pages 0–0, 2019.
  * [47] Hao Luo, Pichao Wang, Yi Xu, Feng Ding, Yanxin Zhou, Fan Wang, Hao Li, and Rong Jin.  Self-supervised pre-training for transformer-based person re-identification.  arXiv preprint arXiv:2111.12084, 2021.
  * [48] Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang.  Pose-guided feature alignment for occluded person re-identification.  In ICCV, pages 542–551, 2019.
  * [49] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  In ECCV, pages 529–544, 2022.
  * [50] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  NeurIPS, 2019.
  * [51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, pages 8748–8763, 2021.
  * [52] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In CVPR, pages 779–788, 2016.
  * [53] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  NeurIPS, 2015.
  * [54] Ergys Ristani, Francesco Solera, Roger S. Zou, Rita Cucchiara, and Carlo Tomasi.  Performance measures and a data set for multi-target, multi-camera tracking.  In ECCV Workshops, 2016.
  * [55] Nikolaos Sarafianos, Xiang Xu, and Ioannis A Kakadiaris.  Deep imbalanced attribute classification using visual attention aggregation.  In ECCV, pages 680–697, 2018.
  * [56] Nikolaos Sarafianos, Xiang Xu, and Ioannis A Kakadiaris.  Adversarial representation learning for text-to-image matching.  In ICCV, pages 5814–5824, 2019.
  * [57] M Saquib Sarfraz, Arne Schumann, Yan Wang, and Rainer Stiefelhagen.  Deep view-sensitive pedestrian attribute inference in an end-to-end model.  arXiv preprint arXiv:1707.06089, 2017.
  * [58] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [59] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In ICCV, pages 618–626, 2017.
  * [60] Zhiyin Shao, Xinyu Zhang, Meng Fang, Zhifeng Lin, Jian Wang, and Changxing Ding.  Learning granularity-unified representations for text-to-image person re-identification.  In ACM MM, pages 5566–5574, 2022.
  * [61] Xiujun Shu, Wei Wen, Haoqian Wu, Keyu Chen, Yiran Song, Ruizhi Qiao, Bo Ren, and Xiao Wang.  See finer, see more: Implicit modality alignment for text-based person retrieval.  In ECCV workshops, pages 624–641, 2023.
  * [62] Xiaoxiao Sun and Liang Zheng.  Dissecting person re-identification from the viewpoint of viewpoint.  In CVPR, pages 608–617, 2019.
  * [63] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang.  Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline).  In ECCV, pages 480–496, 2018.
  * [64] Wei Suo, Mengyang Sun, Kai Niu, Yiqi Gao, Peng Wang, Yanning Zhang, and Qi Wu.  A simple and robust correlation filtering method for text-based person search.  In ECCV, pages 726–742, 2022.
  * [65] Chufeng Tang, Lu Sheng, Zhaoxiang Zhang, and Xiaolin Hu.  Improving pedestrian attribute recognition with weakly-supervised multi-scale attribute-specific localization.  In ICCV, pages 4997–5006, 2019.
  * [66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 2017.
  * [67] Dongkai Wang and Shiliang Zhang.  Unsupervised person re-identification via multi-label classification.  In CVPR, pages 10981–10990, 2020.
  * [68] Guan’an Wang, Shuo Yang, Huanyu Liu, Zhicheng Wang, Yang Yang, Shuliang Wang, Gang Yu, Erjin Zhou, and Jian Sun.  High-order information matters: Learning relation and topology for occluded person re-identification.  In CVPR, pages 6449–6458, 2020.
  * [69] Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou.  Learning discriminative features with multiple granularities for person re-identification.  In ACM MM, pages 274–282, 2018.
  * [70] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In CVPR, pages 3024–3033, 2021.
  * [71] Zhe Wang, Zhiyuan Fang, Jun Wang, and Yezhou Yang.  Vitaa: Visual-textual attributes alignment in person search by natural language.  In ECCV, pages 402–420, 2020.
  * [72] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian.  Person transfer gan to bridge domain gap for person re-identification.  In CVPR, pages 79–88, 2018.
  * [73] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In CVPR, pages 3733–3742, 2018.
  * [74] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  In CVPR, pages 9653–9663, 2022.
  * [75] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In ICML, pages 2048–2057, 2015.
  * [76] Zizheng Yang, Xin Jin, Kecheng Zheng, and Feng Zhao.  Unleashing potential of unsupervised pre-training with intra-identity regularization for person re-identification.  In CVPR, pages 14298–14307, 2022.
  * [77] Hantao Yao, Shiliang Zhang, Richang Hong, Yongdong Zhang, Changsheng Xu, and Qi Tian.  Deep representation learning with part loss for person re-identification.  IEEE TIP, pages 2860–2871, 2019.
  * [78] Kaiwei Zeng, Munan Ning, Yaohua Wang, and Yang Guo.  Hierarchical clustering with hard-batch triplet loss for person re-identification.  In CVPR, pages 13657–13665, 2020.
  * [79] Xiao Zhang, Yixiao Ge, Yu Qiao, and Hongsheng Li.  Refining pseudo labels with clustering consensus over generations for unsupervised object re-identification.  In CVPR, pages 3436–3445, 2021.
  * [80] Xinyu Zhang, Dongdong Li, Zhigang Wang, Jian Wang, Errui Ding, Javen Qinfeng Shi, Zhaoxiang Zhang, and Jingdong Wang.  Implicit sample extension for unsupervised person re-identification.  In CVPR, pages 7369–7378, 2022.
  * [81] Ying Zhang and Huchuan Lu.  Deep cross-modal projection learning for image-text matching.  In ECCV, pages 686–701, 2018.
  * [82] Zhizheng Zhang, Cuiling Lan, Wenjun Zeng, Xin Jin, and Zhibo Chen.  Relation-aware global attention for person re-identification.  In CVPR, pages 3186–3195, 2020.
  * [83] Haiyu Zhao, Maoqing Tian, Shuyang Sun, Jing Shao, Junjie Yan, Shuai Yi, Xiaogang Wang, and Xiaoou Tang.  Spindle net: Person re-identification with human body region guided feature decomposition and fusion.  In CVPR, pages 1077–1085, 2017.
  * [84] Liming Zhao, Xi Li, Yueting Zhuang, and Jingdong Wang.  Deeply-learned part-aligned representations for person re-identification.  In ICCV, pages 3219–3228, 2017.
  * [85] Kecheng Zheng, Wu Liu, Jiawei Liu, Zheng-Jun Zha, and Tao Mei.  Hierarchical gumbel attention network for text-based person search.  In ACM MM, pages 3441–3449, 2020.
  * [86] Aichun Zhu, Zijie Wang, Yifeng Li, Xili Wan, Jing Jin, Tian Wang, Fangqiang Hu, and Gang Hua.  Dssl: deep surroundings-person separation learning for text-based person retrieval.  In ACM MM, pages 209–217, 2021.
  * [87] Kuan Zhu, Haiyun Guo, Zhiwei Liu, Ming Tang, and Jinqiao Wang.  Identity-guided human semantic parsing for person re-identification.  In ECCV, pages 346–363, 2020.
  * [88] Kuan Zhu, Haiyun Guo, Tianyi Yan, Yousong Zhu, Jinqiao Wang, and Ming Tang.  Pass: Part-aware self-supervised pre-training for person re-identification.  In ECCV, pages 198–214, 2022.

Appendix

Figure 3:  Additional image-text pairs in LUPerson-TA dataset. The textual descriptions in the first row are accurate, whereas the descriptions in the second row are noisy and imprecise.  pre-train | Occluded-Duke | RAP | CUHK-PEDES
---|---|---|---
mAP | Rank-1 | mA | Accu | Prec | Recall | F1 | top-1 | top-5 | top-10
INsup | 43.8 | 55.1 | 80.2 | 68.6 | 79.9 | 81.1 | 80.5 | 56.9 | 77.4 | 84.2
SSL | 51.8 | 60.8 | 79.5 | 68.4 | 79.7 | 80.1 | 80.3 | 58.4 | 78.6 | 85.0
ITC | 49.6 | 59.8 | 80.1 | 68.7 | 80.1 | 81.1 | 80.6 | 60.3 | 79.5 | 85.9
SSL+ITC | 53.7 | 63.2 | 80.6 | 69.2 | 80.3 | 81.5 | 80.9 | 61.3 | 80.0 | 87.0
SSL+ITC+MAC w/o soft | 52.6 | 62.4 | 80.4 | 69.3 | 80.5 | 81.4 | 81.0 | 61.0 | 79.8 | 86.3
SSL+ITC+MAC | 53.9 | 62.9 | 80.8 | 69.3 | 80.4 | 81.6 | 81.0 | 62.4 | 80.9 | 88.3

Table 11: Ablating the different components of VAL-PAT pre-training w.r.t
downstream task performances. For INsup and SSL, we adopt Bert [16] as their
text encoder. For other setting, we direct use text encoder from the pre-
training phase.

##  Appendix A More Image-text pairs in LUPerson-TA

Figure 3 demonstrates additional image-text pairs from the LUPerson-TA
dataset. The textual descriptions of the image-text pairs in the first row
accurately describe the characteristics of the corresponding pedestrians,
including fine-grained details such as “white dots” in row 1, column 1, “a
black handbag” in row 1, column 3, and “long brown hair” in row 1, column 5,
which are advantageous for learning more discriminative visual
representations. However, due to the use of a simple image captioning method
[75] and the limited training samples (a total of 102,782 image-text pairs in
CUHK-PEDES [38] and ICFG-PEDES [17]), some textual descriptions generated by
the image captioning model contain noise, such as incorrect recognition of the
color of the pedestrian’s shirt in row 2, column 2, and omission of the
descriptions of the pedestrian’s pants in row 2, column 4. These imprecise
textual descriptions not only hinder the convergence of image-text contrastive
loss, but also introduce noise into the mined attribute labels. In this paper,
we propose optimization objective MAC to alleviate this issue. Alternatively,
as a future research direction, we may explore more sophisticated image
captioning methods to generate more accurate textual descriptions.

##  Appendix B More Ablation Study

We present more ablation studies on occluded reID, pedestrian attribute
recognition and text-based person search, and report these results in Table
11. For these tasks, we respectively select HOReID [68], RePAR [33] and CMPM/C
[81] (we reimplement CMPM/C with better data augemtation strategies) to
evaluate performance of different variants of VAL-PAT. As can be seen, the
combination of SSL+ITC+MAC (our VAL-PAT) achieves the best performances on
most of metrics across these tasks.

##  Appendix C Visualizing the Pre-trained Image Encoder

In Figure 4, we show the Grad-CAM [59] visualizations for the image encoder
$E_{q}^{I}$ in our VAL-PAT, with different text queries fed into the text
encoder $E_{q}^{T}$. Remarkably, we observe that our image encoder
successfully localizes the relevant regions for various query texts, even for
fine-grained queries such as “phone” and “short black hair”. This indicates
our pre-trained framework effectively captures a wide range of visual concepts
and establishes strong associations between these concepts and their
corresponding texts.

Figure 4:  Grad-CAM visualizations of the image encoder $E^{I}_{q}$, given
different text queries.