∎

11institutetext: Hiren Galiyawala 22institutetext: School of Engineering and
Applied Science,
Ahmedabad University, India.
22email<EMAIL_ADDRESS>33institutetext: Mehul S. Raval
44institutetext: School of Engineering and Applied Science,
Ahmedabad University, India.
44email<EMAIL_ADDRESS>
# Person Retrieval in Surveillance Using Textual Query: A Review

Hiren Galiyawala  Mehul S. Raval

(Received: date / Accepted: date)

###### Abstract

Recent advancement of research in biometrics, computer vision, and natural
language processing has discovered opportunities for person retrieval from
surveillance videos using textual query. The prime objective of a surveillance
system is to locate a person using a description, e.g., a short woman with a
pink t-shirt and white skirt carrying a black purse. She has brown hair. Such
a description contains attributes like gender, height, type of clothing,
colour of clothing, hair colour, and accessories. Such attributes are formally
known as soft biometrics. They help bridge the semantic gap between a human
description and a machine as a textual query contains the person’s soft
biometric attributes. It is also not feasible to manually search through huge
volumes of surveillance footage to retrieve a specific person. Hence,
automatic person retrieval using vision and language-based algorithms is
becoming popular. In comparison to other state-of-the-art reviews, the
contribution of the paper is as follows: 1. Recommends most discriminative
soft biometrics for specific challenging conditions. 2. Integrates benchmark
datasets and retrieval methods for objective performance evaluation. 3. A
complete snapshot of techniques based on features, classifiers, number of soft
biometric attributes, type of the deep neural networks, and performance
measures. 4. The comprehensive coverage of person retrieval from handcrafted
features based methods to end-to-end approaches based on natural language
description.

###### Keywords:

Attribute recognition natural language description person retrieval soft
biometric attributes video surveillance.

##  1 Introduction and motivation

Security of the individual and society is a significant concern in today’s
world of automation. Modern life of human beings involves many Internet-of-
Things (IoT) based devices to interact and operate. While such devices provide
luxury to live, they may also endanger living by compromising privacy. There
are many questions in daily life such as:

  * •

Is this person authorized to enter into specific premises?

  * •

Is this person authorized to access the mobile application?

  * •

Is the right person accessing the system?

  * •

Is the person a legal resident of the country?

Answers to such questions need monitoring using surveillance networks. Most
countries in the world have surveillance cameras in many places. Camera
networks cover many places like housing colonies, small offices, large
corporate offices, retail shops, supermarkets, public places like airports,
railway stations, and sports venues. The world has observed many events like
terrorist attacks and general strikes in recent years. Public safety is the
most critical task to be carried out by a security agency during such an
event. A surveillance network helps investigators during and after the event.
Person localisation plays a significant role during such an investigation and
helps to accelerate the process of investigation. Typically, video operators
carry out the person(s) retrieval task by searching through a video database.
It takes time, and the process is also very inefficient. Thus, intelligent
video surveillance is growing as an active research area.

Some critical definitions for giving the reader a better understanding are as
follows:

##### Person identification:

It is a process to establish an individual’s identity or to identify an
individual uniquely. Biometrics such as fingerprint and face are unique to an
individual, which helps in person identification.

##### Person re-identification (Re-ID):

Re-ID aims to identify a person in a new place and at a different time
captured by another camera. The query can be in the form of an image, video,
and even text description. It is useful to re-identify a person in
surveillance-based applications.

##### Person retrieval

aims to spot the person of interest from an image gallery or video frame given
the query. It does not seek to establish an identity; instead, it reduces the
search space. Thus, the output may contain multiple retrievals matching the
query. For example, a surveillance frame may have multiple persons with a
description, i.e., a man with a blue t-shirt and black jeans.

###  1.1 History and background

An early person identification system developed by Alphonse Bertillon, a
police officer and biometric researcher from Paris, France, identified
criminals in the 19th century. His approach is also known as Bertillonage R1 ,
which was widely accepted in France and other countries. Criminal
identification was made only by employing photographs or names before
Bertillonage. He was the first to use anthropometry for law enforcement using
various physical measurements. He also defined the process of physical
measurements and photograph acquisition to maintain uniformity in the records
R1 ; R2  (see Fig. 1). Bertillon’s system measures physical features like
standing height, length and breadth of head, length of individual fingers,
dimensions of the foot, dimensions of the nose, and ear. It also has
descriptions of eye and hair colour, any scars, marks, and tattoos on the body
R1 .

Figure 1: Anthropometric data sheet of Alphonse Bertillon R3 .

Two persons may likely have the same height, but the chances of other
measurements being similar is highly unlikely. Sizes are not unique to
individuals and hence may fail to identify a person if used without other
attributes. e.g., twins may have similar biological features. Hence, it has
often been superseded by fingerprint-based identification. Biometrics is a
reliable solution for person identification R4  due to properties like
uniqueness, stability, universality, and accessibility. The answer is
dependent on “what you are”which uses face, fingerprint, palm print, hand
geometry. It is independent of “what you possess”like identity card, or “what
you know”e.g., password or the personal identification number R15 ; R16 . Such
traditional biometrics-based systems are successful in many applications like
forensics, employee attendance, and mobile or laptop security. Biometrics-
based retrieval systems have limited usage in surveillance applications due to
the following:

  * •

Biometric samples are difficult to capture from video footage of unconstrained
environments.

  * •

It is challenging to capture physiological samples (e.g., fingerprint) from an
uncooperative person.

  * •

Biometric attributes (e.g., face) are challenging to acquire due to camera
distance and position.

  * •

Captured biometrics have poor quality due to low-resolution cameras.

Traditional biometric-based systems fail to retrieve a person from
surveillance videos due to the above reasons. Fig. 2 shows sample video
surveillance frames from the AVSS 2018 challenge II database R5 . A person of
interest is within a green bounding box. It shows various scenarios where
traditional biometrics-based systems fail to retrieve the person. Fig. 2(a)
shows the environment with poor illumination and a low-resolution frame. Fig.
2(b) shows a scenario with a crowd and a large distance between the camera and
the person. Face recognition-based systems fail to retrieve the person in such
scenarios. However, attributes like colour and type of clothing, height and
gender can help in-person retrieval under such challenging conditions. For
example, green (in Fig. 2(a)) and purple (in Fig. 2(b)) coloured clothes are
identifiable. Jain et al. R6  introduce such personal attributes as soft
biometrics, and many other research articles elaborate on soft biometrics R6 ;
R7 ; R8 ; R9 ; R10 ; R11 ; R12 ; R13 . Boston Marathon bombing R14  is a real-
world example where police investigation took three days to search out the
suspect from hundreds of hours of surveillance videos. Thus, automation of the
person retrieval process saves time during a critical investigation. With the
ever-increasing demand for security through surveillance, researchers are
infusing more interest in developing soft biometric-based person retrieval
from low-quality surveillance videos where traditional biometrics fail.

Figure 2: Video surveillance frame samples R5 . Figure 3: Type of biometrics.

Biometrics are broadly classified as hard and soft biometrics, as shown in
Fig. 3. Hard biometrics have physiological and behavioural attributes of a
person. Physiologically based methods establish a person’s identity using
physiological characteristics like face, iris, fingerprint, palm print, hand
geometry and DNA. Behavioural techniques perform authentication by recognizing
patterns such as voice, gait, and signature. Although biometrics like face,
fingerprint, and palm print are unique to an individual, a single biometric
cannot meet the requirements of all applications R15 ; R16 . For example,
camera-based security systems cannot use fingerprints as a biometric for user
authentication. Thus, biometric attribute selection depends on the
application. One such application is person retrieval from a distance through
a surveillance frame. Here traditional biometrics fail, but soft biometrics
play a critical role in retrieval. Soft biometrics are the ancillary
information deduced from the human body. Dantcheva et al. R12  categorized
soft biometrics into demographic, anthropometric, medical, material, and
behavioural attributes.

The term demographic is related to the accumulation and statistical analysis
of broad characteristics about groups of people and populations. Age,
ethnicity, gender, and race are such characteristics that are used as soft
biometrics and also termed as global traits R17 . Gender, ethnicity and race
usually do not show changes over the person’s lifespan and hence are useful
for search space reduction from surveillance. Gender and age estimation are
the most researched demographic attributes. Researchers estimate gender from
face R18 ; R19 ; R20 ; R21 ; R22 ; R23 ; R24 ; R25 ; R26 , fingerprint R27 ;
R28 ; R29 ; R30 ; R31 ; R32 , iris R33 ; R34 , body R35 ; R36 ; R37 ; R38 ;
R39 ; R40 , hand R41 ; R42 ; R43 ; R44 ; R45 ; R46  and speech R47 ; R48 ; R49
; R50 ; R51 . Gender from face covers a major research segment and achieves
99% R22  classification accuracy. However, performance decreases dramatically
for a realistic and unconstrained environment. It imposes illumination
changes, occlusion, different views, and poses. Surveillance videos cover such
unconstrained scenarios and it is also very difficult to acquire face,
fingerprint, iris, hand, and speech biometrics from surveillance. Gender
classification from the full human body is a more suitable way for
surveillance applications.

Anthropometric measurements are a traditional way of body measurements. Since
the era of Bertillonage R1 ; R2  they have been in security practices and are
currently known as anthropometric and geometric soft biometrics. Facial
landmarks related to chin, mouth, eyes, and nose are anthropometric
measurements related to facial geometry R52 ; R53 ; R54 ; R55 . Body height
R56 ; R57 ; R58 ; R59  is the most researched anthropometric attribute.
Similarly, other body measurements, like torso length, leg length, and step
length R58 , are also useful. Medical characteristics help monitor a person
R12  with the help of a computer vision-based technique. Bodyweight, body mass
index (BMI), and skin quality are soft biometrics used for an application like
early detection and prevention of diseases like skin cancer. Another class of
soft biometrics is material and behavioural. It includes various accessories
worn by a person like a hat, scarf, eyeglasses; different clothes and clothing
colours; scars, marks, and tattoos on the body. Such soft biometrics also play
a crucial role in retrieval. For example, a description, a male with a green
t-shirt and blue jeans wearing a black cap and carrying a backpack, contains
clothing colours, clothing types, accessories (i.e., cap and backpack) as soft
biometrics. Material and behavioural attributes are highly time-inconsistent.
E.g., a person wearing a white shirt and black pants in the morning may have
different clothes (blue t-shirt and white shorts) in the afternoon of the same
day. Thus, descriptions for morning and afternoon are different for person
retrieval. Time consistency discussion is in Sec. 2.1.

###  1.2 Characteristics of soft biometrics

Research interest is now more inclined towards applications where person
retrieval at a distance is necessary, based on ancillary information like soft
biometrics R60 ; R61 ; R62 . Some early research R6 ; R63 ; R64 ; R65 ; R66 ;
R67  shows the use of soft biometrics to improve the performance of the
primary biometric system by narrowing down the search space in the initial
stages. Jain et al. R6  mentioned that soft biometrics are inexpensive to
compute, with no person cooperation requirement, and derived at a distance.
Similarly, various pros and cons of soft biometrics are in further discussion.

##### Registration free / enrollment free:

Traditional biometric systems require prior registration of the biometric
signature for identification in the future. However, prior registration of
soft biometric attributes of a specific person is not required. The training
is offline, and the model is useful for retrieval of the individual.

##### Compatible with human language:

Traditional biometrics features like fingerprints are discriminative. However,
they cannot be describable by linguistic labels. It creates a semantic gap
between human beings and the person retrieval system. Soft biometrics is a
form of biometrics that uses labels people generally use to describe or
identify each other like tall, female, young and blond hair. They help to
bridge the semantic gap by generating descriptions understandable by humans.

##### Cost and computationally effective:

Face, fingerprint, palm print, and iris-like biometrics require dedicated
sensors for acquisition and a constrained environment. Soft biometric
attributes like height, gender, and clothing colours are extractable from
single low-quality surveillance videos where the traditional biometric
acquisition fails. It reduces costs on the overall system development.

##### Recognizable from a distance:

Soft biometrics attributes are identifiable from a distance, e.g., clothing
colour, clothing type, gender.

##### No cooperation from the subject:

Soft biometrics attributes (e.g., clothing colour, gender) acquisition does
not require any cooperation from the subject. Unconstrained and non-intrusive
acquisition is also non-invasive, which makes it suitable for surveillance
applications.

##### Privacy preservation:

Soft biometrics-based description (e.g., a tall female with a blue t-shirt)
only provides partial information. Such descriptions are not unique to an
individual. It offers solutions to privacy issues related to soft biometric
data storing.

##### Uniqueness:

Soft biometric attributes are not unique to an individual, i.e., a single
attribute (e.g., female) does not uniquely retrieve the person. Multiple soft
biometrics together can help to overcome this limitation.

##### Permanence:

Soft biometrics are not permanent attributes of an individual like a
fingerprint. It makes person retrieval more challenging.

###  1.3 Structure of the paper

The paper presents the use of soft biometrics for person retrieval and
suggests their strengths and weaknesses. It recommends the most discriminative
and robust soft biometrics during person retrieval under challenging
conditions. Besides covering the general frame work for retrieval using
textual query it also reviews and compares large-scale datasets. It further
provides classification for retrieval methods and provides state-of-the-art
reviews for soft biometric attribute-based retrieval methods. Further, the
integration of benchmark datasets and methods allows an effective comparison
of their performance. In this way, the presented review is entirely different
from other soft biometrics basics reviews R9 ; R10 ; R11 ; R12 , which focus
mainly on soft biometric attribute retrieval from the person’s image. A review
of recent datasets, state-of-the-art retrieval methods including vision and
natural language processing, and dataset-method integration is not available
in the other review articles.

The contributions of the paper are summarised as follows:

  1. 1.

Recommends the most discriminative soft biometrics under challenging
conditions and show case their use for person retrieval.

  2. 2.

Provides the most comprehensive coverage of soft biometrics datasets.

  3. 3.

Spans the complete spectrum for state-of-the-art methods; image based,
discrete attribute based and natural language description-based retrieval.

  4. 4.

Integrates datasets and methods for quantitative and objective performance
evaluation.

  5. 5.

Discusses open research problems in person retrieval using soft biometrics and
their possible solutions.

##  2 Soft biometrics based person retrieval in surveillance videos

Person retrieval has become the most critical and essential task for
surveillance and security. Various government sectors and private
organizations invest more in computer vision-based research techniques due to
ever-increasing video-based data. It is observable that although the frame in
Fig. 2(a) is at low resolution and captured at a distance, a detailed
description can still be available. For example, one can get a narrative like
a male with a green t-shirt and blue jeans wearing a black cap, and carrying a
backpack. Such human description naturally contains soft biometrics like
gender (male), clothing color (green, blue), clothing type (short sleeve),
accessories (cap, backpack). It indicates that soft biometrics have a strong
relationship with human descriptions. Thus soft biometric attribute-based
person retrieval systems are more suitable for an unconstrained environment.

Computer vision has created enormous opportunities to develop an algorithm for
intelligent video processing which is useful for retrieval. Developing such
algorithms is also tricky due to various problems like video frame resolution,
different camera views, poses, illumination conditions, occlusion, and
background merging with the foreground.

###  2.1 Challenges for person retrieval in surveillance

##### C1. Low-resolution videos:

Surveillance videos have lower resolution since they have to be recording for
long periods of time. Fig. 2 shows such a surveillance frame where it is
challenging to extract face information from the low-resolution frame with the
distance between the camera and the person. A face recognition-based retrieval
system fails to establish identity in such challenging conditions. Also, the
low-resolution structure does not help to extract useful features. Hence, it
creates a problem for person retrieval.

##### C2. Illumination changes / varying illumination conditions:

The video looks good if the illumination is excellent. A proper lighting
condition is the key to better visual information. Fig. 4(a) and (b) show
single-camera surveillance frames with different illumination conditions.
Surveillance frames from left to right are with good to poor illumination
conditions. People and their clothing colours are visible in the frames with
proper lighting conditions. They are challenging to observe in reduced
illumination and create a problem in person detection and colour
classification.

Figure 4: Varying illumination conditions in multiple frames of the single
camera. (a) Cam1 and (b) Cam4 in AVSS 2018 challenge II database R5 . Frames
from left to right are with good to poor illumination.

##### C3. Occlusion:

Occlusion in the surveillance frame is a challenge where a person of interest
is not completely visible. Fig. 5 shows such sample frames where a person of
interest is within a green bounding box in each frame. It also creates issues
for detection and the segmentation process in person retrieval. Occlusion also
affects the estimation of an anthropometric attribute like height.

Figure 5: Occlusion in surveillance frames R5 . The persons of interest are
shown with green bounding box.

##### C4. Merging with background information:

Fig. 6 shows surveillance frames in which the person of interest merges with
the background. It happens due to low illumination or similarity in clothing
colour and background. Persons are not visible though they are not under
occlusion in each frame.

Figure 6: Person of interest merges with background information in
surveillance frames R5 . The persons of interest are shown with green bounding
box.

##### C5. Different illumination in the scene for a single sequence:

Fig. 7 shows a sequence of surveillance frames where illumination varies in
different parts of the frame. The person shown within a green bounding box is
moving from right to left in the frame. It shows that the outside light is
entering the room in the top right corner of the scene. It is observable that
the colour information keeps on changing as the person moves through such a
region.

Figure 7: Different lighting condition in a scene for a single sequence R5 .
The person of interest is shown with green bounding box.

##### C6. Different viewpoints of the same person:

A person may appear with different poses in the same camera scene or scenes of
multiple camera networks. Appearance-based methodologies expect minimal or no
change in the visual appearance of the person. Fig. 8 shows a scenario where
the pose of a person keeps on changing, and hence the appearance in the camera
changes. Retrieval is difficult in such a system. Pose changes also create
difficulties for methods that retrieve a person based on his or her gait.

Figure 8: Different viewpoints of same person R107 .

##### C7. Crowded scene:

Sitting areas, canteens, shopping malls, airports and railway stations are
places where a massive crowd is observable. Such frames are in Fig. 9. A
crowded scene contains occlusion and persons with a similar appearance cause
difficulties in retrieval.

Figure 9: Crowded scene R5 .

##### C8. Indoor and outdoor environment:

Surveillance cameras are at different indoor places (e.g., shopping malls,
canteens) and outdoor places (e.g., university campus, public roads). Fig. 4,
5, 6, 7 and 9 show indoor environments, while Fig. 8 shows an outdoor
environment. Both environments create different illumination conditions.
Outdoor environments are affected explicitly by various weather conditions and
day-night illuminations.

##### C9. Time consistency / time duration:

As discussed in Sect. 1.1, material and behavioural attributes like clothing
colour and clothing type are highly time-inconsistent. The same person may
have different clothing even on the same day, i.e., morning (white shirt and
black pants) and afternoon (blue t-shirt and white shorts). Also, attributes
like hairstyle and hair colour may change over the days, and age obviously
varies over years.

##### C10. Type of camera / multiple cameras:

Surveillance feeds captured from different types of cameras have other
characteristics. For example, red clothing colour perception may differ from a
bullet camera to an unmanned aerial vehicle (UAVs) camera. It creates many
concerns for clothing colour-based retrieval.

##### C11. Unavailability of a generic dataset:

The available datasets (Sec. 2.4) do not cover all the challenges, e.g., a
dataset may have only indoor environment images. Also, some of the soft
biometric (e.g., clothing colour) annotation may be missing. Such limitations
incur the challenge of developing a universal end-to-end system.

The conditions under which a surveillance network operates is challenging.
These conditions are responsible for producing unsatisfactory results. The
surveillance system should be robust to create a meaningful impact in the real
world. It is crucial to select soft biometric attributes that help person
retrieval under challenging conditions.

###  2.2 Soft biometric attribute selection

Soft biometric attributes are not unique; for example, there may be multiple
people with a blue torso colour. Thus, it produces numerous matches for the
given textual query. Therefore, it is advantageous to use the most
discriminative attributes for person retrieval. Different soft biometrics have
certain advantages, and they are discussed below. For example, a surveillance
video may contain different view angles and distance. The person’s height is
invariant to such concerns R68 ; R69 ; R78 . Clothing colour is also one of
the most discriminative attributes. It has the following advantages:

  * •

Colour is more immune to noise.

  * •

It is insensitive to dimension, view angle, and direction.

  * •

Colour is recognizable from a far distance.

Clothing type is also insensitive to view angle. Gender is identifiable from
near as well as far distance and different view angles. Table 1 shows the
strength of soft biometric attributes against challenging conditions.
“✓”indicates that a particular attribute is useful for the retrieval process
against a specific challenging scenario. For example, a standing person’s
height estimation requires head and feet points R68 ; R69 ; R78 . The
extraction of those points is not affected by different view angles, near or
far fields, illumination, and low resolution. However, those points are non-
extractable when a person is occluded or has a pose like sitting. Similarly,
“$\times$”indicates that a particular attribute is not useful for the
retrieval process against the challenge.

Table 1: Soft biometric attribute selection for person retrieval against
different challenges.

| View angles | Far field | Near field | Illumination | Partial occlusion | Pose | Low resolution | Time | Camera
---|---|---|---|---|---|---|---|---|---
Height | ✓ | ✓ | ✓ | ✓ | $\times$ | $\times$ | ✓ | ✓ | ✓
Cloth color | ✓ | ✓ | ✓ | $\times$ | ✓ | ✓ | ✓ | $\times$ | $\times$
Cloth type | ✓ | ✓ | ✓ | ✓ | $\times$ | $\times$ | ✓ | $\times$ | $\times$
Gender | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Torso pattern | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
Body geometry | $\times$ | ✓ | ✓ | ✓ | $\times$ | $\times$ | ✓ | ✓ | ✓
Hair (type & color) | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
Shoe (type & color) | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
Accessories (bag, hat) | $\times$ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$

###  2.3 Vision based person retrieval system

This section discusses a person retrieval system that uses a natural language-
based textual query. Researchers propose methods using handcrafted feature-
based retrieval R61 ; R62 ; R63 ; R64 ; R65 ; R66 ; R67 ; R70 ; R71 ; R72 ;
R73 ; R74 ; R75 ; R76 ; R77 , deep learning feature-based linear filtering R68
; R69 ; R78 , parallel classification of attributes R79 ; R80 ; R81
utilization of Natural Language Processing (NLP) algorithms to process textual
queries R82 ; R83 ; R84 ; R85 ; R86 ; R87 ; R88 ; R89 . Such a plethora of
methods consists of person detection, segmentation, soft biometric attributes
classification, and person identification as crucial steps in the person
retrieval process. These key steps are given in Fig. 10 and discussed as
follows:

Step-I (Person detection):

Person retrieval in surveillance is a challenging task because such scenarios
are usually in the wild, containing various objects (e.g., chair, table, car,
and train) apart from the person. Hence, person detection is the critical
initial step before textual query-based person retrieval. Person detection is
the task of locating all persons in the surveillance frame. It was a
foreground-background separation problem before the era of deep learning. A
person is a foreground object, which is different from the rest of the
background. Early research detected the person by different methodologies like
background subtraction R70 ; R90 , adaptive background segmentation R91 ,
Gaussian Mixture Model (GMM) R92 , and Histograms of Oriented Gradients (HoG)
R93 .

Deep learning-based methodologies are becoming popular in recent years due to
their robust feature extraction and learning ability. The computer vision
community considers person detection as an object detection problem. Some of
the popular object detection frameworks are “You Only Look Once (YOLO)”R94 ,
Single-Shot Multibox Detector (SSD) R126 , Region-based Convolutional Neural
Network (R-CNN) R95 , Fast R-CNN R96 , Faster R-CNN R97  and Mask R-CNN R98 .
Person detection provides a bounding box for each person in the frame.

Figure 10: Person retrieval in surveillance using textual query.

Step-II (Segmentation):

Segmentation follows person detection. Segmentation can be either in the form
of a body part from a bounding box or semantic segmentation within the
bounding box. Such segmentation is shown in Fig. 11. A full-body can be
segmented into three major parts: head and shoulders, upper body, and lower
body (see Fig. 11(a)). Fig. 11(b) shows semantic segmentation in which each
pixel belonging to the person within the bounding box has a label. Mask R-CNN
R98  provides both person detection and semantic segmentation together.

Figure 11: Segmentation: (a) body part segmentation and (b) semantic
segmentation.

Step-III (Soft biometric attribute recognition):

Full-body images and segmentation outputs are supplied to extract soft
biometric attributes. Full-body images are useful for extracting
characteristics like height and gender R68 ; R69 ; R78 . On the other hand,
clothing colour, clothing texture, clothing type, shoes, hair, and accessory
attributes are available from different body parts. Usually, details from
video frames are visual features. Recent development shows multi-attribute
learning for person attribute recognition. Researchers propose various
networks like Deep learning-based Multiple Attribute Recognition (DeepMAR)
R152 , Attribute Convolutional Net (ACN) R153 , and Multi-Label Convolutional
Neural Network (MLCNN) R154 .

Figure 12: Textual query: (a) discrete attribute query and (b) natural
language description

Step-IV (Text feature extraction):

Textual attribute query or natural language description is another input to
the person retrieval system. Such textual query samples are given in Fig. 12.
Textual attribute query (Fig. 12(a)) is cheaper to collect in terms of
attribute wise information and has a less complicated sentence structure in
comparison with natural language description (Fig. 12(b)). The data is
collected separately for all attributes which is useful in the retrieval
process. For example, torso type attribute value is available from one of the
predefined classes {Long Sleeve, Short Sleeve, No Sleeve}. However, such a
discrete attribute query has a fragile appearance, descriptive ability, and
practical usage limitation.

In contrast to textual attribute query, natural language description has more
complex sentence structures. However, it provides a detailed description of
the person. It is essential to extract relevant information from such a human
verbal description. For example, the description in Fig. 12(b), ‘women’,
‘brown hair’, ‘blacktop’are more relevant information than ‘the’, ‘a’, and
‘has’. Such relevant information forms textual features. NLP based algorithms
help to extract textual features from natural language descriptions.

Step-V (Feature fusion and retrieval):

Visual features for each person are available from surveillance videos, and
textual features are extractable from the textual query. Person retrieval from
surveillance using textual query covers two major problem domains: (i)
computer vision and (ii) natural language processing. Hence, cross-modal
feature embedding is applicable in the feature fusion block (see Fig.10).
Finally, a person(s) matching the textual query is retrieved. There is a
possibility of multiple person retrieval as soft biometrics are not unique to
an individual.

The accuracy of person retrieval depends on the complexity of the training and
testing data sets. The retrieval robustness relies on the availability of the
richly annotated data sets. The performance evaluation of the method is done
by evaluation metrics like accuracy, True Positive Rate (TPR) and
Intersection-over-Union (IoU). Therefore, the following sub-sections discuss
person retrieval datasets (available in the public domain) and evaluation
metrics.

###  2.4 Person retrieval datasets

A variety of challenging datasets are an essential entity for any research and
development task. Datasets with extensive annotations and reliable evaluation
strategies help to create a robust algorithm. Researchers have developed many
public datasets for person identification and re-identification in the past
two decades. Table 2 shows only datasets with soft biometric annotations,
which help create textual attribute query or natural language description for
person retrieval.

Dataset comparisons have the following parameters: number of persons, number
of images, resolution of the image (width $\times$ height)), number of soft
biometrics annotations, number of cameras used to collect the dataset, type of
attribute query, challenges covered, and whether the dataset is with full
surveillance frame or cropped person. Abbreviations for challenges and types
of attributes are in Table 2, which are useful for further discussion. Layne
et al. R102  annotate the VIPeR R99  dataset with 15 soft biometric attributes
like shorts, skirt, sandals, backpack, jeans, logo, v-neck, stripes,
sunglasses, open-outerwear, headphones, long-hair, short-hair, gender and
carrying an object. These are binary attributes e.g., gender has value from
set {0, 1}, where 0 = male and 1 = female.

Zhu et al. R110  introduce the Attributed Pedestrians in Surveillance (APS)
database specifically for pedestrian attribute classification, which can also
be useful for person retrieval. The APS consists of 11 binary attributes and
two multi-class (total 35) attributes, i.e., clothing colour of upper-body and
lower-body. APS is the first dataset to use clothing colour and covers more
challenges compared to VIPeR. Layne et al. R103  further extend annotations
for VIPeR R99 , PRID R100 , and GRID R101  from 15 to 21 by introducing shirt
colour (red, blue, green), shirt shade (light and dark) and pant and hair
colour. New datasets like VIPeR and PRID are having fixed image resolution,
limited cameras, C2, C5, and C8. GRID upgrades such parameters and introduces
C1 and C3. However, it is limited to some images. VIPeR, PRID, and GRID
contain photos from outdoor scenes only.

Table 2: Datasets with soft biometrics annotations. Dataset |  Year |  # Person |  # Images |  Resolution (W $\times$ H) |  # Soft attributes |  # Camera |  DA / NLD |  Challenges |  Full frame / cropped person
---|---|---|---|---|---|---|---|---|---
VIPeRa R99 ; R102  |  2012 |  632 |  1,264 |  48 $\times$ 128 |  15 |  2 |  DA |  C2, C6, C8, C10 |  Cropped
APiS R110  |  2013 |  – |  3.661 |  48 $\times$ 128 |  35 |  – |  DA |  C2, C3, C6, C7, C8 |  Cropped
VIPeRb R99 ; R103  |  2014 |  632 |  1,264 |  48 $\times$ 128 |  21 |  2 |  DA |  C2, C6, C8, C10 |  Cropped
PRID R100 ; R103  |  2014 |  934 |  24,541 |  64 $\times$ 128 |  21 |  2 |  DA |  C2, C6, C8, C10 |  Cropped
GRID R101 ; R103  |  2014 |  1,025 |  1,275 |  from 29 $\times$ 67 to 169 $\times$ 365 |  21 |  8 |  DA |  C1, C2, C3, C6, C8, C10 |  Cropped
PETA R104  |  2014 |  – |  19,000 |  from 17 $\times$ 39 to 169 $\times$ 365 |  105 |  – |  DA |  C1, C2, C4, C6, C8 |  Cropped
Mareket-1501 R105  |  2015 |  1,501 |  32,217 |  64 $\times$ 128 |  30 |  6 |  DA |  C2, C3, C6, C8, C10 |  Cropped
SoBiR R106  |  2016 |  100 |  1,600 |  from 60 $\times$ 150 to 191 $\times$ 297 |  12 |  8 |  DA |  C1, C2, C6, C8, C10 |  Cropped
DukeMTMC-reID R107  |  2017 |  1,812 |  36,441 |  from 34 $\times$ 85 to 193 $\times$ 477 |  23 |  8 |  DA |  C1, C2, C3, C6, C8, C10 |  Cropped
CUHK-PEDES R85  |  2017 |  13,003 |  40,206 |  from 13 $\times$ 34 to 374 $\times$ 800 |  – |  – |  NLD |  C1, C2, C3, C6, C8 |  Cropped
PA-100k R117  |  2017 |  – |  1,00,000 |  from 50 $\times$ 100 to 758 $\times$ 454 |  26 |  598 |  DA |  C1, C2, C6, C8, C10 |  Cropped
RAP R108  |  2018 |  2,589 |  84,928 |  from 33 $\times$ 81 to 415 $\times$ 583 |  72 |  25 |  DA |  C1, C2, C3, C4, C6, C7, C8, C10 |  Cropped
AVSS 2018 Challenge II R5  |  2018 |  151 |  20,453 (frames) |  704 $\times$ 576 |  16 |  6 |  DA |  C2, C3, C4, C5, C6, C7, C8, C9, C10 |  Full frame
RAP-LSPRC R109  |  2019 |  2,589 |  67,571 |  from 33 $\times$ 81 to 415 $\times$ 583 |  72 |  25 |  DA |  C1, C2, C3, C4, C6, C7, C8, C10 |  Cropped
P-DESTRE R155  |  2020 |  269 |  – |  3840 $\times$ 2160 |  16 |  – |  DA |  C2, C3, C6, C7, C8, C9 |  Full frame
Abbreviations used:
a Layne et al. R102  introduce soft biometric annotation to VIPeR dataset in 2012. |  C1 –Different resolutions |  C6 –Different viewpoint / poses.
b Layne et al. R103  further added 6 more soft biometric annotation to VIPeR dataset in 2014. |  C2 –Varying illumination conditions |  C7 –Crowded scene.
W = Width of an image, H = height of an image. |  C3 –Occlusion |  C8 –In/outdoor environments
DA = Discrete Attribute textual query. |  C4 –Merging with background |  C9 –Time consistency/duration
NLD = Natural Language Description. |  C5 –Different illumination in the scene for a single sequence |  C10 –Type of camera / multiple camera

Deng et al. R104  introduce the first large-scale PEdesTrian Attribute (PETA)
dataset to overcome the limitations of VIPeR, APS, PRID, and GRID datasets.
They introduce C1, C2, C4, C5, C7, C8 and diversity by annotating 61 binary
and 4 multiclass (total 105) attributes in 19,000 images collected from
surveillance datasets like 3DpeS R111 , CAVIAR4ReID R112 , CUHK R113 , GRID
R101 , i-LIDS R114 , PRID R100 , SARC3D R115 , TownCentre R116  and VIPeR R99
. Martinho et al. R106  create the SoBiR dataset with 12 multi-class
attributes. It is a comparatively smaller dataset with 1,600 images. However,
each soft biometric attribute possesses multiple classes which help
discriminate against people better.

Recent research advancement is increasing rapidly in deep learning domains,
which requires a more diverse and huge amount of data for better
generalization. Datasets like VIPeR, APS GRID, and SoBiR are too small for
deep learning-based frameworks. Datasets published in the last five years
provide a good amount of data to create robust models and algorithms. Large-
scale datasets such as Mareket-1501 R105  have 32,217 images of 1,501 persons
and DukeMTMC-reID R107  provides 36,441 images of 1,812 persons. These small
and large-scale datasets use discrete attribute-based queries (Fig. 12(a)),
which has limited practical usage, as discussed in Sect. 2.3. Thus, Li et al.
R85  propose a large-scale dataset with natural language description-based
annotations. The dataset is known as CUHK Person Description Dataset (CUHK-
PEDES). It provides 40,206 images of 13,003 persons with 80,412 description
sentences. It is the only large-scale dataset with a natural language
description that provides an opportunity to explore the relationship between
language and vision for a person retrieval problem.

Liu et al. R117  propose a large-scale pedestrian attribute (PA) dataset with
100000 images. It is known as PA-100k. It consists of the highest number of
images in all the datasets with a more diverse collection from 598 scenes. It
can only be used for attribute-based person retrieval, not for person re-
identification. Further increasing the scale in all aspects, Li et al. R108
created a Richly Annotated Pedestrian (RAP). Li et al. R109  also make a RAP -
Large-Scale Person Retrieval Challenge (RAP-LSPRC) dataset. RAP-LSPRC is a
subset of RAP with a comprehensive evaluation and benchmark dataset. By far,
RAP contains the highest number of images (84,928) collected from 25
surveillance cameras installed in the unconstrained indoor environment of a
shopping mall. RAP and RAP-LSPRC are useful for both person retrieval and
person re-identification tasks.

Datasets discussed so far contain an image gallery of cropped persons from
surveillance video frames. Hence, in such image problems related to person
detection, occlusion, merging with background, crowded scene, and illumination
variations in a single sequence cannot be overcome. It limits the development
of an end-to-end intelligent surveillance system. Halstead et al. first
developed an AVSS 2018 challenge II dataset with full surveillance frames
(20,453) collected from 6 indoor cameras. It is the only dataset that contains
video sequences of 151 persons (110 training + 41 testing) with varying
surveillance frames from 21 to 290. They provide shallow resolution (704
$\times$ 576)) frames. They cover most real-time surveillance challenges
(Sect. 2.1) to develop an end-to-end solution for person retrieval. Such
surveillance frames are in Fig. 4, 5, 6, 7, and Fig. 9.

Recently, Kumar et al. R155  released a UAV-based dataset for Pedestrian
Detection, Tracking, Re-Identification, and Search (P-DESTRE) from aerial
devices. It also contains full surveillance frame videos. These videos are
captured with 4K spatial resolution (3840 $\times$ 2160) and over different
days and times. Thus, P-DESTRE is the first dataset to provide consistent ID
annotations across multiple days. Other datasets discussed so far do not cover
this challenge. Although this dataset covers rich annotations, it lacks
clothing colour annotations. By reviewing all the datasets, the commonly used
soft biometric attribute set is {gender, upper-body cloth type, lower-body
cloth type, upper-body cloth color, lower-body cloth color, upper-body cloth
texture, clothing style, height, hair color, hairstyle, backpack, carrying an
object, eye-wear, shoe}. Such variety and diversity show the continuous
evolution of challenging datasets proposed by active researchers in the vision
and language fields.

###  2.5 Evaluation metric

Different performance measures evaluate the performance of any methodology.
The section covers such evaluation metrics before discussing different
procedures (Sec. 3) because knowledge of metrics helps to understand the
performance of the method better.

##### Accuracy (ACC):

Classification accuracy usually evaluates machine learning models. It is the
ratio of correct predictions to the total number of samples in the test. This
metric represents the overall efficiency of the system. The metric works well
with a balanced dataset, i.e., the number of positive and negative samples in
the dataset are equal. The accuracy equation is given by eq. 1,

$ACC=\frac{TP+TN}{P+N}=\frac{TP+TN}{TP+TN+FP+FN}$ (1)

Where,

P

= The number of real positive samples in the data

N

= The number of real negative samples in the data

TP

= True positive i.e., correctly identified

FP

= False positive i.e., incorrectly identified

TN

= True negative i.e., correctly rejected

FN

= False negative i.e., incorrectly rejected

Let us consider an example of gender classification for 100 persons to
understand the metric and the same example is going to be used for all the
metric discussions in this section. Here, the positive class is male and
negative class is female.

Gender classification example:
---
True positive (TP):
Class: Male
Model predicted: Male
Number of TP results: 1 |  False Positive (FP):
Class: Female
Model predicted: Male
Number of FP results: 1
False Negative (FN):
Class: Male
Model predicted: Female
Number of FN results: 10 |  True Negative (TN):
Class: Female
Model predicted: Female
Number of TN results: 88

$ACC=\frac{1+88}{1+88+1+10}=0.89$

Accuracy is 89% i.e., 89 correct predictions out of 100 persons. Out of 100
persons, 11 are male, and 89 are female. 88 females out of 89 are correctly
identified. But, only one male out of 11 is correctly identified. Thus, 89% of
model accuracy seems very good. But accuracy alone as a metric can’t help such
a class-imbalanced problem as it gives a false sense of high accuracy. A real
problem can arise when the cost of misclassifying minor samples is very high.
Thus, the metric discussed below is helpful, along with accuracy.

##### Precision:

It is a positive predictive value and decides the fraction of correct positive
predictions in total positive predictions. Thus, it provides the proportion of
positive predictions of the samples that are correct. It is a measure of
quality, and its high value means that the classifier returns more relevant
results, i.e., it suggests how many selected items are appropriate. Precision
is calculated as eq. 2,

$Precision=\frac{TP}{TP+FP}$ (2)

For gender classification example,

$Precision=\frac{1}{1+1}=0.5$

The model has precision of 0.5 i.e., 50% of the time the model is correct when
it predicts the person gender as male.

##### Recall or True Positive Rate (TPR):

It represents the classifier’s sensitivity, and it is the fraction of correct
positive predictions to the total number of relevant samples. It provides the
proportion of positive predictions in the correctly retrieved samples. It is a
measure of the quantity and suggests how many pertinent samples have been
selected. Recall is calculated as eq. 3,

$Recall\;or\;TPR=\frac{TP}{TP+FN}$ (3)

For gender classification example,

$Recall=\frac{1}{1+10}=0.09$

The model has a recall of 0.09 i.e., it correctly identifies 9% of total male
persons.

##### Mean Average Precision (mAP):

It is an average of maximum precisions at different recall values. An object
detector accuracy is measured by mAP. It is calculated as eq. 4,

$mAP=\frac{1}{Total\;Samples}\sum_{Recall_{i}}Precision(Recall_{i})$ (4)

##### Intersection-over-Union (IoU):

The localisation accuracy of the object detector is measurable by
Intersection-over-Union (IoU). It uses the bounding box of the object for
evaluation. IoU is the fraction of the intersecting area to the union area
between the output bounding box and the ground truth. It measures how good the
detector is in localizing objects to the ground truth. Fig. 13 depicts the IoU
metric.

$IoU=\frac{Area\;of\;overlap}{Area\;of\;union}=\dfrac{D\cap GT}{D\cup GT}$ (5)

$D$ = bounding box output of algorithm and $GT$ = ground truth bounding box.

Figure 13: Intersection-over-Union (IoU).

Different person retrieval methodologies reviewed below use the evaluation
metrics discussed above. Methods in R69 ; R74 ; R75 ; R76 ; R78 ; R79 ; R80
use IoU or average IoU, R69 ; R70 ; R78  use TPR and R81 ; R82 ; R84 ; R85 ;
R89 ; R134 ; R135 ; R136 ; R137 ; R138 ; R139  use Top-1 accuracy as
evaluation metrics.

##  3 Person retrieval methodologies

Person retrieval is a subject undergoing intense study for the past two
decades due to its prime application to public safety. A large-scale
intelligent surveillance system requires person retrieval using visual
attributes or available person image(s). Vision community researchers are
proposing a plethora of methodologies for robust person retrieval. They have
two major categories based on the type of input query:

  * •

Image-based retrieval.

  * •

Soft biometric attribute-based retrieval.

Classification of person retrieval methodologies based on the type of input
query is shown in Fig. 14. A person is retrievable from the cropped person
image gallery or full frames of surveillance video. The latter is more
suitable for real-time scenarios. The person retrieval example of each
category is also shown in Fig. 14.

Figure 14: Classification of person retrieval methodologies based on the type
of input query. Each classification category shows example of person retrieval
from cropped person image gallery and full frames of surveillance video.
Images shown in image gallery are adopted from RAP R108  dataset and
surveillance video frames are adopted from PRW R118  dataset to showcase the
example.

Image-based person retrieval methodologies aim to retrieve the target from the
large-scale image gallery. This large-scale image gallery or dataset is from
different non-overlapping cameras. Such person retrieval techniques are known
as person re-identification (Re-ID). These techniques require at least one
image of the target person for querying the system to perform re-
identification of the same identity from different cameras. A person ReID
technique assumes that the image gallery contains at least one image of the
target person captured from a camera other than the query image. Such
methodologies fail when a probe image is not available.

In contrast, soft biometric attribute-based person retrieval methodologies do
not require a probe image. It uses a semantic person description generated
from soft biometric attributes. Such a description (textual query) is input to
the person retrieval system. Soft-biometric attribute-based retrieval is
further divisible into two categories based on the type of textual query (Fig.
12 and 14):

  1. 1.

Discrete attribute-based retrieval.

  2. 2.

Natural language description based retrieval.

A discrete attribute-based person retrieval system searches for the person
using soft biometrics attributes, e.g., clothing colour, clothing type, and
gender. While a natural language description-based system accepts a natural
description, e.g., A woman is with average height wearing a white midi with no
sleeve. She has black open hair.

Applications like obvious question answering, image captioning, and person
retrieval are gaining much attention due to the association of two widely
researched domains, i.e., computer vision and natural language processing.
Such applications expect the learning of discriminative feature
representations from both images and text. The person retrieval algorithm
ranks gallery images or person(s) in the surveillance frame according to their
relevance to the description. The best matching images from the gallery or
surveillance frame return the person of interest. This class of surveillance
problems is known as text-based person search or person retrieval using
natural language descriptions R85 ; R139 . Such a person retrieval problem
aims to enable retrieval of the person of interest using two different domain
modalities, i.e., text and image. It takes one type of data as an input query
(i.e., text). It retrieves the relevant data of another type (i.e., person
image) by mapping textual and visual features (Sec. 2.3). Such kind of
retrieval is also known as cross-modal retrieval R139 ; R140 ; R141 ; R142 .

Among all person retrieval methodologies, image gallery-based techniques are
not preferable for end-to-end system-level evaluations. They do not consider
challenges like occlusion, pose, and illumination in person detection from the
full surveillance frame. Natural language description-based person retrieval
systems are more suited for real-time person retrieval from surveillance
videos. A person retrieved using a textual query is an input query to image-
based person ReID systems for re-identification of the same target in the
camera network. This paper discusses soft biometric attribute-based person
retrieval methodologies in a further section.

###  3.1 Discrete attribute-based person retrieval

Soft biometric attributes like clothing colour and clothing type do not stay
consistent for a given individual for a length of time. Such soft attributes
keep on changing over a short period. Thus, soft biometric attribute-based
person retrieval methods are short-term retrieval methods R71 ; R119 . Such
methods are well suited for applications like criminal investigation and
searching for missing persons for a limited period. Discrete attribute-based
person retrieval methods are further divisible into two categories based on
how the features are extracted, i.e.,

  1. 1.

Handcrafted feature-based person retrieval.

  2. 2.

Deep feature-based person retrieval.

####  3.1.1 Handcrafted feature-based methodologies

Research before the era of deep learning shows promising methods for person
retrieval based on handcrafted features. Fig. 15 shows the general block
diagram for person retrieval methods that use handcrafted features. Person
detection is done using adaptive background segmentation, face detection,
frame differencing, and query-based avatar creation. Feature extraction is a
critical step where hand-engineered features are extracted using popular
algorithms. Further, the feature fusion and classification are done for target
person retrieval.

Vaquero et al. R71  exploit the limitation of the sensitivity of face
recognition technology against illumination changes, low-resolution videos,
and pose variations. They are the first to implement a video-based visual
surveillance system that uses a person’s fine-grained parts and attributes.
Person detection uses a face detector. Further, the body is divisible into
three regions with soft biometrics from the areas, i.e., face (hair type,
eyewear type, facial hair type), torso (clothing color), and leg (clothing
color). The nine facial attributes are extracted by training an individual
Viola-Jones detector using Haar features. A normalized colour histogram is in
hue, saturation and luminance (HSL) space for each body part (i.e., torso and
leg).

Figure 15: Block diagram for handcrafted feature-based methods.

Denman et al. R73  also propose a body part segmentation-based approach that
detects a person using adaptive background segmentation R91  and segments the
body into three parts (head, torso, and leg) using an average gradient across
rows of the image. Calculation of height uses the Tsai R120  camera
calibration approach and colour histograms for clothing colour. These features
are fused using a weighted sum fusion approach. The algorithm evaluation on
the PETS R121  dataset achieves the best results with an Equal Error Rate
(EER) of 26.7% (colour model), the worst EER rate of 39.6% (size model), and
an EER rate of 29.8% (a combination of colour and size).

The methods in R71 ; R73  detect a person first and then match the query
rather than searching the person based on the query. Such limitation is
removable by creating an avatar of the person from discrete soft biometric
attributes R74 . The particle filter is then applied to drive a search in the
surveillance frame. The methods mentioned here propose a new dataset for soft
biometric-based person localisation. The height of a person is broken up into
5 classes (concise, short, average, tall, and very tall) and colour into 11
classes (black, blue, brown, green, grey, orange, pink, purple, red, yellow,
and white). Localisation accuracy is measurable using Intersection-over-Union
(IoU). These methods achieve 24.28% average localisation accuracy.

Pedestrian semantic attribute-based ApiS dataset is in R110  with 11 binary
attributes and two multiclass attributes. AdaBoost classifiers are useful for
binary attribute classification, and K Nearest Neighbours (KNN) classifier is
useful for multiclass attribute classification. This paper focuses on
attribute classification and is useful for person retrieval. Halstead et al.
R75  further extend the surveillance dataset of R74  by annotating 12
multiclass attributes on 110 persons. The authors improve the performance of
R74  by adding more attributes for avatar creation and considering the unequal
reliability of each attribute. A 21% improvement is reported over the baseline
R74 . Avatar further extends in the form of channel representation (CR) R76 .
The algorithm also incorporates shape information in the form of HOG
representation to improve the performance. CR captures the spatial
characteristics of colour and texture by representing them in the form of a
multi-dimensional image. Each channel in CR represents either colour or
texture information. The CR based approach achieves 44% average localisation
accuracy.

Martinho et al. R106  generate a feature vector of length 4704 for each 256
$\times$ 256 pre-processed image. They use CIE-LAB, and the Gabor channel
filters feature for the generation of an Ensemble of Localised Features (ELF)
descriptor. A HOG based descriptor is also useful for the creation of 2304
features. The Extra-Trees (ET) supervised ensemble learning algorithm is
applied for classification. The algorithm achieves 12.5% and 20.1% at rank-1
accuracy for one-shot and multi-shot identification. Shah et al. R70  use
clothing colour and clothing type for person retrieval. They detect the person
using motion segmentation based on frame differencing. Background clutter is
removable by a morphological operation. The ISCC-NBS colour model R122  and
CIEDE2000 R123  distance metrics are useful for colour classification. Their
method achieves a 67.96% TP rate, which is better than the GMM (44.88%)
approach. Table 3 shows an overview of the handcrafted feature-based
methodologies. The performance column shows the highest value reported in the
relevant literature in multiple scenario-based analyses.

Table 3: Overview of soft biometric attribute based person retrieval methodologies based on handcrafted features. Work |  Year |  # Soft attributes |  Features |  Classifier |  Dataset |  Performance
---|---|---|---|---|---|---
Vaquero et al. R71  |  2009 |  5 |  Haar, normalized color histogram |  Euclidean |  Private |  –
Denman et al.R73  |  2009 |  3 |  Height, color histogram |  Weighted sum |  PETS R121  |  EER (26.7%)
Denman et al. R74  |  2012 |  3 |  Height, CIE-LAB |  Weighted average |  Private |  IoU (24.28%)
Zhu et al. R110  |  2013 |  13 |  MB-LBP histogram, HOG |  AdaBoost, KNN |  APiS R110  |  –
Halstead et al.R75  |  2014 |  12 |  Height, CIE-LAB |  Weighted average |  SAIVT R75  |  IoU (29.38%)
Denman et al.R76  |  2015 |  12 |  HOG, Color, shape, height |  Weighted average |  SAIVTR75  |  IoU (30.00%)
Martinho et al.R106  |  2016 |  12 |  CIE-LAB, HOG, ELF |  ETC |  SoBiR R106  |  nAUC (88.1%)
Shah et al.R70  |  2017 |  3 |  Color histogram |  CIEDE2000 |  Private |  TP (67.96%)
Abbreviations used:
EER = Equal Error Rate |  IoU = Intersection-over-Union |  HoG = Histograms of Oriented Gradients
MB-LBP = multi-scale block local binary patterns |  KNN = K - Nearest Neighbors |  TP = True Positive rate
ELF = Ensemble of Localised Features |  ETC = Extra Tree Classification |  nAUC = normalised Area Under Curve

With the advent of deep learning techniques, person retrieval methods have
seen a significant improvement in accuracy measures. Most state-of-the-art
person retrieval methods are based on deep learning architectures.

####  3.1.2 Deep feature-based methodologies

Deep learning-based methodologies are becoming popular in the past few years
due to their efficient feature learning ability. The deep Convolutional Neural
Network (DCNN) based approach has gained more attention in the computer vision
community. Table 4 shows an overview of the deep feature-based person
retrieval methodologies. The performance column shows the highest value
reported in the relevant literature in the case of multiple scenario-based
analyses. Semantic Retrieval Convolutional Neural Network (SRCNN) developed by
Martinho et al. in R124  shows the evaluation of a similar setup of R106 .
Binary Cross-Entropy (BCE) and Mean Squared Error (MSE) loss functions
quantify binary classification and regression. SRCNN achieves 35.7% and 46.4%
at rank-1 accuracy for one-shot and multi-shot identification, respectively.
Thus, a deep feature based SRCNN approach demonstrates a rank-1 accuracy
improvement of 23.2% and 26.3% over a handcrafted feature-based system of R106
.

The baseline method R76  of AVSS 2018 challenge II R5  dataset is
implementable using handcrafted features, while all participants R69 ; R79 ;
R80  of the challenge have evaluations based on deep features. The following
discussions in this section consist of methodologies implemented on AVSS 2018
challenge II R5  dataset as well as some other approaches on large scale
datasets like Market-1501 R105 , DukeMTMC R107 , PA100K R117  and CUHK03 R113
.

Galiyawala et al. R69  use height, clothing colour, and gender for person
retrieval. Person detection and semantic segmentation using Mask R-CNN R98
help to remove the cluttered background. It results in a clutter-free torso
patch for efficient colour classification. Height is available using a camera
calibration approach R120 . Torso clothing colour and gender are classified
using a fine-tuned AlexNet R125  based individual model. Sequential
implementation of height, colour, and gender filter aims to eliminate the
detected persons and leave only the target person. This linear filtering-based
approach achieves an average IoU of 0.36. Schumann et al. R80  detect the
person using the Single-Shot Multibox Detector (SSD) R126 . Early-stage false
positives are eliminated by background modelling based on a mixture of
Gaussian distribution. The strategy of the ensemble of classifiers is adapted,
and predictions are fused by computing mean or weighted mean. The evaluation
is based on the Euclidean distance between a query vector and each detected
person’s attribute probability to produce the final result.

Table 4: Overview of soft biometric attribute based person retrieval methodologies based on deep features. Work |  Year |  # Soft attributes |  Deep network |  Dataset |  Performance
---|---|---|---|---|---
Martinho et al.R124  |  2016 |  12 |  SRCNN R124  |  SoBiR R106  |  nAUC (92.8%)
Galiyawala et al.R69  |  2018 |  3 |  Mask R-CNN R98 , AlexNet R125  |  AVSS 2018 Challenge II R5  |  Avg. IoU (0.363)
Schumann et al.R80  |  2018 |  9 |  SSD R126 , MobileNet-v1 R128 , ResNet-50-v1 R127 , DenseNet-121 R129 , DenseNet-169 R129  |  AVSS 2018 Challenge II R5  |  Avg. IoU (0.503)
Yaguchi et al.R79  |  2018 |  9 |  Mask R-CNN R98 , DenseNet-161 R129  |  AVSS 2018 Challenge II R5  |  Avg. IoU (0.511)
Galiyawala et al.R78  |  2019 |  4 |  Mask R-CNN R98 , DenseNet-169 R129  |  AVSS 2018 Challenge II R5  |  Avg. IoU (0.569)
Sun et al.R81  |  2018 |  30 |  PCB R81 , ResNet-50 R127  |  Market-1501 R105 , DukeMTMC R107 , CUHK03 R113  |  Rank-1 acc. (92.3%)
Dong et al.R83  |  2019 |  30 |  AIHM R83 , ResNet-50 R127  |  Market-1501 R105 , DukeMTMC R107 , PA100K R117  |  Rank-1 mAP. (50.5%)
Abbreviations used:
IoU = Intersection-over-Union |  SRCNN = Semantic Retrieval Convolutional Neural Network |  SSD = Single-Shot Multibox Detector
TP = True Positive rate |  AIHM = Attribute-Image Hierarchical Matching |  PCB = Part-based Convolutional Baseline
nAUC = normalised Area Under Curve |  DenseNet = Densely connected convolutional networks |  ResNet = Residual Neural Network

Yaguchi et al. R79  also use mask R-CNN for person detection and DenseNet-161
for attribute classification. Initially, they estimate all the attributes of
the detected persons, and then the matching score is calculated using a
Hamming loss. The person with the minimum loss is the target. Galiyawala et
al. R78  further improve the linear filtering approach of R69  by introducing
adaptive torso patch extraction and bounding box regression. Torso patch
extraction is undertaken by deciding the torso region according to clothing
type attribute. Thus, it removes noisy pixels from the torso patch and
provides better colour classification. An IoU based box regression predicts
the bounding box in the frame where soft biometric attribute-based retrieval
fails. The approaches in R69  and R78  follow a linear filtering approach that
filters out the non-matching person according to attributes and leaves the
target in the end. The other two methods in R79  and R80  estimate all the
detected attributes of a person in parallel. These methods fuse the
characteristics in the end to retrieve the target. The linear filtering
approach does not need to estimate all the attributes for all detection
persons. However, an error in the first filter will propagate to further
attribute filtering and reduce retrieval accuracy.

The AVSS 2018 challenge II R5  dataset is evaluated based on two metrics; an
average IoU and percent of frames with an IoU $\geq$ 0.4. State-of-the-art
average IoU of 0.569 is achieved by a linear filtering approach R78 . 75.9% of
frames with an IoU $\geq$ 0.4 is achieved by R80 . Some sample qualitative
results of person retrieval using the method in R78  are shown in Fig. 16. It
offers a surveillance frame from AVSS 2018 challenge II datasets for test
sequence 40 and frame 66 with a semantic description, namely, height (very
short; 140-160 cm), torso type (short sleeve), torso colour-1 (yellow), torso
colour-2 (NA) and gender (male). The first image shows the person detection
output using Mask R-CNN. Height filter output shows that many persons match
the given query. Among these persons, the colour filter output produces a
couple of matches and this is further refined by using a gender filter. It is
to be noted that all deep features-based approaches R59 ; R78 ; R79 ; R80
perform better than the handcrafted feature-based baseline approach R76 .

Figure 16: Sample output frames for the person retrieved using semantic
description by approach in R78 : person from Test Sequence 11 and Frame 66
with semantic description height (very short; 140-160 cm), torso type (short
sleeve), torso colour-1 (yellow), torso colour-2 (NA) and gender (male).
Images from left to right are Mask R-CNN person detection, height filtering,
clothing colour filtering and gender filtering.

Sun et al. R81  apply part level features because they provide fine-grained
information for person description. The authors propose a Part-based
Convolutional Baseline (PCB) network for part-based feature extraction. Part-
level features are learnable by conducting uniform partition on the conv-layer
without explicitly partitioning images. The spatial consistency within-part is
exploitable to refine the coarse partition provided by PCB. The improvement in
the uniform partition is made by the Refined Part Pooling (RPP) network. It
achieves 92.3% of rank-1 accuracy with only the PCB. Further, it improves to
93.8% of rank-1 accuracy by employing RPP with PCB network on the Market-1501
R105  dataset. The text attribute combinations exist on a large scale in a
real scenario. However, a minimal amount of combination with sufficient data
is available for training. Except for such combinations, others are never
modelled during training. Thus, Dong et al. R83  formulate a textual attribute
query-based person search problem as a zero-shot (ZSL) learning problem for
the first time. The authors propose Attribute-Image Hierarchical Matching
(AIHM), which matches attributes and images at multiple hierarchical levels.
The algorithm achieves state-of-the-art rank-1 mAP of 43.3% on Market-1501
R105 , 50.5% on DukeMTMC R107  and 31.3% on PA100K R117  datasets. Table 5
shows a comparison of handcrafted and deep feature-based methods with their
advantages and disadvantages.

Table 5: Comparison of handcrafted and deep feature based methods. Category |  Sub class |  Characteristics Advantages/disadvantages
---|---|---
Hand crafted features |  Haar R71  |  Face detection works well for near view but it is difficult to detect for far view and crowded scenarios.
|  Color histogram R70 ; R71 ; R73  |  Simple to implement and supports the less number of colors (i.e. primary colors) for classification but difficult if color classes increases (different shades of color) as it involves quantization process for histogram calculations.
|  HOG R76 ; R110  |  HOG is good descriptor but its coarseness leads to sub-region processing in image which increases computational complexity.
|  Height R73 ; R74 ; R75  |  It is calculated based on camera calibration parameters, but a small error in parameters estimation leads to high error in height calculations. Moreover, the camera parameter estimation need to done again if camera is moved.
Deep features |  AlexNet R69  |  It is small network and 61M parameters to learn, but the classification accuracy is low.
|  MobileNet [80] |  It’s a light weight network and 4.2M parameters to learn, but it is having little high error rate.
|  ResNet-50 [80, 81, 83] |  It is having advantage of residual block which is having skip connection from input for better learning ability. Over 25M parameters to learn.
|  DenseNet [78, 79, 80] |  It is having connection from each layer to every other layer and hence leveraging information from other layers which learns much better compare to other networks.

A query-based personal retrieval system usually provides soft biometric
keywords as the input query. Improvisation in natural language processing now
allows automatic extraction of the keywords from sentences. It will move the
system towards full automation.

###  3.2 Natural language description-based person retrieval

Cross-modal retrieval-based applications are drawing attention due to the
rapid growth of multimodal data like text, image, video, and audio. Features
from different modalities like text and image are not directly comparable as
they lie entirely in other spaces. Hence, it is a challenging problem due to
the sizeable heterogeneous gap between different text and image modalities.
One such issue of person retrieval from surveillance video using natural
language description is in this section. Table 6 shows an overview of
different methodologies for person retrieval using natural language
descriptions.

Zhou et al. R89  develop an attention-based algorithm that localises a person
in the surveillance frame using attributes and natural language query. The
author annotated the cityscapes dataset R132  surveillance frame with
attributes and descriptions because the dataset did not have natural language
descriptions. Matrix representations of sentence expressions use the Skip-gram
model R143 . Attributes and descriptions are by bidirectional Long-Short Term
Memory (BLSTM) R130 ; R131  network. Visual features extraction is done using
Faster R-CNN R97  and ResNet152 R127  with the algorithm achieving 74.6%
recall@1 i.e., 74.6% of the highest scoring box is correct. The Cityscape
dataset contains only street views, i.e., frontal view camera. Hence, it does
not cover various view challenges for surveillance. Also, the description
annotated dataset is not available publicly and it limits the usability.

Li et al. R85  first published a large-scale person dataset with natural
language description, CUHK-PEDES (publicly available), as discussed in Sect.
2.4. The author proposes a Recurrent Neural Network with a Gated Neural
Attention mechanism (GNA-RNN) to establish the baseline on CUHK-PEDES. The
network consists of a visual sub-network for visual feature extraction from an
image and a language sub-network for textual feature extraction from a
description. The visual sub-network consists of VGG-16 R133  as a backbone
network and generates 512 graphical units. Each optical unit determines the
existence of a specific appearance pattern. RNN with Long Short-Term Memory
(LSTM) is useful in language sub-network, which takes words and images as
input. It outputs unit level attention that decides which visual units should
pay more attention to the word. The word-level gate determines the importance
of the word, e.g., the word “white”has more weightage than the word “the”. The
GNA-RNN network is trained in an end-to-end manner and provides a top-1
accuracy of 19.05%. This approach creates the baseline for further research in
cross-modal person retrieval based on natural language descriptions.

Table 6: Overview of natural language description based person retrieval methodologies. Work |  Year |  Deep network |  Method |  Dataset |  Performance
---|---|---|---|---|---
|  |  Visual features from person image |  Text features from natural language description |  |  |
Zhou et al.R89  |  2017 |  Faster R-CNN R97 , Resnet152 R127  |  Bi-LSTM R130 ; R131  |  – |  Cityscapes R132 , Private |  Recall@1 (74.6%)
Li et al.R85  |  2017 |  VGG-16 R133  |  LSTM R130  |  GNA-RNN R85  |  CUHK-PEDES R85  |  Top-1 acc. (19.05%)
Li et al.R84  |  2017 |  VGG-16 R133  |  LSTM R130 , word2vec [144] |  IATV [84] |  CUHK-PEDES R85  |  Top-1 acc. (25.94%)
Chen et al.R82  |  2018 |  ResNet50 R127  |  LSTM R130  |  GLIA R82  |  CUHK-PEDES R85  |  Top-1 acc. (43.58%)
Zheng et al.R134  |  2017 |  ResNet-50 R127  |  Text-CNN R134 , word2vec R144  |  DPCE R134  |  CUHK-PEDES R85  |  Top-1 acc. (44.40%)
Niu et al.R135  |  2019 |  VGG-16 R133 , ResNet-50 R127  |  Bi-GRU R145 ; R146  |  MIA R135  |  CUHK-PEDES R85  |  Recall@1 (48.00%)
Zhang et al.R136  |  2018 |  MobileNet R128  |  Bi-LSTM R130 ; R131  |  CMPC + CMPM R136  |  CUHK-PEDES R85  |  Recall@1 (49.37%)
Wang et al.R137  |  2019 |  MobileNet R128  |  Bi-LSTM R130 ; R131  |  MCCL R137  |  CUHK-PEDES R85  |  Top-1 acc. (50.58%)
Sarafianos et al.R138  |  2019 |  ResNet-101 R127  |  BERT R148 , LSTM R130  |  TIMAM R138  |  CUHK-PEDES R85  |  Top-1 acc. (54.51%)
Aggarwal et al.R139  |  2020 |  MobileNet R128  |  NLTK R149 , Bi-LSTM R130 ; R131  |  CMAAM R139  |  CUHK-PEDES R85  |  Top-1 acc. (56.68%)
Abbreviations used:
BERT = Bidirectional Encoder Representations from Transformers |  CUHK-PEDES = CUHK Person Description Dataset |  MCCL = Mutually Connected Classification Loss
Bi-GRU = Bi-directional gated recurrent unit |  DPCE = Dual-Path Convolutional Image-Text Embeddings |  MIA = Multi-granularity Image-text Alignments
Bi-LSTM = Bidirectional Long-Short Term Memory |  GLIA = Global and Local Image-language Association |  GNA-RNN = Recurrent Neural Network with Gated Neural Attention
CMAAM = Cross Modal Attribute Aided Matching |  IATV = Identity-Aware Textual-Visual Matching |  TIMAM = Text-Image Modality Adversarial Matching
CMPM+CMPC = Cross-Modal Projection Matching loss and a Cross-Modal Projection Classification loss |  LSTM = Long-Short Term Memory |

The approaches discussed below are evaluated on this baseline. Most of the
large-scale dataset contains identity level annotations considered by Li et
al. R84  to match visual and textual domains. Identity-aware textual-visual
matching is done in a two-stage network. Identity-level annotations are
effectively utilized by introducing a Cross-Modal Cross-Entropy (CMCE) loss in
the stage-1 network. The CMCE loss implicitly maximizes inter-identity feature
distances and minimizes intra-identity feature distances. However, the
coupling between visual and textual features generated through CMCE loss is
loose. Hence, the initial matching of stage-1 is further refined by stage-2
CNN-LSTM with an underlying co-attention mechanism that produces the final
textual-visual matching confidence. This two-stage framework achieves 25.94%
of top-1 accuracy.

Global and local level image-language association (GLIA) is proposed by Chen
et al. R82  to exploit the semantic information available in the description.
The GLIA approach gains a significant boost to the top-1 accuracy from the
baselines and achieves 43.58% top-1 accuracy. Zheng et al. R134  focus on the
limitation of ranking loss. The authors do not explicitly consider the feature
distribution in a single modality. They overcome it by considering the problem
as instance-level retrieval and propose the instance loss. Each image-text
query pair is regarded as an instance and observed as a class during training
to learn finer granularity. Instead of considering pre-trained models for
feature extraction, the method view adopts end-to-end learning from the data
itself. Dual-path CNN-CNN i.e. Dual-Path Convolutional Image-Text Embedding
(DPCE) architecture is proposed instead of the CNN-RNN approach for image-text
matching. DPCE R134  achieves 44.40% of top-1 rank accuracy.

Multi-granularity Image-text Alignments (MIA) framework R135  adopts a
multiple granularities (i.e., global-global, global-local, and local-local
alignments) based approach for better similarity evaluations between text and
image. The global context of image and description matches global-global
granularity. On the other hand, relations between the global context and local
components establishes the global-local alignment. Visual human parts fit with
noun phrases in the final local-local granularity. The algorithm achieves
48.00% of recall@1. Zhang et al. R136  focus on learning discriminative
features by proposing two loss functions, i.e., cross-modal projection
matching (CMPM) loss and a cross-modal projection classification (CMPC) loss.
Natural language description is first tokenized into words and processed
sequentially using Bi-LSTM. Visual features are extractable from the last
pooling layer of MobileNet R128 . The association module embeds visual and
textual elements into a shared latent space. 49.37% of recall@1 is achieved by
the algorithm while considering both CMPM and CMPC losses. Similar to work in
R84 , Wang et al. R137  also utilize identity-level information and propose
Mutually Connected Classification Loss (MCCL). They first create a baseline
approach before applying MCCL for better feature embedding. This baseline
approach uses MobileNet pre-trained on ImageNet R147  for visual features, Bi-
LSTM for textual element, and triplet loss function for cross-modal feature
embedding. Triplet loss does not fully exploit feature distribution. The MCCL
classification weight is shared between both modalities. Only the baseline
approach with triplet loss achieves 45.55% of recall@1 while MCCL achieves
50.58% of recall@1.

The majority of approaches so far introduce a new loss function for the
network to learn better feature representations without the complexity of
textual phrases. For example, the word “t-shirt”is an adjective, but it might
be useful as a noun in the description. Such limitations go away by
introducing a Text-Image Modality Adversarial Matching (TIMAM) framework R138
. Sarafianos et al. R138  propose adversarial representation learning, which
helps bring features from different modalities very close. TIMAM attains top-1
accuracy of 54.51%. Aggarwal et al. R139  use attribute classification as an
additional task and identity for bridging the gap between different modalities
to improve representation learning. The method uses Deep Coral loss [150] to
reduce the modality gap. They achieve state-of-the-art top-1 accuracy of
56.61% on CUHK-PEDES dataset. Thus, most of the work R82 ; R84 ; R85 ; R134 ;
R135 ; R136 ; R137 ; R138 ; R139  shows cross-modal person retrieval on the
only publicly available CHUHK-PEDES dataset except R89 . CUHK-PEDES contains
only the image gallery of cropped persons, limiting practical usage in real-
time scenarios to retrieve the person from the input of full surveillance
frames.

##  4 Conclusions and open research challenges

Security of society has been a significant concern over the years, and
biometric-based security has gained prominence due to exhaustive research in
the field. Biometric security has evolved through the following stages:

  * •

Utilizing hard biometrics R15 ; R16 .

  * •

Use of soft biometrics to improve the hard biometric-based systems R6 .

  * •

Soft biometric attribute-based person retrieval R7 ; R8 ; R9 ; R10 ; R11 ; R12
; R13 .

  * •

Natural language description-based person retrieval R82 ; R84 ; R85 ; R134 ;
R135 ; R136 ; R137 ; R138 ; R139  inherently uses soft biometrics in natural
language description.

It has progressed from constrained environment-based fingerprint and face
recognition systems to surveillance video-based person retrieval in the wild.

There are mainly two types of person retrieval methods seen in current
research: discrete attribute-based (handcrafted feature-based and deep
feature-based) methods and natural language description-based methods. Each
has advantages and disadvantages. Handcrafted feature-based methods have
limited performance due to less discriminative power of hand-engineered
features and limited support to more classes for classification. Deep features
are much more robust and produce better retrieval results but require a high
amount of training data and high computation resources. Discrete attributes
are easy to acquire, but do not suit real-time applications where the natural
language descriptions are typically used. Most natural language description-
based methods (Table 6) are not evaluated on full-frame surveillance videos.
Moreover, the dataset is also a key element in any system development,
including real-time scenarios. Hence, the current research hotspot is to
develop an end-to-end person retrieval system from unconstrained surveillance
videos.

Given the comprehensive review, some open problems that require further
research and analysis are as follows:

  1. 1.

Diverse dataset development: This review focuses on person retrieval from
surveillance videos using soft biometrics, which are in the form of either a
discrete attribute or natural language description. Both forms of query are in
the form of text to the person retrieval system. The development of various
person retrieval methodologies would not have been possible without the
continuous evolution of challenging datasets (Table 2) proposed by active
researchers in the vision and language field. The datasets cover the following
challenges:

     * •

Different resolutions

     * •

Varying illumination conditions

     * •

Occlusion

     * •

Merging foreground with background

     * •

Viewpoint variations

     * •

Crowded scene

     * •

Indoor and outdoor environments

     * •

Time consistency/time duration

     * •

Type of camera or multiple cameras

These challenges (discussed in Sec. 2.1) are not present in all datasets, but
they lack one or the other difficulty. For example, CUHK-PEDES is the only
large-scale dataset available for cross-modal person retrieval. However, it
lacks full surveillance frames and samples of interest merging with background
and crowd scenarios.

Issues involved in person detection in regular and crowded scenarios cannot be
evaluated due to non-availability of full surveillance frames. AVSS 2018
challenge II is the only dataset that has low-resolution full surveillance
frames and covers many challenges. However, it also lacks outdoor scenes and
natural language descriptions. Thus, it also cannot be tested against various
natural outdoor conditions and limits the development of a robust end-to-end
cross-modal person retrieval framework. The more recent P-DESTRE dataset
provides 4K videos but lacks in indoor environment. The majority of the
datasets consist of very few samples with occlusion and a mixture of indoor
and outdoor environment samples of the same person. Thus, this review of a
promising large-scale dataset provides an opportunity to develop a more
challenging dataset that can overcome the limitations of an individual
dataset.

  2. 2.

Developing robust person attribute recognition model: As discussed in Sect.
2.3, person attribute recognition is the most critical task for visual
attribute extraction. This task uses a cropped person dataset for model
preparations. But the attribute model trained on one dataset may not achieve a
good performance for other datasets due to lack of diversity. Differences in
images from different datasets in terms of colour, view, pose, and
illumination conditions are readily observable in Fig. 17. The model training
has to be done by merging samples from a different dataset. But annotations to
the images from other datasets are not the same. For example, RAP images are
annotated with clothing type with 10 classes, namely, {ub-Shirt, ub-Sweater,
ub-Vest, ub-Tshirt, ub-Cotton, ub-Jacket, ub-SuitUp, ub-Tight, ub-ShortSleeve,
ub-Other}. In comparison, AVSS 2018 challenge II images are annotated with
clothing type with 3 classes, namely, {long sleeve, short sleeve, no sleeve}.
The solution is that these annotations need to be mapped by annotating one of
the datasets with similar annotations to other dataset classes.

Figure 17: Images from two different dataset: (a) RAP R108 , (b) AVSS 2018
challenge II R5 .

Class imbalance is also a problem, e.g., samples for every clothing type may
not be of the same number. The number of samples for clothing type in RAP
dataset are {14652, 5260, 3209, 15640, 7814, 19209, 2057, 2830, 6549, 1840}
R108 . Such a class imbalance problem can be resolved by weighting each class
during the training. Assign a higher weight to a class with fewer samples and
a lower weight to a class with more samples.

The total number of training images in the RAP dataset is 67943, while AVSS
has approximately 14000 images. It is observed from Fig. 17 that the RAP
dataset images have appropriate illumination conditions and resolutions.
However, AVSS dataset images have challenging illumination conditions and
lower resolutions. Thus, models trained by merging such datasets will produce
good accuracy for clear images like RAP but not for images like the AVSS
dataset. Such a problem can be resolved by generating images (for the dataset
having fewer images) using domain adaption methodologies as discussed in R160
; R161 .

  3. 3.

Robust framework development: Different methodologies are reviewed (Table 3,
4, 5) on the dataset mentioned in Table 2. State-of-the-art performance on
AVSS 2018 challenge II datasets in terms of average IoU is 0.569 achieved by
the linear filtering approach of R78  and an approach in R80  achieves 75.9%
of frames with an IoU $\geq$ 0.4. The method in R78  has a limitation of error
propagation due to the sequential processing of samples and a significantly
lower soft biometric amount. Such a process also require to create a separate
model for individual attributes, e.g., gender model and colour model. It is
not feasible and also a very costly solution to make a different model for
every specific attribute. A single model for attribute recognition by multi-
attribute learning (as discussed Sec. 2.3) helps resolve such a problem by
considering the solutions suggested above to develop a robust person attribute
recognition model.

  4. 4.

Adversarial learning-based approach: Although datasets are available with
increasing complexity, e.g., number of persons, images, and attributes, such a
cross-modal retrieval problem covers many possible real-scenarios, e.g., the
backpack carried by a person in one camera view may not be visible in some
other camera views. It is also expensive and tedious to cover many attribute
combinations with image annotations in a dataset. Thus, image and text, both
modalities together, produce many unseen scenarios (which are not available in
the dataset) for person retrieval. It limits the usability and scalability of
real-time deployments. Researchers in R156 ; R157 ; R158 ; R159  propose
adversarial learning-based approaches to mitigate such issues. Yin et al. R156
suggest that attribute-based concept generation learns a discriminative joint
space for image and text. Wu et al. R157 propose a deep adversarial data
augmentation method with attribute (DADAA), which regularises training by
generating unseen scenarios of person images. Unseen attribute combinations
are used by a symbiotic adversarial learning (SAL) R159  framework to
synthesize features of unseen categories and also to optimise feature
embedding by cross-modal alignment in the common embedding space. Hence, data
augmentation, unseen attribute combination and attribute-based image
generation using adversarial learning for person retrieval are also exciting
areas for further research.

  5. 5.

Backbone network explorations: The majority of the works reviewed in Tables 4,
5 use VGG-16, ResNet, DenseNet, and MobileNet, which are popular and provide
state-of-the-art results. Nevertheless, the supremacy of CNN networks like
EfficientNets R151  is as yet unexplored. None of the methods utilise
EfficientNets, which are smaller and faster than VGG-16, ResNet, DenseNet, and
MobileNet. For example, EfficientNet-B0 uses 5.3M parameters which are much
less in comparison with ResNet-50 (26M) and DenseNet-169 (14M) while
evaluating on ImageNet dataset R147 . Thus, a review of state-of-the-art
methodologies and deep network creates a scope to develop a more robust
network with better feature representation to improve performance. Hence,
person retrieval in surveillance using a textual query is still a challenging
and open research problem.

In summary, soft biometrics bridges the gap between human language
descriptions and machines to automate the person retrieval process. The
application domain ranges from crime investigation, security to missing person
retrieval. There are multiple soft biometric attributes that can be used for
person retrieval, and thus the paper offers a way of selecting an attribute
that will help the naive researcher begin research in the domain. The study
includes the perspective of challenging datasets, leading edge methods, and
deep networks. The paper presents an in-depth review by tracing the
development of retrieval in chronological order using handcrafted features,
simple attribute-based retrieval, and natural language-based retrieval. It
guides beginners to gather essential ingredients for their research journey
and, at the same time, provides an expert bird’s eye view of the domain
through a well-conceived summary. The goal of the comprehensive discussion
undertaken in this paper about large-scale datasets and various existing
problem definitions is to engender motivation to solve problems that exist in
this domain.

###### Acknowledgements.

The Board of Research in Nuclear Sciences (BRNS), Government of India
(36(3)/14/20/2016-BRNS/36020) supports this work. The authors acknowledge the
support of NVIDIA Corporation for a donation of the Quadro K5200 GPU used for
this research. The authors are thankful to Ahmedabad University, India, for
access to resources like GPUs. We would also like to thank the vision and
language domain’s active researchers for creating publicly available
challenging datasets.

## References

  * (1) Bertillon (1889). Instructions for taking descriptions for the identification of criminals and others, by means of anthropometric indications. American Bertillon Prison Bureau.
  * (2) H. T. F. Rhodes(1956). Alphonse Bertillon: Father of scientific detection. Abelard-Schuman, New York.
  * (3) https://en.wikipedia.org/wiki/Alphonse_Bertillon. Accessed 28 April 2020.
  * (4) Woodward JD, Orlans NM, Higgins PT (2003). Biometrics. The McGraw-Hill Companies, Inc.
  * (5) Halstead M, Denman S, Fookes C, Tian Y, Nixon MS (2018). Semantic person retrieval in surveillance using soft biometrics: Avss 2018 challenge II. In Proceedings of 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1-6, Auckland, New Zealand, 2018 Nov 27.
  * (6) Jain AK, Dass SC, Nandakumar K (2004). Soft biometric traits for personal recognition systems. In Proceedings of International Conference on Biometric Authentication (ICBA), pp. 731-738, Springer, Berlin, Heidelberg.
  * (7) Samangooei S, Guo B, Nixon MS (2008). The use of semantic human description as a soft biometric. In Proceedings of 2nd IEEE International Conference on Biometrics: Theory, Applications, and Systems, pp. 1-7, Arlington, USA, 29 Sept.-1 Oct 2008.
  * (8) Reid DA, Nixon MS (2010). Imputing human descriptions in semantic biometrics. In Proceedings of 2nd workshop on Multimedia in forensics, security and intelligence, pp. 25-30, ACM, Firenze, Italy, 29 Oct 2010.
  * (9) Dantcheva A, Velardo C, D’Angelo A, Dugelay JL (2011). Bag of soft biometrics for person identification. Multimedia Tools and Applications, Springer, 51(2):739-777.
  * (10) Reid DA, Samangooei S, Chen C, Nixon MS, Ross A (2013). Soft biometrics for surveillance: an overview. Handbook of statistics, Elsevier, 31:327-352.
  * (11) Nixon MS, Correia PL, Nasrollahi K, Moeslund TB, Hadid A, Tistarelli M (2015). On soft biometrics. Pattern Recognition Letters, Elsevier, 68:218-230.
  * (12) Dantcheva A, Elia P, Ross A (2016). What else does your biometric data reveal? A survey on soft biometrics. IEEE Transactions on Information Forensics and Security, 11(3):441-467.
  * (13) Raval MS (2016). Digital Video Forensics: Description Based Person Identification. CSI Communications, 39(12):9-11.
  * (14) https://www.bbc.com/news/magazine-22191033. Accessed 28 April 2020.
  * (15) Jain, AK, Ross A, Prabhakar S (2004). An introduction to biometric recognition. IEEE Transactions on circuits and systems for video technology, 14(1):4-20.
  * (16) Jain AK, Flynn P, Ross A (2007). Handbook of biometrics. Springer Science & Business Media, USA.
  * (17) Tome P, Fierrez J, Vera-Rodriguez R, Nixon MS (2014). Soft biometrics and their application in person recognition at a distance. IEEE Transactions on information forensics and security. 9(3):464-75.
  * (18) Golomb BA, Lawrence DT, Sejnowski TJ (1990). Sexnet: a neural network identifies sex from human faces. In Proceedings of 3rd International Conference on Neural Information Processing Systems (NIPS), pp. 572-577.
  * (19) Gutta S, Wechsler H, Phillips PJ (1998). Gender and ethnic classification of face images. In Proceedings of 3rd IEEE International Conference on Automatic Face and Gesture Recognition, pp. 194-199.
  * (20) Sun Z, Bebis G, Yuan X, Louis SJ (2002). Genetic feature subset selection for gender classification: A comparison study. In Proceedings of 6th IEEE Workshop on Applications of Computer Vision (WACV), pp. 165-170.
  * (21) Moghaddam B, Yang MH (2002). Learning gender with support faces. IEEE Transactions on Pattern Analysis and Machine Intelligence. 24(5):707-11.
  * (22) Jain A, Huang J (2004). Integrating independent components and linear discriminant analysis for gender classification. In Proceedings of 6th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 159-163, 2004 May 19.
  * (23) Sun N, Zheng W, Sun C, Zou C, Zhao L (2006). Gender classification based on boosting local binary pattern. In International Symposium on Neural Networks, pp. 194-201, 2006 May 28, Springer, Berlin, Heidelberg.
  * (24) Kim HC, Kim D, Ghahramani Z, Bang SY (2006). Appearance-based gender classification with Gaussian processes. Pattern Recognition Letters, Elsevier, 27(6):618-626.
  * (25) Bekios-Calfa J, Buenaposada JM, Baumela L (2010). Revisiting linear discriminant techniques in gender recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(4):858-864.
  * (26) Jia S, Cristianini N (2015). Learning to classify gender from four million images. Pattern recognition letters, Elsevier, 58:35-41.
  * (27) Omidiora EO, Ojo O, Yekini NA, Tubi TA (2012). Analysis, design and implementation of human fingerprint patterns system. Towards age & gender determination, ridge thickness to valley thickness ratio (rtvtr) & ridge count on gender detection. International Journal of Advanced Research in Artificial Intelligence, 1(2):57–63.
  * (28) Gupta S, Rao AP (2014). Fingerprint based gender classification using discrete wavelet transform & artificial neural network. International Journal of Computer Science and mobile computing, 3(4):1289-1296.
  * (29) Badawi AM, Mahfouz M, Tadross R, Jantz R (2006). Fingerprint-based gender classification. In Proceedings of International Conference on Image Processing, Computer Vision and Pattern Recognition, pp. 41–46.
  * (30) Tom RJ, Arulkumaran T, Scholar ME (2013). Fingerprint based gender classification using 2D discrete wavelet transforms and principal component analysis. International Journal of Engineering Trends and Technology, 4(2):199-203.
  * (31) Marasco E, Lugini L, Cukic B (2014). Exploiting quality and texture features to estimate age and gender from fingerprints. In Biometric and Surveillance Technology for Human and Activity Identification XI, (Vol. 9075, p. 90750F). International Society for Optics and Photonics.
  * (32) Rattani A, Chen C, Ross A (2014). Evaluation of texture descriptors for automated gender estimation from fingerprints. In European Conference on Computer Vision,pp. 764-777, Springer, Cham.
  * (33) Thomas V, Chawla NV, Bowyer KW, Flynn PJ (2007). Learning to predict gender from iris images. In 1st IEEE International Conference on Biometrics: Theory, Applications, and Systems, pp. 1-5.
  * (34) Lagree S, Bowyer KW (2011). Predicting ethnicity and gender from iris texture. In IEEE International Conference on Technologies for Homeland Security (HST), pp. 440-445.
  * (35) Cao L, Dikmen M, Fu Y, Huang TS (2008). Gender recognition from body. In Proceedings of the 16th ACM international conference on Multimedia, pp. 725-728.
  * (36) Chen L, Wang Y, Wang Y (2009). Gender classification based on fusion of weighted multi-view gait component distance. In IEEE Chinese Conference on Pattern Recognition, pp. 1-5.
  * (37) Li X, Maybank SJ, Yan S, Tao D, Xu D (2008). Gait components and their application to gender recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):145-155.
  * (38) Yu S, Tan T, Huang K, Jia K, Wu X (2009). A study on gait-based gender classification. IEEE Transactions on image processing, 18(8):1905-1910.
  * (39) Yoo JH, Hwang D, Nixon MS (2005). Gender classification in human gait using support vector machine. In International Conference on Advanced Concepts for Intelligent Vision Systems, pp. 138-145). Springer, Berlin, Heidelberg.
  * (40) Shan C, Gong S, McOwan PW (2008). Fusing gait and face cues for human gender recognition. Neurocomputing, 71(10-12):1931-1938.
  * (41) Kanchan T, Krishan K (2011). Anthropometry of hand in sex determination of dismembered remains-A review of literature. Journal of Forensic and Legal Medicine., 18(1):14-17.
  * (42) Scheuer JL, Elkington NM (1993). Sex determination from metacarpals and the first proximal phalanx. Journal of Forensic Science, 38(4):769-778.
  * (43) Falsetti AB (1995). Sex assessment from metacarpals of the human hand. Journal of Forensic Science, 40(5):774-776.
  * (44) Lazenby RA (1994). Identification of sex from metacarpals: effect of side asymmetry. Journal of Forensic Science, 39(5):1188-1194.
  * (45) Krishan K, Kanchan T, Sharma A (2011). Sex determination from hand and foot dimensions in a North Indian population. Journal of forensic sciences, 56(2):453-459.
  * (46) Amayeh G, Bebis G, Nicolescu M (2008). Gender classification from hand shape. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1-7.
  * (47) Wu K, Childers DG (1991). Gender recognition from speech. Part I: Coarse analysis. The journal of the Acoustical society of America, 90(4):1828-1840.
  * (48) Pronobis M, Magimai-Doss M (2009). Analysis of F0 and cepstral features for robust automatic gender recognition. IDIAP Technical Report.
  * (49) Sorokin VN, Makarov IS (2008). Gender recognition from vocal source. Acoustical Physics, 54(4):571-578.
  * (50) Childers DG, Wu K (1991). Gender recognition from speech. Part II: Fine analysis. The Journal of the Acoustical society of America, 90(4):1841-1856.
  * (51) Walawalkar L, Yeasin M, Narasimhamurthy AM, Sharma R (2002). Support vector learning for gender classification using audio and visual cues: A comparison. In International Workshop on Support Vector Machines, pp. 144-159, Springer, Berlin, Heidelberg.
  * (52) Ding L, Martinez AM (2008). Precise detailed detection of faces and facial features. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-7.
  * (53) Nixon M (1985). Eye spacing measurement for facial recognition. In Applications of Digital Image Processing VIII, 575:279-285, International Society for Optics and Photonics.
  * (54) Ramanathan V, Wechsler H (2010). Robust human authentication using appearance and holistic anthropometric features. Pattern Recognition Letters, 31(15):2425–2435.
  * (55) Ghalleb AE, Sghaier S, Amara NE (2013). Face recognition improvement using soft biometrics. In 10th International Multi-Conferences on Systems, Signals & Devices 2013 (SSD13), pp. 1-6.
  * (56) BenAbdelkader C, Davis L (2006). Estimation of anthropomeasures from a single calibrated camera. In 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pp. 499-504).
  * (57) Madden CS, Piccardi M (2005). Height measurement as a session-based biometric for people matching across disjoint camera views. In Image and Vision Computing Conference, Wickliffe Ltd.
  * (58) Bobick AF, Johnson AY (2001). Gait recognition using static, activity-specific parameters. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 1:I-I).
  * (59) BenAbdelkader C, Cutler R, Davis L (2002). View-invariant estimation of height and stride for gait recognition. In International Workshop on Biometric Authentication, pp. 155-167, Springer, Berlin, Heidelberg.
  * (60) Li SZ, Schouten B, Tistarelli M (2009). Handbook of Remote Biometrics for Surveillance and Security, pp. 3–21, Springer-Verlag, New York, USA.
  * (61) Reid DA, Nixon MS, Stevenage SV (2013). Soft biometrics; human identification using comparative descriptions. IEEE Transactions on pattern analysis and machine intelligence,36(6):1216-1228.
  * (62) Tome P, Fierrez J, Vera-Rodriguez R, Nixon MS (2014). Soft biometrics and their application in person recognition at a distance. IEEE Transactions on information forensics and security, 9(3):464-475.
  * (63) Jain AK, Nandakumar K, Lu X, Park U (2004). Integrating faces, fingerprints, and soft biometric traits for user recognition. In International Workshop on Biometric Authentication, pp. 259-269, Springer, Berlin, Heidelberg.
  * (64) Jain AK, Dass SC, Nandakumar K. Can soft biometric traits assist user recognition? (2004). In Biometric technology for human identification, 5404:561-572. International Society for Optics and Photonics.
  * (65) Park U, Jain AK (2010). Face matching and retrieval using soft biometrics. IEEE Transactions on Information Forensics and Security, 5(3):406-415.
  * (66) Jain AK, Park U (2009). Facial marks: Soft biometric for face recognition. In 16th IEEE International Conference on Image Processing (ICIP), pp. 37-40.
  * (67) Lee JE, Jain AK, Jin R. Scars, marks and tattoos (SMT): Soft biometric for suspect and victim identification (2008). In IEEE Biometrics symposium pp. 1-8.
  * (68) Galiyawala HJ, Raval MS, Laddha A (2020). Person retrieval in surveillance videos using deep soft biometrics. In Deep Biometrics, pp. 191-214, Springer, Cham.
  * (69) Galiyawala H, Shah K, Gajjar V, Raval MS (2018). Person retrieval in surveillance video using height, color and gender. In 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1-6.
  * (70) Shah P, Raval MS, Pandya S, Chaudhary S, Laddha A, Galiyawala H (2017). Description based person identification: use of clothes color and type. In National Conference on Computer Vision, Pattern Recognition, Image Processing, and Graphics (NCVPRIPG) pp. 457-469, Springer, Singapore.
  * (71) Vaquero DA, Feris RS, Tran D, Brown L, Hampapur A, Turk M (2009). Attribute-based people search in surveillance environments. In IEEE workshop on applications of computer vision (WACV), pp. 1-8.
  * (72) Samangooei S, Guo B, Nixon MS (2008). The use of semantic human description as a soft biometric. In 2nd IEEE International Conference on Biometrics: Theory, Applications and Systems, pp. 1-7.
  * (73) Denman S, Fookes C, Bialkowski A, Sridharan S (2009). Soft-biometrics: unconstrained authentication in a surveillance environment. In IEEE Digital Image Computing: Techniques and Applications, pp. 196-203.
  * (74) Denman S, Halstead M, Bialkowski A, Fookes C, Sridharan S (2012). Can you describe him for me? a technique for semantic person search in video. In IEEE International Conference on Digital Image Computing Techniques and Applications (DICTA), pp. 1-8.
  * (75) Halstead M, Denman S, Sridharan S, Fookes C (2014). Locating people in video from semantic descriptions: A new database and approach. In 22nd IEEE International Conference on Pattern Recognition, pp. 4501-4506.
  * (76) Denman S, Halstead M, Fookes C, Sridharan S (2015). Searching for people using semantic soft biometric descriptions. Pattern Recognition Letters, Elsevier, 68:306-15.
  * (77) Doretto G, Sebastian T, Tu P, Rittscher J (2011). Appearance-based person reidentification in camera networks: problem overview and current approaches. Journal of Ambient Intelligence and Humanized Computing, 2(2):127-151.
  * (78) Galiyawala H, Raval MS, Dave S (2019). Visual appearance based person retrieval in unconstrained environment videos. Image and Vision Computing, 92:103816.
  * (79) Yaguchi T, Nixon MS (2018). Transfer learning based approach for semantic person retrieval. In 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1-6.
  * (80) Schumann A, Specker A, Beyerer J (2018), Attribute-based person retrieval and search in video sequences. In 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1-6.
  * (81) Sun Y, Zheng L, Yang Y, Tian Q, Wang S (2018). Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In Proceedings of the European Conference on Computer Vision (ECCV), pp. 480-496.
  * (82) Chen D, Li H, Liu X, Shen Y, Shao J, Yuan Z, Wang X (2018). Improving deep visual representation for person re-identification by global and local image-language association. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 54-70.
  * (83) Dong Q, Gong S, Zhu X (2019). Person search by text attribute query as zero-shot learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 3652-3661.
  * (84) Li S, Xiao T, Li H, Yang W, Wang X (2017). Identity-aware textual-visual matching with latent co-attention. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1890-1899.
  * (85) Li S, Xiao T, Li H, Zhou B, Yue D, Wang X (2017). Person search with natural language description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1970-1979.
  * (86) Lin Y, Zheng L, Zheng Z, Wu Y, Hu Z, Yan C, Yang Y (2019). Improving person re-identification by attribute and identity learning. Pattern Recognition, 95:151-161.
  * (87) Liu H, Feng J, Jie Z, Jayashree K, Zhao B, Qi M, Jiang J, Yan S (2017). Neural person search machines. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 493-501.
  * (88) Yamaguchi M, Saito K, Ushiku Y, Harada T (2017). Spatio-temporal person retrieval via natural language queries. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1453-1462.
  * (89) Zhou T, Chen M, Yu J, Terzopoulos D (2017). Attention-based natural language person retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR), pp. 27-34.
  * (90) Chang TH, Gong S (2001). Tracking multiple people with a multi-camera system. In Proceedings IEEE Workshop on Multi-Object Tracking, pp. 19-26.
  * (91) Denman SP, Chandran V, Sridharan S (2017). Robust real time multi-layer foreground segmentation. In Proceedings of International Association for Pattern Recognition (IAPR) Conference on Machine Vision Applications, pp. 496-499.
  * (92) Huang CH, Wu YT, Shih MY (2009). Unsupervised pedestrian re-identification for loitering detection. In 3rd Pacific-Rim Symposium on Image and Video Technology, pp. 771-783, Springer, Berlin, Heidelberg.
  * (93) Bak S, Corvee E, Bremond F, Thonnat M (2010). Person re-identification using spatial covariance regions of human body parts. In 7th IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 435-440.
  * (94) Redmon J, Divvala S, Girshick R, Farhadi A (2016). You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 779-788.
  * (95) Girshick R, Donahue J, Darrell T, Malik J (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 580-587.
  * (96) Girshick R (2015). Fast r-cnn. In Proceedings of the IEEE International conference on computer vision (ICCV), pp. 1440-1448.
  * (97) Ren S, He K, Girshick R, Sun J (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems (NIPS), pp. 91-99.
  * (98) K. He, G. Gkioxari, P. Dollar, R. Girshick (2017). Mask R-CNN. In IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988.
  * (99) Gray D, Brennan S, Tao H(2007). Evaluating appearance models for recognition, reacquisition, and tracking. In Proceedings of IEEE International workshop on performance evaluation for tracking and surveillance (PETS), 3(5):41–47, Citeseer.
  * (100) Hirzer M, Beleznai C, Roth PM, Bischof H (2011). Person re-identification by descriptive and discriminative classification. In Scandinavian conference on Image analysis, pp. 91-102, Springer, Berlin, Heidelberg.
  * (101) Loy CC, Xiang T, Gong S (2010). Time-delayed correlation analysis for multi-camera activity understanding. International Journal of Computer Vision, 90(1):106-129.
  * (102) Layne R, Hospedales TM, Gong S (2012). Person re-identification by attributes. In British Machine Vision Conference (BMVC), British Machine Vision Association, 2(3):8.R.
  * (103) Layne R, Hospedales TM, Gong S (2014). Attributes-based re-identification. In Person Re-identification, pp. 93–117, Springer, London.
  * (104) Deng Y, Luo P, Loy CC, Tang X (2014). Pedestrian attribute recognition at far distance. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 789-792.
  * (105) Zheng L, Shen L, Tian L, Wang S, Wang J, Tian Q (2015). Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision (ICCV), pp. 1116-1124.
  * (106) Martinho-Corbishley D, Nixon MS, Carter JN (2016). Soft biometric retrieval to describe and identify surveillance images. In IEEE International Conference on Identity, Security and Behavior Analysis (ISBA), pp. 1-6.
  * (107) Ristani E, Solera F, Zou R, Cucchiara R, Tomasi C (2016). Performance measures and a data set for multi-target, multi-camera tracking. In European Conference on Computer Vision (ECCV), pp. 17-35, Springer, Cham.
  * (108) Li D, Zhang Z, Chen X, Huang K(2018). A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios. IEEE transactions on image processing, 28(4):1575-1590.
  * (109) Li D, Zhang Z, Shan C, Wang L, Tan T (2019). A Comprehensive Study on Large-Scale Person Retrieval in Real Surveillance Scenarios. In 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS),pp. 1-8.
  * (110) Zhu J, Liao S, Lei Z, Yi D, Li S (2013). Pedestrian attribute classification in surveillance: Database and evaluation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 331-338.
  * (111) Baltieri D, Vezzani R, Cucchiara, R (2011). 3dpes: 3d people dataset for surveillance and forensics. In Proceedings of the joint ACM workshop on Human gesture and behavior understanding, pp. 59-64.
  * (112) Cheng DS, Cristani M, Stoppa M, Bazzani L, Murino V (2011). Custom Pictorial Structures for Re-identification. In British Machine Vision Conference (BMVC), 1(2):p. 6.
  * (113) Li W., Zhao R., Xiao T, Wang X (2014). Deepreid: Deep filter pairing neural network for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 152-159.
  * (114) Wang T, Gong S, Zhu X, Wang S (2014). Person re-identification by video ranking. In European conference on computer vision (ECCV), pp. 688-703, Springer, Cham.
  * (115) Baltieri D, Vezzani R, Cucchiara R (2011). Sarc3d: a new 3d body model for people tracking and re-identification. In International conference on image analysis and processing, pp. 197-206, Springer, Berlin, Heidelberg.
  * (116) Benfold B, Reid I (2011). Stable multi-target tracking in real-time surveillance video. In In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 3457-3464.
  * (117) Liu X, Zhao H, Tian M, Sheng L, Shao J, Yi S, Yan J, Wang X (2017). Hydraplus-net: Attentive deep features for pedestrian analysis. In Proceedings of the IEEE international conference on computer vision (ICCV), pp. 350-359.
  * (118) Zheng L, Zhang H, Sun S, Chandraker M, Yang Y, Tian Q (2017). Person re-identification in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1367-1376.
  * (119) Anguelov D, Lee KC, Gokturk SB, Sumengen B (2007). Contextual identity recognition in personal photo albums. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-7).
  * (120) Tsai R (1987). A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE Journal on Robotics and Automation, 3(4):323-344.
  * (121) Thirde D, Li L, Ferryman J (2006). An overview of the pets 2006 dataset. In International Workshop on Performance Evaluation of Tracking and Surveillance, pp. 47-50.
  * (122) Kelly KL, Judd DB (1976). Color: universal language and dictionary of names. US Department of Commerce, National Bureau of Standards.
  * (123) Sharma G, Wu W, Dalal EN (2005). The CIEDE2000 color‐difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application: Endorsed by Inter‐Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur, 30(1):21-30.
  * (124) Martinho-Corbishley D, Nixon MS, Carter JN (2016). Retrieving relative soft biometrics for semantic identification. In 23rd IEEE International Conference on Pattern Recognition (ICPR), pp. 3067-3072).
  * (125) Krizhevsky A, Sutskever I, Hinton GE (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (NIPS), pp. 1097-1105.
  * (126) Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016). SSD: Single shot multibox detector. In European conference on computer vision (ECCV), pp. 21-37, Springer, Cham.
  * (127) He K, Zhang X, Ren S, Sun J (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 770-778.
  * (128) Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
  * (129) Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 4700-4708.
  * (130) Hochreiter S, Schmidhuber J (1997). Long short-term memory. Neural computation, 9(8):1735-1780.
  * (131) Schuster M, Paliwal KK (1997). Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673-2681.
  * (132) Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016). The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223.
  * (133) Simonyan K, Zisserman A (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
  * (134) [134]. Zheng Z, Zheng L, Garrett M, Yang Y, Shen YD (2017). Dual-path convolutional image-text embedding with instance loss. arXiv preprint arXiv:1711.05535.
  * (135) Niu K, Huang Y, Ouyang W, Wang L (2020). Improving description-based person re-identification by multi-granularity image-text alignments. IEEE Transactions on Image Processing, 29:5542-5556.
  * (136) Zhang Y, Lu H (2018). Deep cross-modal projection learning for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 686-701.
  * (137) Wang Y, Bo C, Wang D, Wang S, Qi Y, Lu H (2019). Language Person Search with Mutually Connected Classification Loss. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2057-2061).
  * (138) Sarafianos N, Xu X, Kakadiaris IA (2019). Adversarial Representation Learning for Text-to-Image Matching. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 5814-5824.
  * (139) Aggarwal S, RADHAKRISHNAN VB, Chakraborty A (2020). Text-based Person Search via Attribute-aided Matching. In IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2617-2625.
  * (140) Wang K, Yin Q, Wang W, Wu S, Wang L (2016). A comprehensive survey on cross-modal retrieval. arXiv preprint arXiv:1607.06215.
  * (141) Zhen L, Hu P, Wang X, Peng D (2019). Deep supervised cross-modal retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 10394-10403.
  * (142) Hu P, Peng D, Wang X, Xiang Y (2019). Multimodal adversarial network for cross-modal retrieval. Knowledge-Based Systems, 180:38-50.
  * (143) Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (NIPS) pp. 3111-3119.
  * (144) Mikolov T, Chen K, Corrado G, Dean J (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  * (145) Cho K, van Merriënboer B, Bahdanau D, Bengio Y (2014). On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111.
  * (146) Chung J, Gulcehre C, Cho K, Bengio Y (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning.
  * (147) Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009). Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255.
  * (148) Devlin J, Chang MW, Lee K, Toutanova K (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  * (149) Loper E, Bird S (2002). Nltk: The natural language toolkit. In ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, ETMTNLP ’02. Association for Computational Linguistics.
  * (150) Sun B, Saenko K (2016). Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision (ECCV), pp. 443-450. Springer, Cham.
  * (151) Tan M, Le Q (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In International Conference on Machine Learning (ICML), pp. 6105-6114.
  * (152) Li D, Chen X, Huang K (2015). Multi-attribute learning for pedestrian attribute recognition in surveillance scenarios. In IEEE IAPR Asian Conference on Pattern Recognition (ACPR), pp. 111-115.
  * (153) Sudowe P, Spitzer H, Leibe B (2015). Person attribute recognition with a jointly-trained holistic cnn model. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 87-95.
  * (154) Zhu J, Liao S, Yi D, Lei Z, Li SZ (2015). Multi-label cnn based pedestrian attribute learning for soft biometrics. In IEEE International Conference on Biometrics (ICB), pp. 535-540.
  * (155) Kumar SV, Yaghoubi E, Das A, Harish BS, Proença H (2020). The P-DESTRE: A Fully Annotated Dataset for Pedestrian Detection, Tracking, Re-Identification and Search from Aerial Devices. arXiv preprint arXiv:2004.02782.
  * (156) Yin Z, Zheng WS, Wu A, Yu HX, Wan H, Guo X, Huang F, Lai J (2018). Adversarial attribute-image person re-identification. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 1100-1106.
  * (157) Wu Q, Dai P, Chen P, Huang Y (2019). Deep adversarial data augmentation with attribute guided for person re-identification. Signal, Image and Video Processing, 5:1-8, Springer.
  * (158) Zha ZJ, Liu J, Chen D, Wu F (2020). Adversarial attribute-text embedding for person search with natural language query. IEEE Transactions on Multimedia, 22(7): 1836-1846.
  * (159) Cao YT, Wang J, Tao D (2020). Symbiotic Adversarial Learning for Attribute-based Person Search. arXiv preprint arXiv:2007.09609.
  * (160) Wei L, Zhang S, Gao W, Tian Q (2018). Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 79-88.
  * (161) Khatun A, Denman S, Sridharan S, Fookes C (2020). End-to-End Domain Adaptive Attention Network for Cross-Domain Person Re-Identification. arXiv preprint arXiv:2005.03222.