# SSPNet: Scale and spatial priors guided generalizable and interpretable
pedestrian attribute recognition

Jifeng Shen<EMAIL_ADDRESS>Teng Guo  Xin Zuo  Heng Fan  Wankou Yang
School of Electrical and Information Engineering, Jiangsu University,
Zhenjiang, 212013, China  School of Computer Science and Engineering, Jiangsu
University of Science and Technology, Zhenjiang, 212003, China  Department of
Computer Science and Engineering, University of North Texas, Denton, TX 76207,
USA  School of Automation, Southeast University, Nanjing, 210096, China

###### Abstract

Global feature based Pedestrian Attribute Recognition (PAR) models are often
poorly localized when using Grad-CAM for attribute response analysis, which
has a significant impact on the interpretability, generalizability and
performance. Previous researches have attempted to improve generalization and
interpretation through meticulous model design, yet they often have neglected
or underutilized effective prior information crucial for PAR. To this end, a
novel Scale and Spatial Priors Guided Network (SSPNet) is proposed for PAR,
which is mainly composed of the Adaptive Feature Scale Selection (AFSS) and
Prior Location Extraction (PLE) modules. The AFSS module learns to provide
reasonable scale prior information for different attribute groups, allowing
the model to focus on different levels of feature maps with varying semantic
granularity. The PLE module reveals potential attribute spatial prior
information, which avoids unnecessary attention on irrelevant areas and lowers
the risk of model over-fitting. More specifically, the scale prior in AFSS is
adaptively learned from different layers of feature pyramid with maximum
accuracy, while the spatial priors in PLE can be revealed from part feature
with different granularity (such as image blocks, human pose keypoint and
sparse sampling points). Besides, a novel IoU based attribute localization
metric is proposed for Weakly-supervised Pedestrian Attribute Localization
(WPAL) based on the improved Grad-CAM for attribute response mask. The
experimental results on the intra-dataset and cross-dataset evaluations
demonstrate the effectiveness of our proposed method in terms of mean accuracy
(mA). Furthermore, it also achieves superior performance on the PCS dataset
for attribute localization in terms of IoU. Code will be released at
https://github.com/guotengg/SSPNet.

###### keywords:

pedestrian attribute recognition, attribute localization, prior knowledge,
generalizability, interpretability

††journal: Journal of LaTeX Templates

[table]capposition=top capbtabboxtable[][]

##  1 Introduction

Pedestrian Attribute Recognition (PAR) focuses on identifying the physical
characteristics of pedestrians, such as gender, clothing and hairstyle from
images or videos captured in various environments, which is a challenging
problem in computer vision. The PAR model can be used to serve a wide range of
downstream tasks, such as video surveillance systems, intelligent
transportation system and intelligent robots. Recently, significant progress
has been made in PAR as a result of advancements in deep learning, which can
also contribute to closely related topics such as pedestrian retrieval [1] and
pedestrian re-identification [2].

Figure 1: The visualization of previous PAR model (red border in the first
row) and our proposed model (green border in the second row) for the
corresponding attributes. ((a), (b) and (c) with red border show wrong
response locations. (d), (e), (f) with red border show the coexistence of
correct response position and irrelevant position. The images with the green
border are the effect of our proposed method after solving the corresponding
problem.)

Previous studies typically solve the attribute recognition problem through
multi-task learning [3] and multi-label learning [4]. They focus on
discriminative features whose positions are learned implicitly. However, as
illustrated in Fig. 1, several attributes response heatmap (visualized by
Grad-CAM [5]) learned from these SOTA methods [6, 7, 8] are unreasonable and
perplexing. For example, in Fig. 1(a), Fig. 1(b) and Fig. 1(c) with red
border, the attribute response heatmaps for the “Hat”, “Boots” and “Backpack”
attributes indicate a focus on incorrect positions, which are uninterpretable
and also counter-intuitive for human beings. We argue that one plausible
explanation for this phenomenon is the different learning needs of attributes
with various granularities. The failure to accommodate these diverse learning
needs when using feature maps presents a challenge in identifying
discriminative features on unsuitable maps. For instance, the model struggles
to learn discriminative features from low-resolution feature maps when dealing
with fine-grained attributes, while coarse-grained attributes may suffer from
background noise introduced by high-resolution feature maps. Consequently,
this mismatch in feature maps leads to heatmap responses being in incorrect
positions. In Fig. 1(d), Fig. 1(e) and Fig. 1(f) with red border, the heatmaps
of “Shorts”, “Trousers” and “UpperLogo” give response to both the correct and
unrelated locations. These irrelevant regions contain features that stem from
dataset biases, which introduce interference and hinder the model’s
generalization performance. We argue that the absence of spatial prior
guidance during the model’s learning process is the primary cause of this
phenomenon. For instance, in the absence of spatial priors, the model is
compelled to search for the discriminative features of a hat across the entire
feature map, making it susceptible to the influence of irrelevant areas.
However, with the aid of spatial priors, which provide partial features near
the head, the task of locating discriminative features becomes notably more
straightforward. Therefore, the lack of spatial prior guidance makes model
more susceptible to dataset bias, resulting in poor generalization
performance. To this end, we aim to answer the following two questions in
order to address these issues.

How to deal with different learning needs for attributes with various
granularity ? One solution is to provide multi-scale feature maps for
attributes with varying granularity, facilitating the extraction of
discriminative features.

Previous works [9] introduce feature pyramid structure to obtain a multi-scale
feature map, which contains discriminative features with different
granularity. However, they often employ a fixed feature map scale for all
attributes, which fails to fully exploit the sensitivity of features related
to attribute granularity. In this paper, we introduce an Adaptive Feature
Scale Selection (AFSS) module that dynamically selects an appropriate feature
map based on the mean accuracy (mA) metric during model training. It
effectively addresses the challenge of recognizing attributes with varying
granularities, enhancing model interpretability and achieving higher IoU
values in WPAL. In our experiments, it is interesting to find that the AFSS
module selects high-resolution feature maps for fine-grained attributes (such
as “Hat” and “Boots”), while favoring low-resolution feature maps for coarse-
grained attributes (like “Age” and “Gender”)

How to get more reasonable attribute locations with spatial prior guidance ?
Providing spatial prior knowledge is typically necessary to assist the model
in avoiding poor generalization by neglecting irrelevant regions, as well as
to assist the model in focusing on discriminative features in the correct
attribute regions.

Existing studies on such spatial knowledge have focused on developing
learnable modules [10] for acquiring attribute locations or spatially
transforming regions based on human posture [11] to facilitate pedestrian
attribute recognition. However, most of the human attributes (e.g. “glasses”,
“shirt styles” and “shoes”) have relatively stable locations in human body and
have strong spatial characteristics. This significant spatial characteristic
is known as a spatial prior. It acts as a spatial constraint, guiding the
model to concentrate on the accurate attribute locations. This helps in
mitigating learning biases from other regions, ultimately enhancing the
model’s generalization performance. In this paper, these spatial prior
knowledge can be utilized directly without the help of complicated extra
model. Beside, we use learnable offset points based on such spatial prior to
balance the constraint of prior knowledge and the freedom of model learning.

Based on these ideas, we propose a SSPNet to learn generalizable and
interpretable PAR model, which mainly comprise of AFSS and PLE modules based
on the multi-label classification framework. The AFSS module can select
feature map with proper scales for attributes of various granularity, while
the PLE module aims to incorporate the spatial prior knowledge of attributes
into the model learning process. From another perspective, AFSS identifies the
most suitable scale of feature map, while PLE utilizes spatial priors to find
the appropriate location of attribute in the feature map. Besides, both the
AFSS and PLE modules can be easily applied after different backbone in PAR
model.

To summarize, our main contributions are fourfold:

  * 1.

An AFSS module is proposed to select feature map with proper scales for
attributes of varying granularity, leading to improved performance and better
interpretability.

  * 2.

A PLE module is proposed to assist PAR by guiding the model to learn the
correct attribute location and discriminative features for better
generalization.

  * 3.

A weakly supervised attribute localization with IoU values are achieved based
on the attribute response heatmaps, showing better interpretability.

  * 4.

The proposed method has achieved state-of-the-art results on PA100k dataset,
as well as better generalization performance in cross-dataset tests.

The rest of this paper is organized as follows. Section 2 introduces the
related work published in recent years. Section 3 describes our proposed
method in detail. The experimental results are given in Section 4 and we
conclude the paper in Section 5.

##  2 Related work

###  2.1 Pedestrian attribute recognition

Deep learning-based methods have achieved remarkable progress in PAR.
Convolutional Neural Networks (CNNs) have been widely adopted for feature
extraction and classification due to their excellent ability to automatically
learn discriminative features. Jia et al. [12] adopt a classical approach by
leveraging global features to achieve PAR. In order to make the model focused
on the attributes, Wu et al. [13] introduce attention mechanisms and Yang at
al. [14] propose a method that utilizes part features to eliminate
interference from irrelevant regions. Furthermore, Wang et al. [15] and Tang
et al. [16] employ the internal relationships between attributes as a way to
guide model learning.

Scale knowledge. Pedestrians exhibit attributes at various scales in natural
images. The inclusion of multi-scale feature (eg. feature pyramid network
[17]) aids the model in capturing a wide range of feature information across
different scales. Consequently, methods [4, 8] that leverage multi-scale
feature demonstrate enhanced accuracy and robustness in PAR. Yang et al. [18]
use multi-scale features for cascade and split learning in PAR. Zhong et al.
[9] propose a multi-scale spatial calibration module to gather information and
build long-range dependencies among pyramid feature maps. In this paper,
compared to PAR methods that utilize feature pyramid network, our approach
incorporates scale prior to select appropriate scale of feature map for each
attribute group adaptively.

Spatial prior. In addition to using the attribute annotation information, more
spatial prior information can be adopted from pedestrian images and videos to
assist with PAR. For instance, methods leveraging attribute positions [19] and
pedestrian body posture [20] as learnable guides have demonstrated their
utility in PAR. The PGDM proposed by Li et al. [11] incorporates pedestrian
body structure so that the part-based results are fused with global body-based
results for improved attribute prediction. Lu et al. [21] and Fan et al. [22]
employ the human orientation as an aid in PAR. In this paper, our method
introduces a new feature learning mechanism rooted in spatial prior, which can
strike a balance between the constraints imposed by prior knowledge and the
flexibility of model learning.

###  2.2 Weakly supervised attribute localization

The weakly supervised localization with image-level supervision is an
important problem, which greatly reduces the cost of annotations. In this
area, weakly supervised attribute localization is conceptually similar to
weakly supervised object detection. The former one focuses on fine-grained
objects or abstract concepts, such as parts of a pedestrian (hat or shoes) and
age, whereas the latter one focuses on the whole object, such as a
pedestrian’s location.

Previous studies usually implement weakly supervised localization based on CAM
[23] and Grad-CAM, which gives a good inspiration for attribute localization.
As a result, weakly supervised attribute localization has followed in a
similar way. Wang et al. [24] propose a weakly supervised method for learning
scene parts and attributes from a collection of images associated with
attributes in text. Yu et al. [6] use novelly designed detection layers to
discover mid-level attribute features and the rough shapes of pedestrian
attributes are inferred by performing clustering on a fusion of activation
maps. The ALM [25] learns regional features for each attribute at multiple
levels and adaptively discovers the most discriminative regions for attribute
recognition and localization. Weng et al. [26] employ attention mechanisms to
achieve attribute localization, concurrently aiding in the task of PAR.

In this paper, our method employs improved Grad-CAM (called Grad-CAM-P) method
in weakly supervised attribute localization task to output a set of bounding
boxes. Beside, we propose a localization method based on bounding box and also
provide a quantitative metric (IoU) for weakly supervised pedestrian attribute
localization in the first time.

##  3 Method

###  3.1 Problem definition

PAR is commonly formulated as a multi-label classification problem. Given an
attribute dataset with $N$ pedestrian samples
$D=\left\\{\left(X_{i},Y_{i}\right)\right\\}_{i=1}^{N}$,
$X_{i}\in\mathbb{R}^{W\times H\times 3}$ and $Y_{i}\in\\{0,1\\}^{M}$ denote
the $i^{th}$ pedestrian image and its attribute label respectively. $M$
represents the number of attributes, $W$ and $H$ are the width and height of
image. In PAR task, given an input $X_{i}$, and the PAR model $f_{{A}}$
outputs a prediction vector $\bm{a}$, which is formulated in Eq. 1.

$\displaystyle\bm{a}=f_{{A}}(\mathcal{F}(\Psi_{b}({X};\bm{\theta}_{{A}});\bm{\theta}_{{S}});\bm{\theta}_{{P}})$
(1)

where $\Psi_{b}(\cdot)$ is a feature extraction function (eg. ResNet50 [27],
Swin Transformer [28], etc.) with parameter $\bm{\theta}_{{A}}$. The
$\mathcal{F}$ is the AFSS module with scale prior $\bm{\theta}_{{S}}$, while
the $\bm{\theta}_{{P}}$ is the spatial prior used in PLE module.
$\bm{a}\in\mathbb{R}_{[0,1]}^{M}$ is the attribute prediction.

Weakly-supervised Pedestrian Attribute Localization (WPAL) aims at locating of
object parts or abstract concepts based on image-level attribute label,
without bounding box annotations. Given an image $X$ and attribute type $c$,
we aim to obtain a set of bounding boxes $\bm{b}$ using function
$f_{orm}(\cdot)$ as shown in Eq. 3.

$\displaystyle A_{c}(X)=f_{cam}(\mathcal{F}(\Psi_{b}(X)),c)$ (2)
$\displaystyle\bm{b}=f_{orm}(A_{c}(X))$ (3)

where $A_{c}(.)$111Here, we omit the model parameters $\bm{\theta}_{{A}}$ and
$\bm{\theta}_{{S}}$ in Eq. 2 for clarity. is the response heatmap of Grad-
CAM-P function $f_{cam}(.)$ for the $c^{th}$ attribute, and $f_{orm}(.)$ is a
outer rectangle function to enclose binary masks generated from heatmap.

Figure 2: Overview of the proposed pipeline. (The workflow of the proposed PAR
method is shown in the upper part, while the proposed weakly supervised
attribute localization method is shown in the bottom part.)

###  3.2 Proposed method

####  3.2.1 Overview

Our proposed PAR model is based on the multi-label classification framework,
which mainly comprises of Adaptive Feature Scale Selection (AFSS) module and
Prior Location Extraction (PLE) module. As shown in column “Groups” and
“Attributes” of Tab. 11, all pedestrian attributes have been divided into four
groups based on spatial prior knowledge. An overview of the proposed pipeline
is shown in Fig. 2, where a feature pyramid is firstly obtained by extracting
the multi-scale features from the input image through the backbone. Following
that, the feature maps selected by the AFSS module are fed into the PLE module
where we investigate three different strategies for leveraging the spatial
priors and feature maps. Subsequently, the feature vectors obtained from the
PLE module are aggregated and used as input for the hierarchical
representation module, which produces the attribute recognition results.
Furthermore, after obtaining the PAR model, we employ Grad-CAM-P to generate
attribute heatmaps, which can be utilized to output bounding box for the each
attribute.

Figure 3: Schematic workflow of the AFSS module. (This module selects the
feature map that maximizes the mA during the model training.)

####  3.2.2 Adaptive feature scale selection (AFSS)

The AFSS module adaptively selects the appropriate scale of feature map in the
feature pyramid for different attribute groups, based on the best
classification result in terms of mean accuracy (mA). The workflow of the AFSS
module is illustrated in Fig. 3. Given an input of pedestrian image, we obtain
three feature maps from the classical feature pyramid network, which are
denoted as $P_{1}$, $P_{2}$, and $P_{3}$ with down-sampling ratio of $1/4$,
$1/8$ and $1/16$ respectively. Then, the AFSS module evaluates the attribute
recognition performance achieved from feature maps P1, P2, and P3 using the mA
scores. In the model training phase, the AFSS module iteratively updates the
choices of optimal scale based on the maximum mA value for each attribute
group. Subsequently, in the model testing phase, the AFSS module uses the
learned optimal feature map for each attribute group. The primary advantage of
the AFSS module is its dynamic selection of the optimal feature map based on
the model’s performance feedback, seamlessly integrated into the training
process. Eq. 4 shows the selection of optimal feature map $x^{*}$ in the
training phase:

$x^{*}=\arg\max_{x}(mA(f_{A}(x),Y)),{x}\in\\{P_{1},P_{2},P_{3}\\}$ (4)

where $mA(\cdot)$ is the mean accuracy function, $f_{A}(x)$ represents the
attributes prediction result with feature map $x$.

We find that attributes in the same group are closely related with similar
size and location. For example, the attributes listed in the “Head” group are
related to the parts above the shoulders of a person. Since the human head
takes up a smaller proportion of the human body, the head-related attributes
also give response in a smaller area size. Besides, attributes that may
related to the whole human body are divided into the “All” group. These
attributes are commonly more abstract concepts, which can be determined by
observing the full human body. Intuitively, the same group attributes can
share the same scale of feature map, and this idea is supported by Tab. 1 in
section. 4.2.

(a) PLE-R

(b) PLE-K

(c) PLE-S

Figure 4: PLE modules with three different spatial priors. (a) PLE-R (The
rectangles with colors are prior regions designed on the image height), (b)
PLE-K (The circles with colors are the human keypoints), and (c) PLE-S (The
circles are the sparse sampling points for prior regions). Small white squares
indicate learnable offset points. Cubes of different colors represent feature
vectors for different attribute groups.

####  3.2.3 Prior location extraction (PLE)

The PLE module aims to incorporate the spatial priors of attributes into the
model learning process, which avoids unnecessary attention on irrelevant areas
and also lowers the risk of model over-fitting. In this section, three
different spatial priors are investigated which will be detailed in the
following.

Region-based PLE (PLE-R). In this method, the feature maps are divided into
several sub-regions along the vertical direction to take advantage of existing
knowledge of the regions where attributes may exist. We perform an overlay and
averaging operation on the pedestrian images in the dataset, resulting in a
prominent outline of the human body. Motivated by this, the relatively fixed
position of human parts can be used as an important hint during the attribute
feature learning process. More specifically, we can crop image to get prior
regions for pedestrian attributes. In our implementation, the unrelated
feature areas are cropped out, and only the features in the prior regions are
kept as input features to the later stage for attribute recognition. The
overlaid and averaged pedestrian images are showcased in Fig. 9a.

We crop feature map into blocks based on height designed for the four
attribute groups and finally get four rectangular areas with colors as shown
in Fig. 4a. The feature vector of each attribute $F_{g}$ in each group is
formulated in Eq. 5.

$\displaystyle
F_{g}=\Psi_{agr}(\Phi_{reg}(x_{g}^{*},R_{g})),g\in\\{All,Head,Torso,Bottom\\}$
(5)

where $\Phi_{reg}(.)$ is a feature extraction function which outputs feature
from prior region $R_{g}$. $x^{*}_{g}$ is the optimal feature of attribute
group $g$. $\Psi_{agr}(.)$ is the feature aggregation function which outputs
the feature vector $F_{g}$ for attribute group $g$. In this paper, feature
pyramid pooling method can be utilized as $\Psi_{agr}(.)$ to obtain
discriminative features for each attribute group. The specific height data of
the feature maps cropped for each attribute group is displayed in Tab. 11.

Human Keypoints-based PLE (PLE-K). Region-based PLE can be considered as a set
of densely sampled points in a region, which usually includes background
information irrelevant to the attributes. To leverage finer spatial priors,
human keypoints are an excellent choice as they are closely associated with
the regions relevant to attributes. For example, keypoints at the head are
closely related to region associated with attributes “Hat” and “Glasses”.

We begin by inputting the original images into the pre-trained SimCC [29]
model to obtain 16 human keypoints for each pedestrian. As shown in different
coloured points on the body of Fig. 4b, these human keypoints are simply
mapped to the different feature maps determined by the AFSS module. Next, each
keypoint is utilized as a reference point, and keypoints with the same colour
represent in the same attribute group. Besides, each keypoint is associated
with $M$ learnable offset points, as formulated in Eq. 6. These offset points
are shown as small white squares in Fig. 4b.

$\displaystyle\begin{aligned}
\Phi_{kpt}(x_{g}^{*},P_{n}^{K})=\sum_{m=1}^{M}A_{m}\cdot W_{m}\cdot
x_{g}^{*}\left(P_{n}^{K}+\Delta
P_{n}^{K}\right),n\in\\{1,2,...,16\\}\end{aligned}$ (6)

where $\Phi_{kpt}$ is a keypoint based feature extraction function. $A_{m}$ is
the learnable weight parameter and $W_{m}$ is the $m^{th}$ offset weight.
$P_{n}^{K}$ denotes the $n^{th}$ human keypoint, while $\Delta P_{n}^{K}$ is
the corresponding 2D offset vector for the $P_{n}^{K}$. Different from the
deformable convolution [30], Eq. 6 mainly focuses on each individual human
keypoint. The specific keypoint indexes selected for attribute groups are
provided in Fig. 9b.

The offset vectors are responsible for collecting discriminative features that
are relevant to the corresponding attribute group. These features are then
aggregated using the function $\Psi_{agr}(.)$ to generate the attribute group-
specific feature vector $F_{g}$, denoted by the four colorful cubes in Fig.
4b, and is formulated in Eq. 7.

$\displaystyle\begin{aligned}
F_{g}=\Psi_{agr}(\Phi_{kpt}(x_{g}^{*},P_{n}^{K}))\end{aligned}$ (7)

Sparse sampling points-based PLE (PLE-S). Although keypoints based feature
extraction function can obtain attribute-related features, it relies on
additional pose estimation modules which requires additional computation and
also brings unavoidable keypoints location errors. To remedy this, a natural
solution is sparse sampling on the attributes related regions, which is also
simple and easy to implement.

As shown in Fig. 4c, our proposed approach samples points from each prior
region uniformly. The sparse sampling positions are fixed with a sampling
ratio (discussed in Section. 4.2). Then, similar to PLE-K, we also learn $M$
offset points based on each sparse point through Eq. 8 and aggregate features
from sparse sampling points by Eq. 9.

$\Phi_{sps}(x_{g}^{*},P_{n}^{S})=\sum_{m=1}^{M}A_{m}\cdot
W_{m}\cdot{x_{g}^{*}}\left(P_{n}^{S}+\Delta
P_{n}^{S}\right),n\in\\{1,2,...,N\\}$ (8)

$F_{j}=\Psi_{agr}(\Phi_{sps}(x_{g}^{*},P_{n}^{S}))$ (9)

where $\Phi_{sps}(.)$ denotes sparse sampling based feature extraction
function, and $P_{n}^{S}$ is the $n^{th}$ sparse sampling point. $\Delta
P_{n}^{S}$ means a 2D offset vector for $P_{n}^{S}$.

PLE-R does not learn additional information from spatial priors, such as
learnable offset points, which can be considered as a baseline method.
Additionally, PLE-R employs the coarsest spatial prior, which involves image
blocks. PLE-K and PLE-S both require the use of learnable offset points. PLE-K
utilizes the finest spatial prior, but requires additional computations to
obtain each pedestrian’s keypoints. PLE-S adopts a balanced approach, which
sparsely samples the image and leverages learnable offset points to capture
more useful features.

####  3.2.4 Hierarchical representation

In this section, we introduce the hierarchical attribute recognition and
hierarchical loss function that works in conjunction with the AFSS and PLE
modules. Initially, the AFSS selects suitable feature map for the PLE
adaptively which outputs feature vectors for each group of attributes. Then,
feature vectors are fed into corresponding fully connected layers designed for
each group of attributes. As shown in Fig. 2, the four FC layers are
independent with each other for the loss computation during the training. We
refer to these fully connected layers and losses of each group as the
hierarchical attribute recognition and hierarchical loss function
respectively.

Hierarchical attribute recognition. In the previous section, we discussed the
division of attributes into four groups, namely “All,” “Head,” “Torso,” and
“Bottom.” As illustrated in Fig. 2, for attribute recognition within each
group, we utilize feature vectors extracted from the corresponding prior
region or points as input to a dedicated FC layer. This approach ensures that
only relevant feature vectors from the corresponding prior regions are
considered, while excluding irrelevant feature vectors from other regions. The
hierarchical attribute recognition process involves a total of four FC layers,
with each FC layer corresponding to an attribute group.

Hierarchical loss. The PAR task can be considered as multiple binary
classification tasks in this paper. The output of the final layer is processed
by a sigmoid function to obtain the prediction of each attribute. Therefore,
the loss function of our method can be defined as a weighted binary cross-
entropy loss [12] (Eq. 10):

$Loss^{bce}=\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{M}\omega_{j}\left(Y_{ij}\log\left(\sigma\left(\bm{a}_{ij}\right)\right)+\left(1-Y_{ij}\right)\log\left(1-\sigma\left(\bm{a}_{ij}\right)\right)\right)$
(10)

where $\bm{a}_{ij}$ is the output of the model for the $j^{th}$ attribute of
the $i^{th}$ pedestrian picture, $\sigma(\cdot)$ is a sigmoid function and
$\omega_{j}$ is an imbalance weight (Eq. 11), which is calculated based on the
number of positive and negative samples of the $j^{th}$ attribute to alleviate
the imbalance of positive and negative sample distribution.

$\begin{array}[]{c}w_{j}=\left\\{\begin{array}[]{ll}e^{1-r_{j}},&y_{ij}<0\\\
e^{r_{j}},&y_{ij}\geq 0\end{array}\right.\end{array}$ (11)

where $r_{j}$ is the proportion of positive samples of the $j^{th}$ attribute
in the training set. Finally, hierarchical Loss can be defined in following
Eq. 12:

$\displaystyle Loss_{total}=\sum_{g}Loss^{bce}_{g}$ (12)

where $Loss^{bce}_{g}$ and $Loss_{total}$ represent the loss in the attribute
group $g$ and the total loss for all groups respectively.

####  3.2.5 Weakly supervised attribute localization

A improved version of Grad-CAM is employed for weakly supervised attribute
localization, dubbed as Grad-CAM-P (Grad-CAM for pixel). It provides pixel-
level feature gradient responses instead of the channel-level feature gradient
responses obtained by the original Grad-CAM. The computation of Grad-CAM-P is
shown as Eq. 13:

$\displaystyle L_{{Grad-CAM-P}}$
$\displaystyle=\operatorname{ReLU}\left(\sum_{k}{\alpha_{k}^{j}}A^{k}\right)$
(13) $\displaystyle{\alpha_{k}^{j}}$
$\displaystyle=\frac{\partial{\bm{a}}_{j}}{\partial A_{x,y}^{k}}$

where $A^{k}$ represents $k^{th}$ channel of feature map $A$. $\bm{a}_{j}$
means model prediction result for $j^{th}$ attribute. ${\alpha_{k}^{j}}$
represents the response gradient of $A^{k}$ for the $j^{th}$ attribute.
$A_{x,y}^{k}$ means feature value at position coordinates $(x,y)$ of $A^{k}$.

Unlike multi-class classification, the feature maps obtained from the feature
extraction layer contain feature information for multiple categories to ensure
the recognition of all existing labels in multi-label classification tasks.
Then, the non-zero channel-level gradient response will weightedly sum the
features of other category in the feature map, whose pixel-level gradient
response should be smaller or even zero, which results in the final computed
heatmap containing responses of other category. To address this issue, we use
pixel-level response values to weight the feature maps and generate response
heatmaps, which can suppress the appearance of irrelevant regions.

Finally, we use feature map selected by AFSS and Grad-CAM-P to generate
attribute response heatmap. After binarizing the heatmap, we convert the
foreground regions into one bounding box for attribute localization with
simple external rectangle operation.

##  4 Experiments

Our model is implemented with PyTorch 1.9.1 and all the experiments are
conducted on Ubuntu 20.04 with NVIDIA GeForce RTX 3080Ti GPU, 32G Memory and
3.70 GHz Intel Core i9-10900K CPU. Pedestrian images are resized to
256$\times$192 as inputs. The training takes 30 epochs with a batch size of
32. The model parameters are updated using the Adam optimizer with initial
learning rate of 0.0001. The Strong Baseline [12] is used as our baseline
method for the ablation experiments.

###  4.1 Dataset and metrics

The PA100K, RAP v1 and UPAR datasets are used for the experiments in this
paper which are all public available.

PA100k dataset [7]. The PA100k dataset is constructed by images captured from
598 real outdoor surveillance cameras, containing a total of 100,000
pedestrian images. Each image is annotated with 26 attributes [31]. Following
[7, 32], the dataset is randomly divided into 80,000, 10,000 and 10,000 images
for training, validation and testing respectively.

RAP v1 dataset [1]. It is a large-scale person re-identification dataset that
focuses on predicting various attributes such as gender, age and clothing etc.
This dataset comprises 41,585 pedestrian images gathered from 26 surveillance
cameras, encompassing 69 binary attributes. The training set of this dataset
comprises 33,268 images, with the remaining images is used for testing.

PETA dataset. [33] It serves the purpose of recognizing pedestrian attributes
and holds significance in video surveillance applications, especially when
close-up shots of faces and bodies are rarely accessible. It comprises a
collection of 19,000 pedestrian images, encompassing 65 attributes, which
consist of 61 binary attributes and 4 multi-class attributes. These images
collectively depict 8,705 individuals.

UPAR dataset [34]. UPAR is designed for pedestrian attribute recognition,
person retrieval tasks in video surveillance and fashion retrieval. It is
based on four well-known person attribute recognition datasets: PA100K, PETA,
RAP v2 [35], and Market1501 [36]. UPAR unifies these datasets by providing 3.3
million additional annotations to harmonize 40 important binary attributes
over 12 attribute categories across the datasets. Notably, we only use UPAR
dataset to test generalization ability of model instead of using it to train.

People clothing segmentation (PCS) dataset [37, 38]. The dataset comprises of
1000 images with $825\times 550$ pixels and corresponding semantic
segmentation masks. There are 59 semantic label classes, comprising of both
the background and clothing types.

Metrics. Mean accuracy (mA) is employed as an evaluation metric for attribute
recognition algorithms, which is formulated in Eq. 14:

$mA=\frac{1}{2N}\sum_{i=1}^{L}\left(\frac{TP_{i}}{P_{i}}+\frac{TN_{i}}{N_{i}}\right)$
(14)

where $L$ is the number of attributes. $TP_{i}$ and $TN_{i}$ are the number of
true positives and true negatives respectively, $P_{i}$ and $N_{i}$ are the
number of all positive and negative examples respectively.

According to the existing methods, we utilize five metrics to evaluate model:
mean accuracy (mA), accuracy (Accu), precision (Prec), recall (Rec), and F1
score (F1).

###  4.2 Ablation study

Table 1: The effect of AFSS (P1, P2, P3 represent approaches using the three fixed scale feature maps respectively. AFSS∗ and AFSS represent that AFSS select feature map for each attribute and each group of attributes respectively.) Groups | P1 | P2 | P3 | AFSS∗ | AFSS
---|---|---|---|---|---
Head | 80.09 | 79.91 | 79.63 | 80.16 | 80.13
Torso | 87.71 | 88.13 | 87.65 | 88.08 | 88.05
Bottom | 80.11 | 80.42 | 80.07 | 80.38 | 80.40
All | 80.39 | 80.45 | 80.75 | 80.77 | 80.71

Scale selection in AFSS. In order to find the optimal scale of feature map for
each attribute group, we have conducted experiment on SSPNet-R (using PLE-R)
with ResNet50 and mA results are shown in the middle columns of Tab. 1 (P1,
P2, P3). It is clear to find that, the “Head,” “Troso,” “Bottom,” and “All”
group obtains the best mA on feature map of P1, P2, P2, and P3 respectively.
Based on these observations, we think that attributes with small granularity
require high resolution feature map, since it can provide detailed
discriminative information to determine the existence and location of
attribute. On the other hand, attributes with middle or large granularity
prefer feature map P2 and P3 respectively, which is also in common sense for
the design philosophy of feature pyramid network.

The right column of Tab. 1 provides comparison between AFSS and AFSS∗. The
AFSS aims to select feature scale for each group of attributes, while the
AFSS∗ aims to select feature scale for each attribute. It is found that both
of them can achieve similar performance, but AFSS is more generalizable and
friendly in design when facing recognition tasks with a wide range of
attributes. AFSS will keep as the default setting in the following
experiments.

Number of offset points in PLE-K.

Figure 5: Performance with different number of offset points.

Figure 6: Performance with varying sparsity levels of sparse points.

Fig. 6 illustrate the change curve of mA with different number of offset
points on PLE-K with ResNet50. It is clear to find that the peak performance
is reached when offset points number $M=3$, and we keep it as default value in
the following experiments.

Sampling ratios in PLE-S. Fig. 6 provides the comparison of mA with different
sampling ratio on PLE-S. It is found that taking 75% of the feature points can
achieve the best performance, comparing with sampling ratio of 25%, 75% and
100%. Based on the observation, we infer that the sparsity of the sampled
points has an impact on the performance of the model since less uniform
sampling ratio ($\textless 50\%$) can lead to more discriminative information
loss. In the following experiments, we use 75$\%$ as the default sampling
ratio.

Table 2: The role of detail work (Offset: Offset points. Prior: Spatial prior knowledge. The symbols $\checkmark$ indicate that the corresponding component is included.) Number | AFSS | PLE-S | mA
---|---|---|---
Offset | Prior
1 |  |  |  | 81.97
2 |  | ✓ | ✓ | 82.81
3 | ✓ |  |  | 82.26
4 | ✓ | ✓ |  | 82.60
5 | ✓ |  | ✓ | 82.69
6 | ✓ | ✓ | ✓ | 83.15

Discussion on different components. Tab. 2 provides detailed ablation
experiments of AFSS and PLE modules. The first row is the baseline method,
while the last row is our proposed method equipped with both AFSS and PLE
modules. To make a fair and convincing evaluation, we deploy PLE based on
sparse points as spatial prior knowledge in the Tab. 2. By comparing the first
and third row, AFSS can bring 0.29$\%$ improvements. When we use PLE-S, it can
bring 0.89% improvements from the third and sixth row. Similar results can
also be obtained by comparing the first and second rows which shows that PLE-S
can bring 0.74% improvements. Within PLE-S, there are two components: offset
and prior. When using offset alone, the learned offset points can improve
performance by 0.34% from the third and forth rows. When using prior alone,
the spatial prior can provides a significant 0.43% improvement from the third
and fifth rows. Finally, by comparing the first and sixth row, AFSS+PLE-S can
bring 1.18$\%$ additional improvements, which indicate the complementarity
between two modules.

Compared to AFSS, PLE bring more performance improvement which indicates that
feature extraction from precise attribute related locations is more important.
Based on these observations, we can infer that both the scale prior (used in
AFSS) and spatial prior (used in PLE) are effective in improving the
performance of the model, and the model achieves the best performance when the
two modules are used simultaneously.

Table 3: Comparing different backbone and efficiency on PA100k dataset. Method | Backbone | mA | Accu | Pre | Recall | F1 | Params | GFLOPs | FPS
---|---|---|---|---|---|---|---|---|---
Basline | ResNet50 | 81.97 | 80.20 | 88.06 | 88.17 | 88.11 | 23.56M | 2.69 | 24
Basline | Swin-S | 82.19 | 80.35 | 87.85 | 88.51 | 88.18 | 48.86M | 7.74 | 15
SSPNet-R | ResNet50 | 82.49 | 80.52 | 87.77 | 88.84 | 88.30 | 24.73M | 2.92 | 23
SSPNet-K | ResNet50 | 82.76 | 80.21 | 87.37 | 89.10 | 88.23 | 26.27M | 3.31 | 19
SSPNet-S | ResNet50 | 83.15 | 80.61 | 87.86 | 89.01 | 88.43 | 26.30M | 3.32 | 21
SSPNet-R | Swin-S | 82.95 | 80.50 | 87.50 | 89.27 | 88.38 | 49.96M | 8.13 | 15
SSPNet-K | Swin-S | 83.31 | 80.35 | 87.41 | 89.56 | 88.47 | 52.43M | 8.65 | 11
SSPNet-S | Swin-S | 83.58 | 80.63 | 87.79 | 89.32 | 88.55 | 52.44M | 8.65 | 12

Comparing different backbone and efficiency. We have also conducted
experiments by using ResNet50 and Swin Transformer as backbones and efficiency
comparison on the PA100k dataset which are shown in Tab. 3. We find that our
SSPNet-S with Swin Transformer achieves the best performance in mA, Accu,
Recall, and F1. Compared to the baseline method (ResNet50), our SSPNet-S
(ResNet50) improves 1.18%, 0.41%, 0.01%, and 0.25% in mA, Accu, Recall, and
F1, respectively. Compared to the baseline method (Swin Transformer), our
SSPNet-S (Swin Transformer) improves 1.39%, 0.28%, 0.81%, and 0.37% in mA,
Accu, Recall, and F1, respectively. Both SSPNet-R and SSPNet-K, which utilize
ResNet50 and Swin Transformer as backbones, have demonstrated significant
performance improvements compared to the baseline method. When using the same
backbone, SSPNet-S outperforms SSPNet-K, and SSPNet-K outperforms SSPNet-R.
Moreover, when utilizing the same PLE module, SSPNet with Swin-S achieves
superior performance compared to SSPNet with ResNet50.

As shown in Tab. 3, compared to the baseline method, all SSPNet methods have
few more parameters and a slightly decrease in FPS. Within the SSPNet family,
SSPNet-R utilizes fewer parameters and computational resources in contrast to
SSPNet-K and SSPNet-S, since it does not employ learnable offset points.

Table 4: Comparison with state-of-the-art methods on PA100k dataset.

Method | Backbone | PA100k
---|---|---
mA | Accu | Prec | Recall | F1
HP-Net [7] | InceptionNet | 74.21 | 72.19 | 82.97 | 82.09 | 82.53
PGDM [11] | CaffeNet | 74.95 | 73.08 | 84.36 | 82.24 | 83.29
VRKD [20] | ResNet50 | 77.87 | 78.49 | 88.42 | 86.08 | 87.24
VSGR [39] | ResNet50 | 79.52 | 80.58 | 89.40 | 87.15 | 88.26
JLPLS-PAA [40] | SE-BN-InceptionNet | 81.61 | 78.89 | 86.83 | 87.73 | 87.27
SSC [41] | ResNet50 | 81.87 | 78.89 | 85.98 | 89.10 | 86.87
CAS-SAL-FR [18] | ResNet50 | 82.86 | 79.64 | 86.81 | 88.78 | 87.79
EALC [26] | EfficientNet-B4 | 81.45 | 80.27 | 87.32 | 88.98 | 88.14
SSPNet (Ours) | ResNet50 | 83.15 | 80.61 | 87.86 | 89.01 | 88.43
SSPNet (Ours) | Swin-S | 83.58 | 80.63 | 87.79 | 89.32 | 88.55

Table 5: Comparison with state-of-the-art methods on RAP v1 dataset.

Method | Backbone | RAP v1
---|---|---
mA | Accu | Prec | Recall | F1
HP-Net [7] | InceptionNet | 76.12 | 65.39 | 77.33 | 78.79 | 78.05
PGDM [11] | CaffeNet | 74.31 | 64.57 | 78.86 | 75.90 | 77.35
VRKD [20] | ResNet50 | 78.30 | 69.79 | 82.13 | 80.35 | 81.23
VSGR [39] | ResNet50 | 77.91 | 70.04 | 82.05 | 80.64 | 81.34
JLPLS-PAA [40] | SE-BN-InceptionNet | 81.25 | 67.91 | 78.56 | 81.45 | 79.98
SSC [41] | ResNet50 | 82.77 | 68.37 | 75.05 | 87.49 | 80.43
PARFormer-L [22] | Transformer-based | 84.13 | 69.94 | 79.63 | 88.19 | 81.35
EALC [26] | EfficientNet-B4 | 83.26 | 69.65 | 79.82 | 83.61 | 81.67
SSPNet (Ours) | ResNet50 | 82.79 | 69.85 | 79.96 | 82.88 | 81.39
SSPNet (Ours) | Swin-S | 83.24 | 70.21 | 80.14 | 82.90 | 81.50

Table 6: Comparison with state-of-the-art methods on PETA dataset.

Method | Backbone | PETA
---|---|---
mA | Accu | Prec | Recall | F1
HP-Net [7] | InceptionNet | 81.77 | 76.13 | 84.92 | 83.24 | 84.07
PGDM [11] | CaffeNet | 82.97 | 78.08 | 86.86 | 84.68 | 85.76
VRKD [20] | ResNet50 | 84.90 | 80.95 | 88.37 | 87.47 | 87.91
VSGR [39] | ResNet50 | 85.21 | 81.82 | 88.43 | 88.42 | 88.42
JLPLS-PAA [40] | SE-BN-InceptionNet | 84.88 | 79.46 | 87.42 | 86.33 | 86.87
SSC [41] | ResNet50 | 86.52 | 78.95 | 86.02 | 87.12 | 86.99
CAS-SAL-FR [18] | ResNet50 | 86.40 | 79.93 | 87.03 | 87.33 | 87.18
EALC [26] | EfficientNet-B4 | 86.84 | 81.71 | 88.58 | 88.23 | 88.40
DAFL [42] | ResNet50 | 87.07 | - | - | - | 86.40
SSPNet (Ours) | ResNet50 | 88.14 | 81.96 | 88.53 | 88.27 | 88.40
SSPNet (Ours) | Swin-S | 88.73 | 82.80 | 88.48 | 90.55 | 89.50

###  4.3 Comparison with state-of-the-arts

To validate the effectiveness of the proposed method, we use two public
datasets and five evaluation metrics for comparative evaluation with SOTA PAR
methods (HPNet [7], PGDM [11], VRKD [20], VSGR [39], JLPLS-PAA [40], SSC [41],
CAS-SAL-FR [18], PARFormer-L [22], EALC [26], DAFL [42]) on the PA100k, RAP
v1, and PETA datasets. In this section, we utilize the best-performing
SSPNet-S as a representative of SSPNet for comparison.

Results on PA100k. As shown in Tab. 4, our SSPNet with Swin-S ranks the first
in terms of mA, Accu, Recall and F1 score, and our SSPNet with ResNet50 ranks
the second with Prec. Compared to SSC, which establishes a spatial and
semantic consistency framework to leverage inter-image relations and
incorporate human priors, our proposed SSPNet with Swin-S outperforms it with
improvements of 1.71%, 1.74%, 1.81%, 0.22%, and 1.68% across all five metrics.
When compared to CAS-SAL-FR, SSPNet with Swin-S respectively demonstrates
performance enhancements of 0.72%, 0.99%, 0.98%, 0.54%, and 0.76% across the
same five metrics.

Results on RAP v1. As presented in Tab. 5, our SSPNet with Swin-S attains the
highest Accu and F1 scores compared to other state-of-the-art approaches. In
comparison to SSC, our proposed SSPNet with Swin-S exhibits notable
improvements of 0.47%, 1.84%, 5.09%, and 1.07% for mA, Accu, Prec, and F1,
respectively. Furthermore, our model outperforms PARFormer-L and EALC in terms
of Accu, Prec, and F1. Additionally, SSPNet using ResNet50 also attained
competitive results.

Results on PETA: As shown in Tab. 6, SSPNet with both ResNet50 and Swin-S can
achieve the SOTA results. SSPNet with Swin-S obtains the best performance in
terms of mA, Accu, Recall, and F1. Compared to DAFL, it exhibits an
improvement of 1.66% in mA. Additionally, when compared to EALC, SSPNet
achieves an increase of 1.10% in F1. It is clear to observe that our approach
consistly obtains the best result on the PETA dataset. It is worth to mention
that our SSPNet utilizes less information in the feature maps for using prior
knowledge, compared to the baseline method. However, the learned fewer
features preserve useful information because the model can focus more on the
areas closely related to the attributes.

Table 7: Cross-dataset model test results(PA100k $\rightarrow$ RAP v1) Attributes | Baseline | SSPNet-R | SSPNet-K | SSPNet-S
---|---|---|---|---
Boots | 50.12 | 51.63 (+1.51) | 53.23 (+3.11) | 53.61 (+3.49)
Hat | 63.82 | 65.33 (+1.51) | 65.21 (+1.39) | 65.78 (+1.96)
HandBag | 74.68 | 76.56 (+1.88) | 77.21 (+2.53) | 77.83 (+3.15)
Glasses | 78.52 | 79.12 (+0.60) | 79.97 (+1.45) | 80.87 (+2.35)
Backpack | 84.73 | 85.87 (+1.14) | 86.24 (+1.51) | 87.02(+2.29)
ShortSleeve | 91.39 | 91.48 (+0.09) | 91.41 (+0.02) | 91.53 (+0.14)
Female | 90.20 | 90.95 (+0.75) | 91.26 (+1.06) | 91.31 (+1.11)
Average | 76.21 | 77.28 (+1.07) | 77.79 (+1.58) | 78.28 (+2.07)
Table 8: Cross-dataset model test results(RAP v1 $\rightarrow$ PA100k) Attributes | Baseline | SSPNet-R | SSPNet-K | SSPNet-S
---|---|---|---|---
Boots | 80.52 | 82.18 (+1.66) | 83.01 (+2.49) | 83.24 (+2.72)
Hat | 68.54 | 70.18 (+1.64) | 71.26 (+2.72) | 71.31 (+2.77)
HandBag | 72.69 | 73.77 (+1.08) | 74.63 (+1.94) | 75.28 (+2.59)
Glasses | 75.81 | 76.31 (+0.50) | 77.28 (+1.47) | 78.49 (+2.68)
Backpack | 82.78 | 82.96 (+0.18) | 83.84 (+1.06) | 84.13 (+1.35)
ShortSleeve | 76.37 | 77.12 (+0.75) | 77.96 (+1.59) | 78.47 (+2.10)
Female | 89.15 | 89.59 (+0.44) | 89.42 (+0.27) | 89.81 (+0.66)
Average | 77.98 | 78.87 (+0.89) | 79.63 (+1.65) | 80.10 (+2.12)
Table 9: Cross-dataset model test results(RAP v1 $\rightarrow$ UPAR datasets) Attribute | Baseline | SSPNet-R | SSPNet-K | SSPNet-S
---|---|---|---|---
BaldHead | 69.71 | 70.69 (+0.98) | 71.08 (+1.37) | 71.73 (+2.02)
Glasses | 78.65 | 80.21 (+1.56) | 80.58 (+1.93) | 81.37 (+2.72)
Hat | 79.81 | 81.07 (+1.26) | 82.49 (+2.68) | 82.21 (+2.40)
Backpack | 82.64 | 83.59 (+0.95) | 83.60 (+0.96) | 83.82 (+1.18)
LongHair | 86.88 | 88.29 (+1.41) | 88.17 (+1.29) | 89.37 (+2.49)
Female | 95.02 | 95.97 (+0.95) | 96.18 (+1.16) | 97.22 (+2.20)
Average | 82.12 | 83.30 (+1.19) | 83.68 (+1.57) | 84.29 (+2.17)

###  4.4 Generalizability

To evaluate the generalization capability of the proposed method, we have
conducted three cross-dataset experiments. The mA of these shared attributes
on the test datasets are reported separately.

PA100k $\rightarrow$ RAP v1 dataset. As shown in Tab. 7, the PAR results are
tested on RAP v1 using the model trained on PA100k with a total of 7
attributes in common. We observe that the SSPNet-S gets the highest mA on all
shared attributes and also the average mA among all methods. The SSPNet-R and
SSPNet-K also get significant improvements compared to baseline method. The
biggest boost comes when the SSPNet-S model recognizes the “Boots”, which we
believe is a gain from the combination of the prior region and the offset
points.

RAP v1 $\rightarrow$ PA100k dataset. As shown in Tab. 8, we use the PA100k to
test model which is trained on the RAP v1. The SSPNet-S also gets the highest
mA on all shared attributes and the highest average mA among all methods,
which is similar to results of PA100k $\rightarrow$ RAP v1 cross-dataset test.
The SSPNet-R, SSPNet-K and SSPNet-S get 0.89%, 1.65% and 2.12% improvements on
average mA of shared attributes, respectively. The largest boost comes from
“Hat” and the second largest from “Boots,” both of which are fine-grained
attributes and exhibit strong spatial priors. This indicates that our model
excels at fine-grained attributes with strong spatial priors.

RAP v1 $\rightarrow$ UPAR dataset. We have conducted additional cross-dataset
experiments on the UPAR dataset with the model trained on the RAP v1 dataset.
To avoid data leaks, we exclude all the data from RAP dataset in the original
UPAR dataset. As shown in Tab. 9, SSPNet-S shows the best results except for
the “Hat” attribute. Compared to baseline method, average mA of SSPNet-S gets
2.17$\%$ improvements which is consistent with the results of the above. So we
can infer that all SSPNet families exhibit better generalization performance
by using prior knowledge, which demonstrates the potential for real-world
applications.

To sum up, we find that our method favors fine-grained attributes with strong
spatial prior, such as “Boots,” “LongHair,” and “Hat”. While for coarse-
grained attributes, our method can also obtain similar or higher performance
compared to the baseline method on cross-datasets tests.

(a) Hat (b) Boots

(c) Shorts (d) Trousers

Figure 7: Qualitative analysis of weakly supervised attribute
localization(From left to right for each set of figures indicate baseline
method, SSPNet-R, SSPNet-K, and SSPNet-S respectively. The red rectangular
boxes are the attributes bounding boxed obtained from our method)

###  4.5 Weakly supervised attribute localization

Qualitative analysis. Fig. 7 displays the heatmaps and weakly supervised
attribute localization results generated using Grad-CAM-P. It can be seen that
the attribute localization with baseline method is not satisfactory, with a
phenomena of localization to irrelevant regions and insufficient fine
localization. However, our methods that leverage prior knowledge can improve
the localization of pedestrian attributes. After incorporating such prior
knowledge, weakly supervised attribute localization results become more
precise. Particularly, the model encounters a minor occlusion issue while
attempting to localize “Trousers” in Fig. 7d. Compared to the baseline method,
SSPNet is able to locate the attribute more accurately.

The SSPNet-R can remove irrelevant regions because of prior regions, so we can
see that heatmaps and bounding boxes more focus on attribute positions.
SSPNet-K can further narrow the prior region by using more accurate human
keypoints. Besides, heatmaps and bounding boxes generated from SSPNet-S are
also more accuracy due to the sparse sampling points compared to baseline
method.

In summary, all the SSPNet families can produce different results for weakly
supervised attribute localization, which demonstrates that the effectiveness
of using spatial prior knowledge in different ways.

(a) Hat

(b) Boots (c) Skirt

Figure 8: Quantitative analysis of weakly supervised attribute localization.
(From left to right, for each image, it indicates the original segmentation
annotation, baseline method, SSPNet-R, SSPNet-K, and SSPNet-S, respectively.
The red and green rectangular boxes are the attribute bounding boxes obtained
from our method and the ground truth, respectively.)

Quantitative analysis. To further evaluate the performance of our method on
weakly supervised attribute localization quantitatively, we utilize the
attribute segmentation annotation of the PCS dataset to generate accurate
bounding boxes for the attributes. Similar to object detection, these bounding
boxes can be used to compute IoU values to evaluate the performance of weakly
supervised attribute localization.

As shown in Fig. 8, the IoU values are shown at the bottom of each image. For
attribute “Hat” and “Boots,” the bounding boxes from the baseline method cover
a large area of regions due to the use of global features. However, the other
three methods based on the spatial prior knowledge make the model focus on the
correct location and obtain higher IoU values.

Compared to the “Hat” and “Boots,” the “Skirt” takes up a large area of the
image which is less difficult to locate. It seems that the baseline method
localization results are influenced by the background near the legs, leading
to a lower IoU value. However, All of our methods obtain higher IoU, because
the prior region excludes some of the distracting regions in advance. So our
methods based on spatial prior knowledge exhibit superior weakly supervised
attribute localization capabilities. Examples of our failure cases can be
found in the appendix.

Table 10: Quantitative result. (We computed the mean IoU, and the Pearson correlation coefficient (PCCs) about IoU and prediction confidence within the same attribute. B, R, K, and S represent baseline method, SSPNet-R, SSPNet-K, and SSPNet-S, respectively.) Attributes | IoU | PCCs
---|---|---
B | R | K | S | B | R | K | S
Hat | 0.06 | 0.13 | 0.17 | 0.16 | 0.26 | 0.40 | 0.41 | 0.43
Boots | 0.09 | 0.25 | 0.30 | 0.28 | 0.08 | 0.26 | 0.21 | 0.27
Glasses | 0.04 | 0.27 | 0.34 | 0.28 | 0.13 | 0.23 | 0.28 | 0.26
Skirt | 0.26 | 0.34 | 0.47 | 0.47 | 0.25 | 0.37 | 0.42 | 0.39
Jeans | 0.30 | 0.37 | 0.39 | 0.39 | 0.33 | 0.33 | 0.36 | 0.38
Dress | 0.25 | 0.27 | 0.29 | 0.30 | 0.14 | 0.18 | 0.18 | 0.18
Shorts | 0.15 | 0.19 | 0.24 | 0.26 | 0.15 | 0.52 | 0.55 | 0.60
Sunglasses | 0.04 | 0.06 | 0.12 | 0.12 | 0.10 | 0.39 | 0.41 | 0.43
T-shirt | 0.14 | 0.15 | 0.20 | 0.24 | 0.43 | 0.48 | 0.51 | 0.52
Average | 0.15 | 0.23 | 0.28 | 0.28 | 0.21 | 0.35 | 0.37 | 0.38

The comparison of average IoU results on the PCS dataset are shown in Tab. 10
(column “IoU”). We have observed that SSPNet-K and SSPNet-S achieves the
highest IoU for attribute localization, which is better than that of SSPNet-R.
This can be well explained by the fact that SSPNet-K and SSPNet-S utilize more
accurate spatial information compared to the whole prior region. Localizing
fine-grained attributes, such as “Hat,” “Boots,” and “Glasses,” is a more
challenging task and the baseline method yields poor results. However, SSPNet
utilizing spatial priors leads to significant improvements in localization
accuracy. Besides, localizing coarse-grained attributes, such as Skirt, SSPNet
still shows significant performance improvements when compared to the baseline
method.

It is worth noting that SSPNet-K performs better in localizing fine-grained
attributes, while SSPNet-S performs better in localizing coarse-grained
attributes from Tab. 10 (“K” and “S” of column “IoU”). This is because
SSPNet-K benefits from the spatial prior of human keypoints, which provides
more useful information for fine-grained attribute localization. However, when
it comes to coarse-grained attribute localization, the limited number of human
keypoints does not provide sufficient feature information compared with sparse
points.

Correlation analysis. The comparison of the Pearson product-moment correlation
coefficient (PCCs) 222The PCCs is a numerical indicator that measures the
strength of the linear relationship between two continuous variables, with
values that range from -1 to 1. about each sample IoU and attribute prediction
confidence within the same attribute are shown in Tab. 10 (column “PCCs”). All
PCCs in our study were found to be positive, indicating a positive correlation
between IoU values and attribute recognition confidence.

From Tab. 10 (column “PCCs”), we can observe that SSPNet exhibits a
significantly higher correlation compared to the baseline method. This
indicates that SSPNet establishes a stronger positive correlation between
attribute localization and attribute recognition confidence, thanks to the
utilization of priors. Our proposed SSPNet approach effectively integrates
attribute localization, prior knowledge, and attribute recognition confidence,
resulting in a closer association. Notably, SSPNet-S achieves the highest
average PCCs.

Furthermore, higher PCCs indicate a stronger positive correlation between
localization IoU values and attribute recognition confidence. Therefore,
improving attribute localization accuracy positively impacts attribute
recognition. This insight can motivates us to explore the possible way of
using both attribute localization (IoU) and recognition (mA) as a new
evaluation metric for PAR task.

##  5 Conclusion

We propose a novel PAR method guided by scale and spatial prior knowledge, to
address the issues of interpretability for attribute locations and
generalization performance for cross-dataset tests.

Most relevant research overlooks the scale prior, and our experimental results
suggest that choosing the appropriate feature map related to attribute
granularity is beneficial to the PAR model. Additionally, it is noteworthy to
mention that spatial prior knowledge can effectively guide the model to learn
the proper attribute location and discriminative features, which improve the
accuracy of PAR model and generalization ability. Furthermore, our
hierarchical recognition framework and attribute grouping enable close-
distance attributes to share features within the same group and prevent
dependencies among different grouped attributes, thereby reducing the risk of
over-fitting. Finally, we have also conducted a preliminary quantitative
analysis of weakly supervised attribute localization. The experimental results
on the public datasets validate the effectiveness of our proposed method in
terms of recognition accuracy, generalization performance and location
interpretability.

Our future work will try to extend our method to other attributes recognition
tasks, such as face attributes, vehicle attributes and image retrieval.

## Acknowledgement

This work was supported in part by NSF of China under Grant No. 61903164 and
in part by NSF of Jiangsu Province in China under Grants BK20191427.

## References

## References

  * [1] D. Li, Z. Zhang, X. Chen, K. Huang, A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios, IEEE transactions on image processing 28 (4) (2018) 1575–1590.
  * [2] W. Yang, H. Huang, Z. Zhang, X. Chen, K. Huang, S. Zhang, Towards rich feature discovery with class activation maps augmentation for person re-identification, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 1389–1398.
  * [3] P. Sudowe, H. Spitzer, B. Leibe, Person attribute recognition with a jointly-trained holistic cnn model, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, 2015, pp. 87–95.
  * [4] Y. Li, F. Shi, S. Hou, J. Li, C. Li, G. Yin, Feature pyramid attention model and multi-label focal loss for pedestrian attribute recognition, IEEE Access 8 (2020) 164570–164579.
  * [5] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-cam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 618–626.
  * [6] K. Yu, B. Leng, Z. Zhang, D. Li, K. Huang, Weakly-supervised learning of mid-level features for pedestrian attribute recognition and localization, arXiv preprint arXiv:1611.05603.
  * [7] X. Liu, H. Zhao, M. Tian, L. Sheng, J. Shao, S. Yi, J. Yan, X. Wang, Hydraplus-net: Attentive deep features for pedestrian analysis, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 350–359.
  * [8] N. Sarafianos, X. Xu, I. A. Kakadiaris, Deep imbalanced attribute classification using visual attention aggregation, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 680–697.
  * [9] J. Zhong, H. Qiao, L. Chen, M. Shang, Q. Liu, Improving pedestrian attribute recognition with multi-scale spatial calibration, in: 2021 International Joint Conference on Neural Networks (IJCNN), IEEE, 2021, pp. 1–8.
  * [10] P. Liu, X. Liu, J. Yan, J. Shao, Localization guided learning for pedestrian attribute recognition, in: British Machine Vision Conference 2018, BMVC 2018, 2019\.
  * [11] D. Li, X. Chen, Z. Zhang, K. Huang, Pose guided deep model for pedestrian attribute recognition in surveillance scenarios, in: 2018 IEEE international conference on multimedia and expo (ICME), IEEE, 2018, pp. 1–6.
  * [12] J. Jia, H. Huang, X. Chen, K. Huang, Rethinking of pedestrian attribute recognition: A reliable evaluation under zero-shot pedestrian identity setting, arXiv preprint arXiv:2107.03576.
  * [13] M. Wu, D. Huang, Y. Guo, Y. Wang, Distraction-aware feature learning for human attribute recognition via coarse-to-fine attention mechanism, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 34, 2020, pp. 12394–12401.
  * [14] L. Yang, L. Zhu, Y. Wei, S. Liang, P. Tan, Attribute recognition from adaptive parts, arXiv preprint arXiv:1607.01437.
  * [15] J. Wang, X. Zhu, S. Gong, W. Li, Attribute recognition by joint recurrent learning of context and correlation, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 531–540.
  * [16] Z. Tan, Y. Yang, J. Wan, G. Guo, S. Z. Li, Relation-aware pedestrian attribute recognition with graph convolutional networks, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 34, 2020, pp. 12055–12062.
  * [17] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125.
  * [18] Y. Yang, Z. Tan, P. Tiwari, H. M. Pandey, J. Wan, Z. Lei, G. Guo, S. Z. Li, Cascaded split-and-aggregate learning with feature recombination for pedestrian attribute recognition, International Journal of Computer Vision 129 (2021) 2731–2744.
  * [19] D. Li, X. Chen, Z. Zhang, K. Huang, Learning deep context-aware features over body and latent parts for person re-identification, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 384–393.
  * [20] Q. Li, X. Zhao, R. He, K. Huang, Pedestrian attribute recognition by joint visual-semantic reasoning and knowledge distillation., in: IJCAI, 2019, pp. 833–839.
  * [21] W.-Q. Lu, H.-M. Hu, J. Yu, Y. Zhou, H. Wang, B. Li, Orientation-aware pedestrian attribute recognition based on graph convolution network, IEEE Transactions on Multimedia.
  * [22] X. Fan, Y. Zhang, Y. Lu, H. Wang, Parformer: Transformer-based multi-task network for pedestrian attribute recognition, IEEE Transactions on Circuits and Systems for Video Technology.
  * [23] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discriminative localization, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2921–2929.
  * [24] S. Wang, J. Joo, Y. Wang, S.-C. Zhu, Weakly supervised learning for attribute localization in outdoor scenes, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 3111–3118.
  * [25] C. Tang, L. Sheng, Z. Zhang, X. Hu, Improving pedestrian attribute recognition with weakly-supervised multi-scale attribute-specific localization, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4997–5006.
  * [26] D. Weng, Z. Tan, L. Fang, G. Guo, Exploring attribute localization and correlation for pedestrian attribute recognition, Neurocomputing 531 (2023) 140–150.
  * [27] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [28] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10012–10022.
  * [29] Y. Li, S. Yang, P. Liu, S. Zhang, Y. Wang, Z. Wang, W. Yang, S.-T. Xia, Simcc: A simple coordinate classification perspective for human pose estimation, in: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI, Springer, 2022, pp. 89–106.
  * [30] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, Y. Wei, Deformable convolutional networks, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [31] X. Wang, S. Zheng, R. Yang, A. Zheng, Z. Chen, J. Tang, B. Luo, Pedestrian attribute recognition: A survey, Pattern Recognition 121 (2022) 108220.
  * [32] C. Su, J. Li, S. Zhang, J. Xing, W. Gao, Q. Tian, Pose-driven deep convolutional model for person re-identification, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 3960–3969.
  * [33] Y. Deng, P. Luo, C. C. Loy, X. Tang, Pedestrian attribute recognition at far distance, in: Proceedings of the 22nd ACM international conference on Multimedia, 2014, pp. 789–792.
  * [34] A. Specker, M. Cormier, J. Beyerer, Upar: Unified pedestrian attribute recognition and person retrieval, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 981–990.
  * [35] D. Li, Z. Zhang, X. Chen, K. Huang, A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios, IEEE transactions on image processing: a publication of the IEEE Signal Processing Society 28 (4) (2019) 1575–1590.
  * [36] Y. Lin, L. Zheng, Z. Zheng, Y. Wu, Z. Hu, C. Yan, Y. Yang, Improving person re-identification by attribute and identity learning, Pattern recognition 95 (2019) 151–161.
  * [37] W. Yang, P. Luo, L. Lin, Clothing co-parsing by joint image segmentation and labeling, in: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, IEEE, 2013.
  * [38] R. Lakshmanamoorthy, People clothing segmentation dataset, https://www.kaggle.com/datasets/rajkumarl/people-clothing-segmentation (2021).
  * [39] Q. Li, X. Zhao, R. He, K. Huang, Visual-semantic graph reasoning for pedestrian attribute recognition, in: Proceedings of the AAAI conference on artificial intelligence, Vol. 33, 2019, pp. 8634–8641.
  * [40] Z. Tan, Y. Yang, J. Wan, H. Hang, G. Guo, S. Z. Li, Attention-based pedestrian attribute analysis, IEEE transactions on image processing 28 (12) (2019) 6126–6140.
  * [41] J. Jia, X. Chen, K. Huang, Spatial and semantic consistency regularizations for pedestrian attribute recognition, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 962–971.
  * [42] J. Jia, N. Gao, F. He, X. Chen, K. Huang, Learning disentangled attribute representations for robust pedestrian attribute recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, 2022, pp. 1069–1077.

##  Appendix A Details of attribute group and spatial priors in PLE

Table 11: Attributes grouping and Spatial prior region details. (The “$\times$” symbol is preceded by the range of height and followed by the width of the feature map.) Input size | Groups | Attributes | Prior region
---|---|---|---
P1: [0$\sim$63]$\times$48 P2: [0$\sim$31]$\times$24 P3: [0$\sim$15]$\times$12 | Head | Hat, Glasses |  |  P1: [0$\sim$24]$\times$48
---
P2: [0$\sim$12]$\times$24
P3: [0$\sim$6]$\times$12
Torso |  | ShortSleeve, LongSleeve, UpperStride,
---
UpperLogo, UpperPlaid, UpperSplice,
Backpack
|  P1: [8$\sim$40]$\times$48
---
P2: [4$\sim$20]$\times$24
P3: [2$\sim$10]$\times$12
Bottom |  | LowerStripe, LowerPattern, Trousers,
---
Shorts, Boots
|  P1: [32$\sim$63]$\times$48
---
P2: [16$\sim$31]$\times$24
P3: [8$\sim$15]$\times$12
All |  | LongCoat, Skirt&Dress, HandBag,
---
ShoulderBag, HoldObjectsInFront, AgeOver60,
Age18-60, AgeLess18, Female, Front, Side, Back
|  P1: [0$\sim$63]$\times$48
---
P2: [0$\sim$31]$\times$24
P3: [0$\sim$15]$\times$12

We mention attribute groups in the AFSS module, and the detailed contents of
the attribute groups are shown in the column “Attributes” of Tab. 11. In the
PLE-R module, we mention the height design of image blocks for each attribute
group, and the specific data is displayed in the column “Prior region” of Tab.
11.

Our inspiration for image block partitioning to obtain prior regions comes
from the overlaid and averaged pedestrian images, as shown in Fig. 9a. These
images exhibit clear outlines of the human body.

As illustrated in Fig. 9b, we present the 16 human keypoints along with their
corresponding indexes in the PLE-K module. The keypoints surrounded by
rectangles of different colors represent the reference points selected for the
respective attribute groups. For detailed information regarding the selected
human keypoints in each attribute group, please refer to Tab. 12.

(a) Average image

(b) Points selection

Figure 9: Average image on PA100K dataset. (We overlay add all pedestrian images and average them to obtain the distribution of pedestrian bodies in the dataset.) and Points selection (It shows the selection of human keypoints through colored rectangular boxes for each attribute group, and details of point indexes through numbers in the points.) Table 12: Selected points of PLE-K. (Four attributes group of human parts and keypoints index) Group | Parts name and keypoints index
---|---
head |  head(7$-$9), shoulder(12, 13)
torso |  shoulder(12, 13), arms(10$-$15), hip(2, 3, 6)
bottom |  hip(2, 3, 6), legs(0$-$5)
all |  whole body(0$-$15)

##  Appendix B Case analysis

###  B.1 Discussion on hard cases

Figure 10: Hard cases. (The upper column presents the baseline results, while
the lower column displays the results for SSPNet.)

Fig. 10 presents results on hard cases. In (a), (b), and (c), attributes
suffer from occlusion. In (b), (e), and (f), scale deformation occurs under
the same attribute due to varying distances. (d) and (e) demonstrate
deformation caused by pedestrian movement, and (g) showcases a fine-grained
attribute. SSPNet consistently outperforms the baseline method by accurately
localizing attributes while avoiding irrelevant regions. Our model excels in
attribute localization, addressing issues related to occlusion, deformation,
and fine-grained details.

Figure 11: Fail cases. (These failed cases involve attribute labels that are
predicted correctly, but the predicted locations significantly deviate from
our expectations. Boots, Glasses, Hat, and Skirt visualization heatmaps are
obtained from SSPNet-K, SSPNet-R, SSPNet-S, and SSPNet-R, respectively)

###  B.2 Failure cases and limitation

Fig. 11 illustrates the failure cases and reflects the limitations of our
method in weakly supervised attribute localization. “Boots” in Fig. 11
indicates that when the attributes requiring localization are widely spread
(e.g., two boots), our method may miss some positions. This is partly due to
the weak supervision signal provided by the label level. “Glasses” in Fig. 11
demonstrates the limitations of our method in localizing extremely fine-
grained attribute positions. One possible reason is that even when AFSS
selects the feature map at the maximum scale (downsampled by a factor of 4)
for attribute recognition, it may still miss the fine-grained features of the
attributes. “Hat” in Fig. 11 indicates that SSPNet-S has room for improvement
in terms of its ability to filter complex backgrounds that are similar to the
attributes being recognized and localized. “Skirt” in Fig. 11 demonstrates
that the localization performance is not satisfactory when dealing with
special samples where the attribute is heavily occluded.


# Observation of concentrating paraxial beams

Andrea Aiello  1 Martin Paúr  2 Bohumil Stoklasa  2 Zden ̌ek Hradil  2
Jaroslav Řeháček  2 and Luis L. Sánchez-Soto1,3  1Max-Planck-Institut für die
Physik des Lichts, Staudtstraße 2, 91058 Erlangen, Germany
2Department of Optics, Palacký University, 17\. listopadu 12, 77146 Olomouc,
Czech Republic
3Departamento de Óptica, Facultad de Física, Universidad Complutense, 28040
Madrid, Spain<EMAIL_ADDRESS>
###### Abstract

We report,to the best of our knowledge, the first observation of concentrating
paraxial beams of light in a linear nondispersive medium. We have generated
this intriguing class of light beams, recently predicted by one of us, in both
one- and two-dimensional configurations. As we demonstrate in our experiments,
these concentrating beams display unconventional features, such as the ability
to strongly focus in the focal spot of a thin lens like a plane wave, while
keeping their total energy finite.

††journal: osac††articletype: Research Article

##  1 Introduction

Legend says that during the Siege of Syracuse the Greek mathematician
Archimedes set the Roman fleet on fire using mirrors that concentrated
sunlight upon the ships. Modern versions of Archimedes mirrors are solar
furnaces that can achieve light intensities of about $10$ MW/m2 at the focus.

Focusing light is the province of mirrors and lenses. A converging lens
focuses a collimated beam onto a bright spot centered about the focal point,
at a distance $f$ from the lens. Reversing the direction of light, the same
lens can be used to convert a small spotlike source at the focus in a
collimated beam. Therefore, if such a source covered an arbitrarily small area
$d^{2}$, we would expect that in the reversed operation a converging lens
could focus a collimated light beam on a spot of the same area $d^{2}$.

However, things are not quite so simple. As a matter of fact, in 1873 Abbe
found that a lens cannot focus light of wavelength $\lambda$ to a spot of area
smaller then $\lambda^{2}$ [1]. Since then, this result has been a tenet of
wave optics and it is often presented as a fundamental constraint arising from
the properties of the Fourier transform, which relates the transverse spatial
spread of a wavepacket to its angular spread in the corresponding wavevector
space [2].

Only recently various techniques aiming at beating this limit were proposed
and eventually demonstrated [3]. In particular, Pendry suggested that the
physical origin of such $d^{2}>\lambda^{2}$ limitation resides in the
impossibility of a lens to act upon those evanescent waves that would
contribute to the formation of a pointlike spot in the focal plane [4]. To
overcome this problem Pendry proposed the use of a suitably engineered
_superlens_ that would amplify evanescent waves.

Here, we follow a completely different approach based on a very elementary
observation: if evanescent waves constitute the origin of the problem, then it
would be sufficient to consider only light waves that do not contain them.
Paraxial optics deals precisely with such a kind of waves [5]. That a paraxial
plane wave can be concentrated into a pointlike spot of infinite brightness by
a thin lens, is a textbook result [6]. However, a _bona fide_ plane wave has
an infinite transverse extension and convey infinite energy. Plane waves are
not the only ones that require an infinite amount of energy to be generated:
Bessel [7] and Airy [8] beams are very famous examples thereof. Therefore,
plane waves, Bessel beams and Airy beams, just to name a few popular
nondiffracting beams, are abstractions which can be implemented only
approximately in the real world.

In this work, we report the first observation of concentrating paraxial beams
of light focused by a thin lens. As shown in Ref. [9], these beams have an
infinite lateral extent like, e.g., plane waves or Bessel beams, but they
carry finite energy. Of course, in practice, these beams are typically
truncated by an aperture. However, if the maximal transverse size of such
aperture falls behind the focal length of the lens, paraxial regime of
propagation is maintained and the truncated beams are still strongly
concentrated in the focal point of the lens, thus preserving the major
characteristics of ideal concentrating beams. In fact, our experiments
demonstrate that even though the beams are truncated they still greatly
outperform standard Gaussian beams in terms of focalized intensity.

Paraxial optics is often erroneously considered synonymous of Gaussian optics.
In this paper, we challenge this preconception with these concentrating beams,
which do not follow the rules of Gaussian beam optics. This concept is
exemplified in Fig. 1, where the focusing properties of a Gaussian and a
concentrating beam are compared. The minimum waist $w_{0}^{\prime}$ of a
Gaussian beam of waist $w_{0}$ focused by a thin lens of focal length $f$ is

$\frac{w_{0}^{\prime}}{w_{0}}=\frac{1}{\sqrt{1+z_{R}^{2}/f^{2}}}\,,$ (1)

and this occurs at a distance

$s^{\prime}=\frac{f}{1+f^{2}/z_{\mathrm{R}}^{2}}$ (2)

from the lens, with $z_{\mathrm{R}}$ being the Rayleigh range [10]. This shows
that $w_{0}^{\prime}/w_{0}\to 0$ only when the focal length $f$ goes to zero
and then $s^{\prime}\to 0$. This explains why extreme optical microscopy
requires small focal length and samples close to the microscope, thus breaking
the paraxial conditions. As we will see, for a concentrating beam the
situation is the opposite: a zero-width waist is obtained at distance $f$ of
the lens, where $f$ can be arbitrarily large.

Figure 1: Illustration of the focusing behavior of a Gaussian beam (left) and
a concentrating beam (right) after passing through a thin lenses.

Therefore, the new message we wish to convey is that, surprisingly, physically
realizable paraxial waves that carry a _finite_ amount of energy and that,
notwithstanding, can be focused onto a spot of (theoretical) infinite
brightness, do exist.

Focusing optical beams has always been an issue of great practical importance.
However, we stress that the significance of our results goes beyond the realm
of optics. Actually, in quantum mechanics it is not difficult to construct
states that are represented, at $t=0$, by a continuous wave function and then
evolve (under the Schrödinger equation) into a singular one [11]. In view of
the exact equivalence between the paraxial wave equation and the Schrödinger
equation [12], our results can be seen as an experimental confirmation of such
a singular blow-up, that has never been observed thus far.

##  2 Concentrating beams: Ideal case

To begin with, let us set the stage for our simple theory. We consider an
electromagnetic wave of frequency $\omega$, which in the scalar approximation
is described by its spatial amplitude ${U}(\mathbf{r})$. We denote by $U(x,y)$
the value of this amplitude at the plane $z=0$, wherein the field has a
Cartesian symmetry, so it factorizes as
$U(x,y)=\mathfrak{a}(x)\mathfrak{a}(y)$. An ideal thin lens of focal length
$f$ and transmission function

$t_{\mathrm{lens}}(x,y)=\exp[-{ikx^{2}}/{(2f)}]\;\exp[-{iky^{2}}/{(2f)}]$ (3)

is placed at $z=0$. Using the angular spectrum representation [13], the
amplitude at an arbitrary plane $z>0$ is

$\displaystyle
U(x,y,z)=\iint\limits_{-\infty}^{\quad+\infty}u(p,q)\,e^{ik(px+qy)}\,\,e^{ik\sqrt{1-p^{2}-q^{2}}\,z}\,dpdq\,,$
(4)

where

$\displaystyle u(p,q)=$
$\displaystyle\;\frac{k^{2}}{(2\pi)^{2}}\iint\limits_{-\infty}^{\quad+\infty}t_{\mathrm{lens}}(x,y)\,U(x,y)e^{-ik(px+qy)}\,dxdy$
(5)

is the Fourier transform of the field at plane $z=0$ immediately after the
lens.

In the paraxial regime waves are characterized by small angular deviations
with respect to the mean propagation axis, say $z$. In such a case,
$p^{2}+q^{2}\ll 1$ and, to a good approximation, we can replace the unpleasant
exact transfer function $\exp(ikz\sqrt{1-p^{2}-q^{2}})$ with the friendly
Fresnel propagator $\exp\\{ikz[1-(p^{2}+q^{2})/2]\\}$. This gives

$\displaystyle
U(x,y,z)=\iint\limits_{-\infty}^{\quad+\infty}u(p,q)\,e^{ik(px+qy)}\,e^{ikz\left[1-(p^{2}+q^{2})/2\right]}\,dpdq\,.$
(6)

To check this expression, let us first consider the somewhat simpler example
of a plane wave of unit amplitude incident along the $z$ axis. The spectrum
$u(p,q)$ can be now readily calculated; the final result reads
$U_{\mathrm{pl}}(x,y,z)=\mathfrak{a}_{\mathrm{pl}}(x,z)\;\mathfrak{a}_{\mathrm{pl}}(y,z)$,
with

$\mathfrak{a}_{\mathrm{pl}}(x,z)=\frac{e^{ikz}}{\sqrt{2\pi
ikf}}\int\limits_{-\infty}^{\quad+\infty}\exp\left[i\left(p^{2}\frac{1-z/f}{2kf}\,+p\frac{x}{f}\right)\right]\,dp\,,$
(7)

and an analogous expression for $\mathfrak{a}_{\mathrm{pl}}(y,z)$. When the
lens is absent, $u(p,q)=\delta(p)\delta(q)$ and this yields
$U(x,y,z)=\exp(ikz)$, as it should be. In the focal plane of the lens,
${z}=f$, and Eq. (7) gives at once

$\displaystyle U(x,y,z=f)=\frac{2\pi
e^{2ikf}}{ikf}\,\delta(x/f)\,\delta(y/f)\,,$ (8)

which means that, as one might anticipated, a plane wave is focused by a thin
lens into a pointlike spot of infinite brightness.

Figure 2: For $1/4<\nu$ the function $\mathfrak{a}_{\nu}(x,0)$ is square
integrable. For $1/4<\nu<1/2$ (represented as a red band)
$\mathfrak{a}_{\nu}(x,z)$ becomes infinite at the focal point $z=f$. For
$1/2<\nu$, $\mathfrak{a}_{\nu}(x,z)$ is finite everywhere and the first moment
$\langle x\rangle$ of the intensity distribution
$|\mathfrak{a}_{\nu}(x,z)|^{2}$ exists and it is equal to $0$. Finally, the
second moment $\langle x^{2}\rangle$ is finite for $3/4<\nu$ and gives a
faithful measure of the beam waist. For $\nu=1$ the concentrating beam becomes
a so-called Lorentz beam [14].

The ideal focusing of a plane wave is well understood, but a plane wave is not
a normalizable field and thus is not physically feasible. Our goal is to study
a family of paraxial fields, square integrable and almost everywhere smooth,
that spontaneously generate a singularity along the propagation axis in the
focal point of a thin lens. We will refer to them as concentrating beams.
According to the ideas developed in Ref. [9] we consider the family (up to a
global constant) $U_{\nu}(x,y)=\mathfrak{a}_{\nu}(x)\,\mathfrak{a}_{\nu}(y)$,
with

$\mathfrak{a}_{\nu}(x)=\frac{1}{(1+x^{2}/\sigma_{\nu}^{2})^{\nu}}\,,$ (9)

where $\sigma_{\nu}>0$ determines the spot size of the beam and $\nu>0$ is a
positive constant that can be varied to obtain different kind of
singularities. For $\nu=1$ they reduce to the so-called Lorentz beam [14]. A
similar expression holds for $\mathfrak{a}_{\nu}(y)$. Using again the paraxial
approximation (6), we obtain $U_{\nu}(x,y,z)={k}/{(2\pi
iz)}\;\mathfrak{a}_{\nu}(x,z)\,\mathfrak{a}_{\nu}(y,z)$, where now

$\mathfrak{a}_{\nu}(x,z)=\sqrt{\frac{k}{2\pi
iz}}e^{i\frac{kx^{2}}{2z}}\int_{-\infty}^{+\infty}\frac{\exp\left\\{i\frac{k}{2z}\left[-2xs+(1-z/f)s^{2}\right]\right\\}}{(1+s^{2}/\sigma_{\nu}^{2})^{\nu}}\,ds\,.$
(10)

One can check that for $1/4<\nu<1/2$ the beams (9) are square integrable

$\displaystyle\iint\limits_{-\infty}^{\quad+\infty}|U_{\nu}(x,y)|^{2}\,dxdy=\pi\sigma_{\nu}\left[\frac{\Gamma\left(2\nu-1/2\right)}{\Gamma(2\nu)}\right]^{2}\,,$
(11)

but the field $U_{\nu}(x,y,z)$ diverges at the focal point of the lens $z=f$.

##  3 Concentrating beams: realistic case

The theory thus far truly confirms that, in principle, it is possible to
achieve an infinitely bright spot by focusing a nonplane wavefront with an
ideal thin lens. However, this is a speculative situation, for any lens always
has a finite aperture, which causes that the amplitude of our beam remains
finite at $z=f$. In addition, we have ignored any polarization effect.

To cope with these problems, we take the monomochromatic field at $z=0$ as the
vector field $\mathbf{E}(x,y)=U(x,y)\,\hat{\mathbf{e}}_{p}(\alpha,\beta)$,
where $U(x,y)$ is the spatial amplitude and
$\hat{\mathbf{e}}_{p}(\alpha,\beta)=\hat{\mathbf{e}}_{x}\cos\alpha+e^{i\beta}\hat{\mathbf{e}}_{y}\sin\alpha$
is the unit polarization vector, with direction angles $(\alpha,\beta)$. We
take the lens as having an aperture $2L$ in each direction. The field at a
plane $z$ can be exactly computed according to

$\mathbf{E}(x,y,z)=\int_{-L}^{L}\int_{-L}^{L}U(\xi,\eta)\,t_{\mathrm{lens}}(\xi,\eta)\;\mathbf{K}(x-\xi,y-\eta,z)\,d\xi
d\eta\,,$ (12)

where

$\displaystyle\mathbf{K}(x-\xi,y-\eta,z)=\frac{1}{2\pi}\left\\{-\hat{\mathbf{e}}_{p}\frac{\partial
G(\mathbf{r}-\mathbf{r}_{0})}{\partial
z}+\hat{\mathbf{e}}_{z}\left[\hat{\mathbf{e}}_{p}\cdot\hat{\mathbf{e}}_{x}\frac{\partial
G(\mathbf{r}-\mathbf{r}_{0})}{\partial
x}+\hat{\mathbf{e}}_{p}\cdot\hat{\mathbf{e}}_{y}\frac{\partial
G(\mathbf{r}-\mathbf{r}_{0})}{\partial y}\right]\right\\}\,.$ (13)

Here $(\xi,\eta)$ are the coordinates of a point in the aperture of the lens,
$G(\mathbf{r}-\mathbf{r}_{0})={\exp(ik|\mathbf{r}-\mathbf{r}_{0}|)}$$/{|\mathbf{r}-\mathbf{r}_{0}|}$
is the Green function of the problem and
$\mathbf{r}-\mathbf{r}_{0}=\hat{\mathbf{e}}_{x}(x-\xi)+\hat{\mathbf{e}}_{y}(y-\eta)+\hat{\mathbf{e}}_{z}z$.
In this way, $\mathbf{E}(x,y,z)$ verifies the Helmoltz equation, and not only
the paraxial approximation, in every point of the space, as one can easily
check.

Figure 3: Comparison of the intensity profiles $|\mathbf{E}(x,0,f)|^{2}$ at
the focal point of the lens $z=f$. The vertical scale is fixed by the
normalization condition (3) for $L=2.5$ mm. Left panel corresponds to plane
waves, central panel to Gaussian beams, and right panel to concentrating beams
with $\nu=1/3$. In each plot, we represent the results for three different
apertures of $L=0.5$ mm (green), 1 mm (orange) and 2.5 mm (blue). The
continuous lines correspond to the theoretical values, whereas the points
indicate experimental values. The input polarization is always linear and
parallel to the $y$ axis. Figure 4: On-axis intensities at the focal plane
$z=f$ for concentrating and Gaussian beams (normalized to the corresponding
value for a plane wave) as a function of their dimensionless widths (in units
of $L$). In the inset, we show the ratio of the on-axis intensities of the
concentrating to the Gaussian beam in terms of the same parameter.

To gain insight into the problem, we will compare the focusing performance of
a plane wave and a singular Lorentz beam, which henceforth we will take with
$\nu=1/3$. For completeness, we also include a Gaussian beam, since this is
perhaps the most prevalent of all beams. To make a fair comparison we
normalize these fields so as they have unit intensity across the aperture;
that is,

$\int_{-L}^{L}\int_{-L}^{L}\mathbf{E}^{\ast}(x,y,0)\cdot\mathbf{E}(x,y,0)\,dxdy=\int_{-L}^{L}\int_{-L}^{L}\
|U(x,y)|^{2}\,dxdy=1\,,$ (14)

which fixes the field amplitudes to the following forms

$\displaystyle U_{\mathrm{pl}}(x,y)$ $\displaystyle=\frac{1}{2L}\,,$
$\displaystyle U_{\nu}(x,y)$
$\displaystyle=\frac{1}{2L}\frac{(1+x^{2}/\sigma_{\nu}^{2})^{-\nu}(1+y^{2}/\sigma_{\nu}^{2})^{-\nu}}{\/{}_{2}F_{1}(1/2,\nu,3/2;-L^{2}/\sigma_{\nu}^{2})}\,,$
(15) $\displaystyle U_{\mathrm{G}}(x,y)$
$\displaystyle=\frac{\sqrt{2}}{\sqrt{\pi}\,\sigma_{\mathrm{G}}\,\mathrm{erf}(\sqrt{2}L/\sigma_{\mathrm{G}})}\exp[-(x^{2}+y^{2})/\sigma_{\mathrm{G}}^{2}]\,,$

$\/{}_{2}F_{1}(a,b,c;z)$ and $\mathrm{erf}(z)$ being the hypergeometric and
error functions, respectively.

We have numerically integrated the exact result in Eq. (12) for these fields
and three different values of the aperture: $L=0.5,1$ and 2.5 mm. The
resulting intensity profiles at the focal point of the lens are shown in Fig.
3. In all the cases we have chosen linear polarization parallel to the $y$
axis. At $z=f$, the polarization is still linear without any appreciable $z$
component, which confirms that polarization plays no significant role in the
behavior of the singularity. We can appreciate that the size of the Gaussian
spot remains approximately constant with the aperture, whereas the
concentrating field reveals a clear focusing when the aperture increases, and
the same happens for the plane wave.

When the width of both concentrating and Gaussian beams exceed $L$, the
corresponding intensity profiles approach the plane wave (uniform) intensity
distribution. The maximum value corresponding to the plane wave is obtained
asymptotically for both beams, but the concentrating beam approaches it much
faster. This is confirmed in Fig. 4, where we plot the maximum intensity at
the focus $z=f$ for both beams (normalized to the value of the plane wave).

Figure 5: Schematic of the experimental setup for the generation of
concentrating beams. See the description in the text for more details.

##  4 Experimental results

We have checked these predictions in the laboratory (Fig. 5. To build up the
desired beam profile we used an amplitude spatial light modulator (SLM) (CRL
OPTO) illuminated by a collimated beam from linearly polarized He-Ne laser
(633 nm Thorlabs). The modulator was calibrated to s transmitted wavefront
with the final Strehl ratio better than 0.98. Next, calibration of the
amplitude modulation linearity was performed, from where we estimate our
systematic errors. On the SLM, several holograms with different aperture sizes
and shapes were realized. These holograms were generated as an interference
between a tilted reference plane wave and the function of desired beam
profile. The beam complex amplitude is carried by the first diffraction order.
This diffraction order was then Fourier transformed by a spherical lens of
focal length $f=200$ mm and the signal was detected by CMOS camera (Basler)
with $1.85\times 1.85~{}\mu$m2-pixel size.

Figure 6: Ratio of the on-axis intensities $\varrho_{S}/\varrho_{G}$ of the
concentrating to Gaussian beams, as the aperture increases. The experimental
points are marked by red error bars (the error being estimated from the
calibration), corresponding to $L=0.5$, 1, 1.5, 2, and 2.5 mm. The red circles
surrounding the error bars serve to help visualization.

In this way, we reproduced the theoretical predictions of Fig. 3. This
required to select the widths $\sigma_{\nu}$ and $\sigma_{G}$ for each
aperture individually. The experimental results are in excellent agreement
with the theory.

Figure 7: Raw intensity data for the concentrating beam (right) and the
Gaussian beam (left) for the aperture of 2.5 mm. At the bottom, we plot
density plots of the intensities as seen in the CMOS camera at the focal spot
of the lens. The (arbitrary) unit of the vertical scale is the same for both
plots.

Experimental data shows an important increase of on-axis intensity of the
concentrating beam when the energy of the beam is slightly increased. For this
reason, we measured the ratio $\varrho_{S}/\varrho_{G}$ of the on-axis
intensities of the concentrating to the Gaussian beam, as the aperture
increases, but keeping the corresponding beam widths: $\sigma_{\nu}=0.16$ and
$\sigma_{\mathrm{G}}=0.5$. The results are presented in Fig. 6, which brings
out the advantages of the concentrating beam in comparison with its Gaussian
counterpart.

Finally, in Fig. 7 we include raw intensity data at the focal point from this
last experiment for $L=2.5$ mm. The real camera pictures provide a dramatic
evidence of the focusing capabilities of the concentrating beams.

To sum up, we have presented and experimentally realized a family of square-
integrable paraxial beams that spontaneously develop a singularity when they
propagate in free space. We stress that this is a linear property of the beams
and not the outcome of any self-focusing or other nonlinear effects [15].
While in the standard picture of extreme light focusing the minimum focal spot
is achieved with large numerical apertures, in our new paradigm the opposite
is true: strong focusing is achieved moving towards paraxial optics, as in the
case of a flat (plane wave) illumination of an aperture. However, for a given
fixed average input intensity across the aperture, the energy carried by a
plane wave increases quadratically with the linear size of the aperture,
whereas for our concentrating beams the energy remains constant. This
promising field enhancement mechanism may foster further interesting research
in classical and quantum optics.

Funding. European Union Horizon 2020 (732894), Grant Agency of the Czech
Republic (18-04291S), Palacký University (IGA_PrF_2020_004), Ministerio de
Ciencia e Innovación (PGC2018-099183-B-I00)

Disclosures. The authors declare no conflicts of interest.

## References

  * [1] E. Abbe, “Beiträge zur theorie des mikroskops und der mikroskopischen wahrnehmung,” Arch. Mikrosk. Anat. 9, 413–468 (1873).
  * [2] L. J. Wong and I. Kaminer, “Abruptly focusing and defocusing needles of light and closed-form electromagnetic wavepackets,” ACS Photonics 4, 1131–1137 (2017).
  * [3] G. Huszka and M. A. Gijs, “Super-resolution optical imaging: A comparison,” Micro Nano Engineering 2, 7–28 (2019).
  * [4] J. B. Pendry, “Negative refraction makes a perfect lens,” Phys. Rev. Lett. 85, 3966–3969 (2000).
  * [5] J. W. Goodman, _Introduction to Fourier Optics_ (Roberts & Company, Englewood, CO, 2005), 3rd ed.
  * [6] M. Born and E. Wolf, _Principles of Optics_ (Cambridge University Press, Cambridge, 2003), 7th ed.
  * [7] J. Durnin, J. J. Miceli, and J. H. Eberly, “Diffraction-free beams,” Phys. Rev. Lett. 58, 1499–1501 (1987).
  * [8] G. A. Siviloglou, J. Broky, A. Dogariu, and D. N. Christodoulides, “Observation of accelerating Airy beams,” Phys. Rev. Lett. 99, 213901 (2007).
  * [9] A. Aiello, “Spontaneous generation of singularities in paraxial optical fields,” Opt. Lett. 41, 1668–1671 (2016).
  * [10] A. M. Almeida, E. Nogueira, and M. Belsley, “Paraxial imaging: Gaussian beams versus paraxial-spherical waves,” Am. J. Phys. 67, 428–433 (1999).
  * [11] A. Peres, _Quantum theory: Concepts and methods_ (Kluver, San Diego, CA, 1994), 5th ed.
  * [12] A. Yariv, _Optical Electronics_ (Cambridge University Press, New York, 1995), 4th ed.
  * [13] L. Mandel and E. Wolf, _Optical Coherence and Quantum Optics_ (Cambridge University Press, New York, 1995).
  * [14] O. E. Gawhary and S. Severini, “Lorentz beams and symmetry properties in paraxial optics,” J. Opt. A: Pure Appl. Opt. 8, 409–414 (2006).
  * [15] N. K. Efremidis and D. N. Christodoulides, “Abruptly autofocusing waves,” Opt. Lett. 35, 4045–4047 (2010).