# Dark-Skin Individuals Are at More Risk on the Street: Unmasking Fairness
Issues of Autonomous Driving Systems

Xinyue Li<EMAIL_ADDRESS>Peking UniversityBeijingChina ,  Zhenpeng Chen
<EMAIL_ADDRESS>University College LondonLondonUnited Kingdom ,  Jie M.
Zhang<EMAIL_ADDRESS>King’s College LondonLondonUnited Kingdom ,
Federica Sarro<EMAIL_ADDRESS>University College LondonLondonUnited
Kingdom ,  Ying Zhang<EMAIL_ADDRESS>Peking UniversityBeijingChina and
Xuanzhe Liu<EMAIL_ADDRESS>Peking UniversityBeijingChina

(2024)

###### Abstract.

This paper conducts fairness testing on automated pedestrian detection, a
crucial but under-explored issue in autonomous driving systems. We evaluate
eight widely-studied pedestrian detectors across demographic groups on large-
scale real-world datasets. To enable thorough fairness testing, we provide
extensive annotations for the datasets, resulting in 8,311 images with 16,070
gender labels, 20,115 age labels, and 3,513 skin tone labels. Our findings
reveal significant fairness issues related to age and skin tone. The detection
accuracy for adults is 19.67% higher compared to children, and there is a
7.52% accuracy disparity between light-skin and dark-skin individuals. Gender,
however, shows only a 1.1% difference in detection accuracy. Additionally, we
investigate common scenarios explored in the literature on autonomous driving
testing, and find that the bias towards dark-skin pedestrians increases
significantly under scenarios of low contrast and low brightness. We publicly
release the code, data, and results to support future research on fairness in
autonomous driving.

Group fairness, pedestrian detection, autonomous driving

††copyright: none††journalyear: 2024††copyright: acmcopyright††ccs: Software
and its engineering Software creation and management††ccs: Computing
methodologies Machine learning

##  1\. Introduction

Autonomous driving systems are on track to become the predominant mode of
transportation in the future (Stocco et al., 2020). However, these systems are
susceptible to software bugs (Garcia et al., 2020), which can potentially
result in severe injuries or even fatalities for both pedestrians and
passengers. The unfortunate incident in 2018 involving an autonomous vehicle
from Uber serves as a stark reminder of these risks (cas, 2018). Given the
safety-critical nature of autonomous driving systems, they have garnered
substantial attention from the software testing community (Zhang et al.,
2022).

Extensive research efforts have been devoted to the testing of autonomous
driving systems. For example, Tian et al. (Tian et al., 2018) introduced
DeepTest, which applies image transformation to simulate potential camera
noise in autonomous driving scenarios. Zhang et al. (Zhang et al., 2018)
developed DeepRoad, a Generative Adversarial Network (GAN)-based approach that
generates test images from real-world driving scenes. Zhou et al. (Zhou et
al., 2020) proposed DeepBillboard, a system for generating adversarial
billboards to induce potential steering errors in autonomous vehicles.

Although significant testing efforts have been made, to the best of our
knowledge, the study of fairness testing for autonomous driving systems
remains under-investigated in the literature. From a Software Engineering (SE)
perspective, fairness is considered a non-functional software property, making
it an important subject for testing (Zhang et al., 2022). Fairness testing, as
an emerging domain within software testing, seeks to uncover fairness issues
in software systems (Galhotra et al., 2017).

Fairness issues in autonomous driving systems, such as a higher accuracy in
detecting pedestrians of white ethnicity compared to black ethnicity, can
perpetuate discriminatory outcomes and unequal treatment based on race. This
can result in harm to individuals belonging to marginalized groups, further
exacerbating existing social inequalities. Therefore, it is crucial to
prioritize fairness testing in autonomous driving systems.

To fill the knowledge gap, we systematically conduct fairness testing on
autonomous driving systems, with a specific emphasis on the pedestrian
detection component — the human-related module within such systems. Our main
focus is to quantitatively assess performance disparities in current
pedestrian detectors across diverse demographic groups, which are widely
recognized as group fairness issues in the literature (Chen et al., 2022b). By
measuring and highlighting these discrepancies, we contribute to the existing
body of knowledge surrounding well-documented fairness concerns.

In this study, we conduct fairness testing on eight state-of-the-art Deep
Learning (DL)-based pedestrian detectors that have been extensively studied in
the research community. To enable this testing, we require datasets that
include demographic labels (such as gender and skin tone) for pedestrians.
However, commonly-used datasets for pedestrian detection often lack such
labels. To address this problem, we manually enrich four widely-adopted real-
world datasets with gender, age, and skin tone labels, resulting in a
collection of 8,311 real-world images annotated with 16,070 gender labels,
20,115 age labels, and 3,513 skin tone labels. Using these labeled datasets,
we assess the group fairness of existing pedestrian detectors and also
investigate how commonly-studied scenarios in the literature of autonomous
driving testing (including brightness, contrast, and weather conditions)
impact the fairness of these detectors.

Our study reveals the following findings: (1) Overall, state-of-the-art
pedestrian detectors exhibit bias across different age and skin tone groups.
On the four datasets examined, the undetected proportions for children surpass
those for adults by an average of 19.67%. At the same time, undetected
proportions for dark-skin individuals exceed those for light-skin individuals
by an average of 7.52% across all datasets. (2) However, the performance of
these pedestrian detectors in detecting males and females does not exhibit a
large difference, with only a 1.1% gap in undetected proportions. (3)
Regarding the exploration of common scenarios, detection performance for the
dark-skin group decreases under low-brightness and low-contrast conditions
compared to the light-skin group. For instance, the difference in undetected
proportions increases from 7.14% to 9.86% between day time and night time
scenarios.

To summarise, we make the following contributions:

  * •

We conduct a systematic empirical study on fairness testing of autonomous
driving systems, evaluating eight widely-studied pedestrian detectors and
uncovering significant fairness issues.

  * •

We augment four large-scale datasets with manually labeled demographic
information, resulting in 8,311 real-world images with 16,070 gender labels,
20,115 age labels, and 3,513 skin tone labels, to facilitate future research
on fairness in autonomous driving.

  * •

We provide a replication package (git, 2023) containing all the code and
results from this study, facilitating replication and extension of this
research.

##  2\. PRELIMINARIES

This study resides at the intersection of two increasingly important SE
topics: software fairness and autonomous driving testing. To provide the
necessary context, we begin by reviewing the background knowledge and relevant
prior research in these areas.

###  2.1. Software Fairness

Fairness has gained significant attention in the SE community since its
initial exploration by SE researchers in 2008 (Finkelstein et al., 2008).
There have been various definitions of fairness in the literature. In this
paper, we focus on group fairness, a concept extensively studied in software
fairness research (Biswas and Rajan, 2021; Zhang and Harman, 2021; Hort et
al., 2021; Chakraborty et al., 2021, 2020; Gohar et al., 2023; Biswas and
Rajan, 2020; Zhang et al., 2023; Zhang and Sun, 2022). Notably, group fairness
closely aligns with legal regulations on fairness (Barocas and Selbst, 2016),
such as the adherence to the four-fifths rule, a cornerstone of US law (Wick
et al., 2019). Consequently, prioritizing group fairness when building
software has emerged as an essential ethical duty and requirement for software
engineers (Chakraborty et al., 2021).

In the context of group fairness, certain personal characteristics that
require protection against unfairness during decision-making are called
sensitive attributes, also known as protected attributes (Zhang et al., 2022;
Corbett-Davies and Goel, 2018). Well-recognized sensitive attributes include
race, sex, age, pregnancy, familial status, disability status, and more (Chen
et al., 2022a). These sensitive attributes typically partition a population
into distinct groups: a privileged group and an unprivileged group (Chen et
al., 2022a). Group fairness entails the equal treatment of these groups by the
same machine learning model. However, in practice, members of unprivileged
groups often experience systematic disadvantages, resulting from unfair
machine learning models. For instance, in the context of a pedestrian
detection task, if age is deemed a sensitive attribute, the predictive model
may exhibit a bias favoring adult groups over child groups. In this scenario,
the adult group is considered the privileged group, while the child group
becomes the unprivileged one.

Presently, the majority of software fairness testing research primarily
revolves around tabular data, as indicated by a recent survey (Chen et al.,
2022a). This paper centers on fairness testing for pedestrian detection in
autonomous driving systems. We specifically examine three sensitive attributes
(i.e., gender, age, and skin tone) that are more recognizable in autonomous
driving datasets and have been widely considered in the fairness literature
(Soremekun et al., 2022).

###  2.2. Autonomous Driving Testing

Autonomous driving is a hot SE research topic, and researchers have proposed
various testing techniques for autonomous driving systems (Pei et al., 2019;
Tian et al., 2018; Zhang et al., 2018; Zhou et al., 2020). These techniques
mainly involve the automatic generation of artificial driving scenarios to
simulate real-world conditions, enabling the evaluation of system behavior and
the detection of blind spots or corner bugs across various autonomous driving
scenarios.

While a substantial body of knowledge focuses on assessing the robustness and
correctness properties of autonomous driving systems (Zhang et al., 2022), to
the best of our knowledge, only few studies have explored the fairness
properties, particularly in the pedestrian detection domain within autonomous
driving. Pedestrian detection is a crucial process identifies pedestrians
within street-level images by providing their predicted locations along with
corresponding bounding boxes and confidence scores (Dollár et al., 2012; Zhang
et al., 2016). Brandao (Brandao, 2019) discussed fairness in pedestrian
detection but focused solely on traditional detection techniques, which are no
longer dominant in contemporary practice. Wilson et al. (Wilson et al., 2019)
examined skin tone bias but with limitations to one dataset. Similarly, Kogure
et al. (Kogure et al., 2022) only explored age bias with one detection method
and a small-scale dataset, lacking a comprehensive analysis. To summarise,
current fairness research in pedestrian detection of autonomous driving is
still in its early stages and faces limitations in terms of diversity in
pedestrian detectors, testing datasets, and sensitive attributes considered.

To address this knowledge gap, our paper presents the a systematic empirical
study on group fairness testing in pedestrian detection. We conduct
experiments using eight popular detection methods and four diverse testing
datasets, encompassing different scenarios determined by a variety of factors
such as brightness, contrast, and weather conditions. Through this
comprehensive study, we gain insights into the potential bias and risks
associated with using existing pedestrian detectors.

##  3\. EXPERIMENTAL DESIGN

In this section, we introduce our research questions and experimental
settings.

###  3.1. Research Questions

We aim to answer the following research questions (RQs) in this study.

RQ1 (Overall fairness): To what extent do widely-studied pedestrian detectors
exhibit unfairness concerning common sensitive attributes? This RQ explores
the performance difference of widely-studied pedestrian detectors when they
are applied to different demographic groups, characterized by sensitive
attributes including gender, age, and skin tone.

RQ2 (Fairness in different scenarios): What fairness do pedestrian detectors
achieve in different brightness, contrast, and weather conditions? We further
investigate the fairness of commonly-studied pedestrian detectors in different
autonomous driving scenarios by considering a variety of different brightness,
contrast, and weather conditions (Tian et al., 2018; Zhang et al., 2018; Zhou
et al., 2020). Correspondingly, we investigate RQ2 by answering three specific
sub-questions:

RQ2.1 What fairness do pedestrian detectors achieve under different brightness
conditions?

RQ2.2 What fairness do pedestrian detectors achieve under different contrast
conditions?

RQ2.3 What fairness do pedestrian detectors achieve under different weather
conditions?

In the following, we introduce the pedestrian detectors (Section 3.2),
benchmark datasets (Section 3.3), evaluation measures (Section 3.4), and
statistical analysis (Section 3.5) that we use to address the above research
questions.

###  3.2. Pedestrian Detection Methods

In recent years, DL has revolutionized pedestrian detection approaches. We
focus our analysis on eight DL-based pedestrian detectors that are widely
studied in the autonomous driving community (Zou et al., 2023; Cao et al.,
2020). These detectors are pre-trained DL models that researchers and
practitioners can directly use for pedestrian detection tasks. They can be
classified into two categories (Hasan et al., 2020): general object detectors
and pedestrian-specific detectors. Next, we briefly introduce each category
and its pedestrian detectors we study. Table 1 provides an overview of these
detectors (“Detector” shows the name of a pedestrian detector; “Backbone”
represents the pre-trained deep neural network used for extracting features
from input images; “Source” indicates the framework/toolkit name and its
source for a given pedestrian detector).

Table 1. Pedestrian detectors. Type | Detector | Backbone | Source
---|---|---|---
General object detectors | YOLOX | - | MMDetection (MMDetection Developers, nd)
RetinaNet | R-50-FPN
Faster R-CNN | R-50-FPN
Cascade R-CNN | R-50-FPN
Pedestrian -specific detectors | ALFNet | ResNet50 |  ALFNet (liuwei16, 2018)
CSP | ResNet50 | Pedestron (hasanirtiza, 2021)
MGAN | VGG16
PRNet | ResNet50 |  PRNet (sxlpris, 2020)

General object detectors: General detectors can detect various objects such as
cars, traffic lights, and pedestrians. They have great generalization ability
but lack pedestrian-specific adaptation (Hasan et al., 2020). They can be
categorized into two genres (Zou et al., 2023): two-stage and one-stage
detectors. Two-stage detectors propose regions before feature extraction and
classification, achieving high accuracy but slower speed; one-stage detectors
complete all operations in one step, providing faster speed but lower
accuracy. Both types involve trade-offs and are widely used for pedestrian
detection. Hence, this paper selects detectors from both categories. For one-
stage detectors, we adopt the widely-studied YOLOX (Ge et al., 2021) (a faster
extension of the YOLO series (Redmon et al., 2016)) and RetinaNet (Lin et al.,
2017) (which addresses the class imbalance problem). For two-stage detectors,
we employ the Faster R-CNN (Ren et al., 2015) (one of the pioneering detectors
in the R-CNN family) and Cascade R-CNN (Cai and Vasconcelos, 2018) (which
achieves higher accuracy through a cascade of multiple CNNs to refine region
proposals). These detectors have been extensively used in the autonomous
driving literature (Hasan et al., 2020; Cao et al., 2020; Yu et al., 2020;
Braun et al., 2019).

Pedestrian-specific detectors: Pedestrian-specific detectors use additional
pedestrian-related information to improve detection performance (Hasan et al.,
2020). In this study, we investigate state-of-the-art pedestrian-specific
detectors, including ALFNet (Liu et al., 2018), CSP (Liu et al., 2019), MGAN
(Pang et al., 2019), and PRNet (Song et al., 2020). ALFNet uses progressive
detection heads on SSD (Liu et al., 2016) to refine initial anchors for
improved detection accuracy. CSP introduces an anchor-free approach by
locating center points and scaling pedestrians. MGAN uses visible-area
bounding-box information to guide attention mask generation for occluded
pedestrian detection. PRNet presents a novel progressive refinement network
for occluded pedestrian detection.

###  3.3. Benchmark Datasets

####  3.3.1. Dataset Selection

We perform our experiments on four real-world datasets that have been
extensively studied by researchers to evaluate the performance of pedestrian
detectors in autonomous driving (Cao et al., 2020; Hasan et al., 2020). These
datasets consist of street-level images captured by cameras mounted on
autonomous vehicles, showcasing pedestrians in diverse poses, sizes, and
occlusion scenarios. Table 2 presents details about these datasets, including
the sensitive attributes, the number of images in each dataset, and the
respective time of day when these images were captured. Next, we briefly
introduce each dataset:

Table 2. Benchmark datasets. Name | Sensitive Attributes | #Images | Time
---|---|---|---
CityPersons | gender, age | 1,525 | day
EuroCity-Day | gender, age | 2,427 | day
EuroCity-Night | gender, age | 2,126 | night
BDD100k | gender, age, skin | 2,233 | day, night

  * •

CityPersons dataset (Zhang et al., 2017) stands as the most widely-studied
benchmark for evaluating pedestrian detectors (Cao et al., 2020; Hasan et al.,
2020; Liu et al., 2018; Song et al., 2020; Liu et al., 2019; Pang et al.,
2019). Its test set includes 1,525 images captured across six cities,
showcasing diverse weather conditions and street scenes.

  * •

EuroCityPersons dataset (Braun et al., 2019) contains 4,553 images gathered
from seven European cities, encompassing both day and night time captures. The
dataset can be categorized into two sets: 2,427 images captured during the
day, and 2,126 images captured at night, referred to as the EuroCity-Day
dataset and the EuroCity-Night dataset, respectively.

  * •

Berkeley Driving dataset (a.k.a., BDD100k dataset) (Yu et al., 2020) is an
extensive driving dataset, including 2,233 images from 40 classes that are
typical of driving scenes. These images were captured from four different
cities, providing various times of the day. Notably, this dataset showcases a
greater diversity of pedestrians than the CityPersons and EuroCityPersons
datasets, including individuals with varied skin tones.

####  3.3.2. Sensitive Attribute Labeling

As introduced in Section 2.1, we focus on three sensitive attributes (i.e.,
gender, age, and skin tone) of pedestrians that are recognizable in autonomous
driving images and widely studied in the fairness literature (Soremekun et
al., 2022). To enable fairness testing, we need datasets with labels that
identify these sensitive attributes of the humans depicted in the images.
Among the datasets investigated herein, the only sensitive attribute already
labeled is the skin tone (i.e., light-skin tone and dark-skin tone) for the
BDD100k dataset. Therefore, we manually label the other sensitive attributes
for each of the datasets considered in this study.

The labeling process involves two annotators to minimize the influence of
labeling bias. We focus on images that align with the widely-adopted
“reasonable subset” principle (Dollár et al., 2012), meaning that we label
images containing labeled pedestrians with a height of at least 50 pixels and
little to none occlusion. For such images, human annotators can label the
sensitive attributes with high confidence (Brandao, 2019). Using the filtered
datasets, the two annotators independently label the gender and age attributes
for each image. For gender, we follow previous studies (Brandao, 2019) and
consider only two labels: male and female. As for age, in line with the
literature (Brandao, 2019; Kogure et al., 2022), we classify pedestrians into
two labels: child and adult, based on their physical characteristics depicted
in the images.

To ensure the reliability of the labeling procedure, both annotators
independently label the age and gender attributes for each image. We use
Cohen’s Kappa ($\kappa$) (Cohen, 1960), a widely-adopted metric for measuring
inter-rater agreement, during the independent labeling process. The obtained
$\kappa$ values for age and gender attributes in each of the four datasets are
summarized in Table 3. According to the literature (Landis and Koch, 1977), a
$\kappa$ value between 0.81 and 1 signifies almost perfect agreement, while a
value between 0.61 and 0.8 indicates substantial agreement. In our labeling
process, we achieve substantial agreement in gender labeling for the EuroCity-
Day dataset and almost perfect agreement for all other tasks. This high level
of agreement underscores the reliability of our labeling procedure (Wen et
al., 2021; Chen et al., 2021). In cases where the two annotators encounter
conflicts, an arbitrator is involved in the discussion to reach a consensus.
This is a standard procedure in empirical SE studies (Chen et al., 2021; Wu et
al., 2022; Wen et al., 2021). After the labeling process, the summary of the
number of labeled pedestrian instances for each dataset is presented in Table
4.

Table 3. Cohen’s $\kappa$ values for labeling age and gender attributes. Dataset | $\kappa_{age}$ | $\kappa_{gender}$
---|---|---
CityPersons | 0.847 | 0.814
EuroCity-Day | 0.925 | 0.800
EuroCity-Night | 0.847 | 0.870
BDD100k | 0.828 | 0.854
Table 4. Number of labeled pedestrian instances per dataset. | Gender | Age | Skin
---|---|---|---
Dataset | Male | Female | Adult | Child | LS | DS
CityPersons | 2,357 | 1,822 | 4,568 | 233 | - | -
EuroCity-Day | 1,726 | 1,646 | 4,498 | 100 | - | -
EuroCity-Night | 1,265 | 1,318 | 4,165 | 68 | - | -
BDD100k | 3,457 | 2,479 | 6,293 | 190 | 2,724 | 789
Overall | 8,805 | 7,265 | 19,524 | 591 | 2,724 | 789

####  3.3.3. Scenario Labeling

To deeply explore the fairness of pedestrian detectors across various driving
scenarios (i.e., different contrast, brightness, and weather conditions), we
also need images labeled with scenario information. We therefore classify the
images containing labeled pedestrians (5,917 out of the total 8,311 images in
all datasets) into different scenarios, to enable our analysis in RQ2. An
alternative approach to achieving this purpose is via generating images with
different scenarios using existing test generation techniques from the
autonomous driving literature. We do not choose this approach because
generated images are not real images and can suffer from unnaturalness (Zhang
et al., 2022).

Labeling brightness. Brightness represents the overall lightness or darkness
of the image. To distinguish the brightness of the images, we adopt two
methods. Firstly, we categorize the datasets into day time and night time. As
seen in Table 2, images in the CityPersons and EuroCity-Day datasets belong to
day time, while images in the EuroCity-Night dataset belong to night time. For
the BDD100k dataset, Wilson et al. (Wilson et al., 2019) have publicly
provided the labels for day time and night time. We directly use the label
information for study.

Secondly, we divide the images into five brightness levels to achieve a more
refined granularity. For this purpose, we employ the HSV measurement, a
commonly-studied metric for evaluating the brightness of pixels in images
(Omori and Shima, 2019; Foley et al., 1996; Cheng et al., 2001; Sural et al.,
2002; Kim et al., 2023). Specifically, for each image, we utilize the OpenCV
library (OpenCV, nd) to perform a linear transformation of the RGB values of
each pixel into the HSV color space format, which comprises three dimensions.
Among these dimensions, one specifically represents the brightness. We
calculate the mean value of the brightness dimension for all pixels in a image
and use this mean value to represent the overall brightness of the image. A
higher brightness value indicates a greater level of brightness. To classify
the images into different brightness levels, we identify the maximum
brightness value (which is 166) and the minimum brightness value (which is 10)
among all images. Then, we evenly divide this brightness range into five
classes, each covering an interval of 31.2 units (calculated as (166-10)/5).
Each class represents a brightness level, ranging from level 1 to level 5.
Higher levels represent higher brightness. Then we divide the images into
different levels based on their brightness values.

Labeling contrast. Contrast is the difference in brightness between objects in
an image. To quantify the contrast of each image, we use the RMS contrast
measurement (Peli, 1990), a standard measure in the computer vision literature
(Bhuiyan and Khan, 2018; Jakhetiya et al., 2018; Pelli and Bex, 2013). To
apply the RMS measurement, we first need to convert all images into the
grayscale mode (Peli, 1990). Then, we calculate the RMS contrast value for
each image based on the converted version. A higher RMS contrast value
indicates a greater contrast. Following a similar approach used for brightness
classification, we partition the range of RMS contrast values into five
classes, each class featuring equal value intervals. These classes correspond
to distinct levels, labeled from level 1 to level 5, with higher levels
indicating images with higher contrast. Then, we categorize the images into
their respective contrast levels based on their RMS contrast values.

Labeling weather conditions. Common weather conditions studied in the
autonomous driving literature include rain, fog, and snow (Tian et al., 2018;
Zhou et al., 2020; Zhang et al., 2018; Pei et al., 2019). However, our
datasets rarely contain images depicting fog and snow. This is due to the fact
that our datasets are collected from real-world scenarios where fog and snow
are infrequently encountered. The limited samples of snowy or foggy weather
pose challenges for statistical analysis. As a result, we focus on rain as the
weather condition of interest. Two annotators independently classify images
containing labeled pedestrians into two categories: rainy and non-rainy.
During this process, 1,840 images are not annotated because the two annotators
and the arbitrator all cannot accurately distinguish the weather conditions.
To measure inter-rater agreement during this independent labeling, we also use
Cohen’s Kappa ($\kappa$). The obtained $\kappa$ value is 0.813, indicating
almost perfect agreement. This high level of agreement confirms the
reliability of our labeling procedure. After the scenario labeling process,
the summary of the number of images under different brightness, contrast, and
weather conditions is presented in Table 5.

Table 5. Number of images in different brightness levels, contrast levels, and weather conditions. Brightness levels | Contrast levels | Weather conditions
---|---|---
Day-time | Night-time | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 | Rainy | Non-Rainy
4400 | 1517 | 189 | 1128 | 3166 | 1318 | 116 | 246 | 1523 | 2383 | 1667 | 98 | 277 | 3800

###  3.4. Evaluation Measures

There have been well-established quantitative measures for group fairness in
the literature. The most widely-adopted fairness measures include SPD
(Statistical Parity Difference), EOD (Equal Opportunity Difference), and AOD
(Average Odds Difference) (Chen et al., 2022b, a; Zhang and Harman, 2021). SPD
quantifies the difference in probabilities of favorable outcomes between
unprivileged and privileged groups; EOD indicates the difference in the true-
positive rate between unprivileged and privileged groups; AOD calculates the
average difference between the false-positive rate and true-positive rate for
unprivileged and privileged groups.

In the context of pedestrian detection, both SPD and EOD are measures that
represent the disparity in proportions of successfully detected pedestrians
between privileged and unprivileged groups. In other words, SPD and EOD also
both express the difference in miss rates between privileged and unprivileged
groups. Miss Rate (MR) is the most commonly-studied performance metric in
pedestrian detection research (Brandao, 2019; Kogure et al., 2022), which
quantifies the proportion of undetected pedestrians. Formally, it is
calculated as follows:

(1) $MR=1-\frac{TP}{TP+FN},$

where TP (true positives) refers to the number of successfully detected
ground-truth bounding boxes, and FN (false negatives) denotes the number of
undetected ground-truth bounding boxes. Pedestrian detectors generate bounding
box locations and confidence scores for recognized “person” instances in
images. To assess whether a given ground-truth bounding box is successfully
detected, the standard method in the literature is to use the Intersection
over Union (IoU) metric (Dollár et al., 2012). The IoU metric quantifies the
degree of overlap between the ground-truth bounding box and the detected
bounding box. If the IoU value is greater than 50%, the ground-truth bounding
box is considered successfully detected (Dollár et al., 2012). Otherwise, it
is classified as undetected.

The calculation of AOD requires precise false-positive information, referring
to instances where members of the negative class (non-pedestrians) are
incorrectly classified as the positive class (pedestrians). As described in
Section 3.3.2, we adhere to the standard practice in the literature, where we
focus on a “reasonable subset” of pedestrians within the images. This approach
presents challenges in calculating precise false-positives for each group
because the negative class (non-pedestrian) may also include instances that
belong to the positive class (pedestrian). Therefore, we do not consider AOD
in this study.

In summary, we use both SPD and EOD as fairness measures for evaluating
pedestrian detectors. However, since these two measures yield identical values
for pedestrian detection, for the remainder of the paper, we present only EOD.

###  3.5. Statistical Analysis

To assess the extent to which any observed unfairness is statistically
significant (i.e., whether there is a significant difference in the miss rate
between privileged and unprivileged groups) we use the two-proportion z-test
(Casella and Berger, 2007). This statistical test is widely adopted in SE
studies (Niu et al., 2016; Veizaga et al., 2021) to analyze differences
between two proportions. A result is deemed significant only if the obtained
$p$-value falls below a predetermined threshold (in our case, 0.05, a widely-
accepted threshold in the literature (Chen et al., 2022b, 2023)). For
instance, in evaluating whether there exists a difference between the miss
rates for male ($MR_{male}$) and female ($MR_{female}$) individuals detected
by a pedestrian detector, the null hypothesis assumes that $MR_{male}$ is
equal to $MR_{female}$. If the resulting $p$-value is less than 0.05, we
reject the null hypothesis, indicating a significant difference between
$MR_{male}$ and $MR_{female}$.

##  4\. RESULTS

This section answers our RQs based on experimental results.

###  4.1. RQ1: Overall Fairness

RQ1 investigates the overall fairness of eight state-of-the-art pedestrian
detectors with respect to gender, age, and skin tone. First, for each
detector, we compute the miss rate (MR) for different demographic groups and
calculate EOD based on the MR results over all the datasets that we study. We
also use the two-proportion z-test to determine the significance of any
observed unfairness (as described in Section 3.5). Table 6 presents the
results, with significant unfairness results highlighted in shading. Next, we
analyze the results for gender, age, and skin tone, respectively.

Table 6. RQ1: Overall fairness with respect to gender, age and skin tone. Statistically significant unfairness (i.e., MR difference) is highlighted in shading. On average, pedestrian detectors exhibit similar miss rates for female and male pedestrians, with a difference of 1.1%. However, significant unfairness is observed with respect to age, where pedestrian detectors demonstrate 19.67% higher miss rates for children compared to adults. Additionally, a substantial unfairness is evident concerning skin tone, with pedestrian detectors showing a miss rate 7.52% higher for the dark-skin group than the light-skin group on average. Detectors | MR Male | MR Female | EOD (Gender) | MR Adult | MR Child | EOD (Age) | MR Light-skin | MR Dark-skin | EOD (Skin tone)
---|---|---|---|---|---|---|---|---|---
YOLOX | 22.93% | 23.43% | -0.50% | 26.90% | 52.96% | -26.06% | 8.08% | 38.78% | -30.71%
RetinaNet | 17.67% | 18.49% | -0.81% | 20.93% | 48.90% | -27.97% | 10.76% | 38.78% | -28.03%
Faster RCNN | 4.97% | 5.51% | -0.53% | 6.92% | 28.26% | -21.34% | 6.72% | 4.31% | 2.41%
Cascade RCNN | 5.79% | 6.43% | -0.64% | 7.88% | 30.80% | -22.92% | 6.35% | 3.68% | 2.68%
ALFNet | 30.86% | 32.90% | -2.04% | 36.62% | 53.47% | -16.85% | 42.55% | 43.98% | -1.43%
CSP | 33.49% | 35.25% | -1.76% | 37.74% | 50.42% | -12.68% | 61.38% | 64.77% | -3.39%
MGAN | 29.76% | 30.97% | -1.21% | 33.58% | 46.53% | -12.95% | 52.94% | 54.88% | -1.94%
PRNet | 40.28% | 41.62% | -1.34% | 44.69% | 61.25% | -16.56% | 59.65% | 59.44% | 0.21%
Average | 23.22% | 24.32% | -1.10% | 26.91% | 46.57% | -19.67% | 31.05% | 38.58% | -7.52%

Gender. As shown in Table 6, on average, the miss rate difference between
female and male pedestrians is merely 1.1%, with $p$-value $\textgreater$
0.05, indicating that one cannot reject the null hypothesis and cannot assume
that this difference is statistically significant.

Then, we explore the difference across the four datasets used in our study.
Figure 1 shows the miss rates for male and female pedestrians in each dataset,
and significant gender unfairness is indicated by labeled miss rate values. In
the CityPersons and EuroCity-Day datasets, which exclusively contain day-time
data, there are no significant differences in the miss rate between females
and males. However, the results on the EuroCity-Night dataset present a
contrasting observation, where all detectors demonstrate a higher miss rate
for females, thus illustrating bias in detecting females. In the BDD100k
dataset that includes both day-time and night-time images, the performance
difference is less pronounced. These observations motivate us to hypothesize
that brightness conditions may influence the fairness of pedestrian detectors,
which is further investigated in RQ2.

Figure 1. RQ1: Miss rates for female and male pedestrians in each dataset.
Statistically significant gender unfairness (i.e., miss rate difference) is
indicated by labeled miss rate values. We find that pedestrian detectors tend
to have poorer performance in detecting females compared to males within
night-time images (e.g., EuroCity-Night dataset).

Age. As observed in Table 6, the pedestrian detectors that we study exhibit
large unfairness regarding age, with all detectors demonstrating significantly
higher miss rates for children compared to adults (with $p$-value $\textless$
0.05). On average, the miss rate difference between children and adults
amounts to 19.67%.

Furthermore, we illustrate the miss rate difference across the four datasets
that we use in Figure 2, with significant gender unfairness indicated by
labeled miss rate values. From the figure, we observe that the miss rate of
children is consistently higher than that of adults for all the datasets and
all detectors. In particular, out of the total 32 results (combinations of
four datasets and eight pedestrian detection models), 30 exhibit a
statistically significant difference in the miss rate with $p$-value ¡ 0.05.
This indicates that the bias favoring adults is not specific to a particular
dataset or model, highlighting a strong unfairness between adults and
children.

Figure 2. RQ1: Miss rates for children and adults in each dataset.
Statistically significant age unfairness (i.e., miss rate difference) is
indicated by labeled miss rate values. The miss rate of children is
consistently higher than that of adults for all datasets and all detectors.

The age bias can be attributed to the inherent challenge of detecting small
objects in object and person detection research, owing to the limited
information provided by small bounding box sizes (Zhang et al., 2016; Dollar
et al., 2009; Kogure et al., 2022; Cao et al., 2020). Given that children
generally have smaller bodies compared to adults, their bounding boxes in the
images also tend to be smaller. To verify this, we analyze the distribution of
bounding box sizes for pedestrians detected and undetected by all pedestrian
detectors, as well as the distribution of ground-truth bounding box sizes for
both adults and children. The results, presented in Figure 3, reveal a
correspondence between the two distributions. Children as well as undetected
pedestrians tend to have smaller bounding boxes.

Figure 3. Bounding boxes distribution of adults and children (left) and
bounding boxes distribution of undetected and detected pedestrians (right).
Both children and undetected pedestrians tend to have smaller bounding boxes.

Skin tone. As shown in Table 6, current pedestrian detectors exhibit a notable
bias towards the dark-skin group, with an average miss rate difference of
7.52% between the dark-skin and light-skin groups. Among the eight detectors
examined, five detectors demonstrate higher miss rates for the dark-skin group
compared to the light-skin group (with four of them being statistically
significant). Furthermore, two detectors display significant miss rate
disparities of 30.71% and 28.03%, signifying substantial fairness challenges
related to skin tone.

Ans. to  RQ1: On average, the pedestrian detectors under study exhibit no
significant performance disparity across gender, but notable disparity across
age and skin tone. In particular, the miss rate difference between children
and adults is as high as 19.67%; the miss rate difference between dark-skin
pedestrians and light-skin pedestrians is 7.52%. These observations highlight
significantly higher risks for children and dark-skin pedestrians with the
existing autonomous driving systems.

###  4.2. RQ2: Fairness in Different Scenarios

RQ2 investigates the fairness of state-of-the-art pedestrian detectors under
different real-world autonomous driving scenarios.

####  4.2.1. RQ2.1: Different brightness conditions.

As introduced in Section 3.3.3, we consider brightness conditions from two
perspectives. Next, we analyze the results for the two perspectives,
respectively.

Day time and Night time. We first evaluate the miss rates of the eight
pedestrian detectors under day time and night time. The results, presented in
Table 7, indicate a noticeable increase in average miss rates during night
time compared to day time. For example, the average miss rates for males and
females at night are 36.55% and 40.05%, respectively, compared to 20.15% and
20.03% during the day. A similar pattern is observed for age and skin tone.
This indicates that the transition from day time to night time influences the
performance of pedestrian detectors, with statistically significant higher
miss rates observed at night.

Table 7. RQ2.1: MR and EOD of each pedestrian detector under day time and
night time. The statistically significant unfairness results are highlighted
in shading. As the time shifts from day to night, the EOD between children and
adults and between dark-skin and light-skin groups increases (i.e., from
-22.05% to -26.63% for age and from -7.14% to -9.68% for skin tone). The
female group exhibits a higher miss rate at night.

Detectors | Gender | Age | Skin Tone (LS: Light Skin, DS: Dark Skin)
---|---|---|---
Day-time | Night-time | Day-time | Night-time | Day-time | Night-time
MR Male | MR Female | EOD | MR Male | MR Female | EOD | MR Adult | MR Child | EOD | MR Adult | MR Child | EOD | MR LS | MR DS | EOD | MR LS | MR DS | EOD
YOLOX | 22.71% | 22.00% | 0.71% | 24.59% | 28.97% | -4.38% | 24.23% | 50.00% | -25.77% | 34.83% | 76.06% | -41.23% | 7.33% | 35.47% | -28.14% | 11.65% | 57.76% | -46.11%
RetinaNet | 17.18% | 17.37% | -0.19% | 20.39% | 22.90% | -2.51% | 18.63% | 46.35% | -27.72% | 27.75% | 69.01% | -41.26% | 9.75% | 35.17% | -25.43% | 15.95% | 59.48% | -43.53%
Faster RCNN | 4.93% | 4.86% | 0.07% | 6.03% | 8.29% | -2.26% | 5.97% | 26.73% | -20.76% | 10.04% | 40.85% | -30.80% | 5.69% | 3.58% | 2.12% | 9.62% | 4.31% | 5.31%
Cascade RCNN | 5.58% | 5.68% | -0.11% | 7.57% | 9.55% | -1.99% | 6.76% | 29.62% | -22.85% | 11.43% | 40.85% | -29.41% | 5.91% | 3.87% | 2.03% | 10.89% | 6.90% | 3.99%
ALFNet | 24.60% | 24.45% | 0.15% | 57.33% | 63.44% | -6.11% | 26.56% | 49.62% | -23.05% | 65.13% | 83.10% | -17.97% | 37.39% | 38.75% | -1.36% | 72.41% | 73.28% | -0.87%
CSP | 26.85% | 26.43% | 0.42% | 61.41% | 67.05% | -5.64% | 28.69% | 47.12% | -18.42% | 63.37% | 76.06% | -12.69% | 56.19% | 59.91% | -3.72% | 91.14% | 92.24% | -1.10%
MGAN | 25.57% | 26.35% | -0.78% | 47.52% | 47.75% | -0.24% | 27.92% | 43.65% | -15.74% | 49.70% | 69.01% | -19.32% | 47.35% | 49.78% | -2.43% | 85.06% | 83.62% | 1.44%
PRNet | 33.78% | 33.09% | 0.69% | 67.55% | 72.42% | -4.87% | 34.82% | 56.92% | -22.10% | 72.62% | 92.96% | -20.34% | 54.46% | 54.69% | -0.23% | 89.62% | 86.21% | 3.41%
Average | 20.15% | 20.03% | 0.12% | 36.55% | 40.05% | -3.50% | 21.70% | 43.75% | -22.05% | 41.86% | 68.49% | -26.63% | 28.01% | 35.15% | -7.14% | 48.29% | 57.97% | -9.68%

Then, we investigate whether the performance is equally decreased for
different demographic groups. Specifically, we explore the fairness change
from day time to night time. Table 7 shows the results, with statistically
significant unfairness (i.e., miss rate difference) emphasized in shading.

Regarding gender, we observe a shift in the miss rate difference between males
and females from day time to night time. During the day, there is only a
slight 0.12% difference for males and females. However, during night time, the
difference increases to -3.5%, indicating a notable change in fairness. In the
night-time condition, all pedestrian detectors exhibit higher miss rates for
females compared to males, with six of them showing statistically significant
differences.

For age, the miss rate difference for children and adults increases from day
time to night time, with the average difference increasing from 22.05% during
the day to 26.63% at night. This suggests a higher probability of children
being undetected during night time conditions.

For skin tone, the miss rate difference between dark-skin and light-skin
groups increases from day time to night time, with the average difference
increasing from 7.14% at day time to 9.68% at night time. This implies
increased detection difficulty for the dark-skin group at night.

Five brightness levels. To offer a more fine grained perspective, we partition
the images into five brightness levels, as described in Section 3.3.3. After
partitioning the data, we first examine the average miss rate across eight
pedestrian detectors under five brightness levels, as presented in Figure 4
(with a darker line indicating a lower brightness level). Consistent with the
previous findings regarding day time and night time performance, we observe
that pedestrian detectors exhibit their worst performance at the darkest level
1 according to the first three figures, which indicates that all the
pedestrian detectors perform worse in low brightness conditions.

Figure 4. RQ2.1: Average MR of eight detectors and EOD trends under different
brightness levels. The three figures above indicate that lower brightness
levels lead to an increased average miss rate for each attribute. In the three
figures below, explicit bias is observed in YOLOX and RetinaNet for the dark-
skin group as the brightness level decreases. Furthermore, there exists an
increasing EOD at brightness level 2 and level 1, putting females at a
disadvantage by making them more likely to be undetected.

Then, we examine how the fairness changes with decreasing brightness levels
for each sensitive attribute. Figure 4 illustrates the EOD changes under five
brightness levels.

For gender, we observe a notable spike in the EOD change at level 2 and level
1, which represent the two lowest brightness levels. This indicates that as
the brightness level decreases, the overall miss rate increases and pedestrian
detectors tend to perform worse at detecting females.

However, for age, there is no apparent pattern in EOD changes for each
pedestrian detector under specific brightness levels. This may be attributed
to the detectors’ limitations in detecting small-scale pedestrians, which
outweigh the impact of brightness, resulting in less noticeable EOD
variations.

For skin tone, we find that different types of detectors exhibit different
performance under five brightness levels. Specifically, one-stage detectors
(i.e., YOLOX and RetinaNet), display an increase in the miss rate difference
for dark-skin individuals as the brightness decreases.

Ans. to  RQ2.1:  Lower brightness conditions can amplify the bias towards
females, children, and dark-skin individuals in pedestrian detection.
Specifically, for gender, we observe a shift in the miss rate difference from
0.12% during the day to 3.5% at night. For age, the miss rate difference
between children and adults increases from 22.05% during the day to 26.63% at
night. Additionally, for skin tone, the miss rate difference between the dark-
skin and light-skin groups undergoes a change from 7.14% during the day to
9.68% at night. Moreover, the miss rate difference between the light-skin and
dark-skin groups increases as the brightness level decreases, especially for
one-stage pedestrian detectors.

####  4.2.2. RQ2.2: Different contrast levels.

Figure 5. RQ2.2: Average MR of eight pedestrian detectors (the above three
figures) and EOD trends under different contrast levels (the below three
figures). The EOD trend in detecting various skin tone groups remains
consistent across all pedestrian detectors, suggesting a greater challenge in
identifying dark-skin individuals as contrast levels decline. Similarly, there
still exists an increasing EOD trend when decreasing the contrast level.

Following the road map of RQ2, we first compare the average miss rates of all
pedestrian detectors at different contrast levels, as shown in Figure 5. For
skin tone, we observe that the average miss rate increases at lower contrast
levels (level 1 and level 2) compared to the other three levels. However, for
gender and age, the average miss rate of all pedestrian detectors does not
show an obvious variation across different contrast levels.

Then, we investigate the fairness changes, as indicated by the EOD trends in
Figure 5. For gender, an increasing EOD trend is observed. As the contrast
level decreases, pedestrian detectors gradually exhibit a higher miss rate
difference between females and males, especially in the lowest contrast level.
This observation indicates that low contrast conditions negatively impact the
detection of females, thereby contributing another factor to the bias and
unfairness in pedestrian detection.

For age, the EOD between children and adults remains large across all five
contrast levels. Certain pedestrian-specific detectors such as ALFNet, CSP,
and PRNet demonstrate higher miss rate differences at the highest contrast
level. However, no specific pattern in the EOD across the five contrast
conditions is observed. This suggests that contrast levels do not
significantly impact the detection performance difference between adults and
children.

For skin tone, as the contrast level decreases, all pedestrian detectors
display greater difficulty in detecting the dark-skin group. Similarly, one-
stage pedestrian detectors especially show an increased miss rate difference
(with a larger EOD) between light-skin and dark-skin groups. These results
demonstrate that contrast conditions also have a substantial impact on the
detection of different gender and skin tone groups.

Ans. to  RQ2.2:  Low-contrast conditions have a negative influence on the
fairness of existing pedestrian detectors regarding gender and skin tone. As
the contrast level decreases, the miss rate differences between dark-skin and
light-skin groups and between female and male groups increase, posing a
greater challenge in detecting dark-skin and female pedestrians.

####  4.2.3. RQ2.3: Different weather conditions.

Table 8 presents the miss rates of the eight pedestrian detectors under non-
rainy and rainy weather conditions, with statistically significant unfairness
results shaded.

Table 8. RQ2.3: MR and EOD of each pedestrian detector under non-rainy and
rainy weather conditions. The statistically significant unfairness results are
shaded. The disparities between light-skin and dark-skin groups, as well as
between males and females, increase when shifting to rainy weather. However,
these disparities are not statistically significant.

Detectors | Gender | Age | Skin Tone (LS: Light Skin, DS: Dark Skin)
---|---|---|---
Dry-weather | Rainy-weather | Dry-weather | Rainy-weather | Dry-weather | Rainy-weather
MR Male | MR Female | EOD | MR Male | MR Female | EOD | MR Adult | MR Child | EOD | MR Adult | MR Child | EOD | MR LS | MR DS | EOD | MR LS | MR DS | EOD
YOLOX | 27.50% | 26.85% | 0.65% | 33.70% | 24.61% | 9.09% | 28.30% | 52.72% | -24.42% | 30.00% | 50.00% | -20.00% | 6.50% | 33.68% | -27.18% | 19.40% | 60.53% | -41.12%
RetinaNet | 21.52% | 21.23% | 0.28% | 28.49% | 22.27% | 6.23% | 22.33% | 48.37% | -26.04% | 27.12% | 50.00% | -22.88% | 8.77% | 33.16% | -24.39% | 26.12% | 63.16% | -37.04%
Faster RCNN | 7.03% | 6.53% | 0.49% | 6.03% | 7.42% | -1.39% | 7.60% | 28.76% | -21.16% | 6.97% | 25.00% | -18.03% | 4.74% | 2.95% | 1.78% | 16.42% | 13.16% | 3.26%
Cascade RCNN | 8.05% | 7.51% | 0.55% | 7.12% | 7.03% | 0.09% | 8.53% | 30.94% | -22.41% | 7.58% | 25.00% | -17.42% | 5.04% | 2.78% | 2.26% | 17.91% | 15.79% | 2.12%
ALFNet | 27.22% | 24.48% | 2.74% | 48.49% | 41.80% | 6.70% | 26.37% | 50.11% | -23.74% | 46.36% | 75.00% | -28.64% | 34.66% | 36.98% | -2.32% | 64.93% | 63.16% | 1.77%
CSP | 27.31% | 23.92% | 3.39% | 47.40% | 41.80% | 5.60% | 26.82% | 45.53% | -18.71% | 48.03% | 65.00% | -16.97% | 52.90% | 57.12% | -4.22% | 76.87% | 86.84% | -9.98%
MGAN | 27.15% | 25.62% | 1.53% | 48.77% | 48.05% | 0.72% | 26.71% | 44.01% | -17.29% | 50.30% | 70.00% | -19.70% | 44.48% | 47.92% | -3.43% | 71.64% | 76.32% | -4.67%
PRNet | 35.35% | 32.14% | 3.21% | 53.42% | 50.39% | 3.03% | 34.18% | 55.99% | -21.81% | 54.39% | 75.00% | -20.61% | 51.59% | 52.95% | -1.36% | 77.61% | 78.95% | -1.34%
Overall | 22.64% | 21.03% | 1.61% | 34.18% | 30.42% | 3.76% | 22.61% | 44.55% | -21.95% | 33.84% | 54.38% | -20.53% | 26.08% | 33.44% | -7.36% | 46.36% | 57.24% | -10.87%

In terms of the overall detection performance of all pedestrian detectors, the
miss rate for each demographic group increases in the rainy weather condition,
possibly due to droplets covering the camera and disrupting the detectors.
Regarding the fairness, we find that the miss rate difference increases
between the dark-skin group and light-skin group (with EOD from -7.36% to
-10.87%) and between males and females (with EOD from 1.63% to 3.76%),
indicating a higher miss rate for the dark-skin group and males from non-rainy
to rainy conditions. However, for these increases, we cannot observe a
statistically significant difference. As for children and adults, the miss
rates difference experiences only a slight decrease, with EOD moving from
-21.95% to -20.53%. Therefore, it can be concluded that rainy weather
conditions have a minor impact on the fairness of pedestrian detectors.

Ans. to  RQ2.3:  Rainy weather conditions have a minor impact on the fairness
of pedestrian detectors. Although we observe an increase in the miss rate
difference between dark-skin and light-skin groups (from 7.36% to 10.87%) and
between males and females (from 1.63% to 3.3%), these increases are not
statistically significant.

##  5\. Threats to Validity

This section discusses the potential threats to the validity of our results.

Manual labeling. The labeling process involves possible subjectivity, posing
threats to the validity of the analysis. To mitigate this threat, two
annotators and an arbitrator are involved in the labeling process: first each
image is independently labeled by two annotators, and in cases where conflicts
in labeling arise, we seek the expertise of an arbitrator to resolve such
discrepancies and arrive at a consensus. The inter-rater agreement between the
two annotators is high, which demonstrates the reliability of the labeling
schema and procedure adopted herein.

Selection of pedestrian detection models. Our study is based on eight
pedestrian detectors, which may lead to possible selection bias in this study.
To mitigate this threat, we select representative pedestrian detectors based
on two considerations. On one hand, the selected detectors are widely-studied
in pedestrian detection and autonomous driving literature (Cao et al., 2020;
Zhang et al., 2016, 2017; Yu et al., 2020). On the other hand, the selected
models cover the two typical types of pedestrian detection methods (i.e., one-
stage detection and two-stage detection), ensuring a comprehensive
representation of the techniques used in the field.

Selection of evaluation measures.  The fairness measures that we employ may
introduce potential limitations. To mitigate this concern, we consider three
commonly-used fairness measures: SPD, EOD, and AOD. Upon careful discussion,
we conclude that AOD may not be suitable for our study (Section 3.4), leading
us to focus on SPD and EOD. We demonstrate that SPD and EOD lead to equivalent
observations for pedestrian detection. Ultimately, we opt to use EOD (i.e.,
SPD) for our analysis, as it involves the comparison of the miss rates of
different demographic groups. This ensures a consistent evaluation with the
current pedestrian detection research, as miss rate is the most widely-adopted
metric for measuring the performance of pedestrian detectors in the
literature.

##  6\. Implications

The findings derived from our analysis carry significant implications for
researchers, software engineers, and policy makers.

Implication for researchers. The significant unfairness exhibited by existing
pedestrian detectors offers valuable implications for researchers to further
focus their effort on fairness testing and improvement for autonomous driving
systems. Our findings demonstrate that brightness and contrast conditions can
significantly influence detection performance regarding skin tone and gender.
Thus, to improve fairness, one practical solution is to design specific
techniques to adjust contrast and brightness of input images, such as
increasing brightness and contrast levels to counterbalance existing bias
towards dark-skin and female pedestrians in low-brightness and low-contrast
conditions. In addition, our findings highlight that one-stage detectors
display significant discrimination against particularly dark-skin pedestrians.
This suggests that mitigating skin tone bias requires careful consideration of
detector types. For age bias, it is important to address the inherent
limitations of pedestrian detectors, particularly their difficulty in
accurately detecting small pedestrians. These implications underline the
importance of factoring these elements into effective fairness improvement
strategies.

Implication for software engineers. Fairness is a critical non-functional
requirement for software systems, but our study demonstrates the existence of
significant bias in state-of-the-art pedestrian detectors. Nevertheless, there
is a notable knowledge gap in detecting and addressing fairness issues in
autonomous driving systems. This emphasizes that it is imperative for software
engineers to prioritize their efforts in this domain. Addressing these issues
is not just a matter of improving autonomous driving system performance, it
also help software companies avoid potential legal implications linked to
anti-discrimination laws.

Implication for policy makers. Autonomous driving systems play a crucial role
in ensuring human safety. Our results reveal that fairness issues exist in
modern pedestrian detection models. Therefore, it is essential for policy
makers to enact laws and regulations that safeguard the rights of all
individuals and address these concerns appropriately. Policies should also aim
to address safety issues for vulnerable road users such as children and dark-
skin individuals, who are, as revealed by this study, more likely to be
overlooked by the current detection models.

##  7\. Conclusion

This paper presents the first comprehensive study on fairness testing of eight
state-of-the-art pedestrian detectors, using four widely-studied testing
datasets. We investigate the fairness aspects of these detectors with respect
to gender, age, and skin tone. Our findings reveal significant bias in the
current pedestrian detectors, particularly towards children and individuals
with darker skin tones. However, we observe balanced detection performance for
male and female individuals. Furthermore, we conduct an in-depth analysis of
fairness in various driving scenarios. Our investigation demonstrates that the
detection performance for dark-skin individuals is further reduced under low-
brightness and low-contrast conditions, exacerbating the unfairness related to
skin tone. As part of our contribution, we publicly release large-scale real-
world pedestrian detection datasets with gender, age, and skin tone labels.
These datasets aim to facilitate future fairness research in autonomous
driving. Overall, this study sheds light on the fairness issues faced by
existing pedestrian detectors, emphasizing the importance of addressing bias
related to age and skin tone. The insights gained can pave the way for more
fair and unbiased autonomous driving systems in the future.

## Data Availability

Our GitHub repository (git, 2023) contains datasets, sensitive attribute
labels, scripts, and results of this work to facilitate replication and
extension.

## References

  * (1)
  * cas (2018) 2018\.  Self-Driving Uber Car Kills Pedestrian in Arizona, Where Robots Roam.  https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html.
  * git (2023) 2023\.  Replication package.  https://github.com/FairnessResearch/Fairness-Testing-of-Autonomous-Driving-Systems.
  * Barocas and Selbst (2016) Solon Barocas and Andrew D. Selbst. 2016.  Big Data’s Disparate Impact.  _California Law Review_ 104 (2016), 671.
  * Bhuiyan and Khan (2018) Md. Al-Amin Bhuiyan and Abdul Raouf Khan. 2018.  Image quality assessment employing RMS contrast and histogram similarity.  _Int. Arab J. Inf. Technol._ 15, 6 (2018), 983–989.  http://iajit.org/index.php?option=com_content&task=blogcategory&id=134&Itemid=461
  * Biswas and Rajan (2020) Sumon Biswas and Hridesh Rajan. 2020.  Do the machine learning models on a crowd sourced platform exhibit bias? An empirical study on model fairness. In _Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020_. 642–653.
  * Biswas and Rajan (2021) Sumon Biswas and Hridesh Rajan. 2021.  Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning Pipeline. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Athens, Greece) _(ESEC/FSE 2021)_. Association for Computing Machinery, New York, NY, USA, 981–993.  https://doi.org/10.1145/3468264.3468536
  * Brandao (2019) Martim Brandao. 2019\.  Age and gender bias in pedestrian detection algorithms.  _ArXiv_ abs/1906.10490 (2019).
  * Braun et al. (2019) Markus Braun, Sebastian Krebs, Fabian Flohr, and Dariu M. Gavrila. 2019.  EuroCity Persons: A Novel Benchmark for Person Detection in Traffic Scenes.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 41, 8 (2019), 1844–1861.  https://doi.org/10.1109/TPAMI.2019.2897684
  * Cai and Vasconcelos (2018) Zhaowei Cai and Nuno Vasconcelos. 2018.  Cascade r-cnn: Delving into high quality object detection. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6154–6162.
  * Cao et al. (2020) Jiale Cao, Yanwei Pang, Jin Xie, Fahad Shahbaz Khan, and Ling Shao. 2020.  From Handcrafted to Deep Features for Pedestrian Detection: A Survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44 (2020), 4913–4934.
  * Casella and Berger (2007) G. Casella and R.L. Berger. 2007.  _Statistical Inference_ (2 ed.).  Duxbury Press.
  * Chakraborty et al. (2021) Joymallya Chakraborty, Suvodeep Majumder, and Tim Menzies. 2021\.  Bias in machine learning software: why? how? what to do?  _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021_ (2021), 429–440.
  * Chakraborty et al. (2020) Joymallya Chakraborty, Suvodeep Majumder, Zhe Yu, and Tim Menzies. 2020.  Fairway: A Way to Build Fair ML Software.  https://doi.org/10.6084/m9.figshare.12363347
  * Chen et al. (2021) Zhenpeng Chen, Huihan Yao, Yiling Lou, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, and Xuanzhe Liu. 2021.  An empirical study on deployment faults of deep learning based mobile applications. In _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021_. 674–685.
  * Chen et al. (2022a) Zhenpeng Chen, Jie M. Zhang, Max Hort, Federica Sarro, and Mark Harman. 2022a.  Fairness Testing: A Comprehensive Survey and Analysis of Trends.  arXiv:2207.10223 [cs.SE]
  * Chen et al. (2022b) Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2022b.  MAAT: a novel ensemble approach to addressing fairness and performance bugs for machine learning software. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022_. 1122–1134.
  * Chen et al. (2023) Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2023.  A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers.  _ACM Trans. Softw. Eng. Methodol._ (feb 2023).  https://doi.org/10.1145/3583561 Just Accepted.
  * Cheng et al. (2001) Heng-Da Cheng, Xihua Jiang, Ying Sun, and Jingli Wang. 2001\.  Color image segmentation: advances and prospects.  _Pattern Recognit._ 34 (2001), 2259–2281.
  * Cohen (1960) Jacob Cohen. 1960\.  A Coefficient of Agreement for Nominal Scales.  _Educational and Psychological Measurement_ 20 (1960), 37 – 46.
  * Corbett-Davies and Goel (2018) Sam Corbett-Davies and Sharad Goel. 2018.  The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning.  _ArXiv_ abs/1808.00023 (2018).
  * Dollar et al. (2009) Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona. 2009.  Pedestrian detection: A benchmark. In _2009 IEEE Conference on Computer Vision and Pattern Recognition_. 304–311.  https://doi.org/10.1109/CVPR.2009.5206631
  * Dollár et al. (2012) Piotr Dollár, Christian Wojek, Bernt Schiele, and Pietro Perona. 2012.  Pedestrian Detection: An Evaluation of the State of the Art.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 34 (2012), 743–761.
  * Finkelstein et al. (2008) Anthony Finkelstein, Mark Harman, S. Afshin Mansouri, Jian Ren, and Yuanyuan Zhang. 2008.  “Fairness Analysis” in Requirements Assignments. In _Proceedings of the 16th IEEE International Requirements Engineering Conference, RE 2008_. 115–124.
  * Foley et al. (1996) James D Foley, Foley Dan Van, Andries Van Dam, Steven K Feiner, and John F Hughes. 1996.  _Computer graphics: principles and practice_. Vol. 12110.  Addison-Wesley Professional.
  * Galhotra et al. (2017) Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou. 2017\.  Fairness testing: testing software for discrimination. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017_. 498–510.
  * Garcia et al. (2020) Joshua Garcia, Yang Feng, Junjie Shen, Sumaya Almanee, Yuan Xia, and Qi Alfred Chen. 2020\.  A comprehensive study of autonomous vehicle bugs. In _Proceedings of the 42nd International Conference on Software Engineering, ICSE 2020_. 385–396.
  * Ge et al. (2021) Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. 2021.  Yolox: Exceeding yolo series in 2021.  _arXiv preprint arXiv:2107.08430_ (2021).
  * Gohar et al. (2023) Usman Gohar, Sumon Biswas, and Hridesh Rajan. 2023\.  Towards Understanding Fairness and its Composition in Ensemble Machine Learning. In _Proceedings of the 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023_. 1533–1545.
  * Hasan et al. (2020) Irtiza Hasan, Shengcai Liao, Jinpeng Li, Saad Ullah Akram, and Ling Shao. 2020.  Generalizable Pedestrian Detection: The Elephant In The Room.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2020), 11323–11332.
  * hasanirtiza (2021) hasanirtiza. 2021\.  _Pedestron: Pedestrian Detection in PyTorch_.  https://github.com/hasanirtiza/Pedestron
  * Hort et al. (2021) Max Hort, Jie M. Zhang, Federica Sarro, and Mark Harman. 2021\.  Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation Methods. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Athens, Greece) _(ESEC/FSE 2021)_. Association for Computing Machinery, New York, NY, USA, 994–1006.  https://doi.org/10.1145/3468264.3468565
  * Jakhetiya et al. (2018) Vinit Jakhetiya, Weisi Lin, Sunil Prasad Jaiswal, Ke Gu, and Sharath Chandra Guntuku. 2018.  Just Noticeable Difference for natural images using RMS contrast and feed-back mechanism.  _Neurocomputing_ 275 (2018), 366–376.  https://doi.org/10.1016/j.neucom.2017.08.031
  * Kim et al. (2023) Hyungjung Kim, Hyunsu Lee, Semin Ahn, Woo-Kyun Jung, and Sung-Hoon Ahn. 2023.  Broken stitch detection system for industrial sewing machines using HSV color space and image processing techniques.  _J. Comput. Des. Eng._ 10, 4 (2023), 1602–1614.
  * Kogure et al. (2022) Shunsuke Kogure, Kai Watabe, Ryosuke Yamada, Yoshimitsu Aoki, Akio Nakamura, and Hirokatsu Kataoka. 2022\.  Age Should Not Matter: Towards More Accurate Pedestrian Detection via Self-Training.  _AAAI Workshop on Artificial Intelligence with Biased or Scarce Data (AIBSD)_ (2022).
  * Landis and Koch (1977) J Richard Landis and Gary G. Koch. 1977.  The measurement of observer agreement for categorical data.  _Biometrics_ 33 1 (1977), 159–74.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017.  Focal loss for dense object detection. In _Proceedings of the IEEE international conference on computer vision_. 2980–2988.
  * Liu et al. (2016) Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016.  Ssd: Single shot multibox detector. In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14_. Springer, 21–37.
  * Liu et al. (2018) Wei Liu, Shengcai Liao, Weidong Hu, Xuezhi Liang, and Xiao Chen. 2018.  Learning efficient single-stage pedestrian detectors by asymptotic localization fitting. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 618–634.
  * Liu et al. (2019) Wei Liu, Shengcai Liao, Weiqiang Ren, Weidong Hu, and Yinan Yu. 2019.  High-level semantic feature detection: A new perspective for pedestrian detection. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 5187–5196.
  * liuwei16 (2018) liuwei16. 2018.  _ALFNet: Adaptive Light-weight Face Network_.  https://github.com/liuwei16/ALFNet
  * MMDetection Developers (nd) MMDetection Developers. n.d..  _MMDetection Model Zoo_.  https://mmdetection.readthedocs.io/en/stable/model_zoo.html
  * Niu et al. (2016) Nan Niu, Wentao Wang, and Arushi Gupta. 2016.  Gray links in the use of requirements traceability. In _Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016_. 384–395.
  * Omori and Shima (2019) Yuki Omori and Yoshihiro Shima. 2019.  HSV Color Space Based Lighting Detection for Brake Lamps of Daytime Vehicle Images.  _J. Comput._ 14, 1 (2019), 25–30.
  * OpenCV (nd) OpenCV. n.d..  _Colorspace Conversion - OpenCV 3.4 Documentation_.  https://docs.opencv.org/3.4/d8/d01/group__imgproc__color__conversions.html
  * Pang et al. (2019) Yanwei Pang, Jin Xie, Muhammad Haris Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, and Ling Shao. 2019\.  Mask-guided attention network for occluded pedestrian detection. In _Proceedings of the IEEE/CVF international conference on computer vision_. 4967–4975.
  * Pei et al. (2019) Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2019\.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems.  _Commun. ACM_ 62, 11 (oct 2019), 137–145.  https://doi.org/10.1145/3361566
  * Peli (1990) Eli Peli. 1990.  Contrast in complex images.  _J. Opt. Soc. Am. A_ 7, 10 (Oct 1990), 2032–2040.  https://doi.org/10.1364/JOSAA.7.002032
  * Pelli and Bex (2013) Denis G. Pelli and Peter Bex. 2013.  Measuring contrast sensitivity.  _Vision Research_ 90 (2013), 10–14.  https://doi.org/10.1016/j.visres.2013.04.015 Testing Vision: From Laboratory Psychophysical Tests to Clinical Evaluation.
  * Redmon et al. (2016) Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016.  You only look once: Unified, real-time object detection. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 779–788.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015\.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In _Advances in Neural Information Processing Systems_ , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf
  * Song et al. (2020) Xiaolin Song, Kaili Zhao, Wen-Sheng Chu, Honggang Zhang, and Jun Guo. 2020.  Progressive refinement network for occluded pedestrian detection. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16_. Springer, 32–48.
  * Soremekun et al. (2022) Ezekiel O. Soremekun, Mike Papadakis, Maxime Cordy, and Yves Le Traon. 2022.  Software fairness: An analysis and survey.  _CoRR_ abs/2205.08809 (2022).
  * Stocco et al. (2020) Andrea Stocco, Michael Weiss, Marco Calzana, and Paolo Tonella. 2020.  Misbehaviour prediction for autonomous driving systems. In _Proceedings of the 42nd International Conference on Software Engineering, ICSE 2020_. 359–371.
  * Sural et al. (2002) Shamik Sural, Gang Qian, and Sakti Pramanik. 2002.  Segmentation and histogram generation using the HSV color space for image retrieval. In _Proceedings of the 2002 International Conference on Image Processing, ICIP 2002, Rochester, New York, USA, September 22-25, 2002_. IEEE, 589–592.  https://doi.org/10.1109/ICIP.2002.1040019
  * sxlpris (2020) sxlpris. 2020.  _Progressive Refinement Network for Occluded Pedestrian Detection_.  https://github.com/sxlpris/PRNet
  * Tian et al. (2018) Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018\.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars. In _Proceedings of the 40th International Conference on Software Engineering_ (Gothenburg, Sweden) _(ICSE ’18)_. Association for Computing Machinery, New York, NY, USA, 303–314.  https://doi.org/10.1145/3180155.3180220
  * Veizaga et al. (2021) Alvaro Veizaga, Mauricio Alférez, Damiano Torre, Mehrdad Sabetzadeh, and Lionel C. Briand. 2021\.  On systematically building a controlled natural language for functional requirements.  _Empir. Softw. Eng._ 26, 4 (2021), 79.
  * Wen et al. (2021) Jinfeng Wen, Zhenpeng Chen, Yi Liu, Yiling Lou, Yun Ma, Gang Huang, Xin Jin, and Xuanzhe Liu. 2021.  An empirical study on challenges of application development in serverless computing. In _Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021_. 416–428.
  * Wick et al. (2019) Michael Wick, swetasudha panda, and Jean-Baptiste Tristan. 2019\.  Unlocking Fairness: a Trade-off Revisited. In _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2019/file/373e4c5d8edfa8b74fd4b6791d0cf6dc-Paper.pdf
  * Wilson et al. (2019) Benjamin Wilson, Judy Hoffman, and Jamie H. Morgenstern. 2019\.  Predictive Inequity in Object Detection.  _ArXiv_ abs/1902.11097 (2019).
  * Wu et al. (2022) Jianyu Wu, Hao He, Wenxin Xiao, Kai Gao, and Minghui Zhou. 2022.  Demystifying software release note issues on GitHub. In _Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, ICPC 2022_. 602–613.
  * Yu et al. (2020) Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. 2020\.  BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang and Harman (2021) Jie M. Zhang and Mark Harman. 2021.  ”Ignorance and Prejudice” in Software Fairness. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. 1436–1447.  https://doi.org/10.1109/ICSE43902.2021.00129
  * Zhang et al. (2022) Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2022\.  Machine Learning Testing: Survey, Landscapes and Horizons.  _IEEE Trans. Software Eng._ 48, 2 (2022), 1–36.
  * Zhang and Sun (2022) Mengdi Zhang and Jun Sun. 2022.  Adaptive fairness improvement based on causality analysis. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022_. 6–17.
  * Zhang et al. (2023) Mengdi Zhang, Jun Sun, Jingyi Wang, and Bing Sun. 2023\.  TESTSGD: Interpretable testing of neural networks against subtle group discrimination.  _ACM Transactions on Software Engineering and Methodology_ (2023).
  * Zhang et al. (2018) Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018.  DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ (Montpellier, France) _(ASE ’18)_. Association for Computing Machinery, New York, NY, USA, 132–142.  https://doi.org/10.1145/3238147.3238187
  * Zhang et al. (2016) Shanshan Zhang, Rodrigo Benenson, Mohamed Omran, Jan Hosang, and Bernt Schiele. 2016.  How Far are We from Solving Pedestrian Detection?. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 1259–1267.  https://doi.org/10.1109/CVPR.2016.141
  * Zhang et al. (2017) Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele. 2017\.  CityPersons: A Diverse Dataset for Pedestrian Detection. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 4457–4465.  https://doi.org/10.1109/CVPR.2017.474
  * Zhou et al. (2020) Husheng Zhou, Wei Li, Zelun Kong, Junfeng Guo, Yuqun Zhang, Bei Yu, Lingming Zhang, and Cong Liu. 2020\.  DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. Association for Computing Machinery, New York, NY, USA, 347–358.  https://doi.org/10.1145/3377811.3380422
  * Zou et al. (2023) Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. 2023.  Object Detection in 20 Years: A Survey.  _Proc. IEEE_ 111, 3 (2023), 257–276.  https://doi.org/10.1109/JPROC.2023.3238524