# Reliable Student: Addressing Noise in Semi-Supervised 3D Object Detection

Farzad Nozarian  Shashank Agarwal  Farzaneh Rezaeianaran  Danish Shahzad
Atanas Poibrenski  Christian Müller  Philipp Slusallek
German Research Center for Artificial Intelligence (DFKI)
Saarland Informatics Campus
<EMAIL_ADDRESS>
###### Abstract

Semi-supervised 3D object detection can benefit from the promising pseudo-
labeling technique when labeled data is limited. However, recent approaches
have overlooked the impact of noisy pseudo-labels during training, despite
efforts to enhance pseudo-label quality through confidence-based filtering. In
this paper, we examine the impact of noisy pseudo-labels on IoU-based target
assignment and propose the Reliable Student framework, which incorporates two
complementary approaches to mitigate errors. First, it involves a class-aware
target assignment strategy that reduces false negative assignments in
difficult classes. Second, it includes a reliability weighting strategy that
suppresses false positive assignment errors while also addressing remaining
false negatives from the first step. The reliability weights are determined by
querying the teacher network for confidence scores of the student-generated
proposals. Our work surpasses the previous state-of-the-art on KITTI 3D object
detection benchmark on point clouds in the semi-supervised setting. On 1%
labeled data, our approach achieves a 6.2% AP improvement for the pedestrian
class, despite having only 37 labeled samples available. The improvements
become significant for the 2% setting, achieving 6.0% AP and 5.7% AP
improvements for the pedestrian and cyclist classes, respectively. Our code
will be released at https://github.com/fnozarian/ReliableStudent

##  1 Introduction

Significant progress has been made in image classification [4] and object
detection [13, 1, 33, 8, 27, 15, 16, 17] with recent developments in deep
learning. The availability of large datasets [4, 11, 20, 14] has helped to
accelerate these advancements. However, annotating massive datasets remains a
bottleneck, particularly for 2D and 3D object detection. Semi-supervised
approaches (SSA) have been proposed to address this problem. Unlike supervised
methods, these approaches require only a limited amount of annotated data for
training, with the remaining data being unlabeled.

Several semi-supervised techniques have been proposed for object detection,
including [28, 12, 9, 5, 21, 22]. Self-training using pseudo-labeling is the
most commonly used method and has shown effectiveness in both object detection
[19, 21, 12, 9] and classification [29, 18]. At its core, a student-teacher
framework is used to incrementally train teacher and student models on
unlabeled data in a mutually beneficial manner. The teacher model is initially
trained in a supervised manner on limited labeled data to generate pseudo-
labels (PL) to train the student model on unlabeled data. Mean-teacher-based
techniques [22, 21] use an exponential moving average (EMA) of the student
model’s weights to update the teacher model’s weights, leading to more stable
predictions on the unlabeled data.

Figure 1: Illustrates the need for class-aware foreground thresholds for
foreground/background target assignment. The $\mathrm{IoU_{FG}}$ on the x-axis
shows the IoU of proposals with respect to pseudo-labels that are foreground
relative to ground truths. (a) The default class-agnostic threshold in the PV-
RCNN baseline. (b) Our class-aware thresholds. Lowering the threshold and
including more foreground proposals can benefit challenging and uncommon
classes. It also significantly reduces false negatives with IoUs close to
zero. (Best viewed in color)

Due to its limited pre-training on labeled data, the teacher model fails to
generalize effectively, resulting in noisy pseudo-labels that hinder the
learning of the student model. Existing methods overcome this problem by
filtering out low-quality pseudo-labels with confidence-based thresholds,
acting as a global quality-based filtering mechanism. However, even with
strict filtering, pseudo-labels remain noisy, as shown in Fig. 1 (a). They
have erroneous Intersection over Union (IoU) with proposals that are
foreground relative to ground truths. This poses a significant problem for
downstream tasks such as target assignment in Region Proposal Network (RPN)
and Region-based Convolutional Neural Network (RCNN) modules, which rely on
these noisy IoUs.

The standard target assignment inevitably misclassifies the proposals with
IoUs close to zero, _i.e_., the bar close to the y-axis in Fig. 1 (a), as
background, leading to performance degradation.

Fig. 1 also shows distinct class-specific distributions of IoUs due to the
different levels of difficulty and the unbalanced distribution of classes in
the dataset. Neglecting the difference in distributions poses a challenge for
class-agnostic target assignment methods in detectors such as PV-RCNN. A high-
value class-agnostic threshold will exacerbate false-negative (FN) errors for
difficult classes, such as pedestrians and cyclists, with lower distribution
modes, while lowering the threshold will cause many false positives (FP) for
the car class, which is easier to learn.

We address these challenges from two perspectives: 1) reducing false-negative
and false-positive errors using a new and simple class-aware target assignment
approach, and 2) increasing robustness in training against potential failure
of our initial assignment by weighting the classification loss to suppress
misclassified proposals. These two steps are complementary, with the first
step aiming to minimize assignment errors by considering the difference
between the distribution modes of different classes, while the second step
mitigates residual errors from the first step.

To this end, we first modify the target assignment process in two key areas
where IoU scores are used. We replace the standard foreground/background
random subsampling with a top-k IoU-based subsampler to promote learning from
uncertain or difficult background proposals. We also propose local class-aware
foreground thresholds for target assignment. As shown in Fig. 1 (b), the new
thresholds include more foreground proposals of difficult classes (leading to
higher recall) while preserving a high value for the dominant car class to
ensure learning from high-precision proposals. The foreground and background
thresholds divide proposals into three categories: foreground (FG), background
(BG), and uncertain (UC). We assign hard labels to FG and BG proposals and use
soft labels for those in the UC category to consider their uncertainty.

Second, to address false negative/positive target assignment errors, we
propose to use the teacher to provide reliability scores for the student-
generated proposals. To this end, the teacher’s RCNN head refines the
student’s proposals and assigns confidence scores to them, which we use to
weight the RCNN classification loss on unlabeled data using different FG/UC/BG
weighting options. Our results show that weighting uncertain and background
proposals effectively suppresses false positives and false negatives,
respectively, and outperforms other proposed weighting schemes.

In summary, our key contributions are as follows:

  * •

We thoroughly investigate the impact of noisy pseudo-labels on the IoU-based
target assignment.

  * •

We propose a class-aware target assignment method to address the target
misclassification problem present in recent pseudo-labeling approaches.

  * •

We propose different reliability weighting options to suppress false negatives
and positives using teacher confidence scores.

  * •

We conduct extensive experiments and ablation studies to evaluate the
effectiveness of our approach on the KITTI 3D object detection benchmark in a
semi-supervised setting.

##  2 Related Work

Figure 2: Overview of our Reliable Student framework. It uses a teacher-
student network, where the EMA teacher produces high-quality pseudo-label
boxes $b_{i}$. We compute the IoU $u_{i}$ between $b_{i}$ and the student’s
post-NMS proposals $r_{i}$, followed by a top-k sampling of $r_{i}$ based on
$u_{i}$. The sampled proposals $r_{i}$ are injected into the student and
teacher RCNN heads to predict the objectness scores $\tilde{s}_{i}$ and
$\hat{s}_{i}$, respectively. While $\tilde{s}_{i}$ serves as an input to the
RCNN classification loss $\mathcal{L}^{cls}_{u}$, $\hat{s}_{i}$ are converted
into reliability weights $w_{i}$ for $\mathcal{L}^{cls}_{u}$. The class-aware
target assignment module uses thresholds for different classes on $u_{i}$ to
assign objectness targets $t_{i}$ for $\mathcal{L}^{cls}_{u}$.

###  2.1 3D Object Detection

Research on 3D object detection from point clouds focused on a bird’s eye view
of the lidar point cloud [3, 7]. However, VoxelNet [33] employed a different
approach by dividing the point cloud into 3D voxels and encoding each voxel
using a feature encoding layer. Although 3D convolution layers were applied to
further aggregate features, this method was considered time-consuming due to
the 3D convolutions involved. To address this, SECOND [27] proposed a
spatially sparse convolutional network to improve the speed of previous
methods. PointPillars [8] then suggested using vertical columns instead of
voxels and a 2D convolutional network to encode features. This approach was
found to be faster and more robust than previous methods. Another approach by
PointNet and PointNet++ [15, 16] was to work directly on encoding points
instead of voxels, resulting in more efficient and flexible approaches. In
this study, we use PV-RCNN [17], a robust two-stage detector that combines the
VoxelNet and PointNet approaches and achieves high performance.

###  2.2 Semi-Supervised Object Detection

There have been many studies in the field of semi-supervised 2D object
detection. PseCo [9] combines both pseudo-labeling and consistency approaches.
It uses not only label-level consistency but also feature-level consistency,
which further improves the performance of the final detector. This approach
also uses focal loss similar to [12] to alleviate the class imbalance in
pseudo-labeling. [10] considers the localization task as a classification task
and proposes a certainty-aware pseudo-label approach. By quantifying the
quality score of classification and regression, they adjust the threshold used
for generating pseudo-labels. Instant-Teaching [32] proposes to generate
pseudo annotation for unlabeled data using a weak augmentation in mini-batch,
then using these predicted annotations as ground truth of the same image with
strong augmentation. For strong augmentation, the authors use Mixup [30].

Recent works have also focused on class imbalance and confirmation bias
issues. LabelMatch [2] leverages the labeled data distribution for adaptive
thresholding to filter out unbiased pseudo-labels and recalibrates the high-
quality unreliable pseudo-labels into reliable ones. Unbiased Teacher [12]
attempts to address the class-imbalance problem in pseudo-labeling by
incorporating a focal loss that forces the model to focus on challenging
samples from the underrepresented classes. Humble Teacher [21] achieves
comparable results by using soft labels instead of hard labels with a teacher
ensemble network to improve the reliability of the pseudo-labels.

Soft Teacher [26] deals with the misclassification of foreground proposals by
suppressing the classification loss using the teacher’s confidence scores. Our
approach follows this but additionally considers the reliability of foreground
targets with a foreground reliability weight. Our work also differs from Soft
Teacher in that we use a third category of targets in the RCNN, called the
Uncertain (UC) region, and assign soft labels to them. These targets may
correspond to real foreground or background boxes. Thus, it is crucial to
assign appropriate weights to this region to optimize the precision-recall
trade-off. Combating Noise [25] assumes that background proposals are
accurate, and it suppresses the noisy foreground proposals losses. In
contrast, we show that dealing with both misclassified foreground and
background proposals is important.

There are few works on semi-supervised point-based 3D object detection, such
as SESS[31] and 3DIoUMatch[24]. SESS uses asymmetric data augmentation
techniques and enforces consistency between teacher and student predictions
through different losses. 3DIoUMatch [24] proposes a pseudo-labeling approach
for both indoor and outdoor 3D object detection. Inspired by FixMatch [18],
they introduce a joint confidence-based pseudo-label filtering mechanism using
predicted objectness and class probabilities. Additionally, they estimate IoU
and use it as a localization quality to filter pseudo-labels. Unlike
3DIoUMatch, we employ only an objectness threshold, eliminating the complexity
of using multiple thresholds. Moreover, unlike 3DIoUMatch, we adopt objectness
supervision on unlabeled data. Our findings indicate that this strategy
enhances performance.

##  3 Method

###  3.1 Overview

An overview of our approach is depicted in Fig. 2. Our approach is based on
the mean-teacher framework, where the teacher creates PLs for unlabeled input
to serve as a supervised signal for the student. The student is provided with
the strongly augmented version of the unlabeled input as well as the labeled
input, and its parameters are updated through backpropagation. The teacher’s
parameters, on the other hand, are gradually updated from the student’s
parameters using the exponential moving average strategy. To ensure the
quality of the generated PLs, we filter them based on their confidence scores.
We introduce the Class-aware Target Assignment module (Sec. 3.2) with class-
aware foreground thresholds on IoU of proposals with PLs to improve recall,
particularly for challenging classes. This is based on the understanding that
the learning status of classes depends on their difficulty level and the
availability of their instances in the dataset. Given these foreground
thresholds and the default background threshold, we define hard classification
targets for the foreground and background proposals, while uncertain proposals
whose IoUs lay between the FG and BG thresholds are assigned soft targets.

Due to the noisy IoU signal used for target assignment, some proposals may be
mistakenly assigned to incorrect targets, leading to FPs and FNs. To mitigate
this, we introduce the reliability-based weight assignment module (Sec. 3.3),
which assigns reliability weights to the proposals of each category based on
the dominant error type in that category, making the training more robust. To
obtain the reliability weights, we use the teacher model to refine the
student’s proposals using its RCNN module and use its confidence score
$\hat{s}_{i}$ as additional supervision to improve the student’s performance.
Given the student’s RCNN refinement box and score
$\\{\tilde{b}_{i},\tilde{s}_{i}\\}$ and their corresponding targets, we use
the teacher score $\hat{s}_{i}$ to weight the loss of classification on
unlabeled data.

###  3.2 Class-aware Target Assignment

We investigate the problem of learning from noisy PLs, mainly used to
supervise RPN and RCNN modules in the detector. We focus on the RCNN module
and its classification target assignment, where the proposals are assigned
with foreground/background labels.

Denote $\mathcal{P}=\\{b_{n},c_{n},s_{n}\\}^{N_{\mathit{pl}}}_{n=1}$ as the
set of filtered PLs consisting of bounding box $b_{n}$, category label
$c_{n}$, and the confidence score $s_{n}$. We define $\\{r_{i}\\}$ as the
final proposals or Regions of Interest (RoIs) generated by the student after
the IoU-guided filtering and deduplication of RPN proposals using Non-Maximum
Suppression (NMS). Existing pseudo-labeling approaches use the IoU between
these RoIs and PLs to assign category labels and FG/BG targets to proposals of
unlabeled data in the RPN and RCNN modules of PV-RCNN, respectively. In RCNN,
for a given proposal, if its maximum IoU with PLs, i.e.,
$u_{i}=\max_{p\in\mathcal{P}}\mathrm{IoU}(r_{i},p)$, exceeds a predefined
class agnostic foreground threshold $\tau^{\mathit{fg}}$, it is considered as
a foreground proposal. We define these IoU thresholds used in these two
modules as local thresholds ($\tau_{\mathit{c}}^{\mathit{fg}}$), as opposed to
the global thresholds ($\delta_{\mathit{c}}^{\mathit{fg}}$), used to filter
out low-quality PLs.

We analyze the suboptimal classification target assignment from PLs with the
optimal assignment from GTs. In Fig. 1, we evaluate the mean IoU of proposals
that are foreground with respect to GTs, i.e., their IoUs with GTs are greater
than the evaluation mode class-wise foreground threshold
$\Delta^{\mathit{fg}}_{\mathit{c}}$. We observe two crucial issues when using
the standard target assignment.

First, the classes exhibit distinct mean IoU distributions. Therefore, the
standard target assignment strategy based on a single class-agnostic
foreground threshold, _e.g_., $\mathit{\tau^{\mathit{fg}}=0.75}$, cannot
reliably classify the proposals. For the pedestrian and cyclist classes, which
have lower distribution modes than the car, such a class-agnostic threshold
results in many misclassified foreground proposals whose IoU cannot exceed the
threshold by a small margin. To address this issue, we propose local class-
aware foreground thresholds $\tau_{\mathit{c}}^{\mathit{fg}}$, instead of a
class agnostic $\tau^{\mathit{fg}}$ on $u_{i}$ IoUs, to construct the FG/BG
target $\mathit{\mathit{t}_{\mathit{i}}}$ for the proposal
$\mathit{r}_{\mathit{i}}$ as follows:

$t_{i}=\begin{cases}1,&u_{i}>\tau_{c}^{\mathit{fg}}\\\
\frac{u_{i}-\tau^{\mathit{bg}}}{\tau_{c}^{\mathit{fg}}-\tau^{\mathit{bg}}},&\tau^{\mathit{bg}}\leq
u_{i}\leq\tau_{c}^{\mathit{fg}}\\\ 0,&u_{i}<\tau^{\mathit{bg}}\end{cases}.$
(1)

Background proposals have consistently low IoUs, enabling a single class-
agnostic threshold $\tau^{\mathit{bg}}$ to distinguish them from other
proposals.

Second, the IoUs used for target assignment are unreliable. This is
particularly the case for the pedestrian and cyclist classes, which are
difficult to learn due to their object size and the imbalanced class
distribution of the dataset. Given the presence of noisy IoUs, despite the
implementation of class-specific local thresholds, the assignment carried out
in Eq. 1 will inevitably result in the occurrence of false negative (FN) and
false positive (FP) errors.

Figure 3: Illustrates the density of IoU values of proposals with their
matched PL ($\mathrm{u}_{\mathrm{i}}$) and GT ($\mathrm{v}_{\mathrm{i}}$) on
the x-axis and y-axis, respectively. Denser regions are shown with darker
shades. The red and orange vertical lines denote the local foreground (FG)
($\mathrm{\tau^{fg}_{c}}$) and background (BG) ($\mathrm{\tau^{bg}}$)
thresholds, while the black horizontal line represents the FG threshold
($\mathrm{\Delta_{c}}$) for the evaluation mode, dividing the plot into six
subregions. Subregions (a) and (f) represent false negative and true negative
proposals, respectively. (b) and (e) depict proposals lying in the uncertain
region and are assigned with soft targets, while (c) and (d) depict true
positive and false positive proposals, respectively. The proposals are
obtained from the last few training iterations. We also omit proposals that
are in the background with respect to both GT and PL for better visualization.
All three plots follow the same subregion breakdown. (Best viewed in color)

To examine how proposals in the FG, UC, and BG categories are affected by the
FP and FN errors, we illustrate the density plots in Fig. 3, showing the
distribution of RoI IoUs relative to both PLs and GTs. The FP proposals are
referred to as foreground with respect to PL, but background with respect to
GT, whereas those that are the opposite are referred to as FN proposals. As
shown, each local class-aware threshold divides the plot into three columns
showing FG, UC, and BG sections from right to left.

Ideally, we expect well-calibrated IoU scores such that the IoU of RoIs with
respect to PLs are as close as possible to their corresponding IoUs with
respect to GTs. In practice, however, there exist two sub-densities close to
the axes contributing to the error. More specifically, in the foreground
region, we observe the density of FP proposals in section (d), near the
x-axis, for all classes. However, for the pedestrian class, we have
significantly higher density compared to the other classes. In the background
region, FN proposals are present in (a) near the y-axis. The definitions of FP
and FN have been extended to the uncertain region, i.e., sections (b) and (e),
where FN and FP proposals are located in section (b) and at the bottom of
section (e), close to the x-axis, respectively.

###  3.3 Reliability-based Weight Assignment

To address these FP and FN erroneous proposals, we focus on making the
training robust against a given set of uncertain PLs. We propose weighting the
classification loss of such proposals based on the reliability of their target
assignment, i.e., the IoU between RoI and PL. We seek a reliability score that
can consistently assign a low value to both FN and FP proposals. In this work,
we evaluate the reliability score proposed by Soft Teacher. However, any other
reliability score can also be plugged into our framework.

We estimate the reliability of the student’s proposals based on their
corresponding teacher’s refined confidence scores. We use these scores to
suppress the loss due to FP and FN targets. To this end, we first reverse the
augmentation $h$ on the student proposals before sending them to the teacher.
The teacher refines each student’s proposal $r_{i}$ using its RoI pooling
module and predicts $\hat{y}_{i}=\\{\hat{b}_{i},\hat{s}_{i}\\}$, where
$\hat{b}_{i}$ and $\hat{s}_{i}$ denote the corresponding refined bounding box
and its confidence score, respectively. The confidence score $\hat{s}_{i}$,
represents the foreground probability of the refined bounding box proposal,
which acts as the reliability score for $r_{i}$. We propose different
reliability weighting schemes based on the teacher’s confidence score
$\hat{s}_{i}$, for the RCNN classification loss of unlabeled samples.

Based on our error breakdown in the previous section, we introduce
reliability-based weighting options as follows:

  * •

Background proposals ($\mathrm{BG}$): suppress the FN proposals in subregion
(f) of Fig. 3 by incorporating the teacher’s background score as a weight
($w_{i}=1-\hat{s}_{i}$) for classification loss in subregions (a) and (f).

  * •

Uncertain FN proposals ($\mathrm{UC}_{\mathrm{FN}}$): suppress the FN
proposals in subregions (b) of Fig. 3 by incorporating the teacher’s
background score as a weight ($w_{i}=1-\hat{s}_{i}$) for classification loss
for subregions (b) and (e).

  * •

Uncertain FP proposals ($\mathrm{UC}_{\mathrm{FP}}$): suppress the FP
proposals in subregion (e) of Fig. 3 by incorporating the teacher’s foreground
score as a weight ($w_{i}=\hat{s}_{i}$) for classification loss for subregions
(b) and (e).

  * •

Foreground proposals ($\mathrm{FG}$): suppress the FP proposals in subregion
(d) of Fig. 3 by incorporating the teacher’s foreground score as a weight
($w_{i}=\hat{s}_{i}$) for classification loss for subregions (c) and (d).

In all the weighting options, proposals belonging to the remaining categories
are assigned with the reliability weight $w_{i}=1$. Later in Sec. 4.3.1, we
evaluate the application of different weighting options individually and in
combination and achieve the best performance from $\mathrm{UC_{FP}+BG}$ by
suppressing FPs from uncertain proposals and FNs from background proposals.

We further leverage these reliability-based weights to let the student model
learn more about challenging and uncertain proposals instead of the easy
backgrounds. The student model’s target assignment in RCNN involves computing
the IoU between post-NMS proposals and pseudo-labels. Prior works perform
sampling on these IoUs such that, at most, 50% of the foreground proposals are
randomly sampled before being passed on for refinement. The remaining
background proposals are further randomly subsampled, ensuring that 20% of
them have low IoU ($\emph{e.g}.\hbox{},<0.1$), that are easily classified as
background. Our approach differs in that it avoids subsampling of such easy
backgrounds on unlabeled data and instead uses a top-k sampling strategy on
the IoU. This allows the model to learn more about the challenging
backgrounds.

Methods | 1% | 2%
---|---|---
Car | Pedestrian | Cyclist | Car | Pedestrian | Cyclist
Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard
PV-RCNN† [17] | 87.7 | 73.5 | 67.7 | 32.4 | 28.7 | 26.2 | 48.1 | 28.4 | 27.1 | \ | 76.6 | \ | \ | 40.8 | \ | \ | 45.5 | \
3DIoUMatch† [24] | 89.0 | 76.0 | 70.8 | 37.0 | 31.7 | 29.1 | 60.4 | 36.4 | 34.3 | \ | 78.7 | \ | \ | 48.2 | \ | \ | 56.2 | \
PV-RCNN | 87.6 | 74.1 | 67.9 | 36.5 | 31.7 | 28.9 | 49.9 | 28.8 | 27.3 | 88.9 | 76.8 | 71.9 | 45.1 | 40.4 | 35.6 | 63.0 | 42.3 | 38.9
3DIoUMatch (Baseline) | 89.2 | 76.4 | 71.3 | 41.8 | 35.7 | 32.9 | 59.9 | 36.0 | 33.8 | 90.7 | 78.9 | 74.3 | 52.9 | 47.0 | 41.8 | 74.2 | 53.3 | 49.6
3DIoUMatch + ULB RCNN CLS | 89.8 | 76.6 | 72.0 | 41.9 | 36.0 | 33.1 | 59.0 | 35.6 | 33.3 | 91.1 | 79.3 | 75.3 | 54.6 | 48.6 | 42.8 | 75.9 | 54.4 | 50.7
Reliable Student | 89.7 | 77.0 | 72.5 | 48.0 | 41.9 | 38.4 | 59.1 | 36.4 | 34.2 | 90.9 | 79.5 | 75.0 | 59.3 | 53.0 | 46.9 | 83.1 | 59.0 | 55.1
% Improvement over Baseline | +0.5 | +0.6 | +1.2 | +6.2 | +6.2 | +5.5 | -0.8 | +0.4 | +0.4 | +0.2 | +0.6 | +0.7 | +6.4 | +6.0 | +5.1 | +8.9 | +5.7 | +5.5

Table 1: Results on the KITTI evaluation set based on mAP over 40 recall
positions. PV-RCNN† is the supervised-only baseline, and 3DIoUMatch† is the
original work (both based on OpenPCDet v0.3). 3DIoUMatch (Baseline) is our
adaptation of the original work to OpenPCDet v0.5, and 3DIoUMatch + ULB RCNN
CLS is our modified version of the baseline with objectness supervision from
unlabeled data. (†) denotes borrowed results from [24], (\\) indicates non-
available results, and Bold indicates the best results from OpenPCDet v0.5.

Let $\\{\tilde{b}_{i},\tilde{s}_{i}\\}$ denote the student’s refinement of the
proposal $r_{i}$. The RCNN classification loss on unlabeled data is summarized
as follows:

$\displaystyle\mathcal{L}_{u}^{\mathit{cls}}=\frac{\sum_{i}^{N_{b}}w_{i}l_{\mathit{cls}}(\tilde{s}_{i},t_{i})}{\sum_{i}w_{i}},$
(2)

where $N_{b}$ are the total number of proposals for a single unlabeled sample.

Given $N_{l}$ labeled samples, we define
$\mathcal{D}_{l}=\\{(x^{l}_{i},y^{l}_{i})\\}^{N_{l}}_{i=1}$, where $y^{l}_{i}$
contains the class labels and bounding box coordinates information, and use
$N_{u}$ unlabeled samples for $\mathcal{D}_{u}=\\{x^{u}_{i}\\}^{N_{u}}_{i=1}$.
The unsupervised RCNN loss $\mathcal{L}_{u}$ consists of the classification
loss $\mathcal{L}_{u}^{\mathit{cls}}$ from Eq. 2, and box regression loss
$\mathcal{L}_{u}^{\mathit{reg}}$, which is defined as:

$\mathcal{L}^{\mathit{RCNN}}_{u}=\frac{1}{N_{u}}\sum^{N_{u}}_{i=1}(\mathcal{L}_{u}^{\mathit{cls}}(\tilde{s}^{u}_{i},t^{u}_{i})+\mathcal{L}_{u}^{\mathit{reg}}(\tilde{b}^{u}_{i},b^{u}_{i})),$
(3)

where $t^{u}_{i}$ is the target for classification loss from Eq. 1, and
$b^{u}_{i}$ is the bounding box of the assigned pseudo box based on $u_{i}$,
acting as the regression loss target. We follow 3DIoUMatch for the RCNN box
regression loss $\mathcal{L}_{u}^{\mathit{reg}}$, as well as for the RPN
classification and regression losses, to formulate the unsupervised loss
$\mathcal{L}_{u}$. The supervised loss $\mathcal{L}_{s}$ is calculated
similarly on labeled data using ground truth $y^{l}_{i}$. The overall loss of
the student model is defined as

$\mathcal{L}=\mathcal{L}_{s}+\lambda_{u}\mathcal{L}_{u},$ (4)

where $\lambda_{u}$ is a coefficient balancing the unsupervised loss. The
teacher weights are updated as the exponential moving average of the student
model.

##  4 Experiments

###  4.1 Experimental Setup

We evaluate our method on KITTI [6] dataset, consisting of 7,481 training
samples and 7,518 test samples. The training samples are divided into the
train set (3,712 samples) for training the model and the validation set (3,769
samples) for evaluation. We use 1% and 2% labeled data splits with three folds
each, provided by 3DIoUMatch [24]. For each fold, we carry out three trials
with different random seed values and report the mean Average Precision (mAP)
over all fold-trial combinations. The mAP is computed using a rotated IoU
threshold of 0.7, 0.5, and 0.5 for the car, pedestrian, and cyclist classes,
respectively, at 40 recall positions. Experiments are conducted over all three
object difficulty levels - Easy, Moderate, and Hard.

#### Implementation Details

For a fair comparison with [24], we utilize PV-RCNN [17] as the object
detection backbone. We used the OpenPCDet v0.5 framework [23] to implement our
method and adapted the original 3DIoUMatch from OpenPCDet v0.3 to v0.5 for a
fair comparison. The data augmentation on the student model is based on the
3DIoUMatch settings. Unlike 3DIoUMatch, which uses both RPN classification and
RCNN objectness scores to filter pseudo labels, our approach uses only the
RCNN objectness threshold, i.e., $\tau^{\mathit{pl}}_{\mathit{car}}=0.95$ for
car, and
$\tau^{\mathit{pl}}_{\mathit{ped}}=\tau^{\mathit{pl}}_{\mathit{cycl}}=0.85$
for pedestrian and cyclist. Unlike 3DIoUMatch, both the RPN and RCNN modules
are supervised using labeled and unlabeled data through classification and
regression losses, with the unlabeled loss weight $\lambda_{\mathit{u}}=1$. On
small amounts of data (1% and 2%), we pre-train PV-RCNN over 80 epochs with 10
repeated traversals in each epoch and use 60 epochs with 5 repeated traversals
in each epoch for the training stage, similar to [24]. We use a batch size of
8, consisting of 8 labeled and 8 unlabeled samples in both stages. For the
evaluation stage, we use the student model.

Figure 4: Illustrates the assigned reliability weights for RCNN classification loss based on the IoU of the proposals with PLs ($\mathrm{u}_{\mathrm{i}}$) on the x-axis and GT ($\mathrm{v}_{\mathrm{i}}$) on the y-axis. The red and orange vertical lines depict the local class-aware foreground (FG) ($\mathrm{\tau^{fg}_{c}}$) and background (BG) ($\mathrm{\tau^{bg}}$) thresholds, respectively, while the black horizontal line represents the FG threshold ($\mathrm{\Delta_{c}}$) for the evaluation mode. The color bar on the right shows the intensity of the reliability weights. Plots are based on the last few training iterations for better visualization. Methods | 1% | 2% |  mAP %
---|---|---|---
Car | Ped. | Cycl. | Car | Ped. | Cycl.
Baseline | 76.4 | 35.7 | 36.0 | 78.9 | 47.0 | 53.3 | 54.6
$\mathrm{BG}$ | 76.8 | 40.5 | 36.7 | 79.1 | 53.2 | 57.2 | 57.3 (+2.7)
$\mathrm{UC}_{\mathrm{FN}}+\mathrm{BG}$ | 76.9 | 41.6 | 36.6 | 79.4 | 51.3 | 58.1 | 57.3 (+2.7)
$\mathrm{UC}_{\mathrm{FP}}+\mathrm{BG}$* | 77.0 | 41.9 | 36.4 | 79.5 | 53.0 | 59.0 |  57.8 (+3.2)
$\mathrm{FG}+\mathrm{UC}_{\mathrm{FN}}+\mathrm{BG}$ | 76.8 | 39.9 | 37.2 | 79.6 | 53.0 | 55.5 | 57.0 (+2.4)
$\mathrm{FG}+\mathrm{UC}_{\mathrm{FP}}+\mathrm{BG}$ | 77.0 | 41.4 | 35.9 | 79.5 | 53.2 | 56.8 | 57.3 (+2.7)
Table 2: Ablation study on different reliability-based weighting options on 1%
and 2% data splits for moderate difficulty level. For a fair comparison, we
show the mAP across all classes in the last column, where
$\mathrm{UC_{FP}+BG}$ performs the best. (*) indicates our chosen weighting
option, and Bold indicates the best results.

###  4.2 Main Results

Tab. 1 shows the results of our approach, the original state-of-the-art
3DIoUMatch method referred to as 3DIoUMatch†, and our adapted version of
3DIoUMatch, which is referred to as the baseline. The baseline performs
similarly to the original work, except for the cyclist class in the 2% split,
where there is a minor drop of less than 3%. Note that the baseline does not
use the RCNN classification loss on unlabeled data, while our approach
benefits from it. Hence, for a more accurate comparison, we have also included
the results of our adapted baseline with RCNN classification loss on unlabeled
data, which shows an improvement over the naive baseline. We refer to our
method as the best option selected from the weighting schemes evaluated in
Tab. 2, i.e., $\mathrm{UC_{FP}+BG}$.

Our framework shows superior performance over both 3DIoUMatch and its improved
version across all labeled data splits, specially for pedestrian and cyclist
classes. While we are also successful in improving for the car class, the
margins are relatively small because of two reasons. First, the car class
suffers from a substantial number of FP errors and in Section 4.3.1, we show
that the effectiveness of reliability weights in such a scenario is limited.
Second, the car class being dominant in terms of class distribution is already
learnt well in the pre-train stage itself, leaving small room of improvements
for the second stage.

###  4.3 Ablation Studies

####  4.3.1 Effects of reliability weights

Tab. 2 ablates the performance over different reliability-based weighting
options, improving the mAP over the baseline by 2.7%-3.2%. The
$\mathrm{UC}_{\mathrm{FN}}$ and $\mathrm{UC}_{\mathrm{FN}}+\mathrm{BG}$ were
evaluated to suppress FN errors, while others assess the effect of suppressing
both FN and FP errors. The last two options were assessed to determine
efficient ways to weight UC proposals to suppress FN or FP errors. While the
reliability weights help in all of these options, $\mathrm{UC_{FP}+BG}$ has
the highest gain in mAP of 3.2% over the baseline. Moreover, the teacher’s
foreground score was found to be more efficient as a weight in the BG option
than in the FG option. We believe that
$\mathrm{FG}+\mathrm{UC}_{\mathrm{FN}}+\mathrm{BG}$ has lower performance due
to the down-weighting of truly uncertain proposals. In Fig. 5, we show the
mean reliability weights of all foreground proposals relative to the PLs with
the weighting option of $\mathrm{FG+UC_{FP}+BG}$. As shown, the weights from
this option effectively suppress the loss due to FP and FN proposals at the
cost of suppressing the loss of some true positives (TP). Moreover, the
weights of FPs are relatively higher (close to 1), especially for the car
class, and less effective than those for the FNs. We conjecture that this is
due to the unbalanced number of FG/BG proposals in the RCNN module. Fig. 6
illustrates this by showing the percentage of FG proposals used to train the
RCNN classification branch. Note that the car class is highly skewed, with
almost 95% of the proposals as BGs. As a result, the network is biased towards
the BG class, and the teacher model cannot provide a reliable FG score for the
FP proposals. Whereas, the $\mathrm{UC_{FP}+BG}$ option compensates this by
avoiding the suppression of the loss due to the TP proposals, instead mainly
suppressing the FPs and FNs, as shown in Fig. 4.

Figure 5: Teacher’s mean reliability weights, averaged over every few
iterations, using the $\mathrm{FG+UC_{FP}+BG}$ weighting type. Figure 6: Shows
the percentage of foreground proposals with respect to GT used to train the
FG/BG classification head, highlighting the imbalanced FG/BG ratios across
different classes.

####  4.3.2 Effects of class-aware target assignment

Tab. 3 analyzes the effects of local class-aware foreground thresholds over
class-agnostic thresholds and their sensitivity to different values. We show
that the class-aware thresholds not only perform better than the default
threshold by a large margin, but also they are consistent in performance
across different values. We leverage our previous finding that the pedestrian
and cyclist classes require lower thresholds than the car class by adjusting
our baseline thresholds by 10%.

####  4.3.3 Effects of top-k based sampler

Tab. 4 shows that using the balanced random sampler with the class-aware
target assignment and unreliability weighting scheme improves the results over
the baseline. However, our top-k sampler improves the baseline further by
0.2%-4.4% across different classes.

Methods | Car | Pedestrian | Cyclist
---|---|---|---
Baseline | 76.4 | 35.7 | 36.0
C-Ag | 0.75 | 76.6 | 37.0 | 33.2
C-Aw | 0.75, 0.55, 0.5 | 76.5 | 41.9 | 36.6
0.65, 0.45, 0.4* | 77.0 | 41.9 | 36.4
0.55, 0.35, 0.3 | 76.9 | 41.1 | 36.5
Table 3: Ablation study of local class-aware (C-Aw) and class-agnostic (C-Ag) foreground thresholds. C-Aw thresholds are shown for the car, pedestrian, and cyclist (in the same order). We used 1% labeled data for the moderate difficulty level. (*) indicates our chosen thresholds, and Bold indicates the best results. Methods | Car | Pedestrian | Cyclist
---|---|---|---
Baseline | 76.4 | 35.7 | 36.0
Default sampler | 76.8 | 37.5 | 35.5
Top-k sampler | 77.0 | 41.9 | 36.4
Table 4: Ablation study of default random sampler and our top-k sampler. We
use 1% labeled data for the moderate difficulty level.

##  5 Conclusion

Our research on semi-supervised 3D object detection indicates that while
generating high-quality pseudo-labels via quality-based filtering is
advantageous, the impact of such noisy pseudo-labels on the IoU-based target
assignment module should be considered. We emphasize the significance of
distinct learning curves for different classes and the need for class-specific
target assignments, especially with pseudo-labeling techniques. Moreover, we
utilize the teacher model to obtain a reliability score to suppress inaccurate
target assignment from noisy pseudo-labels and maintain clear supervision from
unlabeled data. Our research offers an error analysis framework that can be
used with other reliability-based metrics to enhance the overall reliability
of the system. We plan to extend it to more autonomous driving datasets and
object detectors in the future.

## Acknowledgment

This work has been funded by the German Ministry for Education and Research
(BMB+F) in the project MOMENTUM.

## References

  * [1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, volume 12346 of Lecture Notes in Computer Science, pages 213–229. Springer International Publishing, 2020.
  * [2] Binbin Chen, Weijie Chen, Shicai Yang, Yunyi Xuan, Jie Song, Di Xie, Shiliang Pu, Mingli Song, and Yueting Zhuang.  Label matching semi-supervised object detection.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2022.
  * [3] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia.  Multi-view 3d object detection network for autonomous driving.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6526–6534, Honolulu, HI, USA, jul 2017. IEEE.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  pages 248–255. IEEE, 2009.
  * [5] Jinhao Dong and Tong Lin.  MarginGAN: Adversarial training in semi-supervised learning.  In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 10440–10449, 2019.
  * [6] A Geiger, P Lenz, C Stiller, and R Urtasun.  Vision meets robotics: The KITTI dataset.  The International Journal of Robotics Research, 32(11):1231–1237, aug 2013.
  * [7] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L. Waslander.  Joint 3d proposal generation and object detection from view aggregation.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–8. IEEE, oct 2018.
  * [8] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12697–12705. IEEE, jun 2019.
  * [9] Gang Li, Xiang Li, Yujie Wang, Yichao Wu, Ding Liang, and Shanshan Zhang.  PseCo: Pseudo labeling and consistency training for semi-supervised object detection.  In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, volume 13669 of Lecture Notes in Computer Science, pages 457–472. Springer Nature Switzerland, 2022.
  * [10] Hengduo Li, Zuxuan Wu, Abhinav Shrivastava, and Larry S. Davis.  Rethinking pseudo labels for semi-supervised object detection.  In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 1314–1322. AAAI Press.
  * [11] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, volume 8693 of Lecture Notes in Computer Science, pages 740–755. Springer International Publishing, 2014.
  * [12] Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda.  Unbiased teacher for semi-supervised object detection.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [13] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9992–10002. IEEE, oct 2021.
  * [14] Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, Jie Yu, Chunjing Xu, and Hang Xu.  One million scenes for autonomous driving: ONCE dataset.  In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
  * [15] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.  PointNet: Deep learning on point sets for 3d classification and segmentation.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 77–85. IEEE, jul 2017.
  * [16] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  arXiv preprint arXiv:1706.02413, 2017.
  * [17] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  PV-RCNN: Point-voxel feature set abstraction for 3d object detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10526–10535. IEEE, jun 2020.
  * [18] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [19] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister.  A simple semi-supervised learning framework for object detection.  arXiv e-prints, page arXiv:2005.04757, May 2020.
  * [20] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov.  Scalability in perception for autonomous driving: Waymo open dataset.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2443–2451. IEEE, jun 2020.
  * [21] Yihe Tang, Weifeng Chen, Yijun Luo, and Yuting Zhang.  Humble teachers teach better students for semi-supervised object detection.  pages 3131–3140. IEEE, 2021.
  * [22] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017.
  * [23] OpenPCDet Development Team.  Openpcdet: An open-source toolbox for 3d object detection from point clouds.  https://github.com/open-mmlab/OpenPCDet, 2020.
  * [24] He Wang, Yezhen Cong, Or Litany, Yue Gao, and Leonidas J. Guibas.  3dioumatch: Leveraging IoU prediction for semi-supervised 3d object detection.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14615–14624. IEEE, jun 2021.
  * [25] Zhenyu Wang, Ya-Li Li, Ye Guo, and Shengjin Wang.  Combating noise: semi-supervised learning by region uncertainty quantification.  Advances in Neural Information Processing Systems, 34:9534–9545, 2021.
  * [26] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu.  End-to-end semi-supervised object detection with soft teacher.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3040–3049. IEEE, oct 2021.
  * [27] Yan Yan, Yuxing Mao, and Bo Li.  SECOND: Sparsely embedded convolutional detection.  Sensors, 18(10):3337, oct 2018.
  * [28] Qize Yang, Xihan Wei, Biao Wang, Xian-Sheng Hua, and Lei Zhang.  Interactive self-training with mean teachers for semi-supervised object detection.  pages 5937–5946, Nashville, TN, USA, 2021. IEEE.
  * [29] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 18408–18419, 2021.
  * [30] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization, Oct. 2018.
  * [31] Na Zhao, Tat-Seng Chua, and Gim Hee Lee.  Sess: Self-ensembling semi-supervised 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11079–11087, 2020.
  * [32] Qiang Zhou, Chaohui Yu, Zhibin Wang, Qi Qian, and Hao Li.  Instant-teaching: An end-to-end semi-supervised object detection framework.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4081–4090. IEEE, jun 2021.
  * [33] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4490–4499. IEEE, jun 2018.


[table]capposition=top 11institutetext:  1Viterbi School of Engineering,
University of Southern California, USA

# Real-time pedestrian recognition on low computational resources

Guifan Weng1∗

###### Abstract

Pedestrian recognition has successfully been applied to security, autonomous
cars, Aerial photographs. For most applications, pedestrian recognition on
small mobile devices is important. However, the limitations of the computing
hardware make this a challenging task. In this work, we investigate real-time
pedestrian recognition on small physical-size computers with low computational
resources for faster speed. This paper presents three methods that work on the
small physical size CPUs system. First, we improved the Local Binary Pattern
(LBP) features and Adaboost classifier. Second, we optimized the Histogram of
Oriented Gradients (HOG) and Support Vector Machine. Third, We implemented
fast Convolutional Neural Networks (CNNs). The results demonstrate that the
three methods achieved real-time pedestrian recognition at an accuracy of more
than 95% and a speed of more than 5 fps on a small physical size computational
platform with a 1.8 GHz Intel i5 CPU. Our methods can be easily applied to
small mobile devices with high compatibility and generality.

##  1 Introduction

Computer vision has been used widely in a variety of applications including
medical, military, industry, services, and scientific research [1]. In
particular, pedestrian recognition in images and videos is a challenging task
that attracts the attention of the scientific community and industry alike. It
is important in a wide range of applications that intersect with many aspects
of our lives: surveillance systems and airport security, autonomous driving
and driver assistance systems in high-end cars [2, 3], human-robot interaction
and immersive, interactive entertainments, smart homes and assistance for
senior citizens that live alone, and people-finding for military applications
[4]. It is also a prerequisite for tasks on mobile devices. For most mobile
devices, however, there are some specific constraints that make pedestrian
recognition particularly challenging, i.e., limited computational resources,
limited physical size, and limited energy. For example, the drones proposed in
[5] have limited space inside to fit a small battery and a small computer with
low computational resources. The same is true for other types of mobile
devices like drones [6, 7], and modular robots [8, 9]. These constraints make
real-time pedestrian recognition on such small physical-size mobile devices a
problematic challenge.

In this paper we focus on pedestrian recognition, i.e., people assuming poses
that are common while standing or walking. Pedestrian recognition is complex
mostly because of the high variability that characterizes the pedestrians’
projections on the camera image plane. The appearance of a pedestrian on the
image is influenced by the person’s pose, his or her clothing, occlusions, and
the atmospheric conditions that contribute to the illumination of the scene
[10]. Background clutter also plays a role in making the detection difficult.
That is, the diversity of pedestrians in nature makes real-time pedestrian
recognition on low computational hardware a difficult challenge.

Although Convolutional Neural Networks (CNNs) became the state-of-the-art
approaches with high accuracy for object recognition [11, 12] including
pedestrian recognition [13], CNNs take expensive computation. A variety of
studies were proposed for real-time pedestrian recognition with faster speed
[7, 6, 14], however, all of them work on the GPU system that outperforms the
regular CPU platform. Therefore, we propose and compare three methods that
work on the CPU system with small physical size and low computation resources
for real-time pedestrian recognition.

After reviewing and considering the state-of-the-art methods in real-time
pedestrian recognition, this paper presents three approaches including
improved LBP features and AdaBoost classifier, improved HOG features and SVM
classifier, and optimized convolutional neural network [15]. First, we use an
improved method based on the LBP features and AdaBoost classifier because the
LBP features perform well to extract the contour features of pedestrians and
the fast speed of the AdaBoost classifier. Second, this paper presents the
improved HOG features and SVM classifier. We optimized the speed of HOG
features, in particular, the improved approach outperform the original method
[16] with the exhaustive search. Third, We achieve a real-time CNN for
pedestrian recognition by optimizing the hyper-parameters.

To assess how our methods work for real-time pedestrian recognition, we
investigate the accuracy and speed under the following conditions:

1) The speed must be above 5 fps to meet the requirement that it is real-time.

2) The accuracy must be higher than $95\%$.

Therefore, the main contribution in this paper presents three methods for
real-time pedestrian recognition on small mobile devices with low
computational and small physical size hardware. The remainder of this paper is
structured as follows. In section II, we describe related work in object
recognition, particularly in real-time pedestrian recognition. A detailed
description of our methods is described in section III. We present the
experimental results and analyze and discuss these results in section IV.
Last, we conclude this paper and provide an outlook on future research.

##  2 Related work

As one of the early real-time object recognition techniques P. Viola and M.
Jones [17, 18] proposed the famous method of simple features and cascade
AdaBoost classifier for rapid object detection. Although it worked fast on a
desktop computer with Intel Pentium III, it was proposed for face detection
and did not work well for pedestrian recognition. In 2005, N. Dalal and B.
Triggs [16] proposed the classic solution of Histograms of Oriented Gradients
(HOG) features and a Support Vector Machine (SVM) classifier for human
detection. This method became popular in object recognition for human
detection. The HOG features and linear SVM classifiers are fast and accurate.
However, the exhaustive search component is computationally expensive so it is
not fast enough on low-performance computational hardware. Therefore, we
propose an optimization for faster speed on a low-performance system.

In recent years, CNNs became the state-of-the-art methods for object
recognition [13], R-CNN[19], Fast R-CNN[20], Faster R-CNN[21], YOLO[22],
SSD[23]. Although these methods have high accuracy, they usually take
expensive computation and work on a GPU system. They are far from working on
low-performance computational hardware [24, 25].

Both early and recent popular methods have been widely and successfully
applied in many areas, such as pedestrian recognition, face recognition, etc.
However, there is a common issue, all of them require expensive computation,
hence they only work on powerful computational hardware like GPU systems. A
real-time solution on the DSP-based embedded system was proposed in [26]. This
solution works only on the dedicated DSP hardware platform, not on a general
CPU system. An exciting solution was proposed based on HOG features and an SVM
classifier for pedestrian detection at 135 fps on a desktop computer equipped
with an Intel Core i7 870 and a GPU [27]. Similarly, [28] presented an
implementation of vehicle recognition at 4 fps at a resolution of 1224 by 370
pixels based on the HOG feature and linear SVM classifier. However, none of
these methods is fast enough on low-performance systems.

An interesting convolutional neural network for object recognition was
proposed in [29]. Although this work optimized fast R-CNN on an embedded
platform for real-time object recognition, it works at 1.85fps speed on the
CPU and GPU system. Similarly, a low-complexity fully convolutional neural
network that works on a weak GPU platform was proposed in [30] for object
recognition based on YOLO. Yet, none of [30] and [29] is fast enough on a
small-size CPU system.

An alternative approach was proposed in [31, 32] that implemented real-time
object recognition using wireless communication between the mobile robot and
the servers. [33] implemented the near real-time object recognition for drones
by offloading the computation onto an off-board computation cloud. Although
using a server with powerful computational resources can be a feasible
solution, several applications need methods that can operate independently
without a server. For instance, the communication time between the drones and
servers is affected by variations in wireless bandwidths and this can form a
severe bottleneck [34].

Although some of the above methods have fast speed and high accuracy, they
still do not work well given the constraints related to small-size CPU
computation systems. In addition, there are many studies that work on
different low computational hardware for real-time object recognition. An end-
to-end deep vision network model was proposed to predict possible good grasps,
which works in real-time on a Baxter robot at a rate of 80 frames per second
using a GPU system [35]. [36] used the exclusive Qualcomm Snapdragon Flight
board embedded in a 2.4GHz processor to implement a visual-inertial drone
system for real-time moving object detection. Although this computational
board has good performance and small size, it is a propriety system exclusive
for the drone and the method only recognizes moving objects.

In summary, existing methods usually rely on powerful computing resources. In
spite of this, some of the algorithms using convolutional neural networks and
linear SVM classifiers are reasonably fast, yet not fast enough to implement
real-time pedestrian recognition on a regular CPU system with low computation
resources.

##  3 Methodology

This work aims to implement Real-time pedestrian recognition on low
computational resources. This paper presents three approaches to achieve the
requirements of the speed at 2 fps and the accuracy at 95%. LBP is a fast
algorithm to extract the contour features of the objects, in particular, it is
very suitable for extracting the features of pedestrians. The Adaboost
classifier is a fast classifier. The method of HOG features and SVM classifier
is popular in pedestrian recognition. But the speed is not fast due to the
exhaustive search [37]. Thus, this paper presents the optimization of the
exhaustive search in HOG and SVM.

The main requirements for our methods are that they work on the low
computational hardware in real-time, and with high accuracy. To this end, we
propose three methods including LBP features and Adaboost classifier, HOG
features and SVM classifier, and Convolutional neural network for real-time
pedestrian detection. In this section, we describe the details of the three
methods and the experimental setup and dataset.

Choosing suitable hardware is a crucial and difficult task. After considering
several alternatives, such as central processing units (CPU), graphic
processing units (GPUs), digital signal processors (DSPs), and field
programmable gate arrays (FPGAs) [38]. We have chosen the Intel NUC6i5SYK
microcomputer as the hardware in this work since it offers a good trade-off
between size, computing power, power dissipation, and price. In addition,
Intel NUC6i5SYK is a general CPU and Linux (optional) platform with good
compatibility that facilitates the portability of the algorithms to other
hardware. It has physical dimensions of $111\times 111\times 35mm$, and embeds
a 1.80 GHz Intel Core i5-6260U Processor. There are few applications where the
NUC does not fit for mobile devices, in particular, drones, and small robots.
In the experiments, we train the classifiers on the CPU computer and test the
trained classifiers on Intel NUC6i5SYK. The experiments of HOG and SVM, LBP
and Adaboost are implemented based on the OpenCV library. The Convolutional
neural network is implemented based on Darknet.

###  3.1 Dataset

Data sets are a fundamental tool for comparing detection algorithms, fostering
advances in the state-of-the-art. The INRIA pedestrian dataset is a
traditional pedestrian dataset. The pedestrians in the image have relatively
regular postures. Most of them are the persons walking or standing captured
from different angles without much deviation from the horizontal perspective.
The INRIA person data set [16] and VOC (VOC2007 [39], VOC2012[39]) are very
popular in the Pedestrian recognition community, both for training detectors
and reporting results. Compared to the INRIA dataset, VOC2007 and VOC2012 are
more challenging. Although the VOC datasets originally provided samples and
labels of 20 classes of objects, one of them is a person. we keep those labels
for the class of person. The pedestrians in the images of VOC have a large
variety of postures and are captured from different angles deviating severely
from the horizontal perspective. To obtain a more comprehensive dataset, in
this work, we combine the pedestrian data sets VOC(VOC2007, VOC2012) and INRIA
as the data sets to train and test our methods. The statistics of the data
sets are shown in table 1.

Table 1: The statistics of the data sets. The training shows the number of images for the training data set. The testing shows the number of images for the testing data set. The $lab_{tr}$ shows the number of total labels on training images. The $lab_{te}$ shows the number of total labels on testing images. |  training |  $lab_{tr}$ |  testing |  $lab_{te}$ |  total
---|---|---|---|---|---
INRIA |  614 |  1958 |  288 |  605 |  2563
VOC |  6095 |  13256 |  2007 |  4528 |  17784
total |  6709 |  15214 |  2295 |  5133 |  20347

As shown in table 1, in this work we use 614 images from the INRIA dataset and
6095 images from the VOC dataset as the training data sets, 288 images from
the INRIA dataset and 2007 images from the VOC dataset as the testing data
sets. For the training process, we use the bootstrap approach to train the SVM
and AdaBoost classifiers until the classifiers perform well.

###  3.2 LBP features and AdaBoost classifier

Originally, the computation of the LBP feature is based on the comparisons of
adjacent single pixels[40]. However, the strategy provided by OpenCV is based
on the comparison between adjacent and small batches consisting of several
adjacent pixels. Since the value of a batch is the sum of adjacent pixels in
rectangle areas, it can be computed by using the integral image. This is also
a new strategy which promotes the speed of the algorithm and it does not
affect the accuracy of detection much.

The original strategy is shown in Fig.1. We generate the binary pattern based
on the single pixel directly. Each pixel around the central pixel, whose value
is 112, compares with the central pixel. If it is bigger, the binary value is
1. Otherwise, the binary value is 0.

The new strategy adopted by OpenCV is shown in Fig.2. Matrix2 is filled with
the summations of every 4 adjacent pixels in matrix1. The binary pattern is
calculated based on the comparisons of 8 non-central summations with the
central summation.

The size of the image is $S=height\times{width}$. The time complexity of
generating the local binary pattern for the image is $f(8S)$ in the original
strategy. In the new strategy, the time complexity for calculating the
integral image is $f(S)$ and for the generation of LBP based on summations is

$f(\frac{S}{4}\times{8})=f(2\times{S})$ (1)

Thus, the total complexity is $f(3S)$. It is smaller than the time complexity
of the original strategy.

###  3.3 HOG features and SVM classifier

Data: 5885 sample ROIs

1 Extract _ROI image features_ Initialise _NEAT with parameters_ Initialise
_$P_{0}$ of ANN_ while  _not $P=P_{\textrm{max}}$ or $f_{g}=f_{\tau}$ for some
$g_{i}$_ do

2 Select 500 random sample ROIs

3 Evaluate $g_{i}\in P_{G}$

4 Select sub-group $g\subset P=\\{g_{i},...,g_{j}\\}$ for reproduction based
on $f_{g}$

5 Recombine cross-over $g$ and create $g^{*}$

6 Mutate $g^{*}$ with $p_{\textrm{mut}}$

7 Define $P_{G+1}$

8

9 end while

Select _ $\max_{g_{i}\in P}f_{g}$_

Algorithm 1 Evolving a real-time object recognition NN.

The initialization of cache and preprocessing for feature computation of HOG
is based on the fact that the range of value of pixels is fixed, which is
[0,255]. Thus, the gradients of $dx$ and $dy$ must be in the range of
[-255,255].

The computation of the 9 bins of the Histogram of Gradient is based on
gradients, which are $dx$ in the x-direction and $dy$ in the y-direction for
each pair of adjacent pixels. The gradient for each point has magnitude and
angle formed by $dx$ and $dy$. Then, for each gradient, there must be 2 bins
adjacent to it. The magnitude votes the weights for the 2 bins based on linear
interpolation.

Figure 1: Gradients and adjacent bins in HOG

In Fig.3, O is a single point in an image and A is its magnitude. bin0 and
bin1 are the 2 bins adjacent to this magnitude. Since we know the angle of the
gradient, we can obtain the angle formed by bin0 and the direction of the
gradient. Thus, the voted weights for the 2 bins can be computed. For each
point, we need to store its 2 voted weights and the indices of the 2 bins.

As we mentioned above, given the $dx$ and $dy$ for a specific point, all the
values above are fixed. And since the values of $dx$ and $dy$ are in the range
of [-255, 255], all the data can be precomputed and stored in the hardware.
Thus, these data can be pre-loaded before the computation of the HOG feature.
This mechanism accelerates the process of feature computation up to 10% faster
than its original speed.

Except utilizing the cache data, we also adjust the parameters for the whole
process of detection by HOG+SVM. We examine and test 3 parameters for the
whole process, including the size of the step of scanning the whole image, the
number of levels in the image pyramid, and the size of the samples for
training and detection. We try different combinations of different parameters
in order to balance the accuracy and efficiency of the program.

###  3.4 Convolutional Neural Network

For the experiments of HOG+SVM and LBP+Adaboost, we modify the dataset to
improve the efficiency of the programs. We resize each sample provided by
INRIA dataset from the size of ${96}\times{160}$ to ${32}\times{64}$ and
resize the sample from VOC to ${32}\times{64}$ strictly despite its large
variety of different sizes. Since HOG focuses on gradients and LBP focuses on
the comparison of adjacent pixels, three channels of RGB have a subtle
influence on the results. Thus, we transform all the RGB images into gray-
scale images. This condition can also reduce the computation expense and
improve the efficiency of the program.

The disadvantage of the models of LBP+Adaboost and HOG+SVM is that the
exhaustive search does not consider the distortion of a sample with the
original size which does not have the standard proportion of ${32}\times{64}$.
In the experiment of HOG+SVM, we use the size of ${32}\times{64}$ as the
standard size of samples. However, images can be distorted since we do not
just consider samples with such size. We collect all the images with a person
inside and resize them to ${32}\times{64}$ without considering the proportion.

##  4 Experiments

###  4.1 The results

Evaluating the quality performance consists of 2 values, including FPPI(false
positive per image) and miss rate. The results of the detections for an image
can only have 3 cases: detection matching the labelled areas(true positive);
detection not matching any labelled areas(false positive); the labelled areas
without any detection(miss positive). We focus on the last 2 cases which are
both negative results. The number of detections not matching any labelled
areas is the value of FPPI. The number of labelled areas without any detection
is the value of the miss rate. The final goal of the pedestrian detection
algorithm is to reduce them to become as low as possible. The lower they are,
the better the results are.

For each image, we have labelled areas and detected areas and both of them are
rectangles.

  * •

To calculate FPPI, we need to count the detections that do not match any
labelled areas in an image. When the overlap area in the labelled area and a
detected area is less than 50%, the detection does not match any labelled
area, which is a false positive result[41].

  * •

To calculate the miss rate, we need to count the labelled areas without
matching any detections in an image.

In order to describe the quality performance exclusively and the relationship
between speed and quality performance specifically, we use 2 kinds of graphs
to describe our experimental results. In this plot, the x-axis represents the
average time consumed by the model, the y-axis denotes the number of
occurrences of matching between the labelled area and the detection
area(Fig.4). It is said that when the intersection area is bigger than half of
the labelled area, it is regarded as match[41]. This plot aims to present the
relationship between speed and quality performance. The higher the value is,
the better the performance is.

Figure 2: Definition of intersection area between labeled area and detection
area

The efficiency of LBP+Adaboost is the highest among all of the three methods.
Since the computation of the feature is the fastest and the process of running
a decision tree is also fast. Moreover, according to the experiment, its
accuracy of detection is close to HOG+SVM. We assume that is because we do not
have enough training data to explore the further differences between them. The
model of Convolutional Neural Network based on darknet has the highest
accuracy.

###  4.2 The Improvements

The experimental results demonstrate that our methods work well for real-time
pedestrian recognition on the low computational CPU platform. However, we are
still inspired by many approaches to improve the speed of pedestrian
recognition. In the next work, we plan to implement the following approaches
for faster speed on the smaller computers with lower computation that work on
the devices with more constraints.

For the model of HOG+SVM, we expect that the accuracy can be further improved
by applying a Gaussian spatial window to the scale of the size of the sample,
no application of a Gaussian spatial window to blocks. This is also a way to
improve its efficiency. The computation of HOG has considered too many
factors, including Gaussian Window Application, magnitude and angle of
gradients, and different sizes of blocks, cells and samples. Some of these
features are redundancy. They need to be further experimented and extracted
the most critical factors of affecting the result. For example, the experiment
in C4[42] has mentioned that the comparison of neighbour pixels is more
important than the comparison of magnitude. Still, we do not know what kind of
information extracted by HOG is the most critical to make it become such
successful for human detection. If we can extract just one or two of the most
significant information for feature computing, we can make a breakthrough in
efficiency and accuracy.

The strategy of computing LBP feature with the basic unit is the batch the
pixels instead of a single pixel. It can also be applied to the computation of
HOG in further research.

The accuracy of convolutional neural networks based on darknet is better than
those of other models obviously. This is mainly attributed to the strategy of
generating features for convolutional neural network models. The feature of
convolutional combining with max pool is more general than other feature
including HOG and LBP, which strictly relies on the shapes and outlines of the
objects.

##  5 Conclusions

Since the computational capacity of the CPU is limited, we can only promote
efficiency by improving some processes in the program. In this work, we
investigate real-time pedestrian recognition on small physical-size computers
with low computational resources for faster speed. This paper presents three
methods that work on the small physical size CPUs system. First, we improved
the Local Binary Pattern (LBP) features and Adaboost classifier. Second, we
optimized the Histogram of Oriented Gradients (HOG) and Support Vector
Machine. Third, We implemented fast Convolutional Neural Networks (CNNs). The
results demonstrate that the three methods achieved real-time pedestrian
recognition at an accuracy of more than 95% and a speed of more than 5 fps on
a small physical size computational platform with a 1.8 GHz Intel i5 CPU. Our
methods can be easily applied to small mobile devices with high compatibility
and generality. In future, we will apply other AI technologies to improve this
work such as knowledge graph [43, 44].

## References

  * [1] G. Lan, Y. Wu, F. Hu, and Q. Hao, “Vision-based human pose estimation via deep learning: A survey,” IEEE Transactions on Human-Machine Systems, 2022.
  * [2] Z. Zhu, X. Wang, Y. Zhao, S. Qiu, Z. Liu, B. Chen, and F.-Y. Wang, “Crowdsensing intelligence by decentralized autonomous vehicles organizations and operations,” IEEE Transactions on Intelligent Vehicles, vol. 7, no. 4, pp. 804–808, 2022.
  * [3] H. Xu, G. Lan, S. Wu, and Q. Hao, “Online intelligent calibration of cameras and lidars for autonomous driving systems,” in 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 3913–3920, IEEE, 2019.
  * [4] J. Wu, N. Liu, C. Geyer, and J. M. Rehg, “${rmC}^{4}$: A real-time object detection framework,” IEEE Transactions on Image Processing, vol. 22, pp. 4096–4107, Oct 2013.
  * [5] G. Lan, Y. Bu, J. Liang, and Q. Hao, “Action synchronization between human and uav robotic arms for remote operation,” in 2016 IEEE International Conference on Mechatronics and Automation, pp. 2477–2481, Aug 2016.
  * [6] G. Lan, J. Sun, C. Li, Z. Ou, Z. Luo, J. Liang, and Q. Hao, “Development of uav based virtual reality systems,” in 2016 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pp. 481–486, IEEE, 2016.
  * [7] T. Xiang, F. Jiang, G. Lan, J. Sun, G. Liu, Q. Hao, and C. Wang, “Uav based target tracking and recognition,” in 2016 IEEE international conference on multisensor fusion and integration for intelligent systems (MFI), pp. 400–405, IEEE, 2016.
  * [8] G. Lan, M. Jelisavcic, D. M. Roijers, and E. Haasdijk, “Directed locomotion for modular robots with evolvable morphologies,” in Parallel Problem Solving from Nature–PPSN XV: 15th International Conference, Coimbra, Portugal, September 8–12, 2018, Proceedings, Part I 15, pp. 476–487, Springer, 2018.
  * [9] G. Lan, M. De Carlo, F. van Diggelen, J. M. Tomczak, D. M. Roijers, and A. E. Eiben, “Learning directed locomotion in modular robots with evolvable morphologies,” Applied Soft Computing, vol. 111, p. 107688, 2021.
  * [10] B. A. Taiana M., Nascimento J.C., “An improved labelling for the inria person data set for pedestrian detection,” in IbPRIA 2013, Iberian Conference on Pattern Recognition and Image Analysis, Berlin, Heidelberg, vol. 7887, pp. 286–295, 2013.
  * [11] Z. Sun, C. Meng, T. Huang, Z. Zhang, and S. Chang, “Marine ship instance segmentation by deep neural networks using a global and local attention (gala) mechanism,” Plos one, vol. 18, no. 2, p. e0279248, 2023.
  * [12] Z. Sun, C. Meng, J. Cheng, Z. Zhang, and S. Chang, “A multi-scale feature pyramid network for detection and instance segmentation of marine ships in sar images,” Remote Sensing, vol. 14, no. 24, p. 6312, 2022.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, June 2016.
  * [14] G. Lan, J. Benito-Picazo, D. M. Roijers, E. Domínguez, and A. Eiben, “Real-time robot vision on low-performance computing hardware,” in 2018 15th international conference on control, automation, robotics and vision (ICARCV), pp. 1959–1965, IEEE, 2018.
  * [15] G. Lan, L. De Vries, and S. Wang, “Evolving efficient deep neural networks for real-time object recognition,” in 2019 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 2571–2578, IEEE, 2019.
  * [16] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 886–893 vol. 1, June 2005.
  * [17] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 1, pp. I–511–I–518 vol.1, 2001.
  * [18] P. Viola and M. J. Jones, “Robust real-time face detection,” International Journal of Computer Vision, vol. 57, pp. 137–154, May 2004.
  * [19] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587, June 2014\.
  * [20] R. Girshick, “Fast r-cnn,” in Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV ’15, (Washington, DC, USA), pp. 1440–1448, IEEE Computer Society, 2015.
  * [21] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, pp. 1137–1149, June 2017.
  * [22] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788, June 2016.
  * [23] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in Computer Vision – ECCV 2016 (B. Leibe, J. Matas, N. Sebe, and M. Welling, eds.), (Cham), pp. 21–37, Springer International Publishing, 2016.
  * [24] G. Lan, M. van Hooft, M. De Carlo, J. M. Tomczak, and A. E. Eiben, “Learning locomotion skills in evolvable robots,” Neurocomputing, vol. 452, pp. 294–306, 2021.
  * [25] G. Lan, J. M. Tomczak, D. M. Roijers, and A. Eiben, “Time efficiency in optimization with a bayesian-evolutionary algorithm,” Swarm and Evolutionary Computation, vol. 69, p. 100970, 2022.
  * [26] C. Arth and H. Bischof, “Real-time object recognition using local features on a dsp-based embedded system,” Journal of Real-Time Image Processing., vol. 3, pp. 233–253, 2008.
  * [27] L. Van Gool, “Pedestrian detection at 100 frames per second,” in Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’12, (Washington, DC, USA), pp. 2903–2910, IEEE Computer Society, 2012.
  * [28] S. Nawaz, “Hog-svm car detection on an embedded gpu,” Master’s thesis, 11 2015\.
  * [29] H. Mao, S. Yao, T. Tang, B. Li, J. Yao, and Y. Wang, “Towards real-time object detection on embedded systems,” IEEE Transactions on Emerging Topics in Computing, vol. PP, no. 99, pp. 1–1, 2017.
  * [30] S. Tripathi, G. Dane, B. Kang, V. Bhaskaran, and T. Nguyen, “Lcdet: Low-complexity fully-convolutional neural networks for object detection in embedded systems,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 411–420, July 2017.
  * [31] G. Lan, J. Chen, and A. Eiben, “Evolutionary predator-prey robot systems: From simulation to real world,” in Proceedings of the genetic and evolutionary computation conference companion, pp. 123–124, 2019.
  * [32] G. Lan, J. Chen, and A. Eiben, “Simulated and real-world evolution of predator robots,” in 2019 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1974–1981, IEEE, 2019.
  * [33] J. Lee, J. Wang, D. Crandall, S. Šabanović, and G. Fox, “Real-time, cloud-based object detection for unmanned aerial vehicles,” in 2017 First IEEE International Conference on Robotic Computing (IRC), pp. 36–43, April 2017.
  * [34] Y. Nimmagadda, K. Kumar, Y. H. Lu, and C. S. G. Lee, “Real-time moving object recognition and tracking using computation offloading,” in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2449–2455, Oct 2010.
  * [35] D. Guo, F. Sun, T. Kong, and H. Liu, “Deep vision networks for real-time robotic grasp detection,” International Journal of Advanced Robotic Systems, vol. 14, no. 1, p. 1729881416682706, 2017.
  * [36] C. Huang, P. Chen, X. Yang, and K. Cheng, “REDBEE: A visual-inertial drone system for real-time moving object detection,” CoRR, vol. abs/1712.09162, 2017.
  * [37] G. Lan, Z. Gao, L. Tong, and T. Liu, “Class binarization to neuroevolution for multiclass classification,” Neural Computing and Applications, vol. 34, no. 22, pp. 19845–19862, 2022.
  * [38] B. Kisacanin, S. S. Bhattacharyya, and S. Chai, Embedded Computer Vision.  Springer Publishing Company, Incorporated, 1 ed., 2008.
  * [39] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge: A retrospective,” International journal of computer vision, vol. 111, no. 1, pp. 98–136, 2015.
  * [40] T. Ojala, M. Pietikäinen, and D. Harwood, “A comparative study of texture measures with classification based on featured distributions,” Pattern recognition, vol. 29, no. 1, pp. 51–59, 1996.
  * [41] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International journal of computer vision, vol. 88, no. 2, pp. 303–338, 2010.
  * [42] J. Wu, C. Geyer, and J. M. Rehg, “Real-time human detection using contour cues,” in Robotics and Automation (ICRA), 2011 IEEE International Conference on, pp. 860–867, IEEE, 2011.
  * [43] G. Lan, T. Liu, X. Wang, X. Pan, and Z. Huang, “A semantic web technology index,” Scientific reports, vol. 12, no. 1, p. 3672, 2022.
  * [44] T. Liu, G. Lan, K. A. Feenstra, Z. Huang, and J. Heringa, “Towards a knowledge graph for pre-/probiotics and microbiota–gut–brain axis diseases,” Scientific Reports, vol. 12, no. 1, p. 18977, 2022.