# RBF Weighted Hyper-Involution for RGB-D Object Detection

Mehfuz A Rahman
Graphics and Spatial Computing Lab
Saint Mary’s University
Halifax, Canada
<EMAIL_ADDRESS>
& Jiju Peethambaran
Graphics and Spatial Computing Lab
Saint Mary’s University
Halifax, Canada
<EMAIL_ADDRESS>
&Neil London
Modest Tree Media Inc.
Halifax, Canada
<EMAIL_ADDRESS>

###### Abstract

A vast majority of conventional augmented reality devices are equipped with
depth sensors. Depth images produced by such sensors contain complementary
information for object detection when used with color images. Despite the
benefits, it remains a complex task to simultaneously extract photometric and
depth features in real time due to the immanent difference between depth and
color images. Moreover, standard convolution operations are not sufficient to
properly extract information directly from raw depth images leading to
intermediate representations of depth which is inefficient. To address these
issues, we propose a real-time and two stream RGBD object detection model. The
proposed model consists of two new components: a depth guided hyper-involution
that adapts dynamically based on the spatial interaction pattern in the raw
depth map and an up-sampling based trainable fusion layer that combines the
extracted depth and color image features without blocking the information
transfer between them. We show that the proposed model outperforms other RGB-D
based object detection models on NYU Depth v2 dataset and achieves comparable
(second best) results on SUN RGB-D. Additionally, we introduce a new outdoor
RGB-D object detection dataset where our proposed model outperforms other
models. The performance evaluation on diverse synthetic data generated from
CAD models and images shows the potential of the proposed model to be adapted
to augmented reality based applications.

_Keywords_ =

RGB-D, Detection, Depth, Convolution, Involution, Fusion

##  1 Introduction

Object detection aims to classify and localize an object of interest from a
two/three-dimensional scenes. Recognition of objects is an integral part of
autonomous robotics and augmented reality (AR) applications, and hence has
attracted a lot of interest among the computer vision community. The research
on this topic has made significant progress over the recent past with the help
of deep learning models. However, most of the existing state-of-the-art object
detection models are built for two-dimensional (2D) RGB (Red, Green, Blue)
images with little or no three-dimensional (3D) perspective of the objects
which is crucial for applications such as autonomous driving and scene
understanding besides augmented/mixed reality applications. A few deep
learning models specifically address 3D object detection from point clouds
such as Light Detection and Ranging (LiDAR) scans Pan et al. (2021); Qi et al.
(2021); Tian et al. (2021). However, LiDAR sensors for point cloud generation
are expensive and produce sparse output that requires a lot of pre-processing.

Figure 1: Few instances where the usefulness of depth for object detection are
visible. Image courtesy: Nathan Silberman and Fergus (2012); Polseno (2020);
Ranftl et al. (2021); Rankuzz.com (2020); Starecat.com (2022)

There has been a rapid improvement and increased availability of affordable
commercial depth sensors over the last decade. Depth sensors have also become
a conventional part of many modern AR headsets (e.g. Microsoft HoloLens 2).
These depth sensors can capture depth images (also known as depth maps) where
each pixel encodes the distance of a discrete point in the scene from the
sensor. When the depth images are used with its corresponding color images, we
get four channel RGB-D (red, green, blue, depth) images. Prior state-of-the-
art research Gupta et al. (2014); Xiao et al. (2021) has already proven the
significance and the performance improvement of RGB-D based object detection
over RGB based detection. Depth images complement RGB based object detection
in multiple ways. Firstly, depth images better visualize object boundaries,
making it easier to locate objects and properly cover them with bounding
boxes. This is particularly important in cases where the object boundaries are
not clear in color images due to poor illumination or heavy shadows, as shown
in Figure 1(a). Secondly, depth images can resolve scale distortions that
often appear in color images due to perspective projections. Depth images
provide useful information to object detectors, making it easier to learn the
relative sizes of objects in a scene. One such phenomenon is illustrated in
Figure 1(b). Thirdly, depth images can detect camouflaged objects that might
not be easily visible in color images due to their similarity to their
background which is demonstrated with the RGB and corresponding depth map of a
penguin in Figure 1(c). Finally, depth images can handle delusive color and
texture in images (Figure 1(d)) that can mislead the object classification if
solely relied on color and texture information.

Despite having conclusive evidence about the benefits of using extra depth
information, it is challenging to process depth map and color image inputs
simultaneously in object detection models due to the fundamental differences
in depth and color images. Consequently, over the past few years, RGB-D based
object detection has been tackled using two stream networks to extract
features from color and depth images separately and then combining these
features at selected stages of the model Gupta et al. (2016, 2014); Ophoff et
al. (2018, 2019). However, most fusion stages of the extracted depth and color
features are naively selected. Further, such fusion schemes employ simple
concatenation of features that lack proper learnable parameters to train with
backpropagation of neural networks. More importantly, the depth and color
feature fusion stage sometimes blocks proper information exchange between
depth and color image features Xiao et al. (2021). Moreover, some researchers
encode the depth map into a different representation Gupta et al. (2014); Li
et al. (2018); Xu et al. (2017) which is time consuming and designed based on
intuition. The standard convolutional operation is designed considering
feature extraction from color image but not from raw depth image. Therefore,
there is a need to find an alternative for standard convolution to directly
process raw depth image. Further, most of the state-of-the-art RGB-D object
detection models rely on two-stage detectors from outmoded RCNN series of
models Girshick (2015); Girshick et al. (2014) which makes them considerably
slower when compared to more recent real-time object detection Bochkovskiy et
al. (2020); Tan et al. (2020); Wang et al. (2021a) models.

We attempt to tackle some of the above mentioned issues using a depth aware
involution based fusion network for RGB-D object detection. The proposed
single stage architecture, shown in Figure 1, works in real-time incorporating
two new components with notable performance. The specific contributions of
this work are listed below.

  * •

We propose a dynamic depth aware hyper-involution module as an alternative to
standard convolution for proper utilization of raw depth information and
spatial specific features.

  * •

We propose an improved encoding-decoding type fusion stage in the middle
layers of the model that can combine the features extracted from depth stream
and RGB stream to extract the most significant semantic information.

  * •

We develop a pipeline to automatically generate realistic RGB-D images from 3D
CAD models and background images for training and testing the performance and
applicability of the detection model in diverse environment.

  * •

We build a completely new outdoor RGB-D dataset with annotations for RGB-D
based object detection.

##  2 Related Works

This section explores the background on three different topics according to
our specific research objectives. In recent years, the research community has
fervently introduced a plethora of state-of-the-art models for conventional
RGB-based object detection. The object detection architectures can be
categorized into two groups namely: single stage and two-stage detectors Zaidi
et al. (2022). Single stage detectors predict the position and class label of
the object within an image in a single pass through the neural network without
the need for additional region proposals or refining components. At the
moment, the leading single stage models Wang et al. (2022, 2021a, 2021b) are
the successors of YOLO Redmon et al. (2016); Redmon and Farhadi (2017, 2018)
and FCOS Tian et al. (2019, 2020) series. Conversely, two-stage detectors use
a combination of two neural networks to detect objects in the image. First the
region proposal network (RPN) generates a set number of potential locations
where objects may be present in the image. These proposals are then passed to
the detection network which refines location and identification of the objects
in the proposals. Some latest addition to state-of-the-art two-stage models
includes Hong et al. (2022); Sun et al. (2021). Overall, these RGB based
detection models mainly introduce various components in their extended
architecture to compete for speed and accuracy ignoring the importance of
cross modal perception. In this paper, we investigate research challenges of
RGB-D based object detection and develop an improved model for RGB-D based
object detection. Therefore, this section first describes various existing
RGB-D object detection architectures including their limitations followed by
brief studies on alternatives to standard convolution and hyper-networks which
are core components of our RGB-D based detection.

Convolution Name |  Input |  Specialty |  Output |  Type |  Computation and Parameters
---|---|---|---|---|---
Standard convolution LeCun et al. (1998) |  RGB |  Learns various important image features |  Feature tensor |  Static |  Varies
Deformable convolution Dai et al. (2017) |  RGB |  Learns geometric transformation along with image features |  Feature tensor |  Dynamic |  Higher than standard convolution
DCNv2 Zhu et al. (2019) |  RGB |  Improved version of Deformable convolution and avoids irrelevant regions in image |  Feature tensor |  Dynamic |  Higher than standard convolution
PAC Su et al. (2019) |  RGB |  Adapts according to the content of images |  Feature tensor |  Dynamic |  Higher than standard convolution
CondConv Yang et al. (2019) |  RGB |  Input samples specific learning |  Feature tensor |  Dynamic |  Higher than standard convolution
Dynamic convolution Chen et al. (2020) |  RGB |  Superposition of several convolution filters |  Feature tensor |  Dynamic |  Higher than standard convolution
Depth aware conv Wang and Neumann (2018) |  RGB, Depth map |  Apply a weight based on depth similarity only for semantic segmentation |  Feature tensor |  Dynamic |  Similar to standard convolution
S-convChen et al. (2021) |  RGB, Depth map |  Learns spatial information from depth for better semantic segmentation |  Feature tensor |  Dynamic |  Higher than standard convolution
Depth guided filtering Ding et al. (2020) |  RGB, Depth map |  Filters and dilations are varied according to specific pixels for monocular object detection task |  Feature tensor |  Dynamic |  Higher than standard convolution
Depth-wise convolution Ma et al. (2018); Sandler et al. (2018); Tan and Le (2019); Chollet (2017) |  RGB |  Helps to improve efficiency of convolutional network |  Feature tensor |  Dynamic |  Less than standard convolution
Involution Li et al. (2021) |  RGB |  Spatial specific and channel agnostic |  Feature tensor |  Dynamic |  Significantly less than standard convolution
Table 1:  A brief summary of existing alternatives to standard convolution.

###  2.1 RGB-D Object Detection

####  2.1.1 HHAs

Gupta et al.Gupta et al. (2014) introduced a fusion-based model for the task
of RGB-D based object detection. This is the first work that verified
important arguments in favor of depth-aware methods to improve object
detection performance. Moreover, they also introduced a geocentric embedding
technique to convert raw depth images to three-channel HHA format (Horizontal
disparity, Height above ground, and Angle with respect to gravity direction)
before giving input to their model for extracting depth features. However, the
depth image to HHA conversion process is hand designed which is unnecessarily
time consuming Hazirbas et al. (2016). In a sequel work, Gupta et al.Gupta et
al. (2016) addressed scarcity of depth data for the training of RGBD models.
The authors utilized supervision transfer which basically train the depth
feature extraction backbone by teaching the network to regenerate the semantic
representations at intermediate level learned from RGB based backbone pre-
trained with a massive RGB image dataset. Although this strategy improved the
accuracy when compared to their previous work Gupta et al. (2014), it relies
on two-stage Fast RCNN detector Girshick (2015) which is not suitable for real
time applications. Xu et al. Xu et al. (2017) also utilized the concept of
supervision transfer Gupta et al. (2016) and proposed a three-stream model
that slightly improved the performance of RGB-D detection. Nevertheless, this
model also relies on the time consuming HHA conversion Girshick et al. (2014)
of raw depth map and the three parallel backbones of the model, inspired from
AlexNet Krizhevsky et al. (2012), has its own Region Proposal Network (RPN)
and separate Faster RCNN head which further adds to the training time and
computational cost. Cross-Modal Attentional Context (CMAC) algorithm proposed
by Li et al. Li et al. (2018) utilized Long Short Term Memory (LSTM)
Hochreiter and Schmidhuber (1997) to extract global context features from each
region proposals and Spatial Transformer Networks (STN) Jaderberg et al.
(2015) to accurately identify different parts of an object. However, this
model also relies on HHA conversion of depth and there are some disadvantages
of using LSTM as it consumes more memory, prone to easy over-fitting and
sensitive to random weight initialization.

####  2.1.2 Raw Depth Maps

Some of the recent work on RGB-D object detection use raw depth map instead of
converting it to HHA with their specific limitations. For instance, Zhang et
al. Zhang et al. (2020a) introduced a model that consists of three major
streams including a backbone for feature extraction from raw depth map and
Channel Weights Fusion (CWF) that process the concatenated RGB-D features.
However, in this model depth feature extraction prior is designed based on
several intuition and considering only human depth image pattern inside indoor
environment which potentially limits its capacity to extract depth information
in diverse environments. In another work, Ophoff et al. Ophoff et al. (2018)
explored three different stages of feature fusion for RGB-D based pedestrian
detection. For each fusion stages of the model, a single stage object detector
is utilized making it suitable for real time applications. Despite the real-
time advantage, this work has drawbacks including naïve concatenation of the
depth and RGB image features without any special trainable operation and the
issue of increase in dimensions after feature concatenation. In Ophoff et al.
(2019), the authors extended the model for multi-class object detection in
real time and proposed simple fusion layer, i.e., use of convolution after
concatenation to reduce the combined feature dimension. However, this model
requires separate pre-training of depth and RGB network before training the
main model and hence, making it redundant. A recent work Xiao et al. (2021)
introduced two components to improve the information flow between depth and
RGB features in RGB-D object detection. These two components help to bring
significant performance improvements compared to state-of-the-art. Overall,
despite several strategies to improve feature fusion of depth and RGB images,
none of these works explored the effectiveness or alternatives of standard
convolution operation to properly extract depth data.

###  2.2 Alternatives to Standard Convolution

In the recent past, different flexible and effective alternatives of standard
convolution operation LeCun et al. (1998) have been proposed. A few of them
dynamically adapt using pixel information while others adapt using depth. For
instance, deformable convolution Dai et al. (2017) learns geometric
transformations of images such as scale, pose and deformation of parts. Then a
faster and lightweight deformation convolution called deformable ConvNets v2
(DCNv2) Zhu et al. (2019) were introduced, that remains unaffected by features
from irrelevant regions of the image which was an issue in Dai et al. (2017).
Pixel Adaptive Convolution (PAC) Su et al. (2019) is adapted according to the
contents of images while maintaining several favorable properties of standard
convolution. A conditionally parameterized convolution, named CondConv Yang et
al. (2019), can be learned based on specific input samples. Similarly, dynamic
convolution Chen et al. (2020) adapts based on input samples and can be
described as a superposition of multiple convolution kernels. Before applying
the superposition, the kernels are aggregated by a value found by applying an
attention model function on the input. Several studies attempted to utilize
depth maps to manipulate convolution kernels. For example, Wang and Neumann
(2018) introduced two modules which are referred to as depth-aware convolution
and depth-aware average pooling where the output gets more impacted by pixels
with similar depth value. Chen et al. introduced a different convolution
module Chen et al. (2021), called s-conv, that improve segmentation
performance by applying dimensional information to its filter weights and
generating location adaptive filters. ShapeConv Cao et al. (2021) is another
recent work that use depth map to extract information about the content of the
patch besides its whereabout to improve the accuracy of semantic segmentation.
For 3D object detection from images and depth map, authors in Ding et al.
(2020) introduced a depth guided filtering scheme where the convolution
filters and dilations are varied according to specific pixels and channels of
different images. Another line of research deals with depth-wise convolution
Ma et al. (2018); Sandler et al. (2018); Tan and Le (2019); Chollet (2017)
which aims to improve efficiency of neural network, but this area of research
should not be confused with convolution that are manipulated by depth input.

Figure 2: The proposed two streams and single stage detection architecture for
real-time applications.

Contrary to the above facts, each of these alternative to the standard
convolution has their own set of limitations. Like DCNv2 Zhu et al. (2019) is
much slower and has more parameters compared to standard covolution kernel
while CondConv Yang et al. (2019) and Dynamic convolution Chen et al. (2020)
are less effective at lower layers of a model compared to higher layers.
Moreover, the depth based convolutional operations were only designed for task
like semantic segmentationWang and Neumann (2018); Chen et al. (2021); Cao et
al. (2021) or 3D monocular object detection Ding et al. (2020). On a seperate
note, a recently introduced concept called Involution Li et al. (2021)
reversed the fundamental concept of standard convolution to overcome problems
like inter-channel redundancy and inability to learn long distance visual
interactions. This approach shows great promise, as it is dynamic and requires
significantly fewer parameters than other types of standard convolutions.
Therefore in this research, we chose to modify involution to dynamically deal
with raw depth input. Table 1 summarizes standard convolution alternatives.

###  2.3 Hyper-networks

Increasing the filter size of convolutional layers are proven to be useful for
better capturing the long range information of neural networks Krizhevsky et
al. (2017); Ronneberger et al. (2015). In other words, the larger kernels
helps to increase the expressiveness of convolution. However, the problem is
that the trainable parameters of convolution layers increases significantly
with filter size and hence increasing the computational cost. To this end, Ha
et al. Ha et al. (2016) introduced a useful concept called Hyper-networks that
can improve the expressiveness of neural network model without increasing the
parameters count. The key idea here is to use a secondary neural network to
generate weights for the main network. Using this concept Ha et al. achieved
decent classification performance while reducing the number of parameters. Two
other research from Wang et al. Wang et al. (2021c) and Hoopes et al. Hoopes
et al. (2021) has showed the efficacy of hyper-networks for training deep
neural networks that is compatible to the extent of regularization. Most
recently Ma et al. Ma et al. (2022) introduced hyper-convolution that uses
hyper-network to generate filter weights which help them to increase filter
size without affecting the parameters of convolution. This hyper-convolution
helped them to create a parameter efficient model for the task of biomedical
image segmentation. However, the parameters of their hyper-network still
depends on the number of input channels, output channels and number of nodes
in the final layer of the hyper-network. Similarly, Nirkin et al. Nirkin et
al. (2021) developed a patch-wise hyper-network, called HyperSeg, which
generates the weights of each block in the decoder immediately before they are
consumed to solve a segmentation task.

##  3 The Model

In this section, we first introduce the main RGB-D detection architecture.
Then we discuss the two main modules namely the depth aware hyper-involution
and fusion designed specifically for the RGB-D detection. Finally, the
synthetic RGB-D data generation pipeline is described in detail.

###  3.1 The Two Streams Architecture

As discussed in Section 2, most of the existing state-of-the-art RGB-D object
detection models rely on two-stage detection architecture which negatively
impact their real-time speed. Therefore, we design a single stage detector
architecture which unlike a two-stage detector, does not require a separate
sparse prediction stage and predicts bounding boxes in a single pass through
the neural network. As demonstrated in Figure 2, first this model takes color
image and its corresponding depth map as input to two different streams of the
network. One stream of the network containing the depth aware hyper-involution
(described in Section III. B) followed by a pooling layer are responsible for
extracting the color image features with parallel attention to object’s depth.
The second stream of the network processes complementary semantic features
from the corresponding depth map using a hyper-involution (which has the same
filter generator as the depth aware hyper-involution, described in III. B.4,
but excluding the depth aware filter) followed by pooling layer to make shapes
compatible prior to the information fusion. The information extracted from the
two streams of network is then combined using fusion stage described in
Section III. C. The model after the fusion consists of a backbone network with
13 convolutional layers, shown in Figure 2, which is inspired by the success
of Simonyan and Zisserman (2014). An interesting feature of this backbone
structure is that instead of having a large number of hyper-parameters, it has
convolution layers of 3x3 filter with stride 1 and always use the same padding
and maxpool layer of 2x2 filter of stride 2. This backbone structure plays a
crucial role in significantly reducing the overall computational complexity of
the detection model. Final stage of the detection model comprise a detection
head that provides the final classification and localization prediction via
non-max suppression layer. We use the loss function suggested by Redmon and
Farhadi (2017) because of its compatibility with this model output and success
in state-of-the-art single stage detectors Huang et al. (2018); Jo et al.
(2017).

###  3.2 Depth Aware Hyper-involution

Depth aware hyper-involution, as shown in Figure 4, is a module that we design
as an alternative to the standard convolution to ensure that spatial and depth
information is accounted while processing the color image features. To get an
idea of this module, first we need to understand the basic operation and
difference of convolution and involution.

####  3.2.1 Standard Convolution

A standard convolution LeCun et al. (1998) is the weighted sum of local
regions as a fixed sized filter moves in a sliding window fashion over an
image. To elaborate this further, imagine an image tensor $I$ of height $H$,
width $W$ and channels $C_{i}$. Each pixels inside the tensor can be denoted
as $I_{i,j}\in\mathbb{R}^{C_{i}}$ representing different image features. Let
us denote a set of convolution kernels of size $F\times F$ as
$\mathcal{K}\in\mathbb{R}^{C_{n}\times C_{i}\times F\times F}$ where $C_{n}$
represents the number of kernels. When a set of convolution kernels undergo
element wise multiplication and addition while sliding over the image tensor,
the final output feature tensor can be defined using Equation 1.

$O_{i,j,k}=\sum_{c=1}^{C_{i}}\sum_{m=\lfloor\frac{-F}{2}\rfloor}^{\lfloor\frac{F}{2}\rfloor}\sum_{n=\lfloor\frac{-F}{2}\rfloor}^{\lfloor\frac{F}{2}\rfloor}\mathcal{K}_{k,c,m+\lfloor\frac{F}{2}\rfloor,n+\lfloor\frac{F}{2}\rfloor}I_{i+m,j+n,c}$
(1)

In Equation 1, $k\in[1,C_{n}]$ and $m$ and $n$ index the offset positions in
the kernel. One can notice that the problem with the convolution operation is
that it applies a fixed convolution filter at every spatial positions in the
image, also referred to as spatial agnostic feature, which suggests that it
does not account for the difference in different spatial position in the
image. Moreover, it applies separate filters for separate channels of the
input image, referred to as channel specific feature, which is considered as a
redundant operation adding to the computational cost.

Figure 3: The working mechanism of Involution. The involution kernel
$\mathcal{H}^{i,j}$ (where G=1 for simplicity) is obtained by applying the
function $\phi$ on a single pixel located at $(i,j)$ and then rearranging the
channels to form a spatial neighborhood. The element wise multiplication and
addition operation in involution is split into two steps as shown by the
$\otimes$ and $\oplus$, respectively, Image courtesy: Li et al. (2021).

####  3.2.2 Involution

Figure 4: The working mechanism of Depth Aware Hyper-involution. The depth
similarity is calculated from the depth map to produce a depth aware filter.
Meanwhile, the filter generating hyper-network generate learned filter weights
efficiently for each spatial region of the color image. These filters then
undergo multiply and add operation with the input to generate the value of the
output pixel.

To address the above issues of standard convolution, involution operation Li
et al. (2021) has been put forward. The main difference between the involution
and the convolution is the spatial specific and channel agnostic features.
Involution basically incorporates a generalized version of self-attention
mechanisms that enable them to focus on specific regions of the input image
and capture long-range dependencies. This enhances the module’s ability to
model complex spatial relationships in the data, making it a potentially more
effective approach for image processing tasks. Additionally, the channel
agnostic aspect helps to efficiently reduce parameters while still maintaining
its ability to capture complex visual pattern in the data. Precisely, an
involution kernel of size $F\times F$ can be denoted as
$\mathcal{H}\in\mathbb{R}^{H\times W\times F\times F\times G}$ where $G$
indicates the group of channels ($C$) in the input tensor that shares the same
involution kernel. When such involution kernels undergo element wise
multiplication and addition on the image tensor, the final output feature
tensor can be defined as in Equation 2,

$O_{i,j,k}=\sum_{m=\lfloor\frac{-F}{2}\rfloor}^{\lfloor\frac{F}{2}\rfloor}\sum_{n=\lfloor\frac{-F}{2}\rfloor}^{\lfloor\frac{F}{2}\rfloor}\mathcal{H}_{m+\lfloor\frac{F}{2}\rfloor,n+\lfloor\frac{F}{2}\rfloor,\lceil\frac{kG}{C}\rceil}^{i,j}I_{i+m,j+n,k}$
(2)

In Equation 2, $\mathcal{H}^{i,j}$ represents the involution kernel which is
dynamically sampled from pixel position $I_{i,j}$ in the input tensor.
Therefore, unlike the fixed filter of convolution operation, the involution
filter is dynamically generated based on each spatial position of the input
images as shown in $\mathbf{Figure}$ 3. This characteristic helps the
involution operation to give distinct focus on each spatial position in the
image. Moreover, involution applies the same filter for a group channels in
the input image thereby using much less parameters compared to a standard
convolution and hence, reduces the memory consumption.

Figure 5: Difference between pixels of RGB image and its corresponding depth
map.

####  3.2.3 Depth Aware Involution

Nevertheless, involution was designed specifically considering the feature
extraction from color image. It remains unaware about the depth of each pixel
or spatial information while extracting feature from the color image. For
example, the RGB image in Figure 5 highlights three pixels where pixels L, M
and N have the same pixel color as the chair and table has the same dark
color. However, upon examining the depth map shown in Figure 5, it becomes
clear that the depth of pixel L differs from that of pixels M and N. This is
because the depth of pixel L is influenced by the chair, which is closer to
the sensor than the part of the desk that pixels M and N correspond to.

To alleviate the effects of such unaccounted depth disparities in the
detection accuracy, we redesign the involution operation to consider the
spatial and geometric patterns from the depth map. Given an input image tensor
$I$ and depth map $D$, the output of our depth aware hyper-involution
operation is formulated as follows (Equation 3).

$O_{i,j,k}=\sum_{m=\lfloor\frac{-F}{2}\rfloor}^{\lfloor\frac{F}{2}\rfloor}\sum_{n=\lfloor\frac{-F}{2}\rfloor}^{\lfloor\frac{F}{2}\rfloor}\mathcal{P}_{m+\lfloor\frac{F}{2}\rfloor,n+\lfloor\frac{F}{2}\rfloor,\lceil\frac{kG}{C}\rceil}^{i,j}\\\
\mathbf{W}_{m+\lfloor\frac{F}{2}\rfloor,n+\lfloor\frac{F}{2}\rfloor}^{i,j}I_{i+m,j+n,k}$
(3)

where $\mathcal{P}^{i,j}$ represents the kernel that is dynamically generated
via a new parameter-efficient filter generation hyper-network (described in
Section 3.2.4) which is conditioned on the pixel $I_{i,j}$.
$\mathbf{W}_{m+\lfloor\frac{F}{2}\rfloor,n+\lfloor\frac{F}{2}\rfloor}^{i,j}$
is a weighing function that captures the depth similarity between two pixels
$D_{i,j}$ and $D_{i+m,j+n}$ as in Equation 4.

$\mathbf{W}_{p,q}^{i,j}=\frac{1}{\sqrt{1+(\gamma\cdot(d(D_{i,j})-d(D_{p,q})))^{2}}}$
(4)

Figure 6: Detection accuracy comparison using different depth similarity
weighting functions on three different datasets.

In Equation 4, $d(D_{i,j})$ and $d(D_{p,q})$ denotes the corresponding depth
values at position $D_{i,j}$ and $D_{i+m,j+n}$, respectively. The choice of
Equation 4 is based on the idea that the depth differences of various spatial
location and objects in the real scene should be addressed by using depth
pixels from the depth map instead of solely relying on color that can often
mislead like the one in Figure 5. Additionally, the function decay rate is
controlled by the parameter $\gamma$. Section B.4 discusses a performance
comparison that aims to investigate the impact of different depth weighing
function options for the depth aware-hyper involution. The value of $\gamma$
is a constant which can be tuned until the detection model reaches desired
accuracy. In our case, the optimal value of $\gamma$ was 9.5 after testing in
the range 0.5 to 10 with an interval of 0.5. More importantly, Equation 4
calculation does not add any extra parameter to equation 3. Furthermore, it is
important to know that almost all the RGB-D datasets used for this research
rely on different existing algorithms to deal with missing depth pixel values.
For example, NYU Depth v2 uses in-painting algorithm Levin et al. (2004) while
SUN RGBD uses a different depth map improvement algorithm to estimate missing
depth values Song et al. (2015). Therefore, this equation is not affected by
missing depth pixels. Note that the hyper-involution shown in Figure 2 in our
main object detection algorithm has the same filter generation technique like
the depth aware hyper-involution but does not have the depth aware part
$\mathbf{W}_{m+\lfloor\frac{F}{2}\rfloor,n+\lfloor\frac{F}{2}\rfloor}^{i,j}$
since it is used to extract complementary semantic features from depth map.

Figure 7: Graph plots of various RBF functions discussed.

####  3.2.4 Depth Weighting Functions

We considered radial basis function (RBF) as our depth weighing function. An
RBF is a function that calculates a real number output solely based on the
distance between the input and a constant reference point. This reference
point can be either the origin or a specific center point buhmann2003radial.
To quantitatively verify the usefulness of the proposed RBF depth weighing
function in Equation 4, we compare the performance with three other RBF
kernels. First, we evaluate with a Gaussian function, shown in Equation 5,
where the value decreases as the difference between two depth values increases
and vise-versa.

$\mathbf{W}_{p,q}^{i,j}=e^{-(\gamma|d(D_{i,j})-d(D_{p,q})|)^{2}}$ (5)

Figure 8: The filter generation hyper-network. This network samples each
pixels in the RGB to learn the filter weights individually for each spatial
region of the image.

In Equation 5, $d(D_{i,j})$ and $d(D_{p,q})$ denotes the corresponding depth
values at $D_{i,j}$ and $D_{i+m,j+n}$, respectively. The exponent is used in
Equation 5 because it allows the function to decay rapidly as the difference
between two depth values increases. To put it simply, when there is a greater
difference in depth, the function returns a smaller value. Additionally, the
exponential function decay rate is controlled by the parameter $\gamma$. Next,
we tried Triangular function (Equation 6).

$\mathbf{W}_{p,q}^{i,j}=\max(1-|d(D_{i,j})-d(D_{p,q})|,0)$ (6)

For Equation 6, the depth similarity value will always remain in range
$[0,1]$. Then we test our model with Equation 7 that was first introduced in
wendland1995piecewise, which is also referred as the Wendland $c^{2}$
function.

$\mathbf{W}_{p,q}^{i,j}=(1-(d(D_{i,j})-d(D_{p,q}))^{4}+(4\cdot(d(D_{i,j})-d(D_{p,q}))+1)$
(7)

The graph plots in Figure 7 demonstrate how the weighting on these kernels
varies with depth similarity. Equation 4 contributes to the optimal detection
performance when compared on various datasets, as illustrated in Figure 6.

####  3.2.5 Filter Generation Hyper-network

We utilize a new function to map each 2D input kernel coordinate to the kernel
value as demonstrated in Figure 8. The function is basically a parameter
efficient hyper-network. The depth aware hyper-involution kernel weights are
thus generated by a neural network (hyper-network) instead of independent
learning. The trained weights of the kernel of a specific spatial location
$\theta_{ij}$ can be represented using the following function (Equation 8).

$\theta_{i,j}=N_{2}\cdot\lambda(N_{1}\cdot X_{i,j})$ (8)

In Equation 8, $N_{1}$ and $N_{2}$ represent two linear transformations that
collectively constitute a hyper-network. $N_{1}$ is implemented via 3 layers
of 1$\times$1 convolution where first two layers contains 8 filters with non-
linear activation functions and the last layer consist of 6 filters.
Meanwhile, $N_{2}$ is implemented using a single filter of 1$\times$1
convolution followed by a broadcasting of the output based on the size of the
kernel. $\lambda$ implies batch normalization and non-linear activation
functions that interleave two linear projections. The main advantage of using
this hyper-network in our depth aware hyper-involution is that the number of
trainable parameters remains independent of the choice of the kernel size
which is not possible in involution Li et al. (2021) and standard convolution
LeCun et al. (1998). Thus, the expressiveness of our depth aware hyper-
involution can be increased with larger kernel size while keeping the number
of trainable parameters constant. Note that the hyper-network used in Ma et
al. (2022) is also independent of kernel size but it still depends on the
number of input channels, output channels and number of nodes in the final
layer of their hyper-network. Whereas our hyper-network does not rely on the
the number of channels or number of nodes as these values remains constant. It
is also worth mentioning that the name hyper-involution is motivated by the
use of such an efficient hyper-network Ha et al. (2016).

####  3.2.6 Visual Analysis

Figure 9: The heat maps in each row interpret the generated filters for an
image instance from the NYU Depth v2 dataset. The columns after the input
images illustrate the kernels of convolution, involution and depth aware
hyper-involution respectively.

To visually analyze the trained depth aware hyper-involution kernel, we pick
the sum of F $\times$ F values from each kernel (here F represent the height
and width of the kernel) as its representative value and compare it with
similarly trained convolution and involution kernels. All the representatives
at various geometric positions represent the corresponding heat map. A number
of these heatmaps are demonstrated in Figure 9 where the columns following the
input images represent mapping of learned kernels of convolution, involution
and our depth aware hyper-involution respectively. From Figure 9, it is
visible that depth aware hyper-involution is better at capturing various
important semantic features of the input images by using the extra information
from depth map. To be more specific, if one notice the heatmap of first row
last column of Figure 9 the bookshelf at the back is properly mapped capturing
all its sharp edges by the depth aware hyper-involution while the right corner
of the bookshelf are obscured in the respective mapping of convolution and
involution due to darkness. This clearly bolster the idea that our depth aware
hyper-involution can highlight sharp edges regardless of darkness by utilizing
depth information. Another observation can be the second row last column of
Figure 9 where the depth aware hyper-involution clearly maps and differentiate
the darker regions of the input by highlight it with yellow color when
compared with involution and convolution. In this image one can also notice
that the depth aware hyper-involution also gives similar color coding in the
heat map to the pairs of chairs that are in the same depth from the camera
viewpoint which is possible by the extra information from depth. Moreover,
depth aware hyper-involution seems to be more superior at capturing objects
outer surface detail of the input compared to involution and convolution as
shown in the image in last row and forth column of Figure 9 where the flower
texture in the bed are better mapped by the depth aware hyper-involution than
the other two filters. Depth aware hyper-involution is also superior at
preserving the texture information at different spatial regions of the
original image which can be deduced from the image in third row last column of
Figure 9 where the texture details of the floor is mapped with greater detail
by the depth aware hyper-involution kernel which is an advantage of its
spatial specific feature.

###  3.3 Fusion Stage

Figure 10: The working mechanism of the fusion stage module.

The fusion stage combines the extracted features from color image with the
extracted depth features. This stage is important considering that we use two
separate streams of neural network structures to process the inputs where one
stream extract complementary semantic information from depth map and the other
extract features from color image. Hence, this module must ensure that the two
different stream of information combines without losing any information. As
discussed in Section 2.1.2, previous state-of-the-art research has limitations
in their fusion, as the flow of information between RGB and depth features is
blocked. This is because the information is only combined at a specific stage
in the model, which hinders the backbone network from learning modality-
specific representations. Moreover, some of the work uses simple concatenation
operation to combine the RGB and depth feature map with no trainable
parameters. Therefore, these networks cannot learn to adapt while combining
modality specific information. To this end, we propose a unique fusion
strategy that can train in parallel with the network and minimizes information
loss while combining the two streams of information. In our fusion module
demonstrated in Figure 10, we first try to address the modality specific
difference between depth and RGB information by using a residual mapping.
Residual mapping is used in the module to allow the network to learn the
transformation of depth feature map into a compatible version that can undergo
element-wise addition with RGB feature map. Then it performs element-wise
addition to combine the residual mapping of depth and the RGB feature tensors.
However, simply combining the two tensors with element-wise addition will not
make this dynamically trainable with the model because of the lack of
trainable weights. Moreover, simple element-wise addition of tensors may also
produce a coarse representation of the combined feature map. Therefore, we
follow an encoder-decoder structure after the element-wise addition stage,
inspired by the success of this kind of networks for semantic segmentation
tasks Siddique et al. (2021); Zhou et al. (2018). The encoder part normally
takes the added feature tensor and encode rich feature information via an up-
sampling layer followed by a down-sampling convolution. Meanwhile, the decoder
is responsible in generating more representative visual for the later part of
the detector. The decoder can use fully connected layers for this purpose but
it becomes computationally expensive. So we utilize transposed convolution
operation which increases the dimensions of the input tensor by using a filter
bigger than the input. The final element-wise addition copies the rich encoded
information from the encoder and uses it as a part of the decoder. This
enables the model to preserve information from a richer matrix and produce a
fine grained feature map. Furthermore, as there are several trainable weights
in convolution and transposed convolution of encoder and decoder blocks, it
helps to train the fusion stage while training the detector.

Figure 11: The figures in each row following the row of input images interpret
the generated feature map output after the concatenation and fusion stage
respectively. The image samples are taken from SUN RGB-D and NYU Depth v2.

To understand the effect of fusion against normal concatenation, we visualize
their respective output feature maps. Some of these results are demonstrated
in Figure 11 where the rows following the original images represent the output
feature maps for an image instance after using normal concatenation and
fusions stage respectively. The output feature maps of fusion stage
qualitatively verify the fact that the fusion stage mechanism is much superior
in combining the different modality of information and learn to preserve
greater details from the original image and its depth. To be precise, if one
compares the feature map in the second and third row of first column in Figure
11, the wall with the white board is clearly visible in the fusion feature
whereas it is completely obscured in concatenation output. Similarly, a
comparison of second and third row image of the second column in Figure 11
shows how the fusion feature map output captures the checkerboard texture of
the wall behind the red curtains in the original image while this detail is
missed concatenation output. Therefore, this visually support the idea behind
using the encoder to encode rich semantic feature while the decoder up-sample
the combined feature map. Another important distinction which can be observed
if one compares the second and third row image of the third column in Figure
11 where the outer boundary of the chair and desk can be clearly visible in
the fusion output unlike the concatenation output. Likewise, images in row two
and three of the last column in Figure 11 shows how the fusion output
preserves the outer boundary of the two monitors of the input image while the
monitors look like a single monitor in the concatenation feature map. This
comparison indicates that the fusion stage is better at learning while
combining different modality of feature map input.

###  3.4 The Loss Function

Considering our single stage detector we select the loss function used in
Redmon and Farhadi (2017) for training. This loss function mainly accounts for
three different losses, namely the localization loss, classification loss, and
confidence loss. The classification loss is computed using Equation 9

$\displaystyle Loss_{class}=\sum_{i=0}^{S^{2}}I_{i}^{obj}\sum_{cl\in
classes}(cl_{p}-cl_{g})^{2}$ (9)

Equation 9 utilizes a binary value $I_{i}$ to indicate if an object is present
in the grid cell $i$. The total number of grids present in the output tensor
is denoted by $S^{2}$. Here, $cl_{p}$ and $cl_{g}$ represent the predicted
class and ground truth class, respectively. Equation 10 is used to calculate
the localization loss by using the center coordinates (x and y) and dimensions
(w and h) of both the predicted and ground truth bounding boxes, with a
parameter $\lambda_{coordinate}$ set to 5 to apply more penalty for
localization errors.

$Loss_{local}=\lambda_{coord}\sum_{i=0}^{S^{2}}\sum_{j=0}^{A}I_{i,j}^{obj}[(x_{i,p}-x_{i,g})^{2}+(y_{i,p}-y_{i,g})^{2}]\\\
+\lambda{coord}\sum_{i=0}^{S^{2}}\sum_{j=0}^{A}I_{i,j}^{obj}[(\sqrt{w_{i,p}}-\sqrt{w_{i,g}})^{2}+(\sqrt{h_{i,p}}-\sqrt{h_{i,g}})^{2}]$
(10)

In Equation 10, square root of the bounding boxes height and width are taken
considering the fact that minor differences in the dimensions of larger boxes
are less significant than in smaller boxes. Moreover, $A$ stands for the the
total anchor boxes used which are selected using K-means clustering. The class
confidence loss is determined by Equation 11, where the confidence values of
the prediction $C_{i,p}$ and ground truth $C_{i,g}$ are compared, including a
parameter $\lambda_{No-obj}$ set to 0.5 for minimizing the impact of
confidence loss for cells with no objects present.

$Loss_{conf}=\sum_{i=0}^{S^{2}}\sum_{j=0}^{A}I_{ij}^{obj}(C_{i,p}-C_{i,g})^{2}+\lambda_{No-
obj}\sum_{i=0}^{S^{2}}\sum_{j=0}^{A}I_{ij}^{noobj}(C_{i,p}-C_{i,g})^{2}$ (11)

Overall, the loss function is expressed as Equation 12, which incorporates the
three losses with the respective weightings.

$Loss=Loss_{class}+Loss_{local}+Loss_{conf}$ (12)

###  3.5 Automatic RGB-D Data Generation

Figure 12: Automated RGB-D data generation pipeline.

Prior works on RGB-D object detection mostly relied on benchmark datasets like
SUN-RGBD and NYU Depth V2 to evaluate their model performance. Despite the
fact that these two benchmark datasets have real data from different types of
depth sensors and challenging scenes to evaluate model detection capabilities,
they are limited to indoor scenes with objects captured mostly in homes,
universities, office space, and furniture stores. Therefore, the
generalization capability of the RGB-D detectors and their performance on
other complex real-world scenarios with custom objects of interest are often
unclear. Furthermore, detection of objects where the clients have no or few
images of the objects of interest are common in industry settings. To this
end, we designed a synthetic RGB-D data generation pipeline to further explore
the ability of our model to detect custom objects in diverse environments. As
demonstrated in Figure 12, our RGB-D data generation framework consists of
three main components. Firstly, a 3D-2D foreground projector for generating
the perspective projections of 3D CAD (Computer Aided Design) models. Then, a
generative composition model to create realistic composite images of the
projected foreground image with selected background images. Finally, a depth
map generator that produces the depth maps corresponding to the composite
images. To be precise, 3D-2D foreground projector module takes a 3D CAD model
as input and generates 2D foreground perspective viewpoint images of the
model. This generates 2D images using three important viewpoint parameters
namely: azimuth, elevation, and distance. Besides these viewpoint parameters,
additional orientations of the 3D models using 6 degrees of freedom is also
exploited while generating the silhouettes of the CAD models. Next, we apply
the Spatial Transformer Generative Adversarial Network (ST-GAN) Lin et al.
(2018) to combine our generated foreground image to the background image while
maintaining the geometric correction and the scene semantics. Then we utilize
a hybrid version of the dense vision transformer (DPT-hybrid) Ranftl et al.
(2021) as our final component, i.e., the depth map generator. DPT-hybrid
initially takes the composite RGB images and transform them into tokens using
the ResNet-50 He et al. (2016) feature extractor. This helps to produce
aligned depth maps for each of the generated images. Interestingly, this
pipeline were able to produce 16000 RGB-D data within 3 minutes on top of a
Nvidia Quadro RTX 6000 GPU which suggest its high utility use case for the
industry.

##  4 Experiments

In this section, we first provide an overview of the proposed outdoor
datasets. Afterwards, we present the performance of the model on these
datasets followed by analysis of different components by ablation study. We
evaluate our RGB-D object detection model using the benchmark NYU Depth v2
Nathan Silberman and Fergus (2012) and SUN RGB-D Song et al. (2015) datasets.
The official training test split guideline is followed for both of these
dataset. To further explore the capacity of our model, we also use the
synthetic RGB-D data generated by the automated pipeline containing around
16000 RGB-D data. As customary with all other object detection research, mean
average precision (mAP) and average precision (AP) are used as evaluation
metrics, following same technique proposed by PASCAL VOC Everingham et al.
(2015).

One of the significant limitations of the benchmark RBG-D object detection
dataset used in the literature is that they contain RGB-D data only from
indoor environments. This limitation can leave several questions of the
research community unanswered like the query about the performance of RGB-D
object detection in challenging outdoor lighting condition or is RGB-D data
only useful for indoors. To address such concern, in this thesis, we propose a
new RGB-D dataset which we call the Outdoor RGD-D detect dataset which is
fully annotated. All the RGB-D images pairs in this dataset are only focused
on a variety of outdoor environments. The RGB images in this dataset is
outsourced from three different benchmark dataset that includes the Places
dataset Zhou et al. (2017), Open Images Kuznetsova et al. (2020); Krasin et
al. (2017) and the multi-class wildlife dataset Zhang et al. (2020b). The
corresponding depth maps of the images were predicted using dense vision
transformer (DPT-hybrid) Ranftl et al. (2021). We select three object classes
labels for detection in this dataset that include Human, Animals and Vehicle
classes that are most commonly seen in outdoor environments. Despite having
only three classes for detection it is a very challenging dataset for the
detection model given the fact that the classes have a wide variety of sub
types for example: Vehicle class has instances of bus, truck, suv, bike etc.
while Animal class has images of Kangaroos, Ostrich, Dog, etc. Moreover, the
outdoor environments in the images also have a huge variety which can range
from dense forest to busy downtown area along with different weather and
lighting conditions. The dataset has a total of 1819 RGB-D samples which is
split into 997 samples for training and the remaining 822 samples for testing.
Another important feature of this dataset is that it does not have any class
imbalance unlike the frequently used benchmarks in the literature.

###  4.1 Implementation Details

We implemented our RGB-D detection model using Tensorflow version 2.5. We
train our model on a remote server of ACENET Canada that has NVIDIA Quadro RTX
6000 GPU with 24 Gigabytes of memory. We also utilized MATLAB programming to
decode the compressed NYU Depth v2 and SUN RGB-D dataset. A Python script has
been written and used to organize these dataset folders according to our model
input requirements. As suggested by Xu et al. (2017); Li et al. (2018), we
select 19 furniture classes for object detection in these two datasets which
are: bathtub, bed, bookshelf, box, chair, counter, desk, door, dresser,
garbage bin, lamp, monitor, nightstand, pillow, sink, sofa, table, television,
and toilet. For the synthetic data, we utilze our RGB-D data generation
pipeline, as described in Section 3.4, to synthesize around 16000 RGB-D data.
Then we used this sythetic data to train our RGB-D detection model. We choose
7 different small working object classes to evaluate our model which includes
clamp, pipe, brace, nut, screwdriver, door-stopper and paintbrush.

For training the RGB-D object detection model we use Adam optimizer. It should
be noted that we do not apply any pre-trained Imagenet weights and choose to
train the model from scratch. The input images were resized to a size of 415
$\times$ 415\. We train the SUN RGB-D and NYU Depth v2 dataset with a learning
rate of 0.0005 for 150 epochs and 130 epochs respectively. Similarly, for the
outdoor RGB-D dataset we apply a learning rate of 0.0005 for 160 epochs. For
the synthetic data, we applied a learning rate of 0.00009 and trained for 120
epochs. For the non max suppression we select an IOU threshold of 0.5 because
it strikes a good balance between retaining important information and removing
duplicates.

###  4.2 Results on SUN RGB-D and NYU Depth v2

Classes |  RGB-D RCNN Gupta et al. (2014) |  SuperTransfer Gupta et al. (2016) |  AC-CNN Li et al. (2016) |  CMAC Li et al. (2018) |  FetNet Xiao et al. (2021) |  Ours
---|---|---|---|---|---|---
bathtub |  22.90 |  50.60 |  52.20 |  55.60 |  56.40 |  53.30
lamp |  29.30 |  42.50 |  42.90 |  45.00 |  50.80 |  49.50
bed |  66.40 |  81.00 |  82.40 |  83.90 |  78.30 |  94.09
monitor |  43.60 |  62.90 |  63.60 |  65.80 |  69.50 |  73.37
bookshelf |  21.80 |  52.60 |  52.50 |  54.00 |  57.30 |  52.40
night-stand |  39.50 |  54.70 |  55.20 |  57.60 |  59.00 |  59.60
box |  3.00 |  5.40 |  8.60 |  9.80 |  8.00 |  17.50
pillow |  37.40 |  49.10 |  49.70 |  52.70 |  60.80 |  56.45
chair |  40.80 |  53.00 |  54.80 |  55.40 |  68.20 |  69.46
sink |  24.20 |  50.00 |  51.40 |  53.80 |  60.30 |  52.40
counter |  37.60 |  56.10 |  57.30 |  59.20 |  37.60 |  54.34
sofa |  42.80 |  65.90 |  66.80 |  69.10 |  69.00 |  69.50
desk |  10.20 |  21.00 |  22.70 |  24.10 |  32.50 |  38.73
table |  24.30 |  31.90 |  33.50 |  35.00 |  36.00 |  36.90
door |  20.50 |  34.60 |  34.10 |  36.30 |  44.20 |  41.20
tv |  37.20 |  50.10 |  51.80 |  56.90 |  55.40 |  55.46
dresser |  26.20 |  57.90 |  58.10 |  58.50 |  59.10 |  53.70
toilet |  53.00 |  68.00 |  70.40 |  74.70 |  71.20 |  72.50
garbage-bin |  37.60 |  46.20 |  46.50 |  47.20 |  51.90 |  52.20
mAP |  32.50 |  49.10 |  50.20 |  52.30 |  54.00 |  55.40

Table 2:  Experimental results on NYU Depth v2. The first, second and third
best results are highlighted in green, blue and red color, respectively. Note
that mAP values to be read as percentages.

Classes |  RGB-D RCNN Gupta et al. (2014) |  SuperTransfer Gupta et al. (2016) |  AC-CNN Li et al. (2016) |  CMAC Li et al. (2018) |  FetNet Xiao et al. (2021) |  Ours
---|---|---|---|---|---|---
bathtub |  49.60 |  65.30 |  65.80 |  69.00 |  62.50 |  63.98
lamp |  22.00 |  32.10 |  33.80 |  35.60 |  65.00 |  61.29
bed |  76.00 |  83.00 |  83.30 |  86.10 |  80.90 |  81.42
monitor |  10.80 |  36.80 |  39.50 |  40.50 |  43.10 |  50.46
bookshelf |  35.00 |  54.40 |  56.20 |  57.90 |  47.90 |  53.45
night-stand |  37.20 |  46.60 |  47.10 |  49.80 |  62.00 |  60.93
box |  5.80 |  14.40 |  16.40 |  18.20 |  13.30 |  18.17
pillow |  16.50 |  23.40 |  25.20 |  26.70 |  63.90 |  52.09
chair |  41.20 |  46.90 |  47.50 |  50.30 |  69.30 |  63.10
sink |  41.90 |  43.90 |  45.30 |  46.60 |  65.40 |  66.98
counter |  8.10 |  14.60 |  16.00 |  17.40 |  49.20 |  17.80
sofa |  42.20 |  61.30 |  61.90 |  67.20 |  56.30 |  57.90
desk |  16.60 |  23.90 |  24.90 |  26.80 |  30.40 |  35.40
table |  43.00 |  48.70 |  49.00 |  52.90 |  49.50 |  49.71
door |  4.20 |  15.30 |  16.60 |  17.30 |  52.60 |  51.80
tv |  32.90 |  50.50 |  54.10 |  56.70 |  40.30 |  39.18
dresser |  31.40 |  41.30 |  42.70 |  44.40 |  41.90 |  40.23
toilet |  69.80 |  79.40 |  84.20 |  84.90 |  85.50 |  83.42
garbage-bin |  46.80 |  51.00 |  53.40 |  54.40 |  56.90 |  54.00
mAP |  35.20 |  43.80 |  45.40 |  47.50 |  54.50 |  52.70

Table 3:  Experimental results on SUN RGB-D. The first, second and third best
results are highlighted in green, blue and red color, respectively. Note that
mAP values to be read as percentages.

We compare the detection model with recent state-of-the-art RGB-D object
detection methods. For these, we adopt the results reported in their papers to
ensure fair comparison.

Our detection model achieves the best performance with mAP 55.4 $\%$ on NYU
Depth v2 surpassing all state-of-the-art RGB-D detectors by at least 1
percent, as shown in Table 2. Moreover, the proposed model significantly
improves the performance on several classes such as bed, monitor, desk and
toilet. Low detection accuracy with a few objects were most likely caused
object occlusion and noisy depth map as our model rely heavily on depth map
information.

Figure 13: A few detection results where the top five images shows detection
on SUN RGB-D and the bottom images are detections on NYU Depth v2.

Table 3 reports the object detection accuracies of various models on SUN
RGB-D. From Table 3, it is apparent that our model achieves the second best
results on SUN RGB-D dataset reaching an mAP of 52.7 $\%$. However, our model
achieves significant performance on individual furniture classes like bed,
sofa, toilet and monitor. The heterogeneity of the objects within the box
class, including those of varying sizes like small cereal boxes and large
packages found in a mail room, presents a challenge for accurate detection and
results in a lower accuracy for this class. Furthermore, the desk class in the
object detection benchmark Gupta et al. (2014) is facing an issue with
accuracy due to ambiguous data. Precisely, some desks resemble tables and vice
versa, creating difficulty in distinguishing between the two. Also, the fact
that our model were designed to better utilize the boundary information of
objects so the similar semantic pattern between desks and tables are likely
causing difficulties in proper detection of the desk. Despite these
difficulties, it is noteworthy that our model’s accuracy for the desk class
surpasses that of several other models in the literature. The instances of the
lamp class in the dataset present a challenge for accurate classification due
to the high intensity of light emission from the lamp obscuring the visible
shape in the RGB images. Although the shape of the lamps are comparatively
discernible in the depth maps but they are obtained from four distinct sensors
in SUN RGB-D dataset. These variety in depth information, along with the
differences between the depth maps and the RGB images, can negatively impact
the accuracy of the lamp detection in our model because the depth aware hyper-
involution relies on both RGB and depth data to learn its filter weights.
Figure 13 visualize some of the detection from these two datasets for
qualitative evaluation. The performance on benchmark NYU Depth v2 datasets
indicates the efficacy of our detection architecture and its customized
modules. Furthermore, the precision-recall curves displayed in Figure 14
demonstrate an appropriate equilibrium between precision and recall for
several classes.

Figure 14: Precision recall curves for different classes on SUN RGB-D and NYU
Depth v2. The top row shows classes in NYU Depth v2 while the bottom row shows
classes in SUN RGB-D.

###  4.3 Results on Synthetic Dataset

We select 7 different small working object classes from synthesized data to
evaluate our model which are: clamp, pipe, brace, nut, screwdriver, door-
stopper and paintbrush. Figure 15 shows some of the qualitative detection
results with red boxes on the synthetic data and also gives an overall idea
about the quality of our synthesized data. The model achieved an overall mAP
of 58.7 percent on this dataset, as shown in Table 4. It gets a low mAP for
very small object like nuts which is mostly because of the noise in predicted
depth data. More importantly, the model achieves significantly higher mAP on
several individual small object classes like doorstopper, brace and clamp in
complex synthetic factory environment. This results suggest the superiority of
this model for object detection in complex environments like that of inside a
factory.

Figure 15: A few detection results on the synthesized data.

Method |  mAP |  doorstopper |  pipe |  clamp |  screwdriver |  brace |  paintbrush |  nut
---|---|---|---|---|---|---|---|---
FETNet Xiao et al. (2021) |  56.8 |  80.6 |  71.3 |  59.6 |  68.1 |  49.6 |  54.7 |  14.3
Ours |  58.9 |  84.1 |  74.9 |  67.2 |  62.7 |  53.0 |  52.5 |  17.9

Table 4:  Experimental results on automatically synthesized dataset.

###  4.4 Results on Outdoor RGB-D Dataset

Figure 16: A few detection result on the Outdoor RBG-D. Method |  mAP |  Vehicle |  Human |  Animal
---|---|---|---|---
FETNet Xiao et al. (2021) |  78.4 |  79.5 |  77.7 |  78.1
Ours |  80.2 |  81.1 |  80.7 |  78.8
Table 5:  Experimental results on outdoor RGB-D dataset.

Figure 16 shows some of the qualitative detection results on our outdoor RGB-D
detection dataset. This figure shows that despite having only three classes
for detection, the dataset poses great challenge for object detection due to
the variety of objects in each class. For example, first image of row one and
the first two images of row two of Figure 16 shows that our detector was able
to detect the van, bus and truck as vehicle class despite their differences in
visual features. The figures proves that the detector was able to learn this
variety of feature within a class. The detector was also able to detect
objects that were almost blurred in the image due to their speed just like the
image in row one and column two of Figure 16. The detection results in first
row third column of Figure 16 shows that the human was detected despite
wearing a helmet that cover his head which proves that the detector has the
generalization capacity. The detection of animal from far away in a dense
jungle/forest environment in second row third column also suggest to the
model’s accuracy. In the quantitative experiments as shown in Table 5, our
model achieved an mAP of 80.1 which is also significantly higher FETNet Xiao
et al. (2021). Therefore, both qualitative and quantitative results indicate
the high capacity of our detection model in real world outdoor environments
under a variety of lighting conditions.

###  4.5 Inference

| Input |
---|---|---
Model | RGB | RGB-D | GFLOPs
YOLOv2 Redmon and Farhadi (2017) | ✓ |  | 63.03
Casecade-RCNNCai and Vasconcelos (2018) | ✓ |  | 168.3
Faster-RCNN Ren et al. (2015) | ✓ |  | 140.5
Cascade-RCNN+FEM+MVIT Xiao et al. (2021) |  | ✓ | 158.5
Faster-RCNN+FEM+MVIT Xiao et al. (2021) |  | ✓ | 130.7
FETNet Xiao et al. (2021) |  | ✓ | 279.3
Our model |  | ✓ | 26.72
Table 6:  Inference GFLOPs comparison with state-of-the-art RGB and RGB-D
based detection algorithms.

Inference GFLOPs refers to the number of floating point operations required to
perform a single prediction or inference step on a trained model. This
measurement is often used to evaluate the computational complexity and
performance of a given model, and is usually expressed in GigaFLOPs ($10^{9}$
FLOPs). The calculation of Inference GFLOPs involves counting the number of
additions and multiplications required to compute the activations for each
layer in a network for a given input, and converting that count to FLOPs. We
compare the inference GFLOPs with several state-of-the-art RGB and RGB-D based
object detectors to evaluate the real-time computational performance of our
detection model. For inference time comparison of the RGB-D based detectors we
select FETNet Xiao et al. (2021) and some other implementations of RGB-D based
detection using their proposed module, as reported in their paper. As shown in
Table 6, our detection model achieved the best inference GFLOPs which suggest
our model has the least computational complexity. Moreover, our model also
significantly outperforms real-time single stage detector YOLOv2 Redmon and
Farhadi (2017) in terms GFLOPs. This result also indicate the real-time
performance of our RGB-D detection model. A potential reason for achieving
significantly less inference GFLOPs is that our backbone structure has
convolution layers of 3x3 filter with stride 1 which always use the same
padding and maxpool layer of 2x2 filter of stride 2. The fusion layer uses
mostly 3 filters for its convolution operations. Moreover, we apply just 8
filters for depth aware hyper-involution which also contributes to the
impressive inference times.

###  4.6 Ablation Study

####  4.6.1 Module Test

Figure 17: Comparison of mAP for different version of the model in ablation
study. Figure 18: Comparison of inference for different version of the model
in ablation study. Figure 19: Comparison of parameters for different version
of the model in ablation study.

To consider a baseline for the purpose of ablation study we have modified the
detection architecture by replacing the depth aware hyper-involution operation
with standard convolution and replacing the proposed fusion stage with simple
concatenation of features map. We then modified the baseline by replacing the
concatenation with fusion to identify the performance of the proposed fusion.
As demonstrated in the graph plot in 17 the original model achieved the
highest accuracy which verify the usefulness of the depth aware hyper-
involution. As shown in graph plot 18, our main detection model has the
minimum inference GFLOPs when compared with the baseline and baseline with
just fusion or depth aware-involution. This implies that the fusion stage and
depth aware hyper-involution does not increase computational complexity and
helps to maintain real-time performance of the detection model. Moreover, the
model also has less parameters when compared to the model with only fusion and
standard convolution which suggest the depth aware hyper-involution operation
consume less memory than standard convolution which is shown in plot 19. As
demonstrated in the plot 19, when normal concatenation is replaced with the
suggested fusion in the baseline model, the number of parameters increases
significantly. This indicates that the fusion module has more trainable
parameters, which can enhance the model’s learning ability.

####  4.6.2 Number of Parameters Vs Kernel Sizes

Furthermore, we conducted another ablation study to see the effect on number
of parameters of depth aware hyper-involution for different kernel sizes. We
also compare it with the parameters of standard convolution LeCun et al.
(1998) and involution Li et al. (2021) for similar kernel sizes. As shown in
Table 7 and the graph plot in 20, the parameters of depth aware hyper-
involution remains the same for all kernel sizes which is not the case in
involution and standard convolution. Moreover, the number of parameters in
depth aware hyper-involution is less than that of standard convolution for all
sizes of filters. This clearly indicates the usefulness of the hyper-network
in generating filters for the depth aware hyper-involution. Note that, we
applied 8 filters for all these modules during comparison.

Figure 20: Parameter comparison for different kernel sizes of convolution,
involution and our depth aware hyper-involution.

Layer |  3x3 filter |  5x5 filter |  7x7 filter
---|---|---|---
Standard convolution |  216 |  600 |  1176
Involution |  145 |  289 |  505
Depth aware hyper-involution |  273 |  273 |  273

Table 7:  Comparison of the number of parameters for different kernel sizes.

##  5 Conclusions

In this paper, we delineate the importance of depth maps for object detection
task and investigated into the alternatives of convolution for a better
feature extraction from RGB-D images. Aimed at maximizing the utilization of
the depth information, we design a depth-weighted hyper-involution and a new
fusion mechanism which enables dynamic learning during model training and
prevents information loss. Building on top of these modules, we developed a
single stage RGB-D based object detection model which uses minimal number of
network parameters. The proposed object detection framework exhibits higher
accuracy while maintaining low computational complexity. Qualitative and
quantitative experiments performed using the proposed model on benchmark
datasets suggest the effectiveness of the proposed architecture. Moreover, a
fully automated RGB-D data synthesis pipeline was developed to tackle the
scarcity of large datasets for RGB-D-based object detection research. We also
introduced two new RGB-D datasets providing the research community with more
options to evaluate and compare their RGB-D object detection performance in
diverse environments. Although we designed depth aware hyper-involution module
for RGB-D object detection, this filter has proven to map some important
semantic features that can potentially be a good fit for other tasks such as
object parts segmentation or salient object detection. A more focused
investigation of depth aware hyper-involution module in the context of
specific applications such as robotic surgery or augmented reality is
necessary.

### Acknowledgements

This work was supported by Mitacs through the Mitacs Accelerate program.

## References

  * Pan et al. [2021] Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang.  3d object detection with pointformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7463–7472, 2021.
  * Qi et al. [2021] Charles R Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov.  Offboard 3d object detection from point cloud sequences.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6134–6144, 2021.
  * Tian et al. [2021] Hao Tian, Yuntao Chen, Jifeng Dai, Zhaoxiang Zhang, and Xizhou Zhu.  Unsupervised object detection with lidar clues.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5962–5972, 2021.
  * Nathan Silberman and Fergus [2012] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _ECCV_ , 2012.
  * Polseno [2020] Sheri Polseno.  Forced perspective photography, May 2020.  URL https://www.foxboroughartpass.com/blog/forced-perspective-photography.
  * Ranftl et al. [2021] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12179–12188, 2021.
  * Rankuzz.com [2020] Rankuzz.com.  the most expert animals in camouflage techniques (cripsis), May 2020.  URL https://www.rankuzz.com/en/entertainment/the-most-expert-animals-in-camouflage-techniques-cripsis-214.html.
  * Starecat.com [2022] Starecat.com.  Dog with shadow tiger stripes i almost had a heart attack this morning, May 2022.  URL https://starecat.com/dog-with-shadow-tiger-stripes-i-almost-had-a-heart-attack-this-morning/.
  * Gupta et al. [2014] Saurabh Gupta, Ross Girshick, Pablo Arbeláez, and Jitendra Malik.  Learning rich features from rgb-d images for object detection and segmentation.  In _European conference on computer vision_ , pages 345–360. Springer, 2014.
  * Xiao et al. [2021] Zhibin Xiao, J Xue, Pengwei Xie, and Guijin Wang.  Fetnet: Feature exchange transformer network for rgb-d object detection.  The BMVC, 2021.
  * Gupta et al. [2016] Saurabh Gupta, Judy Hoffman, and Jitendra Malik.  Cross modal distillation for supervision transfer.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2827–2836, 2016.
  * Ophoff et al. [2018] Tanguy Ophoff, Kristof Van Beeck, and Toon Goedemé.  Improving real-time pedestrian detectors with rgb+ depth fusion.  In _2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_ , pages 1–6. IEEE, 2018.
  * Ophoff et al. [2019] Tanguy Ophoff, Kristof Van Beeck, and Toon Goedemé.  Exploring rgb+ depth fusion for real-time object detection.  _Sensors_ , 19(4):866, 2019.
  * Li et al. [2018] Guanbin Li, Yukang Gan, Hejun Wu, Nong Xiao, and Liang Lin.  Cross-modal attentional context learning for rgb-d object detection.  _IEEE Transactions on Image Processing_ , 28(4):1591–1601, 2018.
  * Xu et al. [2017] Xiangyang Xu, Yuncheng Li, Gangshan Wu, and Jiebo Luo.  Multi-modal deep feature learning for rgb-d object detection.  _Pattern Recognition_ , 72:300–313, 2017.
  * Girshick [2015] Ross Girshick.  Fast r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1440–1448, 2015.
  * Girshick et al. [2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 580–587, 2014.
  * Bochkovskiy et al. [2020] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  _arXiv preprint arXiv:2004.10934_ , 2020.
  * Tan et al. [2020] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10781–10790, 2020.
  * Wang et al. [2021a] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Scaled-yolov4: Scaling cross stage partial network.  In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_ , pages 13029–13038, 2021a.
  * Zaidi et al. [2022] Syed Sahil Abbas Zaidi, Mohammad Samar Ansari, Asra Aslam, Nadia Kanwal, Mamoona Asghar, and Brian Lee.  A survey of modern deep learning based object detection models.  _Digital Signal Processing_ , page 103514, 2022.
  * Wang et al. [2022] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.  _arXiv preprint arXiv:2207.02696_ , 2022.
  * Wang et al. [2021b] Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao.  You only learn one representation: Unified network for multiple tasks.  _arXiv preprint arXiv:2105.04206_ , 2021b.
  * Redmon et al. [2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 779–788, 2016.
  * Redmon and Farhadi [2017] Joseph Redmon and Ali Farhadi.  Yolo9000: better, faster, stronger.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7263–7271, 2017.
  * Redmon and Farhadi [2018] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  _arXiv preprint arXiv:1804.02767_ , 2018.
  * Tian et al. [2019] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9627–9636, 2019.
  * Tian et al. [2020] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: A simple and strong anchor-free object detector.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * Hong et al. [2022] Qinghang Hong, Fengming Liu, Dong Li, Ji Liu, Lu Tian, and Yi Shan.  Dynamic sparse r-cnn.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4723–4732, 2022.
  * Sun et al. [2021] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al.  Sparse r-cnn: End-to-end object detection with learnable proposals.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14454–14463, 2021.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Dai et al. [2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 764–773, 2017.
  * Zhu et al. [2019] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9308–9316, 2019.
  * Su et al. [2019] Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, and Jan Kautz.  Pixel-adaptive convolutional neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11166–11175, 2019.
  * Yang et al. [2019] Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam.  Condconv: Conditionally parameterized convolutions for efficient inference.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Chen et al. [2020] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu.  Dynamic convolution: Attention over convolution kernels.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11030–11039, 2020.
  * Wang and Neumann [2018] Weiyue Wang and Ulrich Neumann.  Depth-aware cnn for rgb-d segmentation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 135–150, 2018.
  * Chen et al. [2021] Lin-Zhuo Chen, Zheng Lin, Ziqin Wang, Yong-Liang Yang, and Ming-Ming Cheng.  Spatial information guided convolution for real-time rgbd semantic segmentation.  _IEEE Transactions on Image Processing_ , 30:2313–2324, 2021.
  * Ding et al. [2020] Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo.  Learning depth-guided convolutions for monocular 3d object detection.  In _Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition workshops_ , pages 1000–1001, 2020.
  * Ma et al. [2018] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 116–131, 2018.
  * Sandler et al. [2018] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4510–4520, 2018.
  * Tan and Le [2019] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , pages 6105–6114. PMLR, 2019.
  * Chollet [2017] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1251–1258, 2017.
  * Li et al. [2021] Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, and Qifeng Chen.  Involution: Inverting the inherence of convolution for visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12321–12330, 2021.
  * Hazirbas et al. [2016] Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers.  Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture.  In _Asian conference on computer vision_ , pages 213–228. Springer, 2016.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Jaderberg et al. [2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  _Advances in neural information processing systems_ , 28, 2015.
  * Zhang et al. [2020a] Wenli Zhang, Jiaqi Wang, Xiang Guo, Kaizhen Chen, and Ning Wang.  Two-stream rgb-d human detection algorithm based on rfb network.  _IEEE Access_ , 8:123175–123181, 2020a.
  * Cao et al. [2021] Jinming Cao, Hanchao Leng, Dani Lischinski, Daniel Cohen-Or, Changhe Tu, and Yangyan Li.  Shapeconv: Shape-aware convolutional layer for indoor rgb-d semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7088–7097, 2021.
  * Krizhevsky et al. [2017] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Communications of the ACM_ , 60(6):84–90, 2017\.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 234–241. Springer, 2015.
  * Ha et al. [2016] David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ , 2016.
  * Wang et al. [2021c] Alan Q Wang, Adrian V Dalca, and Mert R Sabuncu.  Regularization-agnostic compressed sensing mri reconstruction with hypernetworks.  _arXiv preprint arXiv:2101.02194_ , 2021c.
  * Hoopes et al. [2021] Andrew Hoopes, Malte Hoffmann, Bruce Fischl, John Guttag, and Adrian V Dalca.  Hypermorph: Amortized hyperparameter learning for image registration.  In _International Conference on Information Processing in Medical Imaging_ , pages 3–17. Springer, 2021.
  * Ma et al. [2022] Tianyu Ma, Adrian V Dalca, and Mert R Sabuncu.  Hyper-convolution networks for biomedical image segmentation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1933–1942, 2022.
  * Nirkin et al. [2021] Yuval Nirkin, Lior Wolf, and Tal Hassner.  Hyperseg: Patch-wise hypernetwork for real-time semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4061–4070, 2021.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Huang et al. [2018] Rachel Huang, Jonathan Pedoeem, and Cuixian Chen.  Yolo-lite: a real-time object detection algorithm optimized for non-gpu computers.  In _2018 IEEE international conference on big data (big data)_ , pages 2503–2510. IEEE, 2018.
  * Jo et al. [2017] KangUn Jo, JungHyuk Im, Jingu Kim, and Dae-Shik Kim.  A real-time multi-class multi-object tracker using yolov2.  In _2017 IEEE International Conference on Signal and Image Processing Applications (ICSIPA)_ , pages 507–511. IEEE, 2017.
  * Levin et al. [2004] Anat Levin, Dani Lischinski, and Yair Weiss.  Colorization using optimization.  In _ACM SIGGRAPH 2004 Papers_ , pages 689–694. 2004.
  * Song et al. [2015] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao.  Sun rgb-d: A rgb-d scene understanding benchmark suite.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 567–576, 2015.
  * Siddique et al. [2021] Nahian Siddique, Sidike Paheding, Colin P Elkin, and Vijay Devabhaktuni.  U-net and its variants for medical image segmentation: A review of theory and applications.  _Ieee Access_ , 9:82031–82057, 2021.
  * Zhou et al. [2018] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang.  Unet++: A nested u-net architecture for medical image segmentation.  In _Deep learning in medical image analysis and multimodal learning for clinical decision support_ , pages 3–11. Springer, 2018.
  * Lin et al. [2018] Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey.  St-gan: Spatial transformer generative adversarial networks for image compositing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 9455–9464, 2018.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Everingham et al. [2015] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes challenge: A retrospective.  _International Journal of Computer Vision_ , 111(1):98–136, January 2015.
  * Zhou et al. [2017] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.
  * Kuznetsova et al. [2020] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  _IJCV_ , 2020.
  * Krasin et al. [2017] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  _Dataset available from https://storage.googleapis.com/openimages/web/index.html_ , 2017.
  * Zhang et al. [2020b] Teng Zhang, Liangchen Liu, Kun Zhao, Arnold Wiliem, Graham Hemson, and Brian Lovell.  Omni-supervised joint detection and pose estimation for wild animals.  _Pattern Recognition Letters_ , 132:84–90, 2020b.
  * Li et al. [2016] Jianan Li, Yunchao Wei, Xiaodan Liang, Jian Dong, Tingfa Xu, Jiashi Feng, and Shuicheng Yan.  Attentive contexts for object detection.  _IEEE Transactions on Multimedia_ , 19(5):944–954, 2016.
  * Cai and Vasconcelos [2018] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6154–6162, 2018.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28, 2015.