# Depth Insight - Contribution of Different Features to
Indoor Single-image Depth Estimation

Yihong Wu$1$, Yuwen Heng$1,2$, Mahesan Niranjan$1$, Hansung Kim$1$
$1$ School of Electronics and Computer Science, University of Southampton
$2$ Baidu Inc.

###### Abstract

Depth estimation from a single image is a challenging problem in computer
vision because binocular disparity or motion information is absent. Whereas
impressive performances have been reported in this area recently using end-to-
end trained deep neural architectures, as to what cues in the images that are
being exploited by these black box systems is hard to know. To this end, in
this work, we quantify the relative contributions of the known cues of depth
in a monocular depth estimation setting using an indoor scene data set. Our
work uses feature extraction techniques to relate the single features of
shape, texture, colour and saturation, taken in isolation, to predict depth.
We find that the shape of objects extracted by edge detection substantially
contributes more than others in the indoor setting considered, while the other
features also have contributions in varying degrees. These insights will help
optimise depth estimation models, boosting their accuracy and robustness. They
promise to broaden the practical applications of vision-based depth
estimation. The project code is attached to the supplementary material and
will be published on GitHub.

##  1 Introduction

Depth estimation, predicting the distance from an object’s surface to the
camera, is a key task in the field of computer vision. It plays a crucial role
in many applications, such as 3D reconstruction [1], autonomous driving [17,
33], virtual reality (VR) [6], augmented reality (AR) [20], etc. The goal of
monocular depth estimation based on deep learning is to infer the depth value
of each pixel by analysing the scene information in a single image.

Due to the ill-posed problem of monocular depth estimation, there is a
fundamental need to move towards a scene understanding of objects in images so
that various characteristics of objects can cue depth information. Recent work
on monocular depth estimation using end-to-end trained deep neural network
models shows that such cues are collectively learnable and satisfactory depth
estimation can be achieved [8, 2, 3, 35]. However, the black-box nature of
such models prohibits the understanding of what cues are exploited in
monocular depth estimation. The mechanism behind monocular depth estimation
based on 2D images in neural networks is still not clearly explained, and the
extent to which these models can approximate the human capability of monocular
depth perception remains uncertain.

Building on this gap in understanding, and inspired by causality analysis
[21], we aimed to investigate the factors that influence depth estimation.
This work will pave the way for the development of versatile models applicable
to a broader spectrum of depth estimation tasks, moving beyond reliance solely
on data-driven approaches. Research has shown that monocular depth cues in 2D
images include phenomena such as blurring, shading and brightness [30]. This
paper investigated and analysed the factors that influence machine-based
monocular depth estimation. To provide a comprehensive understanding, we
investigated the roles of the factors relevant to object recognition [11],
such as colour and texture, in the context of monocular depth estimation.
However, many factors are interrelated and cannot be independently segregated.
In the context of scenarios where it is possible to directly and independently
extract features from 2D images, we have considered colour, saturation,
texture and shape, and each of them holds significant relevance in image
processing, exerting varying degrees of impact on the overall outcome.

Colour. Colour is recognised by the perception and interpretation of different
wavelengths of light by the eye [15]. The visual information humans gather
heavily relies on the presence of colour [22]. Colour helps humans recognise
and remember objects faster [12]. Nevertheless, when defining colour, it is
critical to recognise that RGB images do not only represent a singular colour
but also include various elements in addition to colour, such as shape and
texture. To isolate the pure colour information, we utilised a phase
scrambling approach [11], which effectively separates the colour from these
additional attributes.

Saturation. The second feature of interest is saturation. Saturation refers to
the purity or intensity of a colour. For instance, high saturation indicates a
more vivid and pure colour, while low saturation suggests a lighter or more
desaturated colour with a hint of grey. Aerial perspective, within the domain
of remote viewing, refers to the impact of the atmosphere on the visual
depiction of an object. For instance, in Figure 1(a), a nature photograph is
displayed. We evenly split images into ten rows, and the average saturation
values have been calculated for each row, as depicted in Figure 1(b). As the
object moves away from the camera, it can be observed that the saturation
decreases. Building on this observation, saturation serves as a depth cue for
outdoor single-image depth estimation. We aimed to investigate the utility of
saturation as a depth cue in indoor scenes.

(a) Nature Scene Image

(b) Saturation Distribution

Figure 1: Saturation Analysis for a Nature Scene

Texture. In computer vision, texture is defined by repetitive patterns with
varying intensities present in an image[32]. Prior research has found that
textures are important when influencing a human’s perception of distance [27],
with specific regions in the brain having been found to be activated when
exposed to varying textures [23]. Therefore, we also sought to independently
extract the features pertaining to texture and assess their impact on depth
estimation.

Shape. A shape is generally considered to be a graphical representation of an
object or its external borders, contours or external surfaces. Acquiring
precise boundaries of objects in the 3D world based solely on 2D images is
challenging. To simplify this process, we defined the shape feature as the
edge graph, which corresponds to a greyscale map generated using an edge
detection algorithm designed to preserve the object’s boundaries. Edges are
regarded as one of the primary cues essential for the human visual system [9].
Edge graphs usually represent geometric structures or boundaries between
objects. For depth estimation tasks, the geometric features of an object are
crucial to inferring its depth. The geometric structure aids depth estimation
algorithms in capturing the shapes and relationships between objects [18],
with edge maps providing supplementary geometric information. Edge detectors
can analyse pixel gradients in different areas of the image, thereby assisting
in the estimation of the relative distances between objects.

##  2 Related Works

Interpretability within deep learning is attracting significant and growing
interest. Interest in Convolutional Neural Networks (CNNs) and Visual
Transformers has been rapidly increasing lately, particularly concerning their
interpretability. A study revealed that CNN models trained on ImageNet exhibit
a heightened sensitivity to texture information [13]. In addition, recent
research has undertaken a comparison of the attributes between CNNs and
Transformers across various layers [25] by using Centred Kernel Alignment
(CKA) [4, 19]. According to their claims, the transformer allows the early
gathering of global information in contrast to CNNs. This results in a robust
propagation of features from lower to higher layers in the network.
Nevertheless, the primary focus of these enquiries remains centred on model
analysis.

In the realm of human depth perception, substantial work has been conducted to
investigate cues such as position in the image, texture density and focus blur
[14, 5]. Existing works have demonstrated various methods for indoor single-
image depth estimation that exhibit good performance [8, 3]. Despite this, an
analysis of their operations is still lacking. To the best of our knowledge,
there has been no analysis of the contributions of different types of depth
cues specifically for deep learning-based depth estimation in indoor single-
image scenarios. In the two most relevant prior studies to our work, [16]
conducts attribution analysis to identify pixels that contribute most
significantly to the final depth map. However, these methods can only offer
insights into the low-level workings of CNNs. The analysis in [7] was
primarily confined to specific objects situated in outdoor environments, such
as animals and vehicles on roadways. In contrast, in our study, we focused on
colour, saturation, texture and shape, taking into account that our target
application pertains to indoor scenes and requires the extraction of these
cues from a single image.

Approaches for emulating the human capacity for gauging depth from an indoor
scene still encounter gaps in knowledge. The objective of this paper is to
unveil how neural networks extract depth-related information from a single
indoor image to attain a more profound comprehension of the disparities
between monocular visual depth estimation and the depth perception exhibited
by humans.

Simultaneously, our work offers a foundational framework to facilitate
subsequent investigations into assessing the interdependence among pertinent
variables in the realm of depth estimation. A prior exploration delved into
the causal interplay within 3D reconstruction, deconstructing elements like
perspective and depth while also attempting to substantiate the interlinkages
amid diverse variables [21]. Nonetheless, the model expounded upon in this
enquiry operates on the assumption that the object is symmetric [34]. Through
an autoencoder mechanism, it internally dissects the input image into manifold
components, as opposed to explicitly extracting a corresponding viewpoint,
depth and related insights from the RGB image. In our study, we exclusively
extracted various factors from RGB images while carefully investigating the
significance of these factors within the context of depth estimation. Our work
is set to further enable causal analyses in the field of depth estimation,
paving the way for future advancements in comprehending the causality of depth
estimation.

##  3 Methodology

In this section, We consider four cues for monocular depth estimation. In
order to compare the appearance of these four different features, we use the
same sample in this section. Figure 2 shows the original RGB image and its
corresponding ground truth (GT) depth from the NYU data set [29].

(a) Original RGB

(b) Corresponding GT Depth

Figure 2: A Sample from NYU Dataset

###  3.1 Colour

Figure 3: Heat map of the relationship between the distribution of original
RGB three-channel values and the depth map. The horizontal axis represents the
depth range, while the vertical axis corresponds to the pixel count of the R,
G and B channels within the respective depth ranges. The colour bar values
represent the pixel counts for three respective channels from 500 images
randomly selected from the NYU data set.

Figure 3 illustrates the relationship between the distribution of original RGB
three-channel values and the depth maps. The pixels on original RGB images are
primarily concentrated between 0 and 100 in the corresponding grey-scale depth
maps. The heat map reveals that the values of R, G and B pixels are similarly
distributed across a specific depth range. This shows that the factors
affecting depth are not significantly related to the distribution of pixels on
the RGB channel. More details are shown in the Appendix.

Figure 4: Phase Scrambled H Map and Corresponding Depth Map of Figure 2

Hue from the hue, saturation and luminance value (HSV) colour space can be an
expression of colour. However, hue values represent the projection of the RGB
colour space onto a non-linear chroma angle [31]. If an output pixel value
falls outside the valid range, it necessitates remapping to bring it within
the specified range. The chroma angle represents a non-linear trajectory
within a continuous, uninterrupted space. Here, starting at 0 degrees is the
same as coming full circle to 360 degrees. However, when we apply this idea to
an image, like with H maps, the smooth flow is interrupted, creating a series
of separated points instead. Figure 4 illustrates the images and corresponding
depth maps resulting from the phase scrambling and remapping process applied
to the H map from Figure 2, which are mapped back to specific intervals. Some
discontinuous blocks can be observed in this figure. Therefore, We did not
consider utilising the hue maps as the colour feature.

To examine the contribution of colour to depth, we performed phase scrambling
[11] on original RGB images and their corresponding depth maps to remove
influences from shapes, textures and other geometric features. The resulting
data set was labelled “RGB Phase Scrambled”. Nevertheless, even after the
phase scrambling, the outcome still retains the brightness information, making
it not purely a colour feature. Subsequently, these outputs were converted to
greyscale, effectively removing the colour information, and the resulting data
set was labelled as “Greyscale Phase Scrambled”. To illustrate the role of
colour, a comparison of these two phase-scrambled features is conducted.

(a) RGB Image Phase Scrambling

(b) Gray-scale Image Phase Scrambling

(c) Depth Phase Scrambling

Figure 5: Phase Scrambling Results of Figure 2

###  3.2 Saturation

Figure 6: Average Saturation at Different Depth Intervals for Indoor Scenes
(NYU data set)

We investigated whether saturation varies at different depths in indoor
scenes. We partitioned this depth range 0-255 in the NYU data set into eight
segments and then calculated the average saturation for each by converting RGB
to HSV colour space and extracting the saturation values. Figure 6 shows the
average saturation of the NYU data set in different depth ranges. Based on the
observations, it appears that saturation may have less influence on the
results for indoor scenes, different from the result for outdoor scenes shown
in Figure 1.

Nevertheless, we intended to further investigate the extent to which this
subtle difference can affect depth estimation. In addition, as mentioned in
the Introduction, human depth perception can be influenced by saturation. To
assess the specific contribution of saturation, we extracted the saturation
feature for experimentation independently.

To extract the features pertaining to saturation, we started by converting the
RGB colour space to the HSV colour space and then extracting the saturation
maps. Subsequently, these saturation maps are subjected to phase scrambling to
eliminate features such as shape, texture and other visual characteristics.

$\mathrm{V}\leftarrow\max(\mathrm{R},\mathrm{G},\mathrm{B})$ (1)

$\mathrm{S}\leftarrow\begin{cases}\frac{\mathrm{V}-\min(\mathrm{R},\mathrm{G},\mathrm{B})}{\mathrm{V}}&\text{
if }\mathrm{V}\neq 0\\\ 0&\text{ otherwise }\end{cases}$ (2)

For each pixel, the V maps are obtained by taking the maximum value (Eq.1)
among the RGB channels. Subsequently, the saturation feature is obtained based
on phase scrambling from S maps (Eq.2). As depicted in Figure 7, Figure 7(a)
illustrates the saturation feature, with Figure 7(b) displaying its
corresponding depth map.

(a) Saturation Feature

(b) Corresponding Depth

Figure 7: Saturation with Phase Scrambling of Figure 2

###  3.3 Local Texture

The preference for local textures over global textures stems from the fact
that the extraction of global textures includes the consideration of
additional factors, including shape and other features. To mitigate the
influence of other factors and preserve the texture, the images were segmented
into patches and shuffled, thus eliminating global information such as shapes,
since this information introduces more features than just textures.

###  3.4 Shape

The boundaries of an object define its precise outline, marking the separation
between the object and its immediate environment. Edge maps are generated
through the analysis of gradient variations in image pixel values and identify
changes in these values. Although edge maps do not always faithfully represent
real object boundaries, when dealing with a single 2D image, they offer an
efficient means of simulating object shapes. This feature has been defined as
‘shape’ for the subsequent experiment.

As shown in Figure 8, we utilised the Canny operator instead of the Sobel
operator because the latter will find the gradient in the x and y directions,
reflecting the differential changes of pixels [31]. Therefore, not only the
shape feature is included when using the Sobel operator, but some texture
information may also be introduced.

(a) Shape with Sobel

(b) Shape with Canny

(c) Depth

Figure 8: Shape Maps from Figure 2 and Corresponding Ground Truth Depth

##  4 Experiments

As mentioned in the Introduction, we considered four factors that may
contribute to depth estimation: colour, saturation, local texture and shape.
All of these features were trained using the baseline model, and the obtained
results were analysed.

###  4.1 Data

We used the NYU dataset [29], which serves as a widely employed dataset in
computer vision, particularly for depth estimation research. Comprising images
from diverse indoor scenes, it encompasses a variety of objects and furniture.
The size of the NYU dataset enhances the representativeness of model training
and evaluation. It is derived from 464 scenes in three cities. The resolution
of the images is $640\times 480$. 10% of the data is split as the testing set.

###  4.2 Model

The UNet architecture is preferred for deep learning-based depth estimation
due to its comprehensive design, adept at gathering context and integrating
features across different scales [3, 35, 2, 8]. This preference is rooted in
UNet’s feature pyramid structure and efficient reuse of features, enhancing
depth estimation by capturing diverse scale information while preserving
detail. Our experiments demonstrate that employing ResNet50 as the backbone is
sufficient for model convergence on our dataset. Subsequently, we utilised the
U-Net network with ResNet50 as the backbone in the following experiment.

###  4.3 Evaluation Metrics

We utilised six metrics commonly used in the field of depth estimation, which
include three accuracy metrics and three error metrics. The accuracy metrics
are distinguished by thresholds at $1.25$, $1.25^{2}$ and $1.25^{3}$, each
reflecting different levels of tolerance for deviation from the true values.
Higher values of these accuracy metrics indicate better model performance. For
error metrics, the absolute relative error ($rel$) quantifies the average
deviation of predicted values from the actual values. The root mean squared
error ($rmse$) can amplify the effect of outliers by taking the square root of
the average of the squared deviations from the ground truth, and the
logarithmic error ($log_{10}$) metric mitigates the impact of outliers by
applying a logarithmic scale to the error values. Lower values of these error
metrics signify superior model performance.

###  4.4 Experiments and Analysis

Table 1: Depth Estimation Performance with Different Inputs

Features $a_{1}\uparrow$ $a_{2}\uparrow$ $a_{3}\uparrow$ $log_{10}\downarrow$
$rel\downarrow$ $rmse\downarrow$ Original RGB Images $98.13\pm 0.0013$
$99.64\pm 0.0003$ $99.9\pm 0.0001$ $0.0176\pm 0.0001$ $0.0413\pm 0.0008$
$0.0174\pm 0.0002$ RGB Phase Scrambled $43.5\pm 0.0067$ $72.64\pm 0.0068$
$87.47\pm 0.0042$ $0.1498\pm 0.0021$ $0.4754\pm 0.017$ $0.113\pm 0.0019$
Grayscale Phase Scrambled $36.13\pm 0.0316$ $64.09\pm 0.041$ $81.65\pm 0.033$
$0.1769\pm 0.0143$ $0.5627\pm 0.0585$ $0.1364\pm 0.0149$ Saturation $36.9\pm
0.0094$ $65.35\pm 0.0113$ $82.86\pm 0.0065$ $0.1718\pm 0.0026$ $0.5321\pm
0.0208$ $0.1296\pm 0.0015$ Local Texture $49.95\pm 0.0286$ $77.88\pm 0.0228$
$90.83\pm 0.0114$ $0.1276\pm 0.0068$ $0.3187\pm 0.0215$ $0.1065\pm 0.0047$
Shape $96.46\pm 0.0003$ $99.12\pm 0.0002$ $99.71\pm 0.0002$ $0.0235\pm 0.0001$
$0.0556\pm 0.0004$ $0.0224\pm 0.0001$

Table 1 presents the performance of depth estimation using different input
features, evaluated by several metrics (details shown in Sec.4.3). Original
RGB images performed the best with high accuracy ($a_{1}$, $a_{2}$, $a_{3}$)
and low error ($log_{10}$, $rel$, $rmse$). Phase-scrambled RGB and greyscale
images, along with saturation inputs, showed significantly worse performance,
with greyscale being the least accurate. Inputs of local texture had moderate
accuracy and error rates, while shape features performed close to the original
RGB images.

####  4.4.1 Colour

(a) RGB Image

(b) GT Depth

(c) Estimated Depth

Figure 9: Depth Estimation with a Colour Feature Input. The left and middle
images are the original RGB image and the corresponding ground truth depth
map, respectively. The image on the right depicts the estimated depth map,
which is the result of the model’s output after inverse phase scrambling,
employing the colour feature as the input.

To evaluate the contribution of the colour feature, we trained the model with
phase-scrambled RGB images. Figure 9 displays the original RGB image, ground
truth depth map and the estimated depth map, the latter of which has been
reconstructed from the scrambled image using a pre-stored random matrix. As
aligned to the low accuracy indicated in Table 1, it is hard to recognise the
original scene structure from the estimated depth.

To simulate scenarios where the model output differs from the ground truth, we
added Gaussian noise (mean = 0, std = 25) to the phase scrambled image. Figure
11 shows examples of our phase scrambled image with added Gaussian noise and
their corresponding reconstructions. Figure 10 shows the outcomes of
introducing Gaussian noise to the phase-scrambled image, followed by its
restoration using the pre-stored random matrix. As we can see, despite the
introduction of noise through phase scrambling, this noise does not affect the
shape and position of objects in the recovered images. The performance in
Figure 9(c) can be attributed to the poor performance of the model when
provided with colour phase-scrambled input.

(a) Noise for Whole Image

(b) Noise in Central Region

Figure 10: Noised with Phase Scrambled Images. Figure 10(a) illustrates the
outcome of applying Gaussian noise to the entire image and subsequently
restoring it, while Figure 10(b) depicts the results of adding noise and
restoring only the central area, where both the length and width are half of
the original image’s dimensions.

Furthermore, by comparing the respective performances of “RGB Phase Scrambled”
and “Grayscale Phase Scrambled” inputs as shown in Table 1, it can be observed
that, after excluding the contribution of brightness, colour has a limited
impact on depth estimation.

####  4.4.2 Saturation

(a) Saturation Map

(b) GT Depth

(c) Estimated Depth

Figure 11: Depth Estimation with a Saturation Feature Input. The left and
middle images are the original RGB image and the corresponding ground truth
depth map, respectively. The image on the right depicts the estimated depth
map, which is the result of the model’s output after inverse phase scrambling,
employing a saturation map as the input.

We trained the baseline model with saturation maps as the input and evaluated
the contribution of the saturation feature. Figure 11 illustrates the
saturation map, corresponding ground truth depth and the output from the
restoration process. Similarly, due to the poor performance, the restored
output lacks discernible features such as object contours.

Table 1 shows that the $a_{1}$ is about 37%. Although saturation contributed
to estimating the depth of the indoor scene, its contribution was minor.
Saturation exhibits lower accuracies compared to other features except for
greyscale phase scrambled input. Furthermore, error metrics substantiate this
observation. The $rel$ stands at 0.904, while the root mean square error
($rmse$) is shown as 0.1196. Therefore, using saturation as a measure for
depth estimation clearly introduces a significant error compared to the true
depth values. Despite its poor performance, saturation still plays a role in
assessing depth in indoor scenes. This highlights that saturation can provide
some depth cues in certain contexts, although it comes with a higher error
margin.

####  4.4.3 Local Texture

(a) Original Image

(b) Patch Size 128

(c) Patch Size 64

(d) Patch Size 32

(e) Patch Size 16

(f) Patch Size 4

Figure 12: Local Texture with Different Patch Sizes of a Random Sample

Variations in the field of view and resolution will impact the size of the
patch used to extract local textures. The optimal patch size should align with
the specific data set and scene scale employed. Figure 12 illustrates the
paths with varying patch dimensions. As shown in Figure 12(b), when we use a
large patch size of 128, the texture itself is not isolated because the shape
and context information still present in the patches. As the patch size is
decreased, the shape of the objects in the image becomes less apparent and,
therefore, the textures present in the image are increasingly segregated. As
demonstrated in Figure 12(e), for the data set we used, the $16\times 16$
patch size is particularly well-suited for texture extraction while minimising
the influence of other features (e.g. shape). This size is large enough to
restrict shape details but not so small as to be impractical, unlike the
$4\times 4$ patch depicted in Figure 12(f).

Table 2: Performance with Different Patch Sizes

Size $a_{1}\uparrow$ $a_{2}\uparrow$ $a_{3}\uparrow$ $log_{10}\downarrow$
$rel\downarrow$ $rmse\downarrow$ 4 $40.97\pm 0.0037$ $69.6\pm 0.003$ $85.99\pm
0.0018$ $0.1523\pm 0.0008$ $0.3812\pm 0.0026$ $0.1239\pm 0.0007$ 16 $49.95\pm
0.0286$ $77.88\pm 0.0228$ $90.83\pm 0.0114$ $0.1276\pm 0.0068$ $0.3187\pm
0.0215$ $0.1065\pm 0.0047$ 32 $53.27\pm 0.042$ $80.64\pm 0.0253$ $92.46\pm
0.0104$ $0.1185\pm 0.0087$ $0.3012\pm 0.0266$ $0.1009\pm 0.0094$ 64 $74.22\pm
0.0166$ $92.48\pm 0.0122$ $97.51\pm 0.0057$ $0.0747\pm 0.0039$ $0.1885\pm
0.0199$ $0.0629\pm 0.002$ 128 $93.12\pm 0.0066$ $98.47\pm 0.0019$ $99.5\pm
0.0007$ $0.0358\pm 0.0016$ $0.0863\pm 0.0039$ $0.0338\pm 0.0015$

Table 2 shows the performance of texture inputs (shuffle patches) in different
patch sizes. As can be seen in the figure, the accuracy rate gradually
increases with the increase of patch sizes in height. This is because a larger
patch contains more information besides the texture, such as the shape of the
object.

(a) Gray-scale Image

(b) GT Depth

(c) Estimated Depth

Figure 13: Depth Estimation with a Local Texture Input with Patch Size
$16\times 16$. The left and middle images are the original RGB image and the
corresponding ground truth depth map, respectively. The image on the right
depicts the estimated depth map, which is the result of the model’s output
after inverse shuffle by using a pre-stored random matrix, employing a local
texture feature as input.

A sample of the original greyscale image, along with the corresponding depth
map and the estimated depth map, is displayed in Figure 13, demonstrating the
results of the model trained with local texture inputs. To focus on local
textures during training, Figure 13(a) and Figure 13(b) are split into
$16\times 16$ patches and these patches are shuffled using the same random
matrix to eliminate global scene information, such as object shapes. As shown
in Figure 13(c), the estimated depth map only provides a coarse approximation
of the scene’s depth, distinguishing between nearer and farther areas but
failing to capture the precise depth details.

The local texture appears as a minor factor in depth estimation, yielding an
$a_{1}$ accuracy of a mere 50% in Table 1. Error metrics also show this trend
although they are slightly better than the colour and saturation features.
This issue happens because the position changes to the patches weaken their
connections, thereby making it harder for the model to understand objects and
context. This proposition finds corroboration in the robust performance
observed upon deploying the shape feature as the input data source in Sec
4.4.4.

####  4.4.4 Shape

As we noted in Table 1, the shape comes across as the most dominant feature in
these experiments, significantly outperforming other cues taken in isolation.
We suggest this is because the data set contains indoor scenes of objects such
as furniture with accurately extractable edges whose relative orientations and
geometric forms can serve as powerful cues as seen in Figure 14 (b and c).

The outcome aligns with the finding presented in [16], suggesting that CNNs
are capable of deducing the depth map using merely a limited subset of pixels
from the input image. This hypothesis aligns with human perceptual abilities,
which allow for the extraction of approximate distance assessments from images
that depict geometric shapes.

(a) Shape Map

(b) GT Depth

(c) Estimated Depth

Figure 14: Depth Estimation with a Shape Feature Input. The left and middle
images are the original RGB image and corresponding ground truth depth map.
The right is the estimated depth map from the model trained with shape maps.

####  4.4.5 Generalisation

In light of the fact that models using shape maps as input exhibit performance
approximating that of models employing original RGB images as the input, we
have assessed the generalisation capacity of shape models trained with shape
maps on the NYU data set. We applied it to a diverse set of indoor
environments from a different data set [24] that includes kitchens, bedrooms,
bathrooms and various other scenes. The performance of the shape model is
depicted in Figure 15, illustrating its ability to predict depth maps even for
scenes from a different domain, and the performance is similar to that of the
original RGB model. However, shape maps, as input for depth estimation, still
have their limitations. For instance, in the fourth-row images in Figure 15,
the sink only has partial edges, leading to poor depth prediction. Additional
results are presented in the Appendix.

Shape maps require significantly less memory storage compared to original RGB
images, while still providing comparable performance. In a similar vein, event
cameras are designed to only detect rapid changes in pixel intensity [26, 28]
which often occur at the edges of objects or where there is texture variation,
which is similar to a shape map. Moreover, event cameras have previously been
applied in the field of depth estimation [10]. Given these considerations, our
research may offer supporting evidence for the application of event cameras in
monocular depth estimation.

Figure 15: Performance of Shape ONLY model with New Indoor Scenes from other
Domains. The left column displays original RGB scene images, the second column
presents corresponding edge maps and the third column showcases the results
generated by the pre-trained shape-input model. The right column exhibits the
outcomes produced by the pre-trained original-RGB-image-input model.

####  4.4.6 Discussion

Different types of input data have varying effects on the performance of depth
estimation. Comparative analysis of diverse evaluation metrics clearly
highlights the superior role of shape information in the depth estimation
task. Colour, saturation and local texture collectively enhance the indoor
scene depth estimation, although the influence of colour and saturation
appears relatively circumscribed.

For phase-scrambled and local texture inputs, human vision finds it difficult
to interpret images when their phase information is scrambled or only shuffled
patches are present. In contrast, machines are adept at using these inputs to
predict depth maps. Given that the models can output corresponding depth maps
when employing these as inputs, their performance, albeit not optimal, is
still noteworthy compared to the ability of humans.

##  5 Limitations

Throughout our study, we sought to isolate each feature we were evaluating.
However, it is difficult to entirely isolate individual features. For
instance, during the extraction of shape features, the edge detector might
inadvertently capture some texture information.

##  6 Conclusion

In this work, we have sought to decouple and quantify the relative
contributions of various depth cues in monocular depth estimation. Whereas
good results have been demonstrated in the literature by the end-to-end
training of deep neural network models to achieve this task, ours is the first
attempt to understand the degree to which some known cues of depth contribute
when taken in isolation. Our results show that, in a data set of indoor
scenes, shape extracted by edge detection is relatively the most significant
contributor, while other cues (colour, saturation and texture) also play a
role. In achieving these conclusions, this work sought to carefully design
feature extraction techniques that aimed to isolate a single feature from the
other known ones, which is non-trivial. We speculate (and this is the subject
of our current research) that, on different depth inference problems (e.g.
outdoor scenes), the relative contributions of texture and saturation are
likely to play a greater role. This kind of decomposition which we have
extracted can serve to shift research more in the direction of understanding
and explaining how powerful models, such as deep neural networks, work in
scene understanding as opposed to simply offering estimation performance as
black-box function approximators.

## References

  * Alawadh et al. [2022] Mona Alawadh, Yihong Wu, Yuwen Heng, Luca Remaggi, Mahesan Niranjan, and Hansung Kim.  Room acoustic properties estimation from a single 360° photo.  In _2022 30th European Signal Processing Conference (EUSIPCO)_ , pages 857–861. IEEE, 2022.
  * Alhashim and Wonka [2018] Ibraheem Alhashim and Peter Wonka.  High quality monocular depth estimation via transfer learning.  _arXiv preprint arXiv:1812.11941_ , 2018.
  * Bhat et al. [2021] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.  Adabins: Depth estimation using adaptive bins.  In _Proc. CVPR_ , pages 4009–4018, 2021.
  * Cortes et al. [2012] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh.  Algorithms for learning kernels based on centered alignment.  _The Journal of Machine Learning Research_ , 13(1):795–828, 2012.
  * Cutting and Vishton [1995] James E Cutting and Peter M Vishton.  Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth.  In _Perception of space and motion_ , pages 69–117. Elsevier, 1995.
  * Dickson et al. [2021] Anthony Dickson, Alistair Knott, and Stefanie Zollmann.  Benchmarking monocular depth estimation models for vr content creation from a user perspective.  In _2021 36th International Conference on Image and Vision Computing New Zealand (IVCNZ)_ , pages 1–6. IEEE, 2021.
  * Dijk and Croon [2019] Tom van Dijk and Guido de Croon.  How do neural networks see depth in single images?  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2183–2191, 2019.
  * Eigen et al. [2014] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  _Proc. NeurIPS_ , 27, 2014.
  * Farid et al. [2013] Muhammad Shahid Farid, Maurizio Lucenteforte, and Marco Grangetto.  Edges shape enforcement for visual enhancement of depth image based rendering.  In _2013 IEEE 15th International Workshop on Multimedia Signal Processing (MMSP)_ , pages 406–411. IEEE, 2013.
  * Gallego et al. [2020] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al.  Event-based vision: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(1):154–180, 2020.
  * Ge et al. [2022] Yunhao Ge, Yao Xiao, Zhi Xu, Xingrui Wang, and Laurent Itti.  Contributions of shape, texture, and color in visual recognition.  In _European Conference on Computer Vision_ , pages 369–386. Springer, 2022.
  * Gegenfurtner and Rieger [2000] Karl R Gegenfurtner and Jochem Rieger.  Sensory and cognitive contributions of color to the recognition of natural scenes.  _Current Biology_ , 10(13):805–808, 2000.
  * Geirhos et al. [2018] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  _arXiv preprint arXiv:1811.12231_ , 2018.
  * Gibson [1950] James J Gibson.  _The perception of the visual world._ Houghton Mifflin, 1950.
  * Grzybowski and Kupidura-Majewski [2019] Andrzej Grzybowski and Konrad Kupidura-Majewski.  What is color and how it is perceived?  _Clinics in dermatology_ , 37(5):392–401, 2019.
  * Hu et al. [2019] Junjie Hu, Yan Zhang, and Takayuki Okatani.  Visualization of convolutional neural networks for monocular depth estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3869–3878, 2019.
  * Janai et al. [2020] Joel Janai, Fatma Güney, Aseem Behl, Andreas Geiger, et al.  Computer vision for autonomous vehicles: Problems, datasets and state of the art.  _Foundations and Trends® in Computer Graphics and Vision_ , 12(1–3):1–308, 2020.
  * Jin et al. [2020] Lei Jin, Yanyu Xu, Jia Zheng, Junfei Zhang, Rui Tang, Shugong Xu, Jingyi Yu, and Shenghua Gao.  Geometric structure based and regularized depth estimation from 360 indoor imagery.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 889–898, 2020.
  * Kornblith et al. [2019] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton.  Similarity of neural network representations revisited.  In _International conference on machine learning_ , pages 3519–3529. PMLR, 2019.
  * Lee et al. [2011] Wonwoo Lee, Nohyoung Park, and Woontack Woo.  Depth-assisted real-time 3d object detection for augmented reality.  In _ICAT_ , pages 126–132, 2011.
  * Liu et al. [2022] Weiyang Liu, Zhen Liu, Liam Paull, Adrian Weller, and Bernhard Schölkopf.  Structural causal 3d reconstruction.  In _European Conference on Computer Vision_ , pages 140–159. Springer, 2022.
  * Neitz and Neitz [2000] Maureen Neitz and Jay Neitz.  Molecular genetics of color vision and color vision defects.  _Archives of ophthalmology_ , 118(5):691–700, 2000.
  * Puce et al. [1996] Aina Puce, Truett Allison, Maryam Asgari, John C Gore, and Gregory McCarthy.  Differential sensitivity of human visual cortex to faces, letterstrings, and textures: a functional magnetic resonance imaging study.  _Journal of neuroscience_ , 16(16):5205–5215, 1996.
  * Quattoni and Torralba [2009] Ariadna Quattoni and Antonio Torralba.  Recognizing indoor scenes.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 413–420. IEEE, 2009.
  * Raghu et al. [2021] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do vision transformers see like convolutional neural networks?  _Advances in Neural Information Processing Systems_ , 34:12116–12128, 2021.
  * Rebecq et al. [2018] Henri Rebecq, Daniel Gehrig, and Davide Scaramuzza.  Esim: an open event camera simulator.  In _Conference on robot learning_ , pages 969–982. PMLR, 2018.
  * Rowland [1999] James H Rowland.  _The effects of texture on distance estimation in synthetic environments_.  PhD thesis, Monterey, California; Naval Postgraduate School, 1999.
  * Scheerlinck et al. [2020] Cedric Scheerlinck, Henri Rebecq, Daniel Gehrig, Nick Barnes, Robert Mahony, and Davide Scaramuzza.  Fast image reconstruction with an event camera.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 156–163, 2020.
  * Silberman et al. [2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12_ , pages 746–760. Springer, 2012.
  * Swain [1997] Cassandra T Swain.  Integration of monocular cues to create depth effect.  In _1997 IEEE International Conference on Acoustics, Speech, and Signal Processing_ , pages 2745–2748. IEEE, 1997.
  * Szeliski [2010] Richard Szeliski.  _Computer vision: algorithms and applications_.  Springer Science & Business Media, 2010.
  * Tuceryan and Jain [1993] Mihran Tuceryan and Anil K Jain.  Texture analysis.  _Handbook of pattern recognition and computer vision_ , pages 235–276, 1993.
  * Wang et al. [2019] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger.  Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.  In _Proc. CVPR_ , pages 8445–8453, 2019.
  * Wu et al. [2020] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3d objects from images in the wild.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1–10, 2020.
  * Wu et al. [2023] Yihong Wu, Yuwen Heng, Mahesan Niranjan, and Hansung Kim.  Depth estimation for a single omnidirectional image with reversed-gradient warming-up thresholds discriminator.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE, 2023.