# Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective
Distillation and Unlabeled Data Augmentation

Ning-Hsu Wang
<EMAIL_ADDRESS>
&Yu-Lun Liu
Department of Computer Science
National Yang Ming Chiao Tung University
<EMAIL_ADDRESS>

###### Abstract

Accurately estimating depth in 360-degree imagery is crucial for virtual
reality, autonomous navigation, and immersive media applications. Existing
depth estimation methods designed for perspective-view imagery fail when
applied to 360-degree images due to different camera projections and
distortions, whereas 360-degree methods perform inferior due to the lack of
labeled data pairs. We propose a new depth estimation framework that utilizes
unlabeled 360-degree data effectively. Our approach uses state-of-the-art
perspective depth estimation models as teacher models to generate pseudo
labels through a six-face cube projection technique, enabling efficient
labeling of depth in 360-degree images. This method leverages the increasing
availability of large datasets. Our approach includes two main stages: offline
mask generation for invalid regions and an online semi-supervised joint
training regime. We tested our approach on benchmark datasets such as
Matterport3D and Stanford2D3D, showing significant improvements in depth
estimation accuracy, particularly in zero-shot scenarios. Our proposed
training pipeline can enhance any 360 monocular depth estimator and
demonstrates effective knowledge transfer across different camera projections
and data types. See our project page for results:
albert100121.github.io/Depth-Anywhere.

##  1 Introduction

In recent years, the field of computer vision has seen a surge in research
focused on addressing the challenges associated with processing 360-degree
images. The widespread use of panoramic imagery across various domains, such
as virtual reality, autonomous navigation, and immersive media, has
underscored the need for accurate depth estimation techniques tailored
specifically for 360-degree images. However, existing depth estimation methods
developed for perspective-view images encounter significant difficulties when
applied directly to 360-degree data due to differences in camera projection
and distortion. While many methods aim to address depth estimation for this
camera projection, they often struggle due to the limited availability of
labeled datasets.

To overcome these challenges, this paper presents a novel approach for
training state-of-the-art (SOTA) depth estimation models on 360-degree
imagery. With the recent significant increase in the amount of available data,
the importance of both data quantity and quality has become evident. Research
efforts on perspective perceptual models have increasingly focused on
augmenting the volume of data and developing foundation models that generalize
across various types of data. Our method leverages SOTA perspective depth
estimation foundation models as teacher models and generates pseudo labels for
unlabeled 360-degree images using a six-face cube projection approach. By
doing so, we efficiently address the challenge of labeling depth in 360-degree
imagery by leveraging perspective models and large amounts of unlabeled data.

Our approach consists of two key stages: offline mask generation and online
joint training. During the offline stage, we employ a combination of detection
and segmentation models to generate masks for invalid regions such as sky and
watermarks in unlabeled data. Subsequently, in the online stage, we adopt a
semi-supervised learning strategy, loading half of the batch with labeled data
and the other half with pseudo-labeled data. Through joint training with both
labeled and pseudo-labeled data, our method achieves robust depth estimation
performance on 360-degree imagery.

To validate the effectiveness of our approach, we conduct extensive
experiments on benchmark datasets such as Matterport3D and Stanford2D3D. Our
method demonstrates significant improvements in depth estimation accuracy,
particularly in zero-shot scenarios where models are trained on one dataset
and evaluated on another. Furthermore, we demonstrate the efficacy of our
training techniques with different SOTA 360-degree depth models and various
unlabeled datasets, showcasing the versatility and effectiveness of our
approach in addressing the unique challenges posed by 360-degree imagery.

Our contributions can be summarized as follows:

  * •

We propose a novel training technique for 360-degree imagery that harnesses
the power of unlabeled data through the distillation of perspective foundation
models.

  * •

We introduce an online data augmentation method that effectively bridges
knowledge distillation across different camera projections.

  * •

Our proposed training techniques significantly benefit and inspire future
research on 360-degree imagery by showcasing the interchangeability of state-
of-the-art (SOTA) 360 models, perspective teacher models, and unlabeled
datasets. This enables better results even as new SOTA techniques emerge in
the future.

Figure 1: Our proposed training pipeline improves existing 360 monocular depth
estimators. This figure demonstrated the improvement of our proposed training
pipeline tested on the Stanford2D3D [2] dataset in a zero-shot setting.

##  2 Related Work

#### 360 monocular depth.

Depth estimation for 360-degree images presents unique challenges due to the
equirectangular projection and inherent distortion. Various approaches have
been explored to address these issues:

  * •

_Directly Apply_ : Some methods directly apply monocular depth estimation
techniques to 360-degree imagery. OmniDepth [55] leverages spherical geometry
and incorporates SphConv [31] to improve depth prediction with distortion.
[56, 37] use spherical coordinates to overcome distortion with extra
information. SliceNet [21] and ACDNet [54] propose advanced network
architectures tailored for omnidirectional images. EGFormer [50] introduces a
transformer-based model that captures global context efficiently, while [33,
32] focuses on integrating geometric priors into the learning process.

  * •

_Cube_ : Other approaches use cube map projections to mitigate distortion
effects. 360-SelfNet [34] is the first work to self-supervised 360 depth
estimation leveraging cube-padding [7]. BiFuse [35] and its improved version
BiFuse++ [36] are two-branch architectures that utilize cube maps and
equirectangular projections. UniFuse [11] combines equirectangular and cube
map projections and simplifies the architecture.

  * •

_Tangent Image_ : Tangent image projections are also popular. 360MonoDepth
[26] and OmniFusion [16] convert equirectangular images into a series of
tangent images, which are then processed using conventional depth estimation
networks. PanoFormer [29] employs a transformer-based architecture to handle
tangent images, while SphereNet [9] and HRDFuse [1] enhance depth prediction
by collaboratively learning from multiple projections.

#### Unlabeled / Pseudo labeled data.

Utilizing unlabeled or pseudo-labeled data has become a significant trend to
mitigate the limitations of labeled data scarcity. Techniques like [15, 57,
30, 44] leverage large amounts of unlabeled data to improve model performance
through semi-supervised learning. In the context of 360-degree depth
estimation, our approach generates pseudo labels from pre-trained perspective
models, which are then used to train 360-degree depth models effectively.

#### Zero-shot methods.

Zero-shot learning methods aim to generalize to new domains without additional
training data. [6, 42] target this directly with increasing training data.,
MiDaS [24, 4, 23] and Depth Anything [45] are notable for their robust
monocular depth estimation across diverse datasets leveraging affine-invariant
loss. [47] takes a step further to investigate zero-shot on metric depth.
Marigold [12] leverages diffusion models with image conditioning and up-to-
scale relative depth denoising to generate detailed depth maps. ZoeDepth [3]
further these advancements by incorporating scale awareness and domain
adaptation. [10, 38] leverage camera model information to adapt cross-domain
depth estimation.

#### Foundation models.

Foundation models have revolutionized various fields in AI, including natural
language processing and image-text alignment. In computer vision, models like
CLIP [22] demonstrate exceptional generalization capabilities. [19] proposed a
foundation visual encoder for downstream tasks such as segmentation,
detection, depth estimation, etc. [13] proposed a model that can cut out masks
for any objects. Our work leverages a pre-trained perspective depth estimation
foundation model [45] as a teacher model to generate pseudo labels for
360-degree images, enhancing depth estimation by utilizing the vast knowledge
embedded in these foundation models.

##  3 Methods

Figure 2: Training Pipeline. Our proposed training pipeline involves joint
training on both labeled 360 data with ground truth and unlabeled 360 data.
(a) For labeled data, we train our 360 depth model with the loss between depth
prediction and ground truth. (b) For unlabeled data, we propose to distill
knowledge from a pre-trained perspective-view monocular depth estimator. In
this paper, we use Depth Anything [45] to generate pseudo ground truth for
training. However, more advanced techniques could be applied. These
perspective-view monocular depth estimators fail to produce reasonable
equirectangular depth as there exists a domain gap. Therefore, we distill
knowledge by inferring six perspective cube faces and passing them through
perspective-view monocular depth estimators. To ensure stable and effective
training, we propose generating a valid pixel mask with Segment Anything [13]
while calculating loss. (c) Furthermore, we augment random rotation on RGB
before passing it into Depth Anything, as well as on predictions from the 360
depth model.

In this work, we propose a novel training approach for 360-degree monocular
depth estimation models. Our method leverages a perspective depth estimation
model as a teacher and generates pseudo labels for unlabeled 360-degree images
using a 6-face cube projection. Figure 2 illustrates our training pipeline,
incorporating the use of Segment Anything to mask out sky and watermark
regions in unlabeled data during the offline stage. Subsequently, we conduct
joint training using both labeled and unlabeled data, allocating half of the
batch to each. The unlabeled data is supervised using pseudo labels generated
by Depth Anything, a state-of-the-art perspective monocular depth foundation
model. With the benefit of our teacher model, the 360-degree depth model
demonstrates an observable improvement on zero-shot dataset 1.

###  3.1 Unleashing the Power of Unlabel 360 data

#### Dataset statistics.

360-degree data has become increasingly available in recent years. However,
compared to perspective-view depth datasets, labeling depth ground truths for
360-degree data presents greater challenges. Consequently, the availability of
labeled datasets for 360-degree data is considerably smaller than that of
perspective datasets.

Table 1 presents the data quantities available in some of the most popular
360-degree datasets, including Matterport3D [5], Stanford2D3D [2], and
Structured3D [51]. Additionally, we list a multi-modal dataset,
SpatialAudioGen [20], which consists of unlabeled 360-degree data used in our
experiments. Notably, the amount of labeled and unlabeled data used in the
perspective foundation model, Depth Anything [45], is significantly larger,
with 1.5 million labeled images [17, 40, 8, 46, 43, 39] and 62 million
unlabeled images [27, 49, 41, 48, 28, 14, 52, 13], making the amount in
360-degree datasets approximately 170 times smaller.

Table 1: 360 monocular depth estimation lacks a large amount of training data. The number of images in 360-degree monocular depth estimation datasets alongside perspective depth datasets from the Depth Anything methodology. Perspective | Equirectangular
---|---
Labeled | 1.5M | Unlabeled | 62 M | Labeled | 34K | Unlabeled | 344K

#### Data cleaning and valid pixel mask generation

Unlabeled data often contains invalid pixels in regions such as the sky and
watermark, leading to unstable training or undesired convergence. To address
this issue, we applied the GroundingSAM [25] method to mask out the invalid
regions. This approach utilizes Grounded DINOv2 [18] to detect problematic
regions and applies the Segment Anything [13] model to mask out the invalid
pixels by segmenting within the bounding box. While Depth Anything [45] also
employs a pre-trained segmentation model, DINOv2, to select sky regions. Brand
logos and watermarks frequently appear after fisheye camera stitching.
Therefore, additional labels are applied to enhance the robustness of our
training process. We also remove all images with less than 20 percent of valid
pixels to stablize our training progress.

Figure 3: Valid Pixel Masking. We used Grounded-Segment-Anything [25] to mask
out invalid pixels based on two text prompts: “sky” and “watermark.” These
regions lack depth sensor ground truth labels in all previous datasets. Unlike
Depth Anything [45], which sets sky regions as 0 disparity, we follow ground
truth training to ignore these regions during training for two reasons: (1)
segmentation may misclassify and set other regions as zero, leading to noisy
labeling, and (2) watermarks are post-processing regions that lack geometrical
meaning.

#### Perspective foundation models (teacher models).

To tackle the challenges posed by limited data and labeling difficulties in
360-degree datasets, we leverage a large amount of unlabeled data alongside
state-of-the-art perspective depth foundation models. Due to significant
differences in camera projection and distortion, directly applying perspective
models to 360-degree data often yields inferior results. Previous works have
explored various methods of projection for converting equirectangular to
perspective depth as stated in Sec 2. Among these, cube projection and tangent
projection are the most common techniques. We selected cube projection to
ensure a larger field of view for each patch, enabling better observation of
relative distances between pixels or objects during the inference of the
perspective foundation model and enhancing knowledge distillation.

In our approach, we apply projection to unlabeled 360-degree data and then run
Depth Anything on these projected patches of perspective images to generate
pseudo-labels. We explore two directions for pseudo-label supervision:
projecting the patch to equirectangular and computing in the 360-degree domain
or projecting the 360-degree depth output from the 360 model to patches and
computing in the perspective domain. Since training is conducted in an up-to-
scale relative depth manner, stitching the patch of perspective images back to
equirectangular with an aligned scale is an additional research aspect. We opt
to compute the loss in the perspective domain, facilitating faster and easier
training without the need for additional alignment optimization.

###  3.2 Random Rotation Processing

Directly applying Depth Anything on cube-projected unlabeled data does not
yield improvements due to ignorance of cross cube-face relation, leading to
cube artifacts (Figure 4). This issue arises from the separate estimation of
perspective cube faces, where monocular depth is estimated based on semantic
information, potentially lacking a comprehensive understanding of the entire
scene. To address this, we propose a random rotation preprocessing step in
front of the perspective foundation model.

As depicted in Figure 2, the rotation is applied to equirectangular projection
RGB images using a random rotation matrix, followed by cube projection. This
results in a more diverse set of cube faces, capturing relative distances
between ceilings, walls, windows, and other objects more effectively. With the
proposed random rotation technique, knowledge distillation becomes more
comprehensive as the point of view is not static. The inference by the
perspective foundation model is performed on the fly, with parameters frozen
during the training of the 360 model.

In order to perform random rotation, we apply a rotation matrix on the
equirectangular coordinates, noted as $(\theta,\phi)$, and rotation matrix as
$\mathcal{R}$.

$(\hat{\theta},\hat{\phi})=\mathcal{R}\cdot(\theta,\phi).$ (1)

For equirectangular to cube projection, the field-of-view (FoV) of each cube
face is equal to 90 degrees; each face can be considered as a perspective
camera whose focal length is $w/2$, and all faces share the same center point
in the world coordinate. Since the six cube faces share the same center point,
the extrinsic matrix of each camera can be defined by a rotation matrix
$R_{i}$. $p$ is then the pixel on the cube face

$p=K\cdot R^{T}_{i}\cdot q,$ (2)

where,

$q=\begin{bmatrix}q_{x}\\\ q_{y}\\\
q_{z}\end{bmatrix}=\begin{bmatrix}sin(\theta)\cdot\cos(\phi)\\\ \sin(\phi)\\\
\cos{\theta}\cdot\cos{\phi}\end{bmatrix},K=\begin{bmatrix}w/2&0&w/2\\\
0&w/2&w/2\\\ 0&0&1\\\ \end{bmatrix},$ (3)

where $\theta$ and $\phi$ are longitude and latitude in equirectangular
projection and q is the position in Euclidean space coordinates.

Figure 4: Cube Artifact. As shown in the center row of the figure, an
undesired cube artifact appears when we apply joint training with pseudo
ground truth from Depth Anything [45] directly. This issue arises from
independent relative distances within each cube face caused by a static point
of view. Ignoring cross-cube relationships results in poor knowledge
distillation. To address this, as shown in Figure 2 (c), we randomly rotate
the RGB image before inputting it into Depth Anything. This enables better
distillation of depth information from varying perspectives within the
equirectangular image.

###  3.3 Loss Function

The training process closely resembles that of MiDaS, Depth Anything, and
other cross-dataset methods. Our goal is to provide depth estimation for any
360-degree images. Following previous approaches that trained on multiple
datasets, our training objective is to estimate relative depth. The depth
values are first transformed into disparity space using the formula $1/d$ and
then normalized to the range $[0,1]$ for each disparity map.

To adapt to cross-dataset training and pseudo ground truths from the
foundation model, we employed the affine-invariant loss, consistent with prior
cross-dataset methodologies. This loss function disregards absolute scale and
shifts for each domain, allowing for effective adaptation across different
datasets and models.

$\mathcal{L}1=\frac{1}{HW}\sum^{HW}_{i=1}{\rho(d^{*}_{i},d_{i})},$ (4)

where $d^{*}_{i}$ and $d_{i}$ are the prediction and ground truth,
respectively. $\rho$ represents the affine-invariant mean absolute error loss:

$\rho(d^{*}_{i},d_{i})=|\hat{d}^{*}_{i}-\hat{d}_{i}|.$ (5)

Here, $\hat{d_{i}}$ and $\hat{d}^{*}_{i}$ are the scaled and shifted versions
of the prediction $d^{*}_{i}$ and ground truth $d_{i}$:

$\hat{d_{i}}=\frac{d_{i}-t(d)}{s(d)},$ (6)

where $t(d)$ and $s(d)$ are used to align the prediction and ground truth to
have zero translation and unit scale:

$t(d)=\text{median}(d),\quad s(d)=\frac{1}{HW}\sum^{HW}_{i=1}{|d_{i}-t(d)|}.$
(7)

##  4 Experiments

These notations apply for all tables: M: Matterport3D [5], SF: Stanford2D3D
[2], ST: Structured3D [51], SP: Spatialaudiogen [20], -all indicates using the
entire train, validation, and test sets of the specific dataset, and (p)
denotes using pseudo ground truth generated by Depth Anything [45]. Due to
space limits, we provide the experimental setup in the appendix, including
implementation details and evaluation metrics.

###  4.1 Baselines

Recent state-of-the-art methods [1, 50, 35, 36, 11, 21, 29] have emerged. We
chose UniFuse and BiFuse++ as our baseline models for experiments, as many of
the aforementioned methods did not release pre-trained models or provide
training code and implementation details. It’s worth noting that PanoFormer
[29] is not included due to incorrect evaluation code and results, and
EGFormer [29] is not included since its experiments are mainly conducted on
other datasets and benchmarks. Both selected models are re-implemented with
affine-invariant loss on disparity for a fair comparison and to demonstrate
improvement. We conduct experiments on the Matterport3D benchmark to
demonstrate improvement within the same dataset/domain and apply zero-shot
evaluation on the Stanford2D3D test set to demonstrate the generalization
ability of our proposed training technique.

###  4.2 Benchmarks evaluation

We conducted our in-domain improvement experiment on the widely used
360-degree depth benchmark, Matterport3D [5], to showcase the results of
perspective foundation model distillation on the two selected baseline models,
UniFuse [11] and BiFuse++[36]. In Table 2, we list the metric depth evaluation
results from state-of-the-art methods on this benchmark. Subsequently, we
present the re-trained baseline models using affine-invariant loss on
disparity to ensure a fair comparison with their original depth metric
training. Finally, we demonstrate the improvement achieved with results
trained on the labeled Matterport3D training set and the entire Structured3D
dataset with pseudo ground truth.

Table 2: Matterport3D Benchmark. The upper section lists 360 methods trained with metric depths in meters using BerHu loss. All numbers are sourced from their respective papers. The lower section includes selected methods retrained with relative depth (disparity) using affine-invariant loss. Method | Loss | Train | Test | Abs Rel ↓ |  $\delta_{1}$ ↑ |  $\delta_{2}$ ↑ |  $\delta_{3}$ ↑
---|---|---|---|---|---|---|---
BiFuse | BerHu | M | M | - | 0.8452 | 0.9319 | 0.9632
UniFise | BerHu | M | M | 0.1063 | 0.8897 | 0.9623 | 0.9831
SliceNet | BerHu | M | M | - | 0.8716 | 0.9483 | 0.9716
BiFuse++ | BerHu | M | M | - | 0.879 | 0.9517 | 0.9772
HRDFuse | BerHu | M | M | 0.0967 | 0.9162 | 0.9669 | 0.9844
UniFuse | Affine-Inv | M | M | 0.10187 | 0.89311 | 0.97044 | 0.98902
UniFuse | Affine-Inv | M, ST-all (p) | M | 0.08938 | 0.91143 | 0.97512 | 0.99078
BiFuse++ | Affine-Inv | M | M | 0.0938 | 0.9142 | 0.97363 | 0.9894
BiFuse++ | Affine-Inv | M, ST-all (p) | M | 0.085 | 0.917 | 0.976 | 0.991

###  4.3 Zero-shot evaluation

Our goal is to estimate depths for all 360-degree images, making zero-shot
performance crucial. Following previous works [35, 11], we adopted their zero-
shot comparison setting, where models trained on the entire Matterport3D [5]
dataset are tested on the Stanford2D3D [2] test set. In Table 3, the upper
section lists methods trained with metric depth ground truth, with numbers
sourced from their respective papers. The lower section includes models
trained with affine-invariant loss on disparity ground truth. As shown in
Figure 5, [11, 36] demonstrate generalization improvements with a lower error
on the Stanford2D3D dataset.

Depth Anything [45] and Marigold [12] are state-of-the-art zero-shot depth
models trained with perspective depths. As shown in Table 3, due to the domain
gap and different camera projections, foundation models trained with
perspective depth cannot be directly applied to 360-degree images. We
demonstrated the zero-shot improvement on UniFuse [11] and BiFuse++ [36] with
models trained on the entire Matterport3D [5] dataset with ground truth and
the entire Structured3D [51] or SpatialAudioGen [20] dataset with pseudo
ground truth generated using Depth Anything [45].

As Structured3D provides ground truth labels for its dataset, we also evaluate
our models on its test set to assess how well they perform with pseudo labels.
Table 4 shows the improvements achieved on the Structured3D test set when
using models trained with pseudo labels. It’s worth noting that even when the
360 model is trained on pseudo labels from SpatialAudioGen, it performs
similarly well. This demonstrates the success of our distillation technique
and the model’s ability to generalize across different datasets.

Table 3: Zero-shot Evaluation on Stanford2D3D. We perform zero-shot evaluations with models trained on other datasets. Following BiFuse and UniFuse’s settings, we train the 360 models on the entire Matterport3D dataset and then test on Stanford3D’s test set. Method | Loss | train | test | Abs Rel ↓ |  $\delta_{1}$ ↑ |  $\delta_{2}$ ↑ |  $\delta_{3}$ ↑
---|---|---|---|---|---|---|---
BiFuse | BerHu | M-all | SF | 0.1195 | 0.8616 | - | -
UniFuse | BerHu | M-all | SF | 0.0944 | 0.9131 | - | -
BiFuse++ | BerHu | M-all | SF | 0.107 | 0.914 | 0.975 | 0.989
Depth Anything | Affine-Inv | Pers. | SF | 0.248 | 0.635 | 0.899 | 0.97
Marigold | Affine-Inv | Pers. | SF | 0.195 | 0.692 | 0.942 | 0.982
UniFuse | Affine-Inv | M-all | SF | 0.09 | 0.914 | 0.976 | 0.99
UniFuse | Affine-Inv | M-all, ST-all (p) | SF | 0.086 | 0.924 | 0.977 | 0.99
UniFuse | Affine-Inv | M-all, SP-all (p) | SF | 0.09 | 0.92 | 0.978 | 0.99
BiFuse++ | Affine-Inv | M-all | SF | 0.09 | 0.921 | 0.976 | 0.99
BiFuse++ | Affine-Inv | M-all, ST-all (p) | SF | 0.082 | 0.931 | 0.979 | 0.991
BiFuse++ | Affine-Inv | M-all, SP-all (p) | SF | 0.858 | 0.926 | 0.979 | 0.991
Figure 5: Zero-shot qualitative with UnFuse [11] (_left 2 columns_) and BiFuse++ [36] (_right 2 columns_) tested on Stanford2D3D. Table 4: Structured3D Test Set. We demonstrate the improvement on the Structured3D test set using pseudo ground truth for training. The lower section shows enhancements with models trained on pseudo ground truth from Matterport3D and SpatialAudioGen, indicating similar improvements. This highlights the successful distillation of Depth Anything. Method | Loss | train | test | Abs Rel ↓ |  $\delta_{1}$ ↑ |  $\delta_{2}$ ↑ |  $\delta_{3}$ ↑
---|---|---|---|---|---|---|---
UniFuse | Affine-Inv | M-all | ST | 0.202 | 0.759 | 0.932 | 0.97
UniFuse | Affine-Inv | M-all, ST-all (p) | ST | 0.13 | 0.887 | 0.953 | 0.977
UniFuse | Affine-Inv | M-all, SP-all (p) | ST | 0.152 | 0.864 | 0.946 | 0.972

###  4.4 Qualtative results in the wild

We demonstrated the qualitative results in Figure 7 and Figure 6 360-degree
images that were either captured by us or downloaded from the internet111Stig
Nygaard, https://www.flickr.com/photos/stignygaard/49659694937, CC BY 2.0 DEED
Dominic Alves, https://www.flickr.com/photos/dominicspics/28296671029/, CC BY
2.0 DEED
Luca Biada, https://www.flickr.com/photos/pedroscreamerovsky/6873256488/, CC
BY 2.0 DEED
Luca Biada, https://www.flickr.com/photos/pedroscreamerovsky/6798474782/, CC
BY 2.0 DEED. These examples showcase the zero-shot capability of our model
when applied to data outside the aforementioned 360-degree datasets.

Figure 6: Generalization ability in the wild with depth map visualization: We
showcase zero-shot qualitative results using a combination of images captured
by us and randomly sourced from the internet to assess the model’s
generalization ability. For privacy reasons, we have obscured the camera-man
in the images. Figure 7: Generalization ability in the wild with point cloud
visualization: We showcase zero-shot qualitative results in point cloud using
a combination of images captured by us and randomly sourced from the internet
to assess the model’s generalization ability.

##  5 Conclusion

Our proposed method significantly advances 360-degree monocular depth
estimation by leveraging perspective models for pseudo-label generation on
unlabeled data. The use of cube projection with random rotation and affine-
invariant loss ensures robust training and improved depth prediction accuracy
while bridging the domain gap between perspective and equirectangular
projection. By effectively addressing the challenges of limited labeled data
with cross-domain distillation, our approach opens new possibilities for
accurate depth estimation in 360 imagery. This work lays the groundwork for
future research and applications, offering a promising direction for further
advancements in 360-degree depth estimation.

#### Limitations

Our work faces limitations due to its heavy reliance on the quality of
unlabeled data and pseudo labels from perspective foundation models. The
results are significantly impacted by data quality (Sec. 3.1). Without data
cleaning, the training process resulted in NaN values.

## References

  * Ai et al. [2023] Hao Ai, Zidong Cao, Yan-Pei Cao, Ying Shan, and Lin Wang.  Hrdfuse: Monocular 360deg depth estimation by collaboratively learning holistic-with-regional depth distributions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13273–13282, 2023.
  * Armeni et al. [2017] I. Armeni, A. Sax, A. R. Zamir, and S. Savarese.  Joint 2D-3D-Semantic Data for Indoor Scene Understanding.  _ArXiv e-prints_ , February 2017.
  * Bhat et al. [2023] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth.  _arXiv preprint arXiv:2302.12288_ , 2023.
  * Birkl et al. [2023] Reiner Birkl, Diana Wofk, and Matthias Müller.  Midas v3.1 – a model zoo for robust monocular relative depth estimation.  _arXiv preprint arXiv:2307.14460_ , 2023.
  * Chang et al. [2017] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  _International Conference on 3D Vision (3DV)_ , 2017.
  * Chen et al. [2016] Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng.  Single-image depth perception in the wild.  _Advances in neural information processing systems_ , 29, 2016.
  * Cheng et al. [2018] Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, and Min Sun.  Cube padding for weakly-supervised saliency prediction in 360 $\\{$$\backslash$deg$\\}$ videos.  _arXiv preprint arXiv:1806.01320_ , 2018.
  * Cho et al. [2021] Jaehoon Cho, Dongbo Min, Youngjung Kim, and Kwanghoon Sohn.  Diml/cvl rgb-d dataset: 2m rgb-d images of natural indoor and outdoor scenes.  _arXiv preprint arXiv:2110.11590_ , 2021.
  * Coors et al. [2018] Benjamin Coors, Alexandru Paul Condurache, and Andreas Geiger.  Spherenet: Learning spherical representations for detection and classification in omnidirectional images.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 518–533, 2018.
  * Guizilini et al. [2023] V. Guizilini, I. Vasiljevic, D. Chen, R. Ambrus, and A. Gaidon.  Towards zero-shot scale-aware monocular depth estimation.  In _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9199–9209, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society.  doi: 10.1109/ICCV51070.2023.00847.  URL https://doi.ieeecomputersociety.org/10.1109/ICCV51070.2023.00847.
  * Jiang et al. [2021] Hualie Jiang, Zhe Sheng, Siyu Zhu, Zilong Dong, and Rui Huang.  Unifuse: Unidirectional fusion for 360∘ panorama depth estimation.  _IEEE Robotics and Automation Letters_ , 2021.
  * Ke et al. [2024] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler.  Repurposing diffusion-based image generators for monocular depth estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2024.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv:2304.02643_ , 2023.
  * Kuznetsova et al. [2020] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  _IJCV_ , 2020.
  * Lee [2013] Dong-Hyun Lee.  Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks.  2013\.  URL https://api.semanticscholar.org/CorpusID:18507866.
  * Li et al. [2022] Yuyan Li, Yuliang Guo, Zhixin Yan, Xinyu Huang, Duan Ye, and Liu Ren.  Omnifusion: 360 monocular depth estimation via geometry-aware fusion.  In _2022 Conference on Computer Vision and Pattern Recognition (CVPR)_ , New Orleans, USA, June 2022.
  * Li and Snavely [2018] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Liu et al. [2023] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * Pedro Morgado and Wang [2018] Timothy Langlois Pedro Morgado, Nuno Vasconcelos and Oliver Wang.  Self-supervised generation of spatial audio for 360$\deg$ video.  In _Neural Information Processing Systems (NIPS)_ , 2018.
  * Pintore et al. [2021] Giovanni Pintore, Marco Agus, Eva Almansa, Jens Schneider, and Enrico Gobbetti.  SliceNet: deep dense depth estimation from a single indoor panorama using a slice-based representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11536–11545, June 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ranftl et al. [2021] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  _ICCV_ , 2021.
  * Ranftl et al. [2022] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(3), 2022.
  * Ren et al. [2024] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang.  Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
  * Rey-Area et al. [2022] Manuel Rey-Area, Mingze Yuan, and Christian Richardt.  360MonoDepth: High-resolution 360 monocular depth estimation.  In _CVPR_ , 2022.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115:211–252, 2015.
  * Shao et al. [2019] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * Shen et al. [2022] Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao.  Panoformer: Panorama transformer for indoor 360 depth estimation.  In _European Conference on Computer Vision_ , pages 195–211. Springer, 2022.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ , 33:596–608, 2020.
  * Su and Grauman [2017] Yu-Chuan Su and Kristen Grauman.  Learning spherical convolution for fast features from 360 imagery.  _Advances in neural information processing systems_ , 30, 2017.
  * Sun et al. [2021a] Cheng Sun, Chi-Wei Hsiao, Ning-Hsu Wang, Min Sun, and Hwann-Tzong Chen.  Indoor panorama planar 3d reconstruction via divide and conquer.  In _CVPR_ , 2021a.
  * Sun et al. [2021b] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Hohonet: 360 indoor holistic understanding with latent horizontal features.  In _CVPR_ , 2021b.
  * Wang et al. [2018] Fu-En Wang, Hou-Ning Hu, Hsien-Tzu Cheng, Juan-Ting Lin, Shang-Ta Yang, Meng-Li Shih, Hung-Kuo Chu, and Min Sun.  Self-supervised learning of depth and camera motion from 360° videos.  In _Asian Conference on Computer Vision_ , 2018.  URL https://api.semanticscholar.org/CorpusID:53290169.
  * Wang et al. [2020a] Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai.  Bifuse: Monocular 360 depth estimation via bi-projection fusion.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020a.
  * Wang et al. [2023] Fu-En Wang, Yu-Hsuan Yeh, Yi-Hsuan Tsai, Wei-Chen Chiu, and Min Sun.  Bifuse++: Self-supervised and efficient bi-projection fusion for 360° depth estimation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(5):5448–5460, 2023.  doi: 10.1109/TPAMI.2022.3203516.
  * Wang et al. [2020b] Ning-Hsu Wang, Bolivar Solarte, Yi-Hsuan Tsai, Wei-Chen Chiu, and Min Sun.  360sd-net: 360 stereo depth estimation with learnable cost volume.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 582–588. IEEE, 2020b.
  * Wang et al. [2021a] Ning-Hsu Wang, Ren Wang, Yu-Lun Liu, Yu-Hao Huang, Yu-Lin Chang, Chia-Ping Chen, and Kevin Jou.  Bridging unsupervised and supervised depth from focus via all-in-focus supervision.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 12621–12631, 2021a.
  * Wang et al. [2021b] Qiang Wang, Shizhen Zheng, Qingsong Yan, Fei Deng, Kaiyong Zhao, and Xiaowen Chu.  Irs: A large naturalistic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation.  In _2021 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1–6. IEEE, 2021b.
  * Wang et al. [2020c] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer.  Tartanair: A dataset to push the limits of visual slam.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 4909–4916. IEEE, 2020c.
  * Weyand et al. [2020] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim.  Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2575–2584, 2020.
  * Xian et al. [2018] Ke Xian, Chunhua Shen, ZHIGUO CAO, Hao Lu, Yang Xiao, Ruibo Li, and Zhenbo Luo.  Monocular relative depth perception with web stereo data supervision.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 311–320, 2018.  URL https://api.semanticscholar.org/CorpusID:52860134.
  * Xian et al. [2020] Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao.  Structure-guided ranking loss for single image depth prediction.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Xie et al. [2020] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10687–10698, 2020.
  * Yang et al. [2024] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao.  Depth anything: Unleashing the power of large-scale unlabeled data.  In _CVPR_ , 2024.
  * Yao et al. [2020] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan.  Blendedmvs: A large-scale dataset for generalized multi-view stereo networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1790–1799, 2020.
  * Yin et al. [2023] Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen.  Metric3d: Towards zero-shot metric 3d prediction from a single image.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9043–9053, 2023.
  * Yu et al. [2015] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_ , 2015.
  * Yu et al. [2020] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2636–2645, 2020.
  * Yun et al. [2023] Ilwi Yun, Chanyong Shin, Hyunku Lee, Hyuk-Jae Lee, and Chae Eun Rhee.  Egformer: Equirectangular geometry-biased transformer for 360 depth estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 6101–6112, October 2023.
  * Zheng et al. [2020] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou.  Structured3d: A large photo-realistic dataset for structured 3d modeling.  In _Proceedings of The European Conference on Computer Vision (ECCV)_ , 2020.
  * Zhou et al. [2017a] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017a.
  * Zhou et al. [2017b] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe.  Unsupervised learning of depth and ego-motion from video.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6612–6619, 2017b.  doi: 10.1109/CVPR.2017.700.
  * Zhuang et al. [2022] Chuanqing Zhuang, Zhengda Lu, Yiqun Wang, Jun Xiao, and Ying Wang.  Acdnet: Adaptively combined dilated convolution for monocular panorama depth estimation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 3653–3661, 2022.
  * Zioulis et al. [2018] Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, and Petros Daras.  Omnidepth: Dense depth estimation for indoors spherical panoramas.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 448–465, 2018.
  * Zioulis et al. [2019] Nikolaos Zioulis, Antonis Karakottas, Dimitris Zarpalas, Federic Alvarez, and Petros Daras.  Spherical view synthesis for self-supervised $360^{o}$ depth estimation.  In _International Conference on 3D Vision (3DV)_ , September 2019.
  * Zoph et al. [2020] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc V. Le.  Rethinking pre-training and self-training.  _ArXiv_ , abs/2006.06882, 2020.  URL https://api.semanticscholar.org/CorpusID:219635973.

##  Appendix A Appendix / supplemental material

###  A.1 Experimental setup

#### Implementation details.

Our work is divided into two stages: (1) offline mask generation and (2)
online joint training. (1) In the first stage, we use Grounded-Segment-
Anything [25], which combines state-of-the-art detection and segmentation
models. We set the BOX_THRESHOLD and TEXT_THRESHOLD to 0.3 and 0.25,
respectively, following the recommendations of the official code. We use “sky”
and “watermark” as text prompts. All pixels with these labels are set to False
to form our valid mask for the second stage of training. (2) In the second
stage, each batch consists of an equal mix of labeled and unlabeled data. We
follow the backbone model’s official settings for batch size, learning rate,
optimizer, augmentation, and other hyperparameters, changing only the loss
function to affine-invariant loss. Unlike Depth Anything, which sets invalid
sky regions to zero disparity, we ignore these invalid pixels during loss
calculation, consistent with ground truth training settings. We average the
loss for ground truth and pseudo ground truth during updates. All our
experiments are conducted on a single RTX 4090, both offline and online.
However, if future 360-degree state-of-the-art methods or perspective
foundation models require higher VRAM usage, the computational resource
requirements may increase.

#### Metrics.

In line with previous cross-dataset works, all evaluation metrics are
presented in percentage terms. The primary metric is Absolute Mean Relative
Error (AbsRel), calculated as:
$\frac{1}{M}\sum^{M}_{i=1}{|a_{i}-d_{i}|/d_{i}}$, where $M$ is the total
number of pixels, $a_{i}$ is the predicted depth, and $d_{i}$ is the ground
truth depth. The second metric, $\delta_{j}$ accuracy, measures the proportion
of pixels where $max(ai/di,di/ai)$ is within $1.25^{j}$. During evaluations,
we follow [11, 35, 36] to ignore areas where ground truth depth values are
larger than 10 or equal to 0. Given the ambiguous scale of self-training
results, we apply median alignment after converting disparity output to depth
before evaluation, as per the method used in [53]:

$d^{\prime}=d\cdot\frac{\text{median}(\hat{d})}{\text{median}(d)},$ (8)

where $d$ is the predicted depth from inverse disparity and $\hat{d}$ is the
ground truth depth. This ensures a fair comparison by aligning the median
depth values of predictions and ground truths.

###  A.2 More qualitative

We demonstrate additional zero-shot qualitative results in Figure 8. In-domain
results on the Matterport3D test sets are showcased in Figure 9 and Figure 10.

Figure 8: More qualitative tested on Stanford2D3D with zero-shot setting.
Figure 9: In-domain qualitative with UniFuse. Figure 10: In-domain qualitative
with BiFuse++.

###  A.3 Dataset statistic

As described in Sec.3.1 of the main paper, there is a significant difference
in the number of images between the perspective and equirectangular datasets.
Detailed statistics of the datasets are listed in Table5.

Table 5: 360 monocular depth estimation lacks a large amount of training data.
This table lists datasets used in 360-degree monocular depth estimation
alongside perspective depth datasets from the Depth Anything methodology. The
volume of training data for 360-degree imagery (_right column_) is
significantly smaller than that for perspective imagery (_left column_), by
about 200 times. This highlights the need for using perspective distillation
techniques to enhance the limited data available for 360-degree depth
estimation. Ground truth (GT) labels are noted where applicable, showing the
available resources for training in these domains.

Perspective | Equirectangular
---|---
Dataset | Venue | # of images | GT labels | Dataset | Venue | # of images | GT labels
MegaDepth [17] | CVPR 2018 | 128K | ✓ | Stanford2D3D [2] | arXiv 2017 | 1.4K | ✓
TartanAir [40] | IROS 2020 | 306K | ✓ | Matterport3D [5] | 3DV 2017 | 10.8K | ✓
DIML [8] | arXiv 2021 | 927K | ✓ | Structured3D [51] | ECCV 2020 | 21.8K | ✓
BlendedMVS [46] | CVRP 2020 | 115 K | ✓ | SpatialAudioGen [20] | NeurIPS 2018 | 344K | -
HRWSI [43] | CVPR 2020 | 20K | ✓ |  |  |  |
IRS [39] | ICME 2021 | 103K | ✓ |  |  |  |
ImageNet-21K [27] | IJCV 2015 | 13.1M | - |  |  |  |
BDD100K [49] | CVPR 2020 | 8.2M | - |  |  |  |
Google Landmarks [41] | CVPR 2020 | 4.1M | - |  |  |  |
LSUN [48] | arXiv 2015 | 9.8M | - |  |  |  |
Objects365 [28] | ICCV 2019 | 1.7M | - |  |  |  |
Open Images V7 [14] | IJCV 2020 | 7.8M | - |  |  |  |
Places365 [52] | TPAMI 2017 | 6.5M | - |  |  |  |
SA-1B [13] | ICCV 2023 | 11.1M | - |  |  |  |


Author Umesh Srinivasan was supported in part by the Department of Science &
Technology, Ministry of Science & Technology, India under project
SR/S3/EECE/0008/2006-SERC-Engg.

# On the relationship between speech and hearing

Srinivasan Umesh  Dept. of Electrical Eng.,IIT Madras-600042, India  Leon
Cohen  City University of New York, 695 Park Avenue, New York, NY 10021
Douglas Nelson  U.S. Dept. of Defense (retired), Ft. Meade, MD 20755

###### Abstract

We present a framework for experimentally linking speech production and
hearing. Using this approach, we describe experimental results, that lead to
the concept that sounds made by different individuals and perceived to be the
same can be transformed into each other by a “speech scale”. The speech scale
is empirically determined using _only_ speech data. We show the similarity of
the speech scale to the MEL scale of Stevens and Volkmann, which was derived
_only_ from hearing experiments. We thus experimentally link speech production
and hearing.

##  1 Introduction

For speech communication, nature has provided both a speech “transmitter” and
“receiver”; therefore, speech production and hearing must be highly coupled
systems. While a particular individual has only one receiver (the ear), one
has to understand and classify sounds from many different transmitters
(speakers). This is because, when different individuals enunciate sounds which
are perceived to be the same, the actual acoustical waveforms are different,
the corresponding spectra are different, and the location and separation of
the formants are different. However, since they are perceived to be the same,
the enunciations must have a commonalty that the ear extracts in recognizing
the similarity of the sounds. We call this the speech-hearing connection and
the understanding of this issue the speech-hearing problem.

In 1940, Stevens and Volkmann established the fundamental psychoacoustic
response of the ear, known as the MEL scale [3]. It relates perceived
frequency, $f_{mel}$, and the actual physical frequency, $f$, of the sound
wave. The MEL scale is solely based on psychoacoustic hearing experiments. We
have performed _speech_ experiments aimed at showing the common attributes of
speech signals of perceptually similar sounds. Our experiments establish a
“speech scale”, which is based _solely_ on speech production and has led us to
the following view of the speech-hearing connection. The spectra of sounds
made by different individuals and perceived to be phonologically identical can
be transformed into each other by a universal frequency-warping function. The
transformation results in identical spectra, except for a speaker dependent
translation factor. We call the universal frequency-warping function the
_speech-scale_. We show that the experimentally obtained “speech scale” is
similar to the MEL scale and, therefore, we argue that we have experimentally
linked speech and hearing. We emphasize that our speech scale was obtained
solely from speech experiments and in no way used any results of the
psychoacoustic MEL scale or any other hearing data.

The organization of the paper is as follows. In the next section we give an
simplified overview of our approach and the basic concepts we have used to
establish a ”speech scale”. In section 3 we describe the mathematical issues
and the details of the method used to estimate speech-scale. In Section 4 we
discuss the similarity between speech-scale and mel-scale. Discussion and
conclusions are made in Section 5.

##  2 An overview of the approach

We first explain the basis and motivation of our approach with a simple
idealized example. Suppose, hypothetically, that we have the utterances of
four persons enunciating a particular vowel, say /aa/. We may say that the
vowel is the same in each instance if critical listeners agree that they hear
the same sound for each of the utterances. Now suppose we take the spectra,
that is Fourier transform, of each person’s utterances and plot them as in
Fig. 1, where for simplicity, we only plot the formants (resonant
frequencies). For the case of illustration here we have generated these
formants for /aa/ by using the two-tube model for speech production [9]. 111In
a two-tube model $L_{1},A_{1}$ represents the length and area of the
“pharyngeal” cavity or the part starting from the vocal cords at the back of
the mouth and $L_{2},A_{2}$ represents the length and area of the “oral”
cavity, the part ending at the lips or the front part of mouth. The ratio of
the pharyngeal cavity to the oral-cavity lengths, that is
$\frac{L_{1}}{L_{2}},$ is largest for men, intermediate for women and smallest
for children. Usually only ${L_{2}}$ changes significantly from speaker to
speaker, while ${L_{1}}$ is similar among different speakers. Therefore, in
our example, we have maintained $L_{1}$ and the ratio of areas constant, while
changing the lengths of $L_{2}$. The lengths of ${L_{2}}$ have been
appropriately varied so that the corresponding formants correspond to that of
different speakers.

Now, as can be seen in Fig. 1, the spectra, that is, the formant frequency
locations are different, but, since the utterances are perceived to be the
same, they must have something in common. What is this communality? In other
words, what do these spectra have in common? We emphasize that the spectra are
not merely translated versions of each other as that would make the issue
trivial. Indeed if they were translated versions of each other that could be
readily determined in a variety of ways. If they are not translated versions
of each other, then one may ask whether there exists a transformation of each
spectra that would indeed make them identical or possibly translated versions
of each other. Such a transformation would be non-linear. Such transformations
are often called warping functions because one can think of a non-linear
transformation as a warping of the axis. What is important here is to
emphasize that, if there exists such a transformation, then it must be
universal to be of any significance. By universal we mean that the _same_
transformation is used for each speaker and it must not be adjusted for
different speakers. For illustration purposes we now do precisely that for the
above formant example and take each spectrum and warp the frequency-axis by a
function which we call the “universal warping function”. We denote the
frequency-warping function as $\nu=g(f)$, and for this illustrative example we
have used the function $\nu=0.9\log(f)+0.6(\log(f))^{2}$. The result of
applying this function to each of the spectra is illustrated in Fig. 2. We
again note that we use the term “universal warping function” to emphasize the
fact that the _spectra_ are transformed by the _same_ warping function
irrespective of the speaker. In the next section, we will discuss how one
arrives at such warping functions for idealized cases and more importantly in
Section 4 we will describe how one can empirically obtain such a function for
real speech. We also emphasize here that whether such a universal function
exists or not is an empirical question and can only be ascertained by the real
data.

Now examine Fig. 2. It can be seen that:

a) the transformed spectra are merely shifted versions of one and another and
hence we have exposed the commonalty of the original spectra of Fig. 1. To
emphasize this, we align the spectra with the first line, as illustrated in
Fig. 3. Now one clearly sees that indeed they are identical. (We note that in
general there are many ways to determine whether spectra are shifted versions
of each other, the most fundamental way is to note that the absolute value of
the Fourier transform of a function and a translated version of a function are
equal.)

b) the shift is different for each transformed spectrum, and, hence, the shift
is speaker dependent.

We again emphasize that although Fig.1 looks similar to Fig.2, it can be
easily verified that simple shifting (or scaling) will not align the spectra
of Fig.1. It is only after the application of the “universal warping function”
that the spectra appear as shifted versions of each other. Of course, for real
speech, we do not know whether such a “universal-warping function” exists,
and, if it does exists, we do not know the form of the universal warping
function. The purpose of this article is to describe experimental evidence
indicating that, indeed, such a universal warping function exists and to
develop the numerical procedures to obtain it from real speech.

Figure 1: The figure shows the formants from four different hypothetical
speakers. The horizontal axis is real frequency, $f$ measured in Hertz. Figure
2: Each of the spectra in Fig. 1 is transformed according to the following
function $\nu=0.9\log(f)+0.6(\log(f))^{2}$ and plotted respectively. The
horizontal axis now is $\nu$. In the new domain, $\nu$, the spectra are
identical except for a translation factor. See next figure. Figure 3: This
plot confirms that after aligning the formants they are indeed identical.

##  3 Mathematical and Experimental issues

Our basic idea is that the spectra of sounds made by different individuals and
perceived to be the same can be transformed into each other by a universal
warping function and that the transformation results in identical warped
spectra except for a translation factor. We denote the transformation from
physical frequency, $f$, to an alternate domain $\nu,$ as the _speech-scale_ ,
and write

$\displaystyle\nu$ $\displaystyle=g(f)\hskip 72.26999pt(\text{speech scale})$
(1) $\displaystyle f$ $\displaystyle=h(\nu)\hskip 72.26999pt(\text{inverse
speech scale})$ (2)

Suppose we have two enunciations by different speakers, $A$ and $B$, with
corresponding spectra $P_{A}(f)$ and $P_{B}(f)$. We warp each spectrum with
the universal function, $\nu=g(f)$, to obtain two new spectra $S_{A}(\nu)$ and
$S_{B}(\nu)$ defined by the following procedure:

$\displaystyle S_{A}(\nu)$ $\displaystyle=P_{A}(h(\nu))$ (3) $\displaystyle
S_{B}(\nu)$ $\displaystyle=P_{B}(h(\nu))$ (4)

then we claim that, for enunciations which are perceived to be the same, the
warped spectra are related by

$S_{A}(\nu)=S_{B}(\nu+\alpha_{AB})$ (5)

where $\alpha_{AB}$ is a constant. In general, we use the notation that
spectra in physical frequency are represented as $P(f)$ and spectra in the
warped domain are represented as $S(\nu)$. If the claim is correct, the
question arises how can one show it and how can one determine the universal
transformation that we call the speech scale. Before describing our approach,
we must emphasize two things. First, even though we have used two speakers in
the above formulation, it is crucial to understand that this must be true for
_all_ speakers. That is, it is crucial that the _same_ universal-warping
function $\nu=g(f)$ is applicable to all speakers and all phonemes or speech
sounds. Secondly, whether this view and mathematical relationship are true, or
not, can only be determined experimentally using actual speech data.

We have been motivated by Fant’s seminal work [1, 2] where he considers
relationship between formants of different speakers. First, we discuss a very
simplified model where we model the vocal tract as a uniform tube. For a
uniform tube the spectra (or formant frequencies) are inversely proportional
to the length of the tube. Hence for different speakers having different tube-
lengths, the spectra are related by

$P_{A}(f)=P_{B}(\kappa_{AB}f)$ (6)

where $\kappa_{AB}$ is the ratio of the lengths of the tubes for the two
speakers. Recalling that multiplication becomes addition after the application
of log operation, the universal warping function for this simplified model is
$\lambda=\log(f)$. This can be easily verified since

$S_{A}(\lambda)=P_{A}(f=e^{\lambda})=P_{B}(\kappa_{AB}e^{\lambda})=P_{B}(e^{\lambda+\ln\kappa_{AB}})=S_{B}(\lambda+\ln\kappa_{AB})$
(7)

Fant [1, 2] pointed out that Eq. 6 is a very crude model and has observed that
the scaling factor $\kappa_{AB}$ changes for different formant frequencies for
any two speakers. Motivated by Fant’s observation we model the relationship
between spectra of two speakers by the following piece-wise linear model

$P_{A}(f)=P_{B}(\gamma_{AB}^{\beta_{l}}f)\;\;\;\;\mathrm{where}\;f\;\in\;[L_{l},U_{l}]$
(8)

where $L_{l},U_{l}$ are lower and higher frequencies of a frequency region.
The parameter $\gamma_{AB}$ depends only on the pair of speakers and not on
frequency. $\beta_{l}$ is a constant that is _independent_ of speakers and
depends on only the frequency region $[L_{l},U_{l}]$. We have estimated the
frequency-dependent parameter $\beta_{l}$ by doing least-squares fit on
_actual_ speech data. In all of the experiments in this paper, we have used
the Hillenbrand speech (vowel) database[5]. The piece-wise linear
approximation to universal-warping function $\nu$, is then given by

$\nu=\frac{1}{\beta_{l}}\ln(f)\;\mathrm{for}\;f\;\in\;[L_{l},U_{l}]$ (9)

The use of this universal-warping function $\nu$ helps separate the speaker-
dependent term as a translation factor and show the commonality in the speech
spectra from different speakers. We refer to this universal-warping function
as the _speech-scale_

##  4 Relationship between MEL scale and the speech scale.

Stevens and Volkmann [3] experimentally obtained a non-linear mapping between
perceived and physical frequency of a tone and referred to it as the mel-
scale. We have obtained a speech scale which has been purely estimated only
from speech data with the primary purpose to show the commonality $between$
speakers for the same utterance. In Fig. 4 we plot both the speech scale and
Stevens and Volkmann [3] data, that is, the MEL scale. (In addition we have
also plotted the von Békésy basilar membrane data but this issue will be
discussed in the conclusion.) From the figure, it can be seen that the two
curves are remarkably similar. We reiterate, that one has obtained from
psychoacoustic experiments and the other only from speech data. The fact that
the two curves are remarkably similar experimental shows the connection
between speech and hearing.

Figure 4: The figure shows the Speech-Scale, Stevens and Volkmann data and
Békésy’s data. The Speech-scale has been obtained empirically from actual
speech data. The Steven and Volkmann data have been obtained from psycho-
physiological study, while the Békésy data has been obtained from experiments
on the basilar membrane. The fact that all the three curves are similar shows
a strong connection between speech and hearing.

To further show the similarities between the the speech scale and MEL scale,
we point out that the widely accepted closed-form approximations to mel-scale
have the functional form

$\eta=a\log_{10}\left(1+\frac{f}{b}\right)$ (10)

where $f$ is in Hz and $\eta$ is in _Mels_. Here we have used the mel-formula
which is most commonly used in speech and defined with $a=2595$ and $b=700$,
i.e.

$\eta_{\text{MEL}}=2595\log_{10}\left(1+\frac{f}{700}\right)$ (11)

We now take our speech scale and fit it to Eq. (10 (with fitting accuracy of
$99.5\%$) to obtain continuous warping function $\Omega(f)$. The warping
function is given by

$\eta_{\text{speech}}=2478.24\log\left(1+\frac{f}{641.94}\right).$ (12)

Thus we see that indeed $\eta_{\text{speech}}$ is approximately equal to
$\eta_{\text{MEL}}.$

As further evidence of the ability of the speech scale to extract the
commonality among different speakers enunciating the same sound, we apply it
to study the formants of different speakers for the vowel /AW/. In Fig. 4a, we
show the natural frequencies of three formants of /AW/ for each of the 139
different speakers in the Hillenbrand database. For each speaker, the three
formant frequencies have been marked by different symbols along the $y$ axis.
As seen, there is considerable variability in the formant frequencies among
different speakers. We now apply the speech scale to the data of Fig. 4a and
translate each of the resulting spectra to obtain the best overall alignment
[12]. The results of this process are shown Fig. 4b. As seen, after the
application of the _speech scale_ , the formants are aligned remarkably well
showing the commonality among speakers for the vowel /AW/. In a future paper
we will present results for all the vowels.

##  5 Discussion and Conclusion

As we have noted, speech and hearing must be a highly coupled. If one could
demonstrate a relationship between the results of hearing experiments and the
results of speech experiments, a fundamental experimental link would be
established. We believe that we have done that. Using real speech data and
only speech we have experimentally obtained a “speech scale”, which is a
warping of the speech spectrum that exposes the communality of sounds which
are perceived to be the same. Our experiments are analogous to the basic
psychoacoustic scale obtained by Stevens and Volkmann [3] which was obtained
solely on hearing data. The fact that they are almost the same and derived
totally independently, one from speech and the other from hearing shows the
coupling or communality between speech and hearing and experimentally
establishes the speech-hearing connection. In addition there is another scale
that is of relevance which we now discuss. It is the frequency to physical
place mapping on the basilar membrane which is based entirely on physical
aspects of the basilar membrane. This place map, representing the behavior of
the basilar membrane as a function of frequency was discovered by von Békésy
[7] in his studies, for which he won a Nobel prize. In his experiments the
stapes was vibrated with a constant amplitude sinusoid, and the frequency
response at various points along the basilar membrane was examined under the
microscope. The mapping relating the frequency of the stimulation to the
position of maximum response on the basilar membrane was thus established. Is
there a relationship between the Mel scale and the place map of von Békésy?
This question was addressed by von Békésy, himself, who noticed the
fundamental significance of the fact that they are similar and noted “If we
plot the distance from the stapes of the point of maximum displacement versus
frequency, the curve obtained has the general form of the curves inferred from
psycho-physiological data”.

We now compare the speech scale, the Mel scale and the place map of von
Békésy. The results are shown in Fig. 4. The fact that the three, independent,
experimentally derived scales, namely the place map, the MEL scale and the
speech scale, are so similar is an indication that they may be the basic
fundamental experimental linking of the speech-hearing connection.

## References

  * [1] C. G. M. Fant, “Acoustic description and classification of phonetic un its,” in _Ericcson Technics_ , no. 1, 1959, reprinted in Speech So end and Features, MIT Press, Cambridge, 1973.
  * [2] G. Fant, “A Non-Uniform Vowel Normalization,” _STL-QPSR_ , no. 2-3, pp. 1–19, 1975.
  * [3] S. S. Stevens and J. Volkman, “The relation of pitch to frequency,” _American Journal of Psychology_ , vol. 53, p. 329, 1940.
  * [4] G. E. Peterson and H. L. Barney,  “Control Methods Used in a Study of the Vowels,”  J. Acoust. Soc. America, vol. 24, pp. 175–184, M arch 1952.
  * [5] J. Hillenbrand, L. Getty, M. Clark, and K. Wheeler,  “Acoustic Characteristics of American English Vowels,”  J. Acoust. Soc. Am., vol. 97, pp. 3099–3111, May 1995.
  * [6] S. Umesh, L. Cohen, and D. Nelson,  “Fitting the Mel Scale”, in _Proc. IEEE ICASSP ’99_ , pages217–220, (1999).
  * [7] G. v. Békésy and W. A. Rosenblith in _Handbook of Experimental Psychology_ (S. S. Stevens, ed.), (New York), pp. 985–1039, John Wiley, 1951.
  * [8] Georg von Békésy, _Experiments in Hearing_ , McGraw-Hill, New York, 1960
  * [9] J. L. Flanagan, _Speech Analysis and Perception_ , Springer-Verlag, New York, 1972
  * [10] L. Cohen, “The Scale Representation,” _IEEE Trans. Signal Proc._ , pp. 3275–3292, Dec. 1993.
  * [11] S. Umesh, L. Cohen, N. Marinovic, and D. Nelson, “Scale Transform In Speech Analysis,” _IEEE Transactions on Speech and Audio Processing_ , January 1999.
  * [12] D.J. Nelson, D.C. Smith, S. Umesh and L. Cohen, “Estimating Speaker Scale Factors From Vowels,” _Proc. SPIE Wavelet Conf._ , pp. 794-800, San Diego, 4-8 Aug., 2003.