# High-Resolution Depth Estimation for 360∘ Panoramas through
Perspective and Panoramic Depth Images Registration

Chi-Han Peng
National Yang Ming Chiao Tung University
<EMAIL_ADDRESS>Jiayao Zhang
ByteDance
<EMAIL_ADDRESS>
###### Abstract

We propose a novel approach to compute high-resolution (2048x1024 and higher)
depths for panoramas that is significantly faster and qualitatively and
qualitatively more accurate than the current state-of-the-art method [23]. As
traditional neural network-based methods have limitations in the output image
sizes (up to 1024x512) due to GPU memory constraints, both [23] and our method
rely on stitching multiple perspective disparity or depth images to come out a
unified panoramic depth map. However, to achieve globally consistent
stitching, [23] relied on solving extensive disparity map alignment and
Poisson-based blending problems, leading to high computation time. Instead, we
propose to use an existing panoramic depth map (computed in real-time by any
panorama-based method) as the common target for the individual perspective
depth maps to register to. This key idea made producing globally consistent
stitching results from a straightforward task. Our experiments show that our
method generates qualitatively better results than existing panorama-based
methods, and further outperforms them quantitatively on datasets unseen by
these methods.

##  1 Introduction

Panoramas with depth information are very useful for 3D computer vision tasks
such as novel view synthesis [12, 4, 40], 3D scene understanding (_e.g_. room
layout estimation [30]), omnidirectional SLAM [36], and virtual reality (VR)
applications [24]. Traditional monocular depth estimation methods built for
perspective images cannot directly work on panoramas, as a panorama cannot be
converted to a perspective image since the latter can not have field-of-view
(FOV) angles exceeding 180 degrees. Therefore, many depth estimation methods
that are specially built for panoramas have been proposed [47, 33, 31, 19, 11,
14, 13].

Most of these methods are based on deep neural networks trained on panorama
datasets with ground-truth depths such as Matterport3D [6] and Stanford2D3D
[2]. The ground-truth depths in such datasets have been calibrated such that
they are globally consistent (i.e., having the same scale and shift) across
all the viewing directions from the camera position. As a result, these
methods estimate consistent depth maps for the whole panoramas.

However, we observe two main shortcomings of these methods. First, due to the
limitation of GPU memory during the training of the CNN architecture being
used, they can only output depth images with resolutions as high as 1024x512.
This is far below the native resolutions of RGB panoramas (most modern
commodity 360∘ cameras can shoot panoramas with resolutions exceeding
4096x2048) and is insufficient for VR applications. For example, a 90∘-by-90∘
perspective view rendered out of a 1024x512 panorama would have a resolution
of only 256x256. Second, we observe that depth maps estimated by panorama-
based methods often lack the same level of detail as the results generated by
perspective methods. This may be because they were trained on panoramic RGB-D
datasets, which are much smaller and less diverse than the traditional
perspective RGB-D datasets that perspective methods were trained on. See
Figure 1 for examples.

To address the shortcomings, the authors of [23] proposed a stitching-based
approach as follows. First, the panorama is partitioned into several
perspective views. They used a 20-view partition mimicking the 20 triangle
faces of an icosahedron (_i.e_. ”tangent images” in [8]). Each view is fed to
a modern perspective depth estimation method to get a perspective disparity
map (they used MiDaS [22]). Each perspective map is projected back to a common
equirectangular domain. A fully panoramic depth map can then be computed by
stitching together the projected perspective disparity maps and then converted
to depth values. The main challenge of such stitching-based methods is that
the perspective maps tend to have different scales and shifts of values
because the perspective depth estimation method was run on different fields of
view of the panorama. To fix this inconsistency problem, they relied on
extensive disparity map alignment and Poisson blending strategies, both become
very expensive for high-resolution images.

Figure 1:  Comparisons of depth estimations by panorama-based methods
(SliceNet [19] and UniFuse [11]), and perspective method (LeRes [43]) of the
same region in a panorama. We find that LeReS produced qualitatively better
results with better levels of details than any panorama-based methods by a
large margin.

Instead, we propose a simple solution to the global consistency problem:
leveraging a panoramic depth map as the common target for the perspective
depth maps to register to. Note that such ”reference” panoramic depth maps can
be generated in real-time by an existing panorama-based method, which is known
to produce globally consistent values. Our stitching pipeline is as follows.

We begin with partitioning the panorama into several rectangular regions. For
each region, we generate a perspective depth map using LeReS [43]. Next, we
generate a full panoramic depth map using a recent panorama-based method such
as SliceNet [19] or Unifuse [11]. For each perspective map, we register a low
degree (_e.g_. quadratic or cubic) transform function by fitting a set of
sampled pixels in the perspective map to the corresponding pixels in the
panoramic map in a least-squared sense. Being an optimization problem with
only a few variables, the computational cost of this step is very low.

The previous step minimized the differences of scales and shifts between the
perspective maps but does not completely erase the visible seams in between.
Therefore, we further blend the registered perspective maps by a Poisson-based
approach similar to [18]. We opt for computing the final panoramic depth map
as the one that fits the Laplacians of the perspective depth maps in a least-
squared sense. As the optimization problem is ill-posed (being shift
invariant), we regularize it by the $L2$ distances to the reference panoramic
depth map with a small weight. Note that our blending approach is similar to
the disparity map blending step in [23] with the main difference that our
approach no longer needs the spatial-varying weights (_e.g_. ”radial” or
”frustum”-shaped) in [23] to enforce smoother transitions between the
perspective maps.

Through experiments, we show that our method produced qualitatively and
qualitatively better results than [23] at a significantly faster speed. Our
contributions are summarized as follows:

  * •

We propose a simple, effective, and cost-efficient solution to the global
consistency problem of stitching-based methods for high-resolution panoramic
depth map generations, namely the registration-based approach.

  * •

Benefited from the effectiveness of the aforementioned step, we find that a
straightforward Poisson blending-based approach is sufficient to blend the
registered perspective depth maps and erases the visible seams in between,
without the needs for complex spatial-varying weights as was done in [23].

##  2 Related Work

###  2.1 Panorama-based 3D Modeling and Datasets

Leveraging the increasing popularity of 360∘ cameras, 3D modeling of indoor
scenes based on panoramic image inputs have become a popular research field in
recent years. Key tasks include depth estimation [47, 33, 19, 11], room layout
estimation [48, 42, 29, 20, 44, 34], object detection and segmentation [37,
31], and more generally 3D reconstruction tasks such as registration of
multiple panoramas [41, 35].

Panoramic image datasets with depth information are summarized as follows.
Matterport3D [6] and Standford2D3D [3] provide real-world panoramas of diverse
indoor scenes with ground-truth depths. Note that although the depths of one
panorama are captured through multiple 3D scans (the 3D scanners have limited
field-of-views and relied on motors to take multiple scans in a controlled
manner), they have been calibrated so that the depth values are consistent
across all viewing directions - we don’t see any seams between the different
views in the depth maps. SunCG [27], Structure3D [45], and Replica360 [28] are
synthetic datasets of indoor scenes. They provide photo-realistically rendered
indoor RGB-D images and annotations of 3D structures including room layouts.
3D60 [46] is a collective dataset consisting of real and synthetic sources.

###  2.2 Monocular perspective depth estimation

Monocular (_i.e_. needed just a single image as input instead of stereo ones)
perspective depth estimation is a very active research topic in which modern
methods can now reliably predict generally accurate depths from arbitrary
images in the wild (_i.e_. in unseen datasets). We list methods proposed in
recent years with competitive performances: Xian et al. 2018 [38], MegaDepth
[15], MiDaS v2 [21] and v3 [22], SGR [39], Huynh et al. 2021 [9], AdaBins [5],
and LeReS [43]. Note that some of the methods predict disparity values (e.g.,
MiDaS), which are the inverse of depths. Authors of [17] proposed a derivative
method that calls an external perspective depth estimation method repeatedly
on carefully chosen subsets of the input image, and blend the partial depth
estimations to form the final output. The downside is that it is much slower
than standard methods.

###  2.3 Panoramic Depth Estimation

Tateno et al. [32] proposed a deformable convolutional filter that maps
perspective views to the equirectangular domain in a distortion-aware manner
and used it to perform panoramic depth predictions via numerous, densely
sampled perspective depth-prediction calls that could be trained on both
panoramic and perspective datasets for perspective depth predictions. In [7]
an elaborate discussion of such filters are discussed. OmniDepth [47] was an
early method to estimate depths specifically for panoramas in an end-to-end
manner. They reported that applying monocular perspective methods directly on
equirectangular images led to inferior results. To solve the problem, they
trained a customized encoder-decoder network (called ”UResNet” in their paper)
directly on panoramas with ground-truth depths from synthetic sources. Later,
BiFuse [33] found that injecting perspective views of the panoramas (in
cubemap formats) into the training process increased the performance
considerably. Inspired by the gravity-aligned nature of indoor scenes,
SliceNet [19] proposed a novel network architecture that encodes the input
panorama into feature vectors that each correspond to a vertical slice of the
panorama. HoHoNet [31] leveraged a similar idea in their method that predicts
room layouts, depths, and semantic labels simultaneously. Unifuse [11]
iterates on the ”two-branch” (one feeds the equirectangular projection and
another feeds the cubemap projection of the input panorama) network design of
BiFuse and proposed a simpler version. PanoDepth [14] uses stereo matching
ideas to improve panoramic depth estimations. OmniFusion [13] is a panoramic
method via aligning and blending tangent depth maps using transformers. A
trend of network design in recent methods is to leverage perspective views
sampled on panoramic images while using various strategies to eliminate the
distortions and discontinuity artifacts caused in the process [26, 13, 25].
Finally, 360MonoDepth [23] proposed a novel derivative approach to generate
high resolution panoramic depth maps by stitching perspective depth maps
generated by external methods.

##  3 Method

Figure 2:  Overview of our method. (a) We first partition the panorama into
several perspective views and feed each perspective image to a monocular depth
estimation method such as LeReS [43]. Note that most panoramic RGB-D datasets
don’t have depths in the top and bottom so our method skips those regions. (b)
The predicted perspective depth maps are projected to the equirectangular
domain and are stitched together to form a panoramic depth map. (c) For each
projected perspective depth map, we solve a low-degree function in a least-
squared sense that transforms the pixels in the perspective map to the
corresponding pixels in a shared ”reference” panoramic depth map, which is
generated in real-time by a panorama-based method such as SliceNet [19] or
UniFuse [11]. (d) We generate a panoramic Laplacian map of the registered
depth maps. (e) Finally, we optimize for a new panoramic depth map that fits
the Laplacians of the registered depth maps with a small regularization term
using L2 distances to the reference panoramic depth map.

We use notations as follows. We assume a 3D world space with right-hand rule
in which the +z axis is the ”straight-up” direction and the +x axis is chosen
arbitrarily. A viewing direction is equivalent to a point on the unit sphere
centered at the origin, which can be denoted by a spherical coordinate,
$(\phi,\theta)$, in which $0^{\circ}\leq\phi<360^{\circ}$ is the azimuth angle
from the +x axis on the x-y plane in counter-clockwise order and
$0^{\circ}\leq\theta\leq 180^{\circ}$ is the zenith angle to the +z axis. An
equirectangular projection maps a viewing direction to a 2D point trivially as
$(\phi,\theta)$. The equirectangular domain denotes the rectangle
$(0^{\circ},0^{\circ})$, $(360^{\circ},0^{\circ})$,
$(360^{\circ},180^{\circ})$, $(0^{\circ},180^{\circ})$ in the 2D plane. A
perspective view is a perspective projection defined by a viewing direction
and field-of-view angles in the horizontal and vertical directions ($FOVx$ and
$FOVy$ angles in short) in which the camera is at the origin and +z axis is
the ”up” direction. An perspective-to-equirectangular (P2E) projection
projects a perspective view to a region in the equirectangular domain. Note
that the projected region would not be a rectangle anymore (see Figure 3). A
summary of our method is given in Figure 2 and its description. We explain the
key steps in the following sub-sections.

###  3.1 Equirectangular-to-Perspective Partitions

Figure 3:  (a) to (d) show four cases of a perspective view with
$FOVx=80^{\circ}$ and $FOVy=60^{\circ}$ in 3D space where in (a) the looking
direction, $M$, ’s spherical coordinate is $(0,90^{\circ})$ (on the equator),
in (b) it is $(0,70^{\circ})$ (slightly tilted up), in (c) it is
$(0,60^{\circ})$ (tilted up and the bottom edge of the image plane lies on the
x-y plane), and in (d) it is $(0,50^{\circ})$ (further tilted up). $c0$ to
$c3$ denote the four corners of the image plane. $Left$ and $Up$ denote the
two axes of the image plane in 3D. In each figure’s upper-right corner, we
show the P2E-projected region of the perspective view in the equirectangular
domain. We show the partition that each view is designated to cover in blue.

Our method produces depths for a rectangular subset of the equirectangular
domain that spans azimuth from $0^{\circ}$ to $360^{\circ}$ and zenith from
$25^{\circ}$ to $155^{\circ}$, where all panorama datasets have depth values.
We denote this region as the ”target domain”. We partition the target domain
into rectangular sub-regions along horizontal and vertical lines, and then use
a perspective depth estimation method such as LeReS [43] to estimate the depth
values for each sub-region. We denote a rectangular region that covers azimuth
from $\phi_{0}^{\circ}$ to $\phi_{1}^{\circ}$ and zenith from
$\theta_{0}^{\circ}$ to $\theta_{1}^{\circ}$ as a ”partition”
$P_{\phi_{0},\phi_{1},\theta_{0},\theta_{1}}$. Recall that we assume all
perspective views have the camera position at origin and the ”up” direction at
+z axis. Even so, there are still infinitely many ways to construct a
perspective view that sufficiently covers a partition. Therefore, we describe
an algorithm to find one such perspective view that would bound the region
tightly as follows.

First we note that the shape of a perspective view’s P2E-projected domain is
invariant to rotations along the z-axis, so we limit our analysis to the case
of azimuth equals zero. Also, there are reflective symmetries vertically
w.r.t. the equator so we need only to analysis the ”tilted-up” cases (zenith
not greater than $90^{\circ}$) and the corresponding tilted-down cases can be
derived trivially. Now, as shown in Figure 3, we observe all possible cases of
a perspective view’s P2E-projected domains and rectangles centered at the
viewing direction (point $M$ in the equirectangular domain) bounded within.
These rectangles are the rectangular partitions that a perspective view may
cover. We find that in the equirectangular domain, the upper edges of the
P2E-projected domain are always closer or equal to the viewing direction ($M$)
than the lower edges along the y-axis. Therefore, we choose the rectangle with
left and right edges horizontally aligned to the bottom two corners (same
azimuths) of the P2E-projected domain that goes as up as possible until
touching the upper edge.

The viewing direction $M$’s 3D coordinate is $(cos(\Theta),0,sin(\Theta))$,
$\Theta$ is the vertical tilted-up angle from the equator. The two axis of the
image plane in 3D are $Left=(0,1,0)$ and $Up=(-sin(\Theta),0,cos(\Theta))$,
respectively. To derive the lower-left corner ($c0$)’s azimuth, we first find
$c0$’s 3D position as:

$M+tan(FOVx/2)*Left-tan(FOVy/2)*Up,$ (1)

$FOVx$ and $FOVy$ are the field-of-view angles of the perspective view. $c0$’s
azimuth can then be computed and denoted as $c0_{\phi}$. Note that the 3D
positions of the other three corners (denoted as $c1$, $c2$, and $c3$, in
counterclockwise order) can be derived similarly. We can now uniquely locate
the point $c$ on the upper edge with azimuth equals to $c0_{\phi}$.

Finally, given a rectangular partition that the perspective view is designated
to cover, we derive the corresponding $c_{\phi}$ and $c_{\theta}$. $c$’s 3D
position is now fixed. We then solve $FOVy$ such that the upper edge would
intersect $c$. Next, we solve $FOVx$ using the formula for $c0$’s azimuth.
$\blacksquare$

For each perspective view of the panorama, the corresponding monocular depth
map is computed by a perspective method such as LeReS. By default, we
partition the target domains into 3 rows and 5 columns (15 total) of
rectangular sub-regions. The rows are divided along zeniths $25^{\circ}$,
$60^{\circ}$, $120^{\circ}$, and $155^{\circ}$. The columns are divided along
azimuths $0^{\circ}$, $72^{\circ}$, $144^{\circ}$, $216^{\circ}$, and
$288^{\circ}$. Comparisons of different ways to partition the target domain
are in Section 4.3.

###  3.2 Perspective-to-Equirectangular Registration

As shown in Figure 2 (b), each partition of the target domain is filled with
depth values from the corresponding P2E-projected perspective depth map, using
looking directions and $FOVx$ and $FOVy$ chosen according to the algorithm
described in Section 3.1. However, each filled partition tends to have
different scales and shifts of values. With the common scale and shift being
unknown, as has been shown in [23], aligning them using optimization could be
an expensive process, especially for high-resolution images.

Instead, we propose the idea of using a ”reference” panoramic depth map, in
which the depth values are known to be consistent across the whole target
domain, as the common target for the perspective depth maps to register to.
The reference panoramic depth map can be generated by one of the existing
panorama-based methods such as SliceNet, or UniFuse. For each partition
$P_{\phi_{0},\phi_{1},\theta_{0},\theta_{1}}$, the registration is summarized
as solving a linear least-squares optimization problem as follows:

$\displaystyle\underset{a,b,c,d}{\text{argmin}}$
$\displaystyle\sum_{i=0}^{N-1}(a(x_{i})^{3}+b(x_{i})^{2}+cx_{i}+x_{i}-X_{i})^{2}$
(2)

where $x_{i}$ and $X_{i}$ denote the depth values of the $i$-th sampled pixel
in the partition and the corresponding pixel in the reference panoramic depth
map, respectively. $N$ denotes the number of sampled pixels in the partition.
We sample pixels at every 1 degree horizontally and vertically. Here, we show
an algorithm design with a cubic function. We also experimented with other
choices (_e.g_. quadratic and linear) and report the finding in Section 4.3.
We use Google Ceres Solver [1] to solve the optimization problem. We then use
the solved registration function to transform every pixel values in a
partition (_i.e_. not just the sampled pixels).

###  3.3 Laplacian-based Poisson Blending

The aforementioned registration process aligned the scales and shifts of the
depth values in every partition (_e.g_. comparing Figure 2 (b) and (c)) but
does not completely remove the visible seams in between. To solve this issue,
we blend the partitions using a Poisson-based approach. Similar to the
traditional gradient-based Poisson blending algorithms (which was used in
[23]), we opt for optimizing for a panoramic depth map that fits the
Laplacians of the registered perspective maps directly.

We use the standard 3x3 discrete Laplacian operator (shown in Equation 4). For
each partition, we also sample ”padded” depth values outside its region in the
equirectangular domain by slightly expanding the equirectangular regions of
each partition and updating the FOV angles respectively. By default, we expand
the region of each partition 5 pixels horizontally and 2 pixels vertically
(for both 2K and 4K cases). This means that there would be small overlaps of
Laplacian values between the partitions and we take the average of Laplacians
at the overlapping pixels. This padding step serves the purpose of slightly
smoothing out the Laplacians between adjacent partitions. The Laplacian-based
blending is expressed as an optimization problem as follows:

$\displaystyle\underset{x_{i,j},\>0\leq i<W,\>0\leq j<H}{\text{argmin}}$
$\displaystyle\sum_{j=1}^{H-2}\sum_{i=1}^{W-2}((l_{i,j}-L_{i,j})^{2}$ (3)
$\displaystyle+\gamma\,(x_{i,j}-X_{i,j})^{2})$ subject to $\displaystyle
x_{i,j}\geq 0,\forall i,j$

where $x_{i,j}$ is the depth value at pixel coordinate $(i,j)$ in the
panoramic depth map to be solved. $W$ and $H$ are the width and height of the
panoramic depth map. $l_{i,j}$ is the Laplacian value at pixel $(i,j)$
computed by the standard 3x3 discrete Laplacian operator:

$\displaystyle l_{i,j}=4x_{i,j}-x_{i-1,j}-x_{i+1,j}-x_{i,j-1}-x_{i,j+1},$ (4)
$\displaystyle\forall\>1\leq i\leq W-2,1\leq j\leq H-2.$

and $L_{i,j}$ are the ”target” Laplacian values computed by the same formula
in the registered depth values at each partitions (one example is shown in
Figure 2 (d)). $X_{i,j}$ is the depth value at pixel coordinate $(i,j)$ in the
reference panoramic map. In short, the objective function has two terms - a
Laplacian term and a regularization term using L2 distances to the reference
panoramic depth map. We set $\gamma$, the weight for the regularization term,
to 1e-4.

The optimization problem is solved using the standard Jacobi iterative method
with the depth values of the reference panoramic depth map as the initial
guess. To speed up solving, we solve the problem in a multi-scale manner. That
is, we first solve the problem in a reduced-resolution version of the image
buffer, pass the solved values to a finer-resolution buffer, solve again, and
so on until the problem is solved in the original resolution. For the
2048x1024 case, we solve in 3 levels (_i.e_. 512x256, 1024x512, and
2048x1024). The iterations for the finest level are chosen to be 50 as we
observed that the problem usually converged (with residuals lower than 0.1% of
initial values) during iteration 40 to 50. We then set iterations for the
subsequent levels to be 100 and 200. For the 4096x2048 case, we solve in 4
levels (50, 100, 150, and 200 iterations).

##  4 Results and Analysis

We tested our method and [23] on a computer with Intel i7-10700 CPU, 32GB ram,
and NVidia RTX 2070 GPU. We compare the timing statistics of our method versus
[23] in Table 1. In summary, our method is 3.05 times faster (2048x1024
outputs), and 1.94 times faster (4096x2048 outputs), than [23]. We draw each
perspective view in a 1024x989 resolution, which is enough to form 4K outputs
when stitched. Same as in prior methods (including OmniDepth, BiFuse,
SliceNet, and UniFuse), when comparing a computed depth map, $\omega$, to a
ground-truth depth map, $\Omega$, a ”median-scaling” step is first taken by
multiplying every depth value in $\omega$ by the median value of $\Omega$
divided by the median value of $\omega$. Note that for the Matterport dataset,
depth values are reported in meters. We choose HoHoNet [31], SliceNet [19],
and UniFuse [11], as the methods to generate the panoramic depth maps for our
method.

Res | Pers.D | Pano.D | Reg. | Blend | Total | [23]
---|---|---|---|---|---|---
2K | 4.27 | 0.122 | 0.22 | 10.89 | 15.50 | 47.3
4K | 4.27 | 0.122 | 0.23 | 34.27 | 38.89 | 75.3
Table 1: Timing comparisons. We list the times (in seconds) for our
perspective-view rendering and depth estimation (LeReS 15 times), panoramic
depth estimation (UniFuse), registration step (cubic), and blending step. We
solved for 100 panoramas and calculated the average. For [23], to reproduce
similar results as shown in their paper, we set the options to: Poisson
blending, 3 levels of disparity map alignments, and MiDaS V2. Note that [23]
also has a one-time ”pre-processing” step which took about 40 seconds.

###  4.1 Quantitative Comparisons

In Table 3, we quantitatively compare our method versus recent panorama-based
methods and [23] on the Matterport dataset testing split in 2K resolution
outputs and the synthetic Replica360 dataset in 2K and 4K resolutions provided
by [23]. Same as in [23], we up-sample 1K results generated by previous
methods to 2K by bilinear interpolation. Our method produced quantitatively
better results than [23] on all three datasets by large margins. Our method,
when paired with SliceNet or HoHoNet, scored nearly as good as the top
performers (SliceNet and HoHoNet) on the Matterport 2K dataset. But when
tested on the Replica360 2K and 4K datasets, in which the previous panorama-
based methods were not trained on, our method outperforms all existing
panorama-based methods. Also note that on Replica360 2K and 4K datasets, our
method improved the quantitative score of the panorama-based method that is
being used in every cases. We find [23] to perform competitively on the
Replica360 2K and 4K datasets (through accuracy metrics), but not on the
Matterport 2K dataset.

Method | RMSE | MAE | AbsRel |  RMSElog
---|---|---|---|---
Reg. only | +3.97% | +8.49% | +14.62% | +8.23%
Blend only | +8.32% | +12.23% | +26.69% | +56.13%
No padding | +0.02% | +0.00% | +0.01% | +0.13%
Avg. s/s | +6.79% | +11.26% | +24.18% | +47.20%
Smoothing | +6.36% | +9.92% | +17.49% | +28.01%
4-fold | +0.36% | +0.70% | +1.51% | +1.47%
3-fold | +0.05% | +0.41% | +0.08% | +0.35%
MiDaS | +0.26% | +1.06% | +1.75% | +2.89%
Linear | -0.52% | -0.77% | -1.53% | -1.60%
Quadratic | -0.51% | -0.75% | -0.88% | -0.77%
Table 2: Quantitative statistics of ablation studies and alternative design choices. We show relative ratios w.r.t. the case of using UniFuse as the panorama-based method on Matterport 2K dataset. ”Reg. only” means only performing the registration step. ”Blend only” means only performing the blending step. ”No padding” means skipping the padding in the blending step. ”Avg. s/s” means simply registering each partition to the average scale and shift. ”Smoothing” means replacing the blending step by smoothing depth values at partition boundaries. ”4-fold” means taking a partition of 3 rows by 4 columns (horizontally divided along azimuths $0^{\circ}$, $90^{\circ}$, $180^{\circ}$, and $270^{\circ}$), and ”3-fold” means taking a partition of 3 rows by 3 columns (along azimuths $0^{\circ}$, $120^{\circ}$, and $240^{\circ}$). ”MiDaS” means using MiDaS for perspective depth estimations. ”Linear” and ”Quadratic” means using alternative degree registration functions. | Error metric $\downarrow$ | Accuracy metric $\uparrow$
---|---|---
DS | Method | RMSE | MAE | AbsRel | RMSElog | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$
Matterport 2K | Bifuse [33] | 0.6350 | 0.3675 | 0.1367 | 0.0901 | 82.77% | 94.46% | 97.53%
HoHoNet [31] | 0.4707 | 0.2620 | 0.0967 | 0.0629 | 90.50% | 97.27% | 97.09%
SliceNet [19] | 0.4463 | 0.2153 | 0.0665 | 0.0513 | 95.17% | 98.07% | 99.54%
UniFuse [11] | 0.6040 | 0.3309 | 0.1110 | 0.0728 | 87.79% | 95.70% | 98.38%
360MonoDepth [23] | 0.7729 | 0.5106 | 0.2653 | 0.1253 | 60.38% | 85.55% | 94.70%
Our (HoHoNet) | 0.4791 | 0.2655 | 0.1004 | 0.0662 | 90.23% | 97.09% | 98.93%
Our (SliceNet) | 0.4949 | 0.2569 | 0.0883 | 0.0648 | 91.51% | 97.21% | 99.10%
| Our (UniFuse) | 0.6107 | 0.3333 | 0.1152 | 0.0766 | 87.08% | 95.36% | 98.20%
Replica360 2K | Bifuse [33] | 0.0555 | 0.0416 | 0.2150 | 0.1121 | 71.56% | 91.39% | 96.32.%
HoHoNet [31] | 0.0300 | 0.0193 | 0.1116 | 0.0671 | 90.31% | 95.90% | 98.11%
SliceNet [19] | 0.0403 | 0.0279 | 0.1590 | 0.0896 | 85.15% | 93.88% | 96.44%
UniFuse [11] | 0.0362 | 0.0248 | 0.1336 | 0.0774 | 86.87% | 95.94% | 97.72%
360MonoDepth [23] | 0.0706 | 0.0456 | 0.1813 | 0.0865 | 78.48% | 93.56% | 98.34%
Our (HoHoNet) | 0.0272 | 0.0182 | 0.1074 | 0.0643 | 90.98% | 96.07% | 98.28%
Our (SliceNet) | 0.0380 | 0.0272 | 0.1553 | 0.0862 | 85.35% | 94.31% | 96.77%
| Our (UniFuse) | 0.0354 | 0.0247 | 0.1334 | 0.0762 | 87.00% | 96.08% | 97.80%
Replica360 4K | Bifuse [33] | 0.0642 | 0.0485 | 0.2446 | 0.1266 | 63.27% | 89.13% | 95.65%
HoHoNet [31] | 0.0357 | 0.0249 | 0.1359 | 0.0744 | 85.17% | 94.63% | 96.61%
SliceNet [19] | 0.0473 | 0.0341 | 0.1891 | 0.0994 | 78.31% | 93.17% | 96.77%
UniFuse [11] | 0.0394 | 0.0289 | 0.1480 | 0.0818 | 82.20% | 96.26% | 98.54%
360MonoDepth [23] | 0.0611 | 0.0400 | 0.1667 | 0.0815 | 80.04% | 95.25% | 98.47%
Our (HoHoNet) | 0.0332 | 0.0239 | 0.1309 | 0.0709 | 86.07% | 94.98% | 96.76%
Our (SliceNet) | 0.0444 | 0.0335 | 0.1831 | 0.0975 | 76.80% | 93.27% | 97.34%
| Our (UniFuse) | 0.0380 | 0.0281 | 0.1447 | 0.0795 | 82.69% | 96.66% | 98.65%
Table 3: Quantitative comparisons of our method (using HoHoNet, SliceNet, or
UniFuse to generate the reference panoramic depth maps) versus previous
panorama-based methods and the stitching-based method proposed in [23]. RMSE,
MAE, AbsRel, and RMSElog measure the root mean squared error, mean absolute
error, mean relative error, and RMSE in log-10 space (same as in UniFuse and
BiFuse), of depth values. $\delta_{1}$, $\delta_{2}$, and $\delta_{3}$ measure
the ratios of pixels with mutual relative errors below $1.25$, $1.25^{2}$, and
$1.25^{3}$, respectively. Highlighting: best, second-best, third-best.

###  4.2 Qualitative Comparisons

We show qualitative comparisons in Figure 5 on the Matterport 2K and
Replica360 2K and 4K datasets. We found UniFuse produced qualitatively best
results among panorama-based methods. For stitching-based methods, we found
[23] generates clearer images than UniFuse, while our method generated
slightly clearer results than [23] with finer details in general. We found
that in [23]’s results, the estimated depths can deviate from the ground truth
when observed at larger scales. For example, observe the inconsistent depths
of the two white walls (at roughly the same distances to the camera) of the
Replica360 2K example in Figure 5. More examples are shown in the
Supplementary Materials.

Figure 4: . Qualitative comparisons of ablation studies and alternative design
choices. Only performing the blending step led to over-saturated results.
Similarly, replacing the registration step with a trivial averaging of scales
and shifts led to similar results. Skipping the padding may lead to glitches
at partition boundaries. The ”smoothing” result shows that a simple color
smoothing cannot effectively remove the inconsistency between partitions.
Figure 5: . Qualitative comparisons of results generated by our method (using
SliceNet or UniFuse to generate the reference panoramic depth maps), previous
panorama-based methods, and the stitching-based method in [23] (360MonoDepth).

###  4.3 Analysis

See Table 2 and Figure 4 for supporting quantitative statistics and
qualitative examples.

Ablation studies: Only performing the registration step led to blocks with
visible seams (_e.g_. Figure 2 (c)). Similarly, only performing the blending
step led to worse quantitative and qualitative results. Skipping the padding
in the blending step may led to glitches at the partition boundaries. Next, we
tried swapping key designs of our method with trivial approaches. First,
instead of using a common reference map, we tried to register the individual
perspective depth maps by transforming them to the average of scale and shift.
Second, we tried blending the registered perspective depth maps by simply
smoothing the depth values at partition boundaries. In both cases, the results
become worse quantitatively and qualitatively.

Alternative degree registration functions: We tried linear and quadratic
registration functions instead of cubic functions. We found that using simpler
functions actually led to slightly better quantitative scores. We still opt
for cubic functions because we think the additional degrees of freedom may be
beneficial for coping with unseen cases.

Switching to MiDaS: To verify that the differences of performance between [23]
and our method are not simply due to the different perspective methods that
were used, we tried using MiDaS instead of LeReS to do the perspective depth
estimations. We find the results to be slightly worse, but do not make up all
the differences between the two methods.

Different partitions: We tried other ways to partition the target domain into
perspective views. We find that sparser partitions (larger FOVs) led to
slightly worse performance. This may be because perspective methods run on
larger FOVs output less detailed estimations in general (see discussions in
[17]). A benefit of using fewer partitions is shorter computation times of the
perspective depth maps generation step (which is not the performance
bottleneck).

Limitations: Our results may be negatively affected by major errors of the
panoramic depth maps being used. For examples, for unseen cases such as
outdoor scenes, some of the existing panorama-based methods may produce very
low quality results and negatively affect our results (two such examples are
OmniDepth [47] and BiFuse run on outdoor scenes, as shown in [23]’s website).
Same as in [23], another limitation is the accuracy of the estimated
perspective depth maps being used. In either cases, our method is poised to
benefit from advances in both panoramic and perspective depth estimation
methods.

##  5 Conclusion

We show that a bottleneck of stitching-based panoramic depth estimation
methods, _i.e_. the global consistency problem, can be solved satisfactorily
and very efficiently by our registration-based approach. Accordingly, we
propose a streamlined stitching pipeline that outperforms current state-of-
the-arts method [23] quantitatively and qualitatively and is much faster. For
future work, our main goal is speed improvement, either by developing GPU-
based Laplacian solvers inspired by [10] to solve the blending step or
alternatively, using style transfer-based approaches. Acknowledgements: this
work is funded by the Ministry of Science and Technology of Taiwan (110N007
and 111R10286C).

## References

  * [1] Sameer Agarwal, Keir Mierle, and The Ceres Solver Team.  Ceres Solver, 3 2022.
  * [2] I. Armeni, A. Sax, A. R. Zamir, and S. Savarese.  Joint 2D-3D-Semantic Data for Indoor Scene Understanding.  ArXiv e-prints, Feb. 2017.
  * [3] Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese.  Joint 2d-3d-semantic data for indoor scene understanding.  arXiv preprint arXiv:1702.01105, 2017.
  * [4] Tobias Bertel, Mingze Yuan, Reuben Lindroos, and Christian Richardt.  OmniPhotos: Casual 360° VR photography.  ACM Transactions on Graphics, 39(6):266:1–12, Dec. 2020.
  * [5] S. Farooq Bhat, I. Alhashim, and P. Wonka.  Adabins: Depth estimation using adaptive bins.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4008–4017, Los Alamitos, CA, USA, jun 2021. IEEE Computer Society.
  * [6] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  arXiv preprint arXiv:1709.06158, 2017.
  * [7] Marc Eder, True Price, Thanh Vu, Akash Bapat, and Jan-Michael Frahm.  Mapped convolutions.  2019\.
  * [8] Marc Eder, Mykhailo Shvets, John Lim, and Jan-Michael Frahm.  Tangent images for mitigating spherical distortion.  In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [9] Lam Huynh, Phong Nguyen-Ha, Jiří Matas Esa Rahtu, and Janne Heikkilä.  Guiding monocular depth estimation using depth-attention volume.  2021\.
  * [10] Stefan Jeschke, David Cline, and Peter Wonka.  A gpu laplacian solver for diffusion curves and poisson image editing.  ACM Trans. Graph., 28(5):1–8, dec 2009.
  * [11] Hualie Jiang, Zhe Sheng, Siyu Zhu, Zilong Dong, and Rui Huang.  Unifuse: Unidirectional fusion for 360∘ panorama depth estimation.  IEEE Robotics and Automation Letters, 2021.
  * [12] Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michael Frahm, Shu Wu, Matthew Yu, Peizhao Zhang, Zijian He, Peter Vajda, Ayush Saraf, and Michael F. Cohen.  One shot 3d photography.  Transactions on Graphics (Proceedings of SIGGRAPH), 39(4), 2020\.
  * [13] Yuyan Li, Yuliang Guo, Zhixin Yan, Xinyu Huang, Duan Ye, and Liu Ren.  Omnifusion: 360 monocular depth estimation via geometry-aware fusion.  In 2022 Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, June 2022.
  * [14] Yuyan Li, Zhixin Yan, Ye Duan, and Liu Ren.  Panodepth: A two-stage approach for monocular omnidirectional depth estimation.  In 2021 International Conference on 3D Vision (3DV), pages 648–658. IEEE, 2021.
  * [15] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In Computer Vision and Pattern Recognition (CVPR), 2018.
  * [16] Yiqun Mei, Yuchen Fan, and Yuqian Zhou.  Image super-resolution with non-local sparse attention.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3517–3526, June 2021.
  * [17] S. Mahdi H. Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, and Yağız Aksoy.  Boosting monocular depth estimation models to high-resolution via content-adaptive multi-resolution merging.  2021\.
  * [18] Patrick Pérez, Michel Gangnet, and Andrew Blake.  Poisson image editing.  ACM Trans. Graph., 22(3):313–318, jul 2003.
  * [19] Giovanni Pintore, Marco Agus, Eva Almansa, Jens Schneider, and Enrico Gobbetti.  Slicenet: deep dense depth estimation from a single indoor panorama using a slice-based representation: Supplementary material.  2021\.
  * [20] Giovanni Pintore, Marco Agus, and Enrico Gobbetti.  AtlantaNet: Inferring the 3D indoor layout from a single 360 image beyond the Manhattan world assumption.  In Proc. ECCV, August 2020.
  * [21] René; Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12159–12168, 2021.
  * [22] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(03):1623–1637, mar 2022.
  * [23] Manuel Rey-Area, Mingze Yuan, and Christian Richardt.  360monodepth: High-resolution 360-degree monocular depth estimation.  In CVPR, 2022.
  * [24] Ana Serrano, Incheol Kim, Zhili Chen, Stephen DiVerdi, Diego Gutierrez, Aaron Hertzmann, and Belen Masia.  Motion parallax for 360∘ rgbd video.  IEEE Transactions on Visualization and Computer Graphics, 2019.
  * [25] Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao.  Panoformer: Panorama transformer for indoor 360 depth estimation.  arXiv e-prints, pages arXiv–2203, 2022.
  * [26] Z. Shen, C. Lin, L. Nie, K. Liao, and Y. Zhao.  Distortion-tolerant monocular depth estimation on omnidirectional images using dual-cubemap.  In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, Los Alamitos, CA, USA, jul 2021. IEEE Computer Society.
  * [27] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser.  Semantic scene completion from a single depth image.  Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [28] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe.  The Replica dataset: A digital replica of indoor spaces.  arXiv preprint arXiv:1906.05797, 2019.
  * [29] Cheng Sun, Chi-Wei Hsiao, Min Sun, and Hwann-Tzong Chen.  Horizonnet: Learning room layout with 1d representation and pano stretch data augmentation.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1047–1056, 2019.
  * [30] Cheng Sun, Chi-Wei Hsiao, Min Sun, and Hwann-Tzong Chen.  Horizonnet: Learning room layout with 1d representation and pano stretch data augmentation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [31] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Hohonet: 360 indoor holistic understanding with latent horizontal features.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2573–2582, 2021.
  * [32] Keisuke Tateno, Nassir Navab, and Federico Tombari.  Distortion-aware convolutional filters for dense prediction in panoramic images.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [33] Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai.  Bifuse: Monocular 360 depth estimation via bi-projection fusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 462–471, 2020.
  * [34] Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai.  Led2-net: Monocular 360deg layout estimation via differentiable depth rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12956–12965, 2021.
  * [35] Haiyan Wang, Will Hutchcroft, Yuguang Li, Zhiqiang Wan, Ivaylo Boyadzhiev, Yingli Tian, and Sing Bing Kang.  Psmnet: Position-aware stereo merging network for room layout estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [36] Changhee Won, Hochang Seok, Zhaopeng Cui, Marc Pollefeys, and Jongwoo Lim.  Omnislam: Omnidirectional localization and dense mapping for wide-baseline multi-camera systems.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 559–566, 2020.
  * [37] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese.  Gibson env: Real-world perception for embodied agents.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9068–9079, 2018.
  * [38] Ke Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, Yang Xiao, Ruibo Li, and Zhenbo Luo.  Monocular relative depth perception with web stereo data supervision.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [39] Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao.  Structure-guided ranking loss for single image depth prediction.  In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [40] Jiale Xu, Jia Zheng, Yanyu Xu, Rui Tang, and Shenghua Gao.  Layout-guided novel view synthesis from a single indoor panorama.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16438–16447, 2021.
  * [41] Sheng Yang, Beichen Li, Yan-Pei Cao, Hongbo Fu, Yu-Kun Lai, Leif Kobbelt, and Shi-Min Hu.  Noise-resilient reconstruction of panoramas and 3d scenes using robot-mounted unsynchronized commodity rgb-d cameras.  ACM Transactions on Graphics (TOG), 39(5):1–15, 2020.
  * [42] Shang-Ta Yang, Fu-En Wang, Chi-Han Peng, Peter Wonka, Min Sun, and Hung-Kuo Chu.  Dula-net: A dual-projection network for estimating room layouts from a single rgb panorama.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3363–3372, 2019.
  * [43] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen.  Learning to recover 3d scene shape from a single image.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. (CVPR), 2021.
  * [44] Wei Zeng, Sezer Karaoglu, and Theo Gevers.  Joint 3d layout and depth prediction from a single indoor panorama image.  In European Conference on Computer Vision, pages 666–682. Springer, 2020.
  * [45] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou.  Structured3d: A large photo-realistic dataset for structured 3d modeling.  In Proceedings of The European Conference on Computer Vision (ECCV), 2020.
  * [46] Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, Federico Alvarez, and Petros Daras.  Spherical view synthesis for self-supervised 360 depth estimation.  In 2019 International Conference on 3D Vision (3DV), pages 690–699. IEEE, 2019.
  * [47] Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, and Petros Daras.  Omnidepth: Dense depth estimation for indoors spherical panoramas.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 448–465, 2018.
  * [48] Chuhang Zou, Alex Colburn, Qi Shan, and Derek Hoiem.  Layoutnet: Reconstructing the 3d room layout from a single rgb image.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2051–2059, 2018.

##  6 Supplementary Materials

###  6.1 Additional Qualitative Results

We show qualitative comparisons of HoHoNet [6]’s results versua our method
using HoHoNet to generate the reference panoramic depth maps in Figure 6 and
Figure 7. Same as in other cases (_e.g_. SliceNet and UniFuse), we can see
that our method significantly improved the levels of details of the respective
panorama-based method.

We show additional qualitative results in Figure 8 and Figure 9. In the first
example of Replica360 4K result in Figure 9, we can see that the result of
[23] has incorrect relative depths of the white wall w.r.t. the two adjacent
walls.

###  6.2 Quantitative metrics in 360MonoDepth [23]

In [23], they reported different quantitative scores on the Matterport 2K
dataset and the Replica360 2K/4K datasets then our experiments. We think the
differences came from the different approaches to align an estimated depth map
and a ground-truth depth map. In our paper, we used the ”median-scaling”
approach that is the common practice in all panorama-based methods since
OmniDepth [47]. After checking [23]’s paper and code, we think they instead
used the least squares-based scaling approach (proposed in the original MiDaS
v3 paper [22]), in which both optimal scaling and offset are computed by
least-squares to align an an estimated depth map and a ground-truth one. We
note that some of their quantitative scores are very different from the ones
reported in other papers. For example, they reported RMSE of BiFuse as 0.994
on the Matterport 2K dataset, but it is 0.6259 on the Matterport 1K dataset
reported in HoHoNet [31], SliceNet [19], UniFuse [11], and OmniFusion [13].
Our evaluation method reported RMSE of BiFuse as 0.6350 on the Matterport 2K
dataset, which is much closer.

Figure 6: . Qualitative comparisons on the Matterport 2K dataset of HoHoNet
and our method using HoHoNet to generate the reference panorama depth maps.
Figure 7: . Qualitative comparisons on the Replica360 2K and 4K dataset of
HoHoNet and our method using HoHoNet to generate the reference panorama depth
maps. Figure 8: . Additional qualitative comparisons on the Matterport 2K
dataset. Figure 9: . Additional qualitative comparisons on the Replica 2K and
4K datasets.

###  6.3 Comparison with OmniFusion [13]

Recently, OmniFusion [13] reported results with new state-of-the-arts
quantitative scores among panorama-based methods. At submission, we could not
get to run their codes correctly on our machines, so quantitative evaluations
on the new Matterport 2K and Replica360 2K/4K datasets are not reported.
Comparing to our method, OmniFusion outputs 1K instead of 2K or higher
resolution outputs. We also don’t find the same levels of details in their
results (shown in their paper) as in stitching-based methods ([23] and ours).
We expect our method to benefit from the improved accuracy of panoramic depth
maps generated by OmniFusion.

###  6.4 Quantitative evaluation of sharp detail preservation

We also measured several Laplacian-based metrics to quantitatively evaluate
sharp details preservation. The results (Table 4) showed that our method
significantly outperformed both panoramic methods and 360MonoDepth [23]. These
quantitative results are in line with the qualitative comparisons shown in the
main paper - both showed that our results outperform other panorama-based
methods and 360MonoDepth (another stitching-based method) in terms of sharp
detail preservation.

DS | Method |  $\|\nabla^{2}\|$ $\downarrow$ |  $\|LoG\|$ $\downarrow$ |
---|---|---|---|---
Matterport 2K | Bifuse | 0.0450 | 0.2134 |  Highlighting: Best, second-best, third-best
SliceNet | 0.0386 | 0.1827
UniFuse | 0.0402 | 0.1843
360MonoDepth | 0.0384 | 0.1681
Our (SliceNet) | 0.0287 | 0.1269
Our (UniFuse) | 0.0281 | 0.1240
Replica360 2K | Bifuse | 0.00093 | 0.0202
SliceNet | 0.00397 | 0.0811 |
UniFuse | 0.00134 | 0.0270 |
360MonoDepth | 0.00064 | 0.0122 |
Our (SliceNet) | 0.00049 | 0.0107 |
Our (UniFuse) | 0.00051 | 0.0110 |
Replica360 4K | Bifuse | 0.00061 | 0.0132 |
SliceNet | 0.00446 | 0.0918 |
UniFuse | 0.00049 | 0.0103 |
360MonoDepth | 0.00038 | 0.0075 |
Our (SliceNet) | 0.00024 | 0.0053 |
Our (UniFuse) | 0.00027 | 0.0058 |
Table 4: $\|\nabla^{2}\|$ and $\|LoG\|$ measure the mean absolute errors of
Laplacian and Laplacian of Gaussian (LoG) using the standard 5x5 mask, which
are proxies to measure how sharp features of estimated and ground-truth depth
maps match.

###  6.5 Using super-resolution approaches to up-sample panoramic depth maps
instead of bilinear filtering

We used a recent super-resolution method [16] to up-sample 1K outputs of
UniFuse and SliceNet to 2K (Matterport3D) and 4K (Replica3604K). The results
are very similar to bilinear up-sampling results qualitatively and
quantitatively ($<1\%$ differences by RMSE, MAE, and AbsRel).


# Semantic Code Classification for Automated Machine Learning

Polina Guseva∗
HSE University
<EMAIL_ADDRESS>
Anastasia Drozdova
HSE University
<EMAIL_ADDRESS>
Natalia Denisenko
HSE University
Daria Sapozhnikova
HSE University
Ivan Pyaternev
HSE University
Anna Scherbakova
HSE University
Andrey Ustuzhanin
HSE University
Equal contribution.

###### Abstract

A range of applications for automatic machine learning need the generation
process to be controllable. In this work, we propose a way to control the
output via a sequence of simple actions, that are called semantic code
classes. Finally, we present a semantic code classification task and discuss
methods for solving this problem on the Natural Language to Machine Learning
(NL2ML) dataset.

##  1 Introduction

Code generation has the potential to be useful in a range of applications,
although there is a variety of limitations (Chen et al. (2021)). Despite the
recent success in this area with Codex models (Chen et al. (2021)), controlled
code generation remains challenging (Xu et al. (2021); Hu et al. (2017)). Code
generation models usually receive a code prompt (Feng et al. (2020); Chen et
al. (2021) or a description of a task in natural language (Clement et al.
(2020)), but the output is not controlled.

Automatic machine learning (AutoML) is a specialized area of code generation.
Controlled generation of machine learning (ML) code is crucial for AutoML
applications because code requirements for the same task may vary drastically
in practice. Some applications focus more on interpretability; others might be
satisfied with uninterpretable models that perform well.

Many machine learning programs follow a specific sequence of steps. Such
sequence is called an ML pipeline. Every machine learning program may be
roughly split into code snippets, such that each snippet is responsible for a
single small action. These small actions are called semantic code classes.

A sequence of semantic classes can control the generation process and provide
a generative model with additional information about the requested pipeline.
Such a sequence serves as a guide and indicates the most important steps.

Training a model to recognize a sequence of semantic actions requires a
labeled dataset. The labeling task is highly specific and requires a lot of
manual labor, hence it might be beneficial to train an additional model for
labeling.

In this work, we present the semantic code classification task and compare the
results of various models. Such representation allows to better understand the
structure of the ML code and the contents of each step of possible pipelines.

##  2 Dataset

Natural Language to Machine Learning (NL2ML) corpus is a collection of code
blocks from Kaggle competition notebooks. It includes 101 071 different
notebooks comprising 2 587 074 cells (or code block), which participants
submitted to 266 Kaggle competitions. 4 748 of those code blocks have passed
through manual assessment of its semantics. Each block is attributed to an
upper-level category, which corresponds to a coarse meaning, and a second-
level category, representing the finer semantic of that block. Publication of
the corpus, along with the detailed description of the annotation process, is
delayed due to legal considerations with Kaggle.

The upper level categories attribute the general kind of code block. One block
can belong to a single category. Here is the list of possible values:
Hypothesis, Environment, Data extraction, Exploratory data analysis, Data
transform, Model train, Model evaluation, Model interpretation, Hyperparameter
tuning, Visualization. The lower-level category represents a more specific
type, for example, upper-level class Data transform contains such items as
create dataframe, remove duplicates, correct missing values, feature
engineering, filter and others.

The dataset is very unbalanced (appendix 1). Some simple classes (show table)
are overrepresented because data contains many similar examples. At the same
time, many classes (load data from zip, remove duplicates) are represented by
less than 10 examples. Another difficulty is that data is heavily skewed
because a large portion of the snippets came from the same competition; hence
the snippets in the same class tend to be similar in general.

##  3 Code classification

This section describes methods we used to classify code blocks into the lower
level semantic classes. The details about metric computation are provided in
the section 4. For all models except neural networks we provide metrics with
error bars that measure randomness in training the model.

The source code is available
online111https://github.com/whatevernevermindbro/nl2ml-mirror.

###  3.1 Baseline solution: SVM

Support vector machines classifier (Boser et al. ) with linear kernel was used
as baseline solution because the method does not require a large amount of
samples. The multiclass support is handled according to a one-vs-one scheme
(Chang & Lin (2011)). Baseline F1-score is 0.676 $\pm$ 0.029.

The F1-score is defined by the following formula:

$F_{1}=2\frac{precision\cdot recall}{precision+recall}$

Due to the imbalance in the data, the F1-score was chosen as the target
metric.

###  3.2 Preprocessing

All the code blocks were modified according to PEP8 using a standard Python
utility, autopep8, to eliminate the problems with code-style differences.

Comments have been removed from the code. Despite the fact that they can
contain a lot of information about the block’s semantic type, their presence,
as a rule, slightly degrades the quality of predictions. This is most likely
because the content of the comments can be both useful and confusing. The text
may contain information unrelated to the code block or unnecessary code that
does not affect the semantic type.

For tokenization, the Byte-pair encoding method was used (Sennrich et al.
(2015)). The tokenizer was trained using unlabeled data. The size of the
dictionary used is 30,000 tokens; during the training, the BPE-Dropout
regularization technique was used (Provilkov et al. (2019)).

This tokenization has proven to be quite helpful for many models. The names of
entities in the program code can consist of several words, written together or
through an underscore "_". Splitting by the spaces cannot distinguish
different words included in the names of entities, and breaking into n-grams
may not be flexible since words in names can have different lengths. The BPE
method was optimal because it can extract information about the donkeys that
are often found in the names of entities.

Term frequency-inverse document frequency (TF-IDF) (Zhang et al. (2011)) was
used as features for models. These statistics show the importance of the token
in the document, given the context of the corpus.

###  3.3 Naive Bayesian Classifier and NB-SVM

The Naive Bayesian Classifier (NB) (Raschka (2014)) is a probabilistic model
based on Bayes’ theorem. NB Classifier is a common pre-neural network method
for classifying text data. As a rule, it is assumed that the data has either a
multinomial distribution or a Bernoulli distribution.

Since the dataset is unbalanced, estimates for the parameters of the
multinomial distribution may be unstable, which negatively affects the quality
of predictions. To combat this problem, a complementary NB was proposed in
Rennie et al. (2003).

The model with the Bernoulli distribution showed the best result among naive
Bayesian classifiers. This is because this model works best with short texts
(Mccallum & Nigam (2001)). Short texts predominate in the training dataset.

Metrics for NB classifier are in rows Multinomial NB, Complementary NB,
Bernoulli NB of table 2.

NB classifier and SVM are often used as basic solutions in text classification
problems. NB tends to perform better on shorter passages, while SVM works best
on longer texts. The article Wang & Manning (2012) discusses the composition
of an NB classifier and SVM, Naive Bayes - Support Vector Machine (NBSVM),
which combines the advantages of both methods. The paper also states that
binarization of NB features or using bigrams should improve results, but in
our case it’s the opposite. Metrics for NBSVM classifier are in rows NBSVM,
NBSVM (binarization), NBSVM birams of table 2.

###  3.4 Neural networks

Neural networks are an important class of machine learning algorithms. It is
due to neural networks that the quality of text data processing has been
improved. Unfortunately, many advanced neural network architectures require a
lot of training data. There was not too much data in this case, but it was
possible to obtain relatively good results.

There is not enough data to train vector representations, so frozen pretrained
CodeBERT (Feng et al. (2020)) was used. CodeBERT has several drawbacks that
are worth noting. Firstly, the important thing is that this model uses its own
tokenizer. The tokens used do not quite match the machine learning code as
they were derived from more general code, which means that machine learning-
specific tokens may be too rare to enter the dictionary. Secondly, CodeBERT
was trained on the program code not only in Python3 but also in several other
languages: JavaScript, Java, PHP, Ruby, Go. Due to this, the connection of
tokens with machine learning is becoming even less.

Another important point when training a neural network is the loss function,
on which the success of the model depends a lot. The standard loss function in
a classification problem is cross-entropy. Conventional cross-entropy is
unstable to unbalanced data; the situation can be improved by adding weights
for examples from different classes. Since there are a lot of classes, a vast
number of hyperparameters appear, which is a big drawback since neural
networks take a long time to learn.

For unbalanced data, there are alternative loss functions:

  1. 1.

Soft-F1. This loss function directly optimizes the target metric in the task
at hand, which is a significant advantage.

The critical drawback of this loss function is that it requires a sizeable
mini-batch size. If the batch size is too small, rare classes will often not
be included in the batch, hence the precision and the recall are equal to 1,
which is the optimal value for these metrics. If there are many such rare
classes, the loss function will be extremely close to the optimum, leading to
the fact that learning does not occur. In the dataset, quite a few classes
contain less than 10 objects, which means that large batch size is required.

  2. 2.

Focal Loss. This loss function comes from computer vision (Lin et al. (2017)).
The main idea is to adaptively reduce the weight of objects for which the
model determines the class correctly and confidently. Due to this, it is
possible to learn patterns even in rare classes.

The following architectures were used:

  1. 1.

Recurrent neural network (RNN). GRU and LSTM were tried as a recurrent layer;
the result is approximately the same. The recurrent network is bidirectional,
since this configuration usually gives the best result, allowing more context
for the tokens. Row RNN in results table (2).

The token sequence is fed into the inputs of the recurrent layer. The output
goes through a dense layer, GELU activation, dropout, and outputs
probabilities through dense layer and Softmax.

Layer sizes are optimized with random search. The model with the best score
uses a single LSTM layer with hidden state size of 180 and a hidden fully
connected layer of size 269. The scheme is in appendix 2.

  2. 2.

RNN with additive attention (Bahdanau et al. (2014)). The model differs from
the usual RNN in only one layer: the attention layer added after the recurrent
layer.

The sizes of all layers were optimized using random search. The best model
uses a single LSTM layer with hidden state size of 100. After attention, there
is a hidden fully connected layer of size 135. The scheme of the model can be
found in appendix 3.

A layer of attention helps to identify the most important values in a
sequence, which leads to a dramatic improvement in the result. Row
RNN+Attention in results table (2).

###  3.5 Augmentation

Since the marked-up data is insufficient, augmentation was used. However, it
is necessary that the code remains compilable and belongs to the same semantic
type after modifications.

Many code blocks contain similar variable names (eg. "df", "train", "test").
They do not always provide additional information about the block type, but
due to the frequency they are used by the classifier. Therefore, augmentation
was applied, which masks some fraction of the variables in the block. the
augmentation does not change the semantic of the block, as the names of user-
defined variables can be almost anything. The variables in code were found
using AST trees and replaced by masks, each variable in a block with a
different one.

The augmentation was used with SVM and Attention RNN. Results are in rows
SVM+Augmentation, RNN+Attention+Augmentation in table (2). The results have
not improved enough(the value is within error bars), so we do not use
augmentations in other experiments.

###  3.6 Hierarchy

The structure by which the code snippets are classified is a two-level graph:
the first is the general class of action, the second is what exactly is done
in the block. Therefore, it was decided to implement this structure as a two-
level classifier, where the upper vertex is predicted first, and then the
lower by a separate classifier, already trained on specific objects belonging
to the child vertices.

The advantage of considering the upper classes separately is that in contrast
to the original dataset, each class is represented by at least several dozen
objects. At the same time, most of the very small original classes go into
such a classification together with other small classes. Thus, the
determination of their upper class is a more realistic task. If we correctly
guessed the upper class, then small objects are likely to have no large
competitors. However, if the class was identified incorrectly in the first
step, the subsequent classification is guaranteed to be inaccurate.

SVM was chosen as the lower classifier since the model demonstrated the best
results in previous experiments, while the upper classifiers were

  1. 1.

RNN containing a bidirectional LSTM, the vector representations were taken
from the frozen pre-trained CodeBERT (Feng et al. (2020)).

  2. 2.

SVM

In table 1 the metrics of the upper level classification are presented. SVM
achieved sufficient results. The overall results of hierarchy classifier are
in 2 \- rows RNN + SVM Hierarchy, SVM + SVM Hierarchy \- this method allowed
to significantly improve metrics.

Table 1: Upper level prediction results. | Metrics
---|---
Model | F1-score | Accuracy
RNN | 0.759 | 0.773
SVM | 0.937 | 0.938

###  3.7 Semi-supervised, pseudo labels

Since the amount of labeled data is extremely small, it was decided to
experiment with semi-supervised models that allow to use unlabeled data (Xie
et al. (2019)).

First, we trained a classic SVM model on the marked-up dataset. It’s
predictions on unlabeled data were used as pseudo labels. Then another SVM
model with separately selected hyperparameters was trained on pseudo-labeled
data and a part of marked-up data. It was tested on the rest of the labeled
data.

We tested this approach using 20%, 40%, and 100% of unlabeled data. The
results are in rows Pseudo labels 20%, Pseudo labels 40%, Pseudo labels 100%
in table (2).

##  4 Results

We evaluated models on marked-up code snippets of the NL2ML dataset. Several
approaches demonstrated some improvements in metrics from the baseline
solution. All of the resulting models are compared in the table (2) by
F1-score and accuracy. The best results are achieved by hierarchy and pseudo-
labels models. The first one utilizes the knowledge of the classes structure,
and the second one uses a great amount of unlabeled data.

The target metrics were computed on a test dataset that contains 20 % of all
labeled data. For each model, we optimized hyperparameters with random search.
For neural networks, we used a validation set that contains 20 % of labeled
data. For other methods, we used cross-validation with 10 folds.

Table 2: Summary table of all the models and their results | Metrics
---|---
Model | F1-score | Accuracy
SVM+Linear (Baseline) | 0.676 $\pm$ 0.029 | 0.678 $\pm$ 0.031
SVM + Poly | 0.620 $\pm$ 0.033 | 0.627 $\pm$ 0.030
SVM + RBF | 0.672 $\pm$ 0.031 | 0.676 $\pm$ 0.031
Multinomial NB | 0.561 $\pm$ 0.031 | 0.583 $\pm$ 0.031
Complementary NB | 0.576 $\pm$ 0.024 | 0.587 $\pm$ 0.023
Bernoulli NB | 0.609 $\pm$ 0.021 | 0.623 $\pm$ 0.026
NBSVM | 0.694 $\pm$ 0.013 | 0.704 $\pm$ 0.015
NBSVM (binarization) | 0.684 $\pm$ 0.016 | 0.694 $\pm$ 0.015
NBSVM (bigrams) | 0.689 $\pm$ 0.14 | 0.697 $\pm$ 0.14
RNN | 0.541 | 0.556
RNN + Attention | 0.639 | 0.648
SVM + Augmentation | 0.686 $\pm$ 0.031 | 0.696 $\pm$ 0.029
RNN + Attention + Augmentation | 0.638 | 0.652
RNN + SVM Hierarchy | 0.291 | 0.282
SVM+SVM Hierarchy | 0.741 | 0.744
Pseudo labels 20 % | 0.713 $\pm$ 0.014 | 0.724 $\pm$ 0.014
Pseudo labels 40 % | 0.727 $\pm$ 0.016 | 0.738 $\pm$ 0.014
Pseudo labels 100 % |  0.755 $\pm$ 0.03 |  0.764 $\pm$ 0.028

##  5 Conclusion

This paper compares methods for semantic classification of code snippets from
NL2ML corpus, a large-scale dataset of data science-specific code harvested
from Kaggle, the largest platform for data science competitions. The dataset
includes only 4 748 manually labeled snippets.

We provided detailed qualitative and quantitative comparisons of the used
classification methods. The final proposed model demonstrated better results
than the baseline SVM solution, which has an F1-score of 0.67. The method with
the best F1-score of 0.755 is an SVM trained on pseudo-labels.

The proposed method allows to automatically markup a program code corpus with
better accuracy and performance, as well as effectively assess the quality of
data science code in Python. We also genuinely hope that this work can
contribute to the development of the next generation of robust AutoML systems,
as semantic classification provides an interlingua that can be used for
translating between natural and programming languages.

## References

  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ , 2014.
  * (2) Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik.  A training algorithm for optimal margin classifiers.  In _Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory_ , pp. 144–152.
  * Chang & Lin (2011) Chih-Chung Chang and Chih-Jen Lin.  Libsvm: a library for support vector machines.  _ACM transactions on intelligent systems and technology (TIST)_ , 2(3):1–27, 2011.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _CoRR_ , abs/2107.03374, 2021.  URL https://arxiv.org/abs/2107.03374.
  * Clement et al. (2020) Colin B. Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan.  Pymt5: multi-mode translation of natural language and python code with transformers.  _CoRR_ , abs/2010.03150, 2020.  URL https://arxiv.org/abs/2010.03150.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou.  Codebert: A pre-trained model for programming and natural languages.  _CoRR_ , abs/2002.08155, 2020.  URL https://arxiv.org/abs/2002.08155.
  * Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing.  Controllable text generation.  _CoRR_ , abs/1703.00955, 2017.  URL http://arxiv.org/abs/1703.00955.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  _CoRR_ , abs/1708.02002, 2017.  URL http://arxiv.org/abs/1708.02002.
  * Mccallum & Nigam (2001) Andrew Mccallum and Kamal Nigam.  A comparison of event models for naive bayes text classification.  _Work Learn Text Categ_ , 752, 05 2001.
  * Provilkov et al. (2019) Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita.  Bpe-dropout: Simple and effective subword regularization.  _CoRR_ , abs/1910.13267, 2019.  URL http://arxiv.org/abs/1910.13267.
  * Raschka (2014) Sebastian Raschka.  Naive bayes and text classification I - introduction and theory.  _CoRR_ , abs/1410.5329, 2014.  URL http://arxiv.org/abs/1410.5329.
  * Rennie et al. (2003) Jason D. M. Rennie, Lawrence Shih, Jaime Teevan, and David R. Karger.  Tackling the poor assumptions of naive bayes text classifiers.  In _Proceedings of the Twentieth International Conference on International Conference on Machine Learning_ , ICML’03, pp. 616–623. AAAI Press, 2003.  ISBN 1577351894.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  _CoRR_ , abs/1508.07909, 2015.  URL http://arxiv.org/abs/1508.07909.
  * Wang & Manning (2012) Sida Wang and Christopher Manning.  Baselines and bigrams: Simple, good sentiment and topic classification.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 90–94, Jeju Island, Korea, July 2012. Association for Computational Linguistics.  URL https://aclanthology.org/P12-2018.
  * Xie et al. (2019) Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le.  Self-training with noisy student improves imagenet classification.  _CoRR_ , abs/1911.04252, 2019.  URL http://arxiv.org/abs/1911.04252.
  * Xu et al. (2021) Frank F. Xu, Bogdan Vasilescu, and Graham Neubig.  In-ide code generation from natural language: Promise and challenges.  _CoRR_ , abs/2101.11149, 2021.  URL https://arxiv.org/abs/2101.11149.
  * Zhang et al. (2011) Wen Zhang, Taketoshi Yoshida, and Xijin Tang.  A comparative study of tf*idf, lsi and multi-words for text classification.  _Expert Systems with Applications_ , 38(3):2758–2765, 2011.  ISSN 0957-4174.  doi: https://doi.org/10.1016/j.eswa.2010.08.066.  URL https://www.sciencedirect.com/science/article/pii/S0957417410008626.

##  Appendix A Distribution of snippets by semantic types

Figure 1: The distribution of labeled snippets by semantic types.

##  Appendix B Architectures of neural nets

Figure 2: Architecture of recurrent neural network Figure 3: Architecture of
recurrent neural network with attention

##  Appendix C Code snippets

Table 3: Examples of code snippets from NL2ML dataset Semantic class | Example
---|---
⬇ train_df.drop("Date", inplace=True, axis=1)  test_df.drop("Date",
inplace=True, axis=1)
⬇ from sklearn import linear_model reg_CC = linear_model.Lasso(alpha=0.1)
reg_Fat = linear_model.Lasso(alpha=0.1)
⬇ parameters = {’lstm_nodes’: [14,16,20],  ’nb_epoch’: [50],  ’batch_size’:
[32],  ’optimizer’: [’adam’]}
⬇ fig = plt.figure()  fig.suptitle("Algorithm␣Comparison")  ax =
fig.add_subplot(111)  plt.boxplot(results)  ax.set_xticklabels(names)
plt.show()
⬇ scaler = MinMaxScaler()  df[’revenue’] = scaler.fit_transform(
df[[’revenue’]]  )
⬇ rfr = RandomForestRegressor(  n_estimators=200,  max_depth=5,
max_features=0.5,  random_state=449,  n_jobs=-1  )  rfr.fit(x_train, y_train)

##  Appendix D Selected hyperparameters for each model

Table 4: Hyperparameters for models without hierarchy and first level of hierarchical models Model | Hyperparameter | Type | Value
---|---|---|---
SVM+Linear (Baseline) | C | numeric | 37.17
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.31
SVM + Poly | C | numeric | 1.43
| Degree of poly kernel | integer | 3
| min_df for TF-IDF | integer | 6
| max_df for TF-IDF | numeric | 0.30
SVM + RBF | C | numeric | 8.71
| min_df for TF-IDF | integer | 7
| max_df for TF-IDF | numeric | 0.39
Multinomial NB | Alpha | numeric | 0.01
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.57
Complementary NB | Alpha | numeric | 0.29
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.30
Bernoulli NB | Alpha | numeric | 0.01
| min_df for TF-IDF | integer | 5
| max_df for TF-IDF | numeric | 0.30
NBSVM | C for SVM | numeric | 0.50
| Kernel type | categorical | Linear
| Alpha for Naive Bayes | numerical | 4.22
| min_df for TF-IDF | integer | 1
| max_df for TF-IDF | numeric | 0.94
NBSVM (binarization) | C for SVM | numeric | 0.10
| Kernel type | categorical | Linear
| Alpha for Naive Bayes | numerical | 0.28
| min_df for TF-IDF | integer | 3
| max_df for TF-IDF | numeric | 0.94
NBSVM (bigrams) | C for SVM | numeric | 0.10
| Kernel type | categorical | Linear
| Alpha for Naive Bayes | numerical | 2.51
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.95
RNN | RNN cell type | categorical | LSTM
| Size of hidden state | integer | 180
| Size of linear layer | integer | 269
RNN + Attention | RNN cell type | categorical | LSTM
| Size of hidden state | integer | 100
| Size of linear layer | integer | 135
SVM + Augmentation | C | numeric | 5.82
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.30
| Percent of masked variables | numeric | 0.93
RNN + Attention + Augmentation | RNN cell type | categorical | LSTM
| Size of hidden state | integer | 169
| Size of linear layer | integer | 255
| Percent of masked variables | numeric | 99.7%
SVM+SVM Hierarchy (1st level) | C | numeric | 149.65
| Kernel type | categorical | Poly
| Degree of poly kernel | integer | 2
RNN+SVM Hierarchy (1st level) | RNN cell type | categorical | LSTM
| Size of hidden state | integer | 143
| Size of linear layer | integer | 185
Pseudo labels 20 % | C | numeric | 98.37
| Kernel type | categorical | linear
| min_df for TF-IDF | integer | 3
| max_df for TF-IDF | numeric | 0.53
Pseudo labels 40 % | C | numeric | 121.59
| Kernel type | categorical | linear
| min_df for TF-IDF | integer | 3
| max_df for TF-IDF | numeric | 0.41
Pseudo labels 100% | C | numeric | 145.56
| Kernel type | categorical | linear
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.26
Table 5: Hyperparameters for SVM+SVM Hierarchy (2nd level). Separate parameters for each upper level class. If upper level class is not present in this table, then it has one subclass and hence does not require a second level classificator. Upper level class | Hyperparameter | Type | Value
---|---|---|---
Data Extraction | C | numeric | 135.49
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 50
| max_df for TF-IDF | numeric | 0.89
EDA | C | numeric | 4.70
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 4
| max_df for TF-IDF | numeric | 0.61
Environment | C | numeric | 232.06
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 6
| max_df for TF-IDF | numeric | 0.81
Model_Train | C | numeric | 9.63
| Kernel type | categorical | RBF
| min_df for TF-IDF | integer | 17
| max_df for TF-IDF | numeric | 0.47
Data_Transform | C | numeric | 298.83
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.20
Model_Evaluation | C | numeric | 0.84
| Kernel type | categorical | Poly
| Degree of poly kernel | integer | 3
| min_df for TF-IDF | integer | 16
| max_df for TF-IDF | numeric | 0.50
Data_Export | C | numeric | 5.35
| Kernel type | categorical | RBF
| min_df for TF-IDF | integer | 23
| max_df for TF-IDF | numeric | 0.83
Hyperparam_Tuning | C | numeric | 20.03
| Kernel type | categorical | RBF
| min_df for TF-IDF | integer | 6
| max_df for TF-IDF | numeric | 0.26
Visualization | C | numeric | 865.58
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.66
Table 6: Hyperparameters for RNN+SVM Hierarchy (2nd level). Separate parameters for each upper level class. If upper level class is not present in this table, then it has one subclass and hence does not require a second level classificator. Upper level class | Hyperparameter | Type | Value
---|---|---|---
Data Extraction | C | numeric | 7.66
| Kernel type | categorical | RBF
| min_df for TF-IDF | integer | 17
| max_df for TF-IDF | numeric | 0.88
EDA | C | numeric | 1.71
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 6
| max_df for TF-IDF | numeric | 0.32
Environment | C | numeric | 462.65
| Kernel type | categorical | RBF
| min_df for TF-IDF | integer | 16
| max_df for TF-IDF | numeric | 0.87
Model_Train | C | numeric | 4.58
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 1
| max_df for TF-IDF | numeric | 0.90
Data_Transform | C | numeric | 204.62
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.45
Model_Evaluation | C | numeric | 2.03
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 3
| max_df for TF-IDF | numeric | 0.63
Data_Export | C | numeric | 26.49
| Kernel type | categorical | Poly
| Degree of poly kernel | numeric | 2
| min_df for TF-IDF | integer | 23
| max_df for TF-IDF | numeric | 0.93
Hyperparam_Tuning | C | numeric | 14.58
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 10
| max_df for TF-IDF | numeric | 0.65
Visualization | C | numeric | 2.48
| Kernel type | categorical | Linear
| min_df for TF-IDF | integer | 2
| max_df for TF-IDF | numeric | 0.21