# FloorLevel-Net: Recognizing Floor-Level Lines with Height-Attention-Guided
Multi-task Learning

Mengyang Wu, Wei Zeng, and Chi-Wing Fu M. Wu, and C.-W. Fu are with the
Chinese University of Hong Kong. e-mail: {mywu<EMAIL_ADDRESS>Zeng
is with Shenzhen Institute of Advanced Technology, Chinese Academy of
Sciences. e-mail<EMAIL_ADDRESS>Zeng and C.-W. Fu are the
corresponding authors.This paper has supplementary downloadable material
available at http://ieeexplore.ieee.org., provided by the author. The material
includes additional results and comparisons. Contact<EMAIL_ADDRESS>for
further questions about this work.

###### Abstract

The ability to recognize the position and order of the floor-level lines that
divide adjacent building floors can benefit many applications, for example,
urban augmented reality (AR). This work tackles the problem of locating floor-
level lines in street-view images, using a supervised deep learning approach.
Unfortunately, very little data is available for training such a network $-$
current street-view datasets contain either semantic annotations that lack
geometric attributes, or rectified facades without perspective priors. To
address this issue, we first compile a new dataset and develop a new data
augmentation scheme to synthesize training samples by harassing (i) the rich
semantics of existing rectified facades and (ii) perspective priors of
buildings in diverse street views. Next, we design _FloorLevel-Net_ , a multi-
task learning network that associates explicit features of building facades
and implicit floor-level lines, along with a height-attention mechanism to
help enforce a vertical ordering of floor-level lines. The generated
segmentations are then passed to a second-stage geometry post-processing to
exploit self-constrained geometric priors for plausible and consistent
reconstruction of floor-level lines. Quantitative and qualitative evaluations
conducted on assorted facades in existing datasets and street views from
Google demonstrate the effectiveness of our approach. Also, we present
context-aware image overlay results and show the potentials of our approach in
enriching AR-related applications. Project website:
https://wumengyangok.github.io/Project/FloorLevelNet.

###### Index Terms:

Multi-task learning, attention mechanism, semantic segmentation, street view,
augmented reality

##  I Introduction

Floor-level lines are line segments that separate adjacent floors on a
building facade; see Fig. 1 (middle). Being able to recognize them in city-
wide street views can benefit various applications,  _e.g._ , urban 3D
reconstruction [29], building topology analysis [42], and urban vitality study
[48]. Intrinsically, floor-level lines for the same building are parallel,
through which we can reconstruct the homography of the facade in a perspective
view [10] and support floor-aware augmented reality (AR) applications; see
Fig. 1 (right).

This work considers the problem of inferring floor-level lines in street-view
images, which requires the recovery of not only geometric priors (_e.g._ ,
positions and vanishing directions) in the image view, but also semantic
information (_e.g._ , floor orders) relevant to the floor-level lines. The
task is relatively intuitive for humans but very challenging for computers. So
far, we are not aware of any work that can robustly detect and recognize
floor-level lines.

Related prior works on 3D reconstruction (_e.g._ , [4, 15]), and facade
parsing (_e.g._ , [27, 44]), typically rely on various assumptions about the
structural regularity on building facades,  _e.g._ , repetitive windows and
balconies (_e.g._ , [31, 41]). Though these extra constraints help
disambiguate the problem and the results might help infer floor-level lines,
they still have various limitations, such as being error-prone when handling
perspective-oriented facades (Fig. 1 top-left) and scenes with occlusions
(Fig. 1 bottom-left). Alternatively, we may infer floor-level lines by
locating line segments roughly in the same direction. However, existing line
detectors (_e.g._ , [14, 34]) can easily generate huge amount of irrelevant
line segments in cluttered street views, and offer no semantic information for
inferring the floor order. Recently, Lee et al. [22] proposed the concept of
semantic lines that characterize the spatial scene structure in images. The
method first finds candidate lines using line features at multi-scale pooling
layers, then filters out semantic lines using local line features. However,
the filtering process (semantic or not) is a binary classification problem,
whilst this work requires to recover not only a multi-class label per line but
also a plausible floor order for lines in the same facade,  _i.e._ , order 1,
order 2,  _etc_.

Figure 1: Left column shows two example street-view images in London (top) and
Hong Kong (bottom), where the camera views are side\- and _front_ -facing
relative to the building, respectively. Note the occlusions introduced by the
advertisement billboard and light post circled in red on bottom left. Middle
column shows floor-level lines recognized by our method with geometric
positions and semantic order labels. Right column shows potential floor-aware
image-overlay results to aid shopping (top) and navigation (bottom).

Recent attempts of using deep neural networks for semantic scene segmentation
(_e.g._ , [8]) and planar surface recognition (_e.g._ , [45, 49]) exploit the
possibility of jointly learning the semantic and geometric attributes in
street views. The methods achieve superior performances over previous deep
learning methods that use solely the semantic features. This work also
leverages a supervised deep learning approach that can jointly infer geometric
priors and semantic information of floor-level lines. This is nevertheless a
challenging task. First, a key requirement for network training is the
availability of large, labeled street-view images with annotations of floor-
level lines. However, existing street-view datasets contain either semantic
annotations that lack geometric attributes, or rectified facades without
perspective priors. To address this issue, we devise a new data augmentation
scheme and compile a new dataset by effectively combining the rich context of
floor-level lines marked on an existing facade dataset, with perspective
priors easily-extracted on street-view images. In this way, we can largely
reduce the manual workload in the dataset construction, while promoting the
generalizability of our network model.

Second, we are not aware of any network architecture that can fulfill the
requirement of recognizing and locating floor-level lines in street-view
images. To fill the gap, we design _FloorLevel-Net_ $-$ a new deep learning
framework with two main components inspired by the characteristics of our
floor-level dataset. (i) _FloorLevel-Net_ leverages multi-task learning that
associates geometric properties of facades orientation to camera, with
semantic features of facade appearance, including windows, doors, and
balconies, etc. (ii) _FloorLevel-Net_ incorporates a height-attention
mechanism to enforce vertical orderings of floor-level lines, as floor order
naturally increases from bottom to top in image space. We further exploit
various geometric properties, including the _vanishing point (VP)_ and _floor
order_ constraints, to enforce a consistent reconstruction of floor-level
lines from piecewise segmentations by _FloorLevel-Net_.

We evaluate the performance of our approach on both building images from
existing facade datasets and self-collected street-view images from Google
Street View (GSV) [3]. Both qualitative and quantitative analysis demonstrate
the effectiveness of the individual components in our approach. The main
contributions of this work include:

  * •

We propose a _data augmentation_ scheme to integrate rich semantics of
rectified facades and geometric priors of buildings in diverse street views,
and compile a new dataset for recognizing floor-level lines (Sec. IV).

  * •

We design _FloorLevel-Net_ $-$a _multi-task CNN architecture with height
attention_ to simultaneously predict a facade segmentation map and a floor-
level distribution map for an input street-view image (Sec. V-A to V-C). We
further develop a post-processing framework to extract floor-level parameters
(including the endpoints and vanishing points), and refine the parameters with
_self-constrained geometric priors_ (Sec. V-D).

  * •

We evaluate the effectiveness of our approach in recognizing floor-level
lines, and demonstrate the applicability in enriching context-aware urban AR
(Sec. VI).

##  II Related Work

The recognition of geometric structures in urban scenes, _e.g._ , surface
layout [18] and driving lanes [50], has been gaining attention in image
processing and vision research. This work targets at floor-level lines that
separate adjacent floors on building facades. A closely-related topic is to
reconstruct 3D building models from a monocular image. Conventional approaches
typically adopt a two-stage approach: first parse a facade into piecewise
regions like windows, balconies, etc. [27, 44], then split the regions into
regular layouts for subsequent modeling,  _e.g._ , by shape grammars [28, 35].
Existing methods, however, rely heavily on detecting repetition [41, 31],
symmetric or rectangular structures on facade layout [36, 39], thus exhibiting
significant challenges for parsing general urban scenes in the wild.
Particular obstacles include perspective-oriented facades (_e.g._ , see Fig. 1
(top) and the red inset in Fig. 2) and scene occlusions by billboards and
traffic lights (_e.g._ , see Fig. 1 (bottom) and the blue inset in Fig. 2),
resulting in failures in matching repetition/symmetry/rectangular constraints.
It is hard to identify common geometric assumptions that can well fit diverse
styles of building facades in different cities.

Recent studies on urban scene recognition focus on deep-learning-based
approaches, benefiting from new datasets of urban scenes and advancements in
neural network architecture, _e.g._ , [26, 51, 6]. Specifically, DeepFacade
[24, 23] parses facades using a fully convolutional network to produce pixel-
wise semantics. Some work also attempts to formulate geometric understanding
of urban scenes as an image segmentation problem. For instance, Haines and
Calway [16] infer surface orientations using spatiograms of gradients and
colors. However, the results are piecewise segments that are typically
discontinuous with coarse and irregular boundaries, hindering the recognition
of geometric priors like the endpoints of floor-level lines. To further
recover geometric priors, a second-stage post-processing framework can be
employed. Zeng et al. [49] employ a vanishing-point-constrained optimization
to enhance piecewise segmentations of planar building facades.

Similarly, we adopt a twofold process to recognize floor-level lines on
building facades. First, we predict a segmentation mask of pixel-wise line
labels (floor-level distribution) using a multi-task CNN architecture. Second,
we refine the piecewise floor-level distribution into line parameters using
self-constrained geometric priors. Nevertheless, the work faces severe
challenges of being lack of _training dataset_ and _suitable network
architecture_. We tackle the challenges from the following perspectives:

  * •

Data augmentation is a vital technique to improve the diversity of the
training data and the network generalizability. There are several publicly
available datasets for urban scene (_e.g._ , KITTI [13] and Cityscapes [9])
and facade (_e.g._ , CMP [37] and LabelMeFacade [12]). Yet, we cannot use them
directly in this work. In contrast to explicit building structures such as
windows and balconies that have obvious visual appearance, floor-level lines
are rather implicit, requiring contextual information for the recognition. To
fill the gap, we propose a new data augmentation scheme that leverages
detailed facade semantics of front-facing facades in CMP [37] and diverse
building perspectives in GSV images. In this way, the synthesized dataset
efficiently captures both the rich context of floor-level lines and geometric
priors of building facades in reality.

Figure 2:  Our two-stage approach: (i) _FloorLevel-Net_ is a multi-task
learning network that segments the input image into building-facade-wise
semantic regions (top) and floor-level distributions (bottom); and (ii) our
method further fits and refines the pixel-wise network outputs into polylines
with geometric parameters. Further, we can take the reconstructed floor-level
lines to support and enrich urban AR applications with floor-aware image
overlay.

  * •

Multi-task learning helps to boost the performance of many deep-learning-based
vision tasks, by learning features from relevant knowledge domains. In the
literature, various efforts have been devoted to exploit multi-task learning
for geometric understanding of urban scenes. For example, Liu et al. [25]
tackle scene recognition and reconstruction tasks in a tightly-coupled
framework; besides, there are assorted works for indoor scene understanding
(_e.g._ , [38, 30, 11]) by fusing features of depth, surface normals, and
objects. Human experience on locating floor-level lines typically relies on
recognizing the facades and its enclosing regions, _e.g._ , doors and windows.
Therefore, we divide our task into subtasks of recognizing facades and floor-
level lines, and design a multi-task learning network to simultaneously
address the two subtasks together.

  * •

Channel-wise attention exploits the inter-channel relationship of features and
scales the feature map according to the importance of each channel. The
mechanism is first proposed in SENet [19] and adopted widely in image
classification (_e.g._ , [40]) and segmentation (_e.g._ , [46]). Recent
attention approaches take more advantages of contextual information inside the
image domains,  _e.g._ , the criss-cross attention module by Huang et al. [20]
and recursive context routing (ReCoR) mechanism by Chen et al. [7]. Choi et
al. [8] utilize the unbalanced class distribution at varying vertical
locations in urban scene images, and design a height-attention module to
emphasize the unbalance. Floor-level lines in our dataset exhibit a similar
property, since floor order on the same facade always increases from bottom to
top in the image space. This drives us to adopt a similar configuration in the
recognition of floor-level lines.

##  III Overview

In this work, our goal is

> to recognize and locate the floor-level lines on each building facade (that
> is close to the camera) in the image space of street-view images.

The input to our method is a single RGB image
$\mathcal{I}\in\mathbb{R}^{W\times H\times 3}$ of width $W$ and height $H$.
Given $\mathcal{I}$, we aim to predict a set of line segments
$\mathcal{L}=\\{L_{1},...,L_{n}\\}$ per facade that marks the separations
between adjacent floors; see Fig. 1 (middle) for two examples. In the
following, we refer line segments as lines for simplicity. Each line $L_{i}$
contains a pair of endpoints ($x_{s}^{i},\,y_{s}^{i}$) and
($x_{e}^{i},\,y_{e}^{i}$) in the image space of $\mathcal{I}$. Also, we aim to
recognize a floor order (denoted as $l^{i}\in\\{1,2,3,...\\}$) per line
$L_{i}$, such that the line that separates the ground floor and the first
floor has a floor order value of one, the next line above has a floor order
value of two, etc. So, each $L_{i}$ is specified as a 5-tuple
$\\{x^{i}_{s},\,y^{i}_{s},\,x^{i}_{e},\,y^{i}_{e},\,l^{i}\\}$. As a
visualization, we use the same coloring scheme (orange for order 1, green for
order 2, etc.) to reveal the floor orders in all our results.

Figure 3:  Illustration of our data augmentation scheme. We take the
advantages of the rich context of facades in the CMP dataset [37] (a1 & a2)
and the perspective-oriented building facades readily extracted from the GSV
(Google Street View) images (b1). From them, we can efficiently obtain
simplified semantics (a3) and annotate floor-level lines (a4), and further
generate a very large amount of augmented image samples, _i.e._ , an augmented
image (b2) with its associated semantic image (b3) and floor-level-lines image
(b4), by pairing up different geometric priors with different CMP facades.

There are three main parts in our approach.

  1. (i)

We compile a set of facade images from the CMP dataset [37], and employ them
to augment building facades in GSV images based on the geometric prior of
facade perspective (Sec. IV). By our new data augmentation scheme, we can
efficiently generate a large amount of image samples to train our network.

  2. (ii)

We develop a multi-task learning network$-$_FloorLevel-Net_ (Sec. V-A to V-C),
to segment the input image into piecewise regions of facades and to detect
candidate pixels possibly associated with floor-level lines; see the multi-
task learning module in Fig. 2. Particularly, _FloorLevel-Net_ encapsulates
semantic (_e.g._ ,  _windows_ , _doors_ , _shops_ , etc.) and geometric
(_e.g._ , _floor orders_ , _facade orientation_ , etc.) information related to
floor-level lines by articulating a fused loss function that is differentiable
by means of pixel-wise convolutions.

  3. (iii)

We infer per-line 5-tuple parameters from the piecewise segmentations in the
geometry post-processing stage (Sec. V-D). Here, we locate facade regions,
group floor-level lines per facade, then regress a polyline per floor-level
line; see the geometry processing module in Fig. 2. Further, we refine the
polylines based on self-constrained geometric priors, _i.e._ , the facade
boundary, floor order, and vanishing points.

So far, we are not aware of any work on recognizing floor-level lines in
street-view images. Existing related works focus on recognizing building
facades as a whole or explicit objects, such as windows, doors, and balconies.
Hence, we show the effectiveness of our approach through comparisons with
ablation techniques (Sec. VI-C) and DeepFacade, which is a closely-related
work on facade parsing (Sec. VI-D), and demonstrate the potential of our work
to support and enrich urban AR applications (Sec. VI-F).

##  IV Dataset Preparation & Analysis

An immediate challenge to recognizing floor-level lines is the lack of
properly-annotated street-view images. On the one hand, existing city-wide
datasets such as Cityscapes [9] provide mainly urban scene segmentations,
_e.g._ , roads, buildings, vehicles, etc., while lacking details of facades,
not to mention floor-level lines. On the other hand, building facade datasets
such as CMP [37] provide detailed semantics of facades, _e.g._ , windows,
doors, balcony, etc. However, the images exhibit mostly rectified views of
front-facing facades, which cannot reflect real-world facades with perspective
orientations.

###  IV-A Data Augmentation

To relax the enormous workload for labeling a new dataset, we propose a data
augmentation scheme (Fig. 3) to take the best advantages of the facade
semantics in the CMP dataset [37] together with the perspective-oriented
building facades extracted from GSV images.

To start, we first analyze the facade images (Fig. 3 (a1)) with their
semantics (Fig. 3 (a2)) provided by the CMP dataset. Here, we make use of the
simplified semantics (Fig. 3 (a3)), _i.e._ , _window_ , _shop_ , and _door_ ,
which offer contextual information for floor-level lines, _i.e._ , windows
usually appear on every floor, whereas shops and doors usually appear on the
ground floor. There may not be obvious lines between adjacent floors, yet
humans can infer them based on the associated contexts,  _e.g._ , windows.
Thanks to the regular front-facing structures in these inputs, manually
labeling floor-level lines (Fig. 3 (a4)) is very fast, compared with labeling
on general street-view images that are perspective. We extract a total of 150
rectified facade images from the CMP dataset.

Next, we employ the GSV API [1] to collect street-view photos. Here, we
randomize the camera headings to obtain perspective-oriented facades, and set
the pitch parameter in the API to zero, so the camera view directions are
always horizontal, _i.e._ , parallel to the ground. Also, for each GSV photo,
we identify nearby planar facades that are feasible for overlaying CMP facade
images, _i.e._ , with an aspect ratio similar to those in the CMP dataset.
Then, we manually annotate a quad to mark the region of each feasible planar
facades (Fig. 3 (b1)) and label also its orientation $\\{left,right,front\\}$
relative to the camera view direction. The facade priors serve as reference
locations for overlaying facade images and semantics. For instance, see Fig. 3
(b1), the blue quad marks the facade region of the left building with
orientation categorized as _front_ (_i.e._ , front facing the camera), whereas
the green quad marks the facade region of the right building with _right_
orientation. By doing so, we can derive an affine transformation matrix for
each facade, and use it to generate an augmented street-view image (Fig. 3
(b2)) with its associated semantic image (Fig. 3 (b3)) and labeled floor-
level-lines image (Fig. 3 (b4)).

To enrich the data sample diversity and to improve the network
generalizability, we collect 200 GSV images in Hong Kong and London, with both
high-rise and relatively low-rise buildings, respectively. Together with the
facade images from the CMP dataset, we compile a new dataset with 3,000 pairs
of augmented street-view images, each associated with facade semantics and
floor-level line labels. We train our _FloorLevel-Net_ using the dataset, and
compare it with an ablation model trained on the CMP dataset of only rectified
facades. The results show a significant boost by the data augmentation scheme
(Sec. VI-C).

Figure 4:  Characteristics of the data samples in our dataset: distributions
of (a) the facade orientation, (b) the highest floor order, and (c) average
number of pixels of each floor order in a single image.

###  IV-B Dataset Characterization

We conduct a preliminary analysis of our augmented dataset to show its strong
generalizability, and later take it to derive feasible components in our
network. By observing the following characteristics of the dataset, some hyper
parameters, _e.g._ , maximum floor order, are determined.

  * •

_Facade orientation_ : Fig. 4 (a) reports the distribution of facade
orientations. As mentioned in the previous subsection, three orientations
$\\{left,right,front\\}$ are considered based on the facade orientation
relative to the camera. The three orientations share similar proportions,
without biasing towards any of the three.

  * •

_Highest floor order_ : Fig. 4 (b) plots the distribution of the highest floor
order in the data samples. Most facades have a floor order of five. Also,
there is no single-floor building, since we omit single-floor buildings when
compiling the data. There are few high-rise buildings ($<5\%$) with the
highest floor order above 10.

  * •

_Average pixel number per floor order_ : Fig. 4 (c) presents the average
number of pixels belonging to each floor order in a single image of size
480$\times$360\. Since the street-view images are taken from the ground, low
floor orders have more pixels, and the number of pixels decreases as floor
order increases. Floor-level lines of orders above 10 are too small to overlay
AR images.

In consideration of the distribution of highest floor order and average pixel
number per floor order in the augmented dataset, we set the maximum floor
order to be 10 in our floor-level line detection module (Sec. V-A).

Figure 5: Overview of our _FloorLevel-Net_ framework. We consume each input
street-view image by a multi-task learning network to jointly segment the
facades and detect floor-level lines. In the module of our line feature
decoder, we incorporate height-attention layers (see bottom left) along with
the original convolutional layers, to help enforce the vertical ordering of
the floor-level lines, where “multi” denotes element-wise multiplication.

##  V FloorLevel-Net Approach

In this section, we first present the framework of _FloorLevel-Net_ (see Fig.
5), which is a multi-task learning network (Sec. V-A) that fully utilizes the
two-stream semantics of facades and floor-level lines, with a height-attention
mechanism (Sec. V-B) to enforce the vertical ordering of floor-level lines.
Then, we present the implementation details of _FloorLevel-Net_ (Sec. V-C),
followed by the add-on geometry post-processing module to generate the final
parameters for each floor-level line (Sec. V-D).

###  V-A Multi-task Learning

Based on the establishment of our augmented dataset, each street-view image
$\mathcal{I}\in\mathbb{R}^{W\times H\times 3}$ for training is coupled with
two label classes: (i) facade semantics $\Psi_{fa}$ and (ii) floor-level
distributions $\Psi_{fl}$. $\Psi_{fa}$ consists of $\\{$_window, door, shop,
left, right, front, other_ $\\}$, where the first three labels are the
context, the subsequent three indicate the facade orientation, and $other$
marks the non-facade pixels. $\Psi_{fl}$ consists of
$\\{l_{1},\cdots,l_{n},other\\}$, where $n$ denotes the highest floor order
(_i.e._ , 10) as suggested by the data characteristics (Sec. IV-B), and
$other$ marks the non-floor-level-lines pixels. The two-stream semantics
complement each other for the goal: $\Psi_{fa}$ provides rich context for
separating floor-level lines in each facade, whereas $\Psi_{fl}$ helps to
estimate the order and position of each floor-level line. This observation on
our training data inspires us to apply a multi-task learning process to
benefit each task.

As illustrated in Fig. 5, _FloorLevel-Net_ first employs a shared encoder to
learn a feature map, then it applies two-branch decoders to gradually upsample
the feature map via deconvolutional layers. The facade feature decoder in the
upper branch outputs segmentation mask
$\bar{\mathcal{M}}_{fa}\in\mathbb{R}^{W\times H}$, such that
$\bar{\mathcal{M}}_{fa}(\psi_{fa}|x)$ indicates the probability of pixel $x$
having label $\psi_{fa}\in\Psi_{fa}$. $\bar{\mathcal{M}}_{fa}$ is computed by

$\bar{\mathcal{M}}_{fa}(\psi_{fa}|x)=softmax(W_{fa}F(x)),$ (1)

where $F(x)$ is the feature vector of pixel $x$ and $W_{fa}$ denotes the
parameters learned by the network. We use the loss function
$\mathcal{L}_{fa}=\mathcal{C}e(\bar{\mathcal{M}}_{fa},\mathcal{M}_{fa})$ to
supervise the network training, where $\mathcal{M}_{fa}$ is the ground truth
map of facades and $\mathcal{C}e(\cdot)$ is a standard softmax cross entropy
function.

On the other hand, the line feature decoder in the lower branch of
_FloorLevel-Net_ outputs floor-level distribution map
$\bar{\mathcal{M}}_{fl}\in\mathbb{R}^{W\times H}$ for floor-level line
detection. Besides the softmax cross entropy
$\mathcal{C}e(\bar{\mathcal{M}}_{fl},\mathcal{M}_{fl})$, where
$\mathcal{M}_{fl}$ denotes the associated ground truth, we also fuse
$\mathcal{L}_{fa}$ here in the loss function for supervising the network
training:

$\mathcal{L}_{fl}=\mathcal{C}e(\bar{\mathcal{M}}_{fa},\mathcal{M}_{fa})+\mathcal{C}e(\bar{\mathcal{M}}_{fl},\mathcal{M}_{fl}).$
(2)

Incorporating $\mathcal{C}e(\bar{\mathcal{M}}_{fa},\mathcal{M}_{fa})$ in
$\mathcal{L}_{fl}$ contributes to the floor-level lines predictions, since
facade context such as windows can help infer the floor levels. Experimental
results also confirm that the fused loss helps to predict more continuous
segmentations and more accurate floor orders (Sec. VI-C).

Figure 6:  Height-related feature: Pixel distribution of floor-level lines in
vertical bounds of the image space.

###  V-B Height-Attention Mechanism

Observing that _“cars can’t fly in the sky”_ , _i.e._ , pixels of _car_ are
generally below those of the _sky_ , Choi et al. [8] suggest a height-
attention mechanism based on the analysis of the CityScape [9] dataset, which
indicates that the class distribution significantly depends on a vertical
position in urban scene. For instance, a lower part of an image is mainly
composed of road, while the upper part, buildings and sky are principal
objects. We hypothesis a similar pattern for pixel vertical distribution of
floor-level lines in our case, _i.e._ , pixels of order $i$ floor-level lines
are generally below those of order $j$ floor-level lines, given $i<j$. Here we
simplify the analysis by dividing the image space (from bottom to top) into
four equal-sized vertical bounds, _i.e._ , _low_ , _mid-low_ , _mid-high_ ,
and _high_ ; see the inset in Fig. 6. For each floor-level line in our data
samples, we count its pixels in each vertical bound, and compute its
distribution ratios in the four bounds. Then, we sum and normalize the
distribution ratios for floor-level lines of same order in the whole data, and
obtain the distribution ratios per floor-level line order. From Fig. 6, we can
see that pixels of low floor-level lines appear more in the lower vertical
bound (_e.g._ , 73% of floor order 1 pixels are in the _low_ bound), whilst
pixels of high floor-level lines have a higher chance of locating in the upper
vertical bound (_e.g._ , 53% of floor order 6 pixels are in the _high_ bound).

Next, we analyze the probabilities of floor-level line pixels in the whole
image and in each vertical bound. Table I presents the probability
distributions of the lowest six dominant floors orders, which take over 97% of
all floor orders (see Fig. 4 (c)). Here $p_{i}$ denotes the probability that
an arbitrary pixel is assigned to the $i$-th floor order. The results further
confirm the hypothesis of the pixel vertical distribution. For example, the
probability of floor order 1 $p_{1}$ is 9.75% in the whole image, and it drops
from 27.7% to 1.8% in vertical bounds from _low_ to _high_. On the contrary,
$p_{6}$ increases from 0.03% to 3.09%, which matches with the distribution in
Fig. 6. We further measure the uncertainty of pixel distribution probabilities
in separate bound by calculating the entropies as
$-\sum\mathit{p}_{i}log\mathit{p}_{i}$. The overall entropy of the entire
images is 0.436, and the _low_ bound has the smallest entropy of 0.298 due to
the dominant probability of floor order 1. The result indicates that if a
pixel falls in the _low_ bound, it shall probably be predicted as floor order
1 but not the other floor orders.

TABLE I: Probability distributions of pixels (in percentage) of the lowest six floor orders in the whole image and in each vertical bound. Given | Probabilities of the lowest six floor orders | Class
---|---|---
$p_{1}$ | $p_{2}$ | $p_{3}$ | $p_{4}$ | $p_{5}$ | $p_{6}$ | Entropy
Image | 9.75 | 11.8 | 9.56 | 4.90 | 2.83 | 1.17 | 0.436
Low | 27.7 | 10.1 | 1.71 | 0.35 | 0.12 | 0.03 | 0.298 |  0.389 (avg)
Mid-low | 6.75 | 20.1 | 8.72 | 2.97 | 1.09 | 0.32 | 0.386
Mid-high | 2.79 | 10.3 | 14.8 | 7.07 | 3.86 | 1.20 | 0.429
High | 1.80 | 6.66 | 13.0 | 9.20 | 6.22 | 3.09 | 0.442

These observations suggest that vertical position in image space can serve as
a good indicator of floor order. Hence, we are motivated to leverage height-
attention layers in the design of _FloorLevel-Net_. In detail, we include an
HA layer between adjacent convolutional layers in the line feature decoder;
see Fig. 5 (bottom left) for the illustration.

Specifically, the HA layer bridges a lower-level feature map of floor-level
lines $\bar{\mathcal{M}}^{l}_{fl}\in\mathbb{R}^{C_{l}\times W_{l}\times
H_{l}}$ and a higher-level feature map
$\bar{\mathcal{M}}^{h}_{fl}\in\mathbb{R}^{C_{h}\times W_{h}\times H_{h}}$
(note: subscripts $l$ and $h$ denotes lower-level and higher-level,
respectively) in the following ways: (i) adopts a width-wise pooling to
extract vertical features from $\bar{\mathcal{M}}^{l}_{fl}$; (ii) employs a
1-D convolutional layer with a bilinear interpolation to generate attention
map $\mathcal{A}_{fl}\in\mathbb{R}^{C_{h}\times H_{h}}$, which shares the same
channel and height size as $\bar{\mathcal{M}}^{h}_{fl}$; and (iii) generates a
refined higher-level feature map $\bar{\bar{\mathcal{M}}}_{fl}^{h}$ by an
element-wise multiplication of $\mathcal{A}_{fl}$ and
$\bar{\mathcal{M}}^{h}_{fl}$. In doing so, $\mathcal{A}_{fl}$ enriches the
per-channel scaling factors for each individual row of vertical positions, and
the refined higher-level feature map $\bar{\bar{\mathcal{M}}}_{fl}^{h}$
embodies the height-wise contextual information for locating and also ordering
the floor-level lines.

To study the effectiveness of the HA mechanism, we extract attention weights
learned by the last HA layer and compare them with floor-level distributions
in street-view images. Fig. 7 shows a typical comparison example using the
input image in the left. Here, we plot two heatmaps with floor orders as the
horizontal axis and image-space height as the vertical axis. The heatmaps are
constructed in the following way. Horizontal axes denote floor order from 1 to
10, whereas vertical axes denote vertical position (height) sub-divided into
10 vertical bounds for visualization. For each cell $(m,n)$, we count the
pixels in floor-level $m$ (middle) or extract the attention weight of the
$m$-th channel (right), inside the $n$-th vertical bound. Also, we normalize
the intensities (in red) in each map. Comparing the two heatmaps, we can see
that they exhibit very similar patterns. Particularly, if we mark the most
frequent height(s) per floor order (see the blue boxes) in both plots, we can
see that the attention weights reveal the heights of the pixels in floor-level
lines of different orders. The comparison together with the ablation analysis
presented in Sec. VI-C demonstrate the effectiveness of the HA mechanism.

Figure 7:  Left is an example input image with annotated floor orders. Middle
plots the floor-level line pixel distributions in the input image, whereas
right plots the attention weights in the last height-attention layer.  Figure
8:  Geometry-constrained post-processing for retrieving line parameters:
grouping floor-level lines based on facade segmentation (a), fitting polylines
according to line predictions (b), refining vanishing point using geometry
constraints (c), and the final output (d).

###  V-C Implementation of FloorLevel-Net

_FloorLevel-Net_ adopts an encoder-decoder structure based on DeepLabV3+ [6]
with ResNet101 [17] as the backbone. Specifically, we employ four ResNet
stages and one atrous spatial pyramid pooling (ASPP) layer to generate the
shared features, and use five convolutional layers in each of the decoders for
facade segmentation and floor-level lines detection. In the line feature
decoder, we arrange one height-attention layer between each pair of adjacent
convolutional layer, and append a residual layer at the end to fuse the
features from the facade feature decoder. We use ReLU for all layers, except
for the final prediction layers, where softmax is applied. We implement
_FloorLevel-Net_ using PyTorch and train it from scratch on a single NVidia
GeForce GTX 1080 Ti GPU card. During the training, we adopt the momentum
optimizer with learning rate 1e-3 and a small batch size of four. Convergence
is reached at about 100K iterations.

###  V-D Geometry Post-Processing

_FloorLevel-Net_ predicts two piecewise segmentation masks
$\bar{\mathcal{M}}_{fa}$ and $\bar{\mathcal{M}}_{fl}$, from which we further
extract line parameters by considering various geometric constraints:

  * (i)

Facade constraint: Floor-level lines are attached to facades, so the detected
lines are valid only if they lie inside a facade region. Since
$\bar{\mathcal{M}}_{fa}$ may contain multiple facades, we process floor-level
line segmentations and generate only one line with a specific floor order per
facade.

  * (ii)

Vanishing point (VP) constraint: Floor-level lines of the same facade are
intrinsically parallel in 3D, so they should meet at a common VP, which is at
a finite location for perspective-oriented facades.

  * (iii)

Order constraint: Assuming that the up direction in input image roughly
matches the up direction in real world, orders of floor-level lines for the
same facade should strictly increase from bottom to top.

To enforce these constraints, we formulate a second-stage geometry post-
processing to generate floor-level line parameters with the following steps
(as illustrated in Fig. 8):

  * (i)

Line grouping (Fig. 8 (a)): First, we group relevant labels in
$\bar{\mathcal{M}}_{fa}$ to form a super-segment per facade. Then we group
floor-level lines segmentation in $\bar{\mathcal{M}}_{fl}$, and identify a
group of lines per facade.

  * (ii)

Polyfit (Fig. 8 (b)): We fit a polyline (in the form of
$\beta_{0}^{i}+\beta_{1}^{i}x$) for pixels of each floor-level line (say
$\\{(x^{i}_{j},y^{i}_{j})\\}^{n}_{j=1}$ for $i$-th line) using a least squares
regression:

$\underset{\beta_{0}^{i},\beta_{1}^{i}}{\arg\min}\\{\sum_{j=1}^{n}(y^{i}_{j}-(\beta_{0}^{i}+\beta_{1}^{i}x^{i}_{j}))^{2}\\}.$
(3)

Note that on the same facade, there could be disjoint predicted regions of the
same floor order; we have to merge these regions before the regression.

  * (iii)

Refinement (Fig. 8 (c)): Polylines derived for the same facade may not
perfectly meet at a common VP, so we regress location $(x_{c},y_{c})$ for the
common $VP_{c}$ and refine each polyline as $\beta^{i}(x-x_{c})+y_{c}$ by
taking $VP_{c}$ as an anchor point. To do so, we formulate a gradient-based
optimization to update parameters $\\{\beta^{i}\\}_{i=1}^{m}$, $x_{c}$,
$y_{c}$ with a global least squares:

$\underset{\\{\beta^{i}\\},x_{c},y_{c}}{\arg\min}\\{\sum_{i=1}^{m}\sum_{j=1}^{n}(y^{i}_{j}-\beta^{i}(x-x_{c})-y_{c})^{2}\\}.$
(4)

  * (iv)

Final result (Fig. 8 (d)): Last, we take the horizontal and vertical ranges of
each facade, to derive the two endpoints ($x_{s}^{i},\,y_{s}^{i}$) and
($x_{e}^{i},\,y_{e}^{i}$) for each polyline. Together with the floor order
$l^{i}$, we can then obtain the five-tuple
$\\{x^{i}_{s},\,y^{i}_{s},\,x^{i}_{e},\,y^{i}_{e},\,l^{i}\\}$ for each floor-
level line.

##  VI Experiment

To the best of our knowledge, this work is the first attempt at recognizing
floor-level lines in street-view images. Hence, there are no benchmark
datasets, evaluation metrics, and existing methods for the task. We fill the
gaps by preparing a new dataset (Sec. VI-A), proposing quantitative metrics
for evaluation (Sec. VI-B), and comparing with ablated techniques (Sec. VI-C)
and some closely-related techniques (Sec. VI-D and Sec. VI-E). Further, we
present some AR-related applications to demonstrate the applicability of this
work (Sec. VI-F).

###  VI-A Testing Dataset

Besides the new training dataset prepared by using our data augmentation
scheme (see Sec. IV),nwe prepare a new testing dataset for evaluating the
effectiveness and generalizability of our approach. Here, we choose street-
view images by considering the followings: (i) the image should contain at
least one (nearby) facade, which is not fully occluded; (ii) the facades
should contain at least one floor-level line; (iii) the facades can be
perspective-oriented in view but cannot be curved in the physical world; and
(iv) the facades should contain some semantic elements,  _e.g._ , windows or
balconies. We collect 600 street-view images for the testing dataset from the
following four different sources:

  * •

We randomly select 150 street views from the eTrims [21] and arcDataset [43].
Each image includes only one single facade, and the buildings are mostly low-
rise.

  * •

We randomly select 150 more images from the LabelMeFacade [12], TSG-20 [2] and
ZuBuD+ [32] datasets. These images have more high-rise buildings, and some
have multiple facades.

  * •

We randomly download 200 GSV images in London and Glasgow (UK), which feature
typical European-style buildings, similar to those in the CMP dataset.

  * •

Lastly, we randomly download another 100 GSV images in Hong Kong (HK). These
images feature high-rise buildings, and many facades are partially occluded by
billboards, advertisements, etc.

On each test image, we manually label the position and order of each floor-
level line on the facades in the form of a quadrilateral region,  _e.g._ , see
the “GT” row in Fig. 9. It takes around 2 minutes to label one image and
around 20 hours in total for all the 600 images. The datasets are available on
the project website: https://wumengyangok.github.io/Project/FloorLevelNet/.

###  VI-B Evaluation Metrics

As a two-stage method, our results include (i) predicted pixel-wise
segmentations from _FloorLevel-Net_ , and (ii) regressed endpoints
$(x^{i}_{s},y^{i}_{s})$ & $(x^{i}_{e},y^{i}_{e})$ and order $l^{i}$ of each
floor-level line from the geometry post-processing. We employ the following
metrics to evaluate the two results separately:

  * •

Pixel-wise accuracy. For each street-view image, we denote the set of pixels
in ground-truth and in predicted line regions as $\mathcal{P}_{gt}$ and
$\mathcal{P}_{pred}$, respectively. We employ the $F_{1}$ score for
quantitative comparisons, which can be measured upon the number of true
positives (TP = $|\mathcal{P}_{gt}\cap\mathcal{P}_{pred}|$), false positives
(FP = $|\mathcal{P}_{pred}-\mathcal{P}_{gt}\cap\mathcal{P}_{pred}|$), and
false negatives (FN =
$|\mathcal{P}_{gt}-\mathcal{P}_{gt}\cap\mathcal{P}_{pred}|$):

$F_{1}=(2TP)/(2TP+FP+FN).$ (5)

  * •

Line-wise accuracy. We join the endpoints of each floor-level line $L_{i}$ to
form a three-pixel wide straight line, yielding a bag of pixels
$\mathcal{P}_{i}$. Then, we can compute the confidence of $L_{i}$ as being
correctly recognized as

$CI(L_{i})=\frac{\sum_{p\in\mathcal{P}_{i}}{\mathbb{1}(\mathcal{M}_{fl}(p)=l^{i})}}{|\mathcal{P}_{i}|},$
(6)

where $\mathbb{1}(\cdot)$ is an indicator function for counting the number of
pixels in $P_{i}$ that appears on the ground truth image $\mathcal{M}_{fl}$
with label $l^{i}$. If $CI(L_{i})>0.5$, we regard the line prediction as a
true positive. For each street-view image, we again count the number of floor-
level lines as being correctly detected (TP), missed (FN), and wrongly
recognized (FP), then employ the $F_{1}$ score (like Eq. (5)) to compute the
line-wise accuracy.

Figure 9:  Qualitative comparison results in the ablation analysis on the
unseen testing dataset. The results illustrate the effects of (i) our data
augmentation scheme by comparing DeeplabV3+ models trained on CMP and on our
training data (green background), (ii) multi-task learning with and without
the height-attention mechanism (red background), and (iii) our full method
further with geometry post-processing (yellow background), vs. the ground
truths (GT).

###  VI-C Ablation Analysis

We carry out an ablation analysis to evaluate the major components in our
approach, including (i) the data augmentation scheme, (ii) the multi-task
learning, (iii) the height-attention mechanism in _FloorLevel-Net_ , and (iv)
the geometry-post-processing. In the analysis, we employ the testing dataset,
in which the images do not appear in the training data.

Ablated techniques. To evaluate components in _FloorLevel-Net_ , we take a
stepwise testing strategy by adding the components one by one to the baseline
DeepLabV3+ [6] model until we have our full architecture. The difference
between consecutive tests indicates the performance enhancement made by the
added component. To start, we evaluate our data augmentation scheme by the
following pair of experiments:

  * •

_Baseline+CMP_ : We train a DeepLabV3+ model (baseline) using the facade
dataset CMP [37] with rectified facades (Fig. 3 (a1)) and our labels of floor-
level lines (Fig. 3 (a4)).

  * •

_Baseline+DataAugm_ : We train another DeepLabV3+ model on the training
dataset of 3,000 pairs of augmented street-view images (Fig. 3 (b2)) and
labels of floor-level lines (Fig. 3 (b4)), which are produced by our data
augmentation scheme, as described in Sec. IV-A.

Next, we evaluate the multi-task learning framework and height-attention
mechanism in _FloorLevel-Net_ by adding these components to
_Baseline+DataAugm_ , as follows:

  * •

_Multi-task_ : This refers to the multi-task learning network that we build to
simultaneously predict facades and floor-level lines, as described in Sec.
V-A.

  * •

_Multi-task+HA_ : We further incorporate a set of height-attention layers, as
described in Sec. V-B, to form the full architecture of _FloorLevel-Net_.

Last, we arrive at _Our Full Method_ by including the second-stage geometry-
post-processing to regress floor-level lines.

Qualitative analysis. Fig. 9 shows the qualitative comparison results. Here,
we select one or two street-view images from each constituting component in
our testing dataset: eTrims (1st column), arcDataset (2nd column), TSG-20 (3rd
column), LabelMeFacade (4th column), ZuBuD+ (5th column), GSVs in UK (6th &
7th columns), and GSVs in Hong Kong (8th & 9th columns). The top row shows the
input images, the bottom row shows the ground-truth images, whereas each of
the other rows from the top show the prediction results obtained by the
ablated methods,  _i.e._ , _Baseline+CMP_ , _Baseline+DataAugm_ , _Multi-task_
, _Multi-task+HA_ , and _Our Full Method_.

Overall, _Baseline+DataAugm_ generates better predictions than _Baseline+CMP_
(comparing 2nd & 3rd rows in Fig. 9), especially for street views with
multiple facades and for facades that are perspective-oriented,  _e.g._ , the
GSV images in UK & Hong Kong. The results indicate the effectiveness of our
data augmentation scheme in generating augmented street-view images for
network training.

Next, comparing 3rd & 4th rows Fig. 9, we can see that _Multi-task_ is able to
produce more continuous and distinctive segmentations of floor-level lines
than _Baseline+DataAugm_. This shows that exploring the rich semantics in
facades indeed facilitates the recognition of floor-level lines. Further, we
can see that the full architecture of _FloorLevel-Net_ with the height-
attention mechanism (5th row in Fig. 9) can produce even better results,
especially in the prediction of floor orders. Particularly, the full
architecture of _FloorLevel-Net_ successfully predicts the floor orders, even
for challenging street-view images shown in the 4th to 9th columns, whilst the
method fails in the absence of the height-attention mechanism. Better
predictions help simplify the line generation and refinement in the subsequent
geometry post-processing stage. Benefiting from the fine predictions by
_FloorLevel-Net_ , the post-processing stage (6th row in Fig. 9) is able to
deduce satisfactory floor-level line parameters, subject to the various
geometric constraints we discussed earlier.

TABLE II: Quantitative comparison with the ablated techniques in terms of pixel-wise accuracy on network prediction results, and line-wise accuracy on geometry post-processing results. Results colored in blue are associated with our full method. |  | Lower floors | Upper floors | Overall
---|---|---|---|---
|  | Pixel-wise | Line-wise | Pixel-wise | Line-wise | Pixel-wise | Line-wise
|  | Accuracy | Accuracy | Accuracy | Accuracy | Accuracy | Accuracy
Facades & buildings | Baseline + CMP | 0.374 | 0.725 | 0.136 | 0.676 | 0.289 | 0.702
Baseline + DataAugm | 0.599 | 0.887 | 0.479 | 0.828 | 0.560 | 0.859
Multi-task | 0.631 | 0.912 | 0.498 | 0.845 | 0.586 | 0.880
Multi-task + HA | 0.676 | 0.926 | 0.586 | 0.860 | 0.644 | 0.894
GSV in UK & HK | Baseline CMP | 0.152 | 0.592 | 0.114 | 0.486 | 0.133 | 0.524
Baseline + DataAugm | 0.530 | 0.781 | 0.449 | 0.708 | 0.505 | 0.742
Multi-task | 0.555 | 0.819 | 0.491 | 0.734 | 0.535 | 0.774
Multi-task + HA | 0.636 | 0.854 | 0.531 | 0.749 | 0.605 | 0.798

Quantitative analysis. Further, we conduct a quantitative evaluation for each
ablated technique. Here, we categorize the testing dataset into two groups:
images from datasets focusing on buildings or facades (eTrims, LabelmeFacade,
etc.) are relatively easier for recognizing floor-level lines as the images
are prepared for facade parsing or building classification task with regular
facade elements, whereas GSVs in UK & HK with complex environments and dynamic
attributes are harder for recognition. We would also like to evaluate the
performance of our approach for different floor-level lines, as many AR
applications need only floor information close to the camera. To do so, we
categorize floors 1-3 as lower floors since most of these floor pixels are in
the low bound of image space, and the remaining floors as upper floors.

Figure 10: Comparison between our approach and DeepFacade-Variant on
orthogonal (left two columns) and perspective (right two columns) facades.
Both methods produce satisfactory results for facades that are orthogonal to
camera view, whilst our approach is more robust to facades in perspective
view, and multiple facades in one image (last row).  Figure 11: The potential
of our approach to support and enrich various AR scenarios,  _e.g._ ,
navigation, advertisement, etc.

We measure the pixel-wise accuracy on the ablation network outputs, and line-
wise accuracy on the line parameters after geometry post-processing. The
quantitative results are presented in Table II, in which our full method,
_Multi-task + HA_ with geometry post-processing, is marked in blue. Based on
the results, we can derive some interesting observations. First, the overall
accuracy drops when scene complexity increases from simple street views in
facade datasets to GSVs in UK and HK. There are more facades and higher floors
in GSVs, especially for those in HK, resulting in less accurate predictions.
Second, predictions for lower floors have better accuracy than those for
higher floor. This is probably because of relatively weaker supervisions for
upper floor-level lines, as upper floor-level lines are typically further away
from the camera than the lower ones and there are relatively less contextual
information. Third, the data augmentation scheme, multi-task learning, and
height-attention layers gradually contribute to the performance enhancement,
yet there are still much room for improvement in terms of pixel-wise accuracy.
The result indicates that floor-level line recognition is more challenging
compared with general urban scene segmentation tasks. Nevertheless, our
predictions provide enough information for the geometry post-processing, which
can correctly recognize and locate most floor-level lines in terms of line-
wise accuracy.

###  VI-D Comparison with a Variant of DeepFacade

As there are no previous works on locating floor-level lines, we consider
those on facade parsing,  _i.e._ , segmenting a building facade into windows,
doors, etc., which can then serve as references for locating floor-level
lines, as competitors. Here, we compare with DeepFacade [24], which is a
state-of-the-art deep-learning-based method for facade parsing. In detail, we
extract floor-level lines from their results as follows: (i) use RANSAC [5] to
rectify a street-view image and record the associated homographic matrix; (ii)
apply DeepFacade [24] on the rectified image to produce a facade segmentation
map, and extract only the window, door, and shop regions as positive regions
in the map; (iii) mark a floor-level line in-between each pair of adjacent
(piecewise) positive regions; and (iv) apply the inverse of the homographic
matrix to revert the original positions of the detected lines to obtain the
final results. From now, we refer the above steps as _DeepFacade-Variant_.

This experiment compares _Our Full Method_ with _DeepFacade-Variant_. However,
we find that _DeepFacade-Variant_ can hardly produce satisfactory results for
GSVs in UK and HK (see supplemental material for examples) for two reasons:
(i) RANSAC rectification is constrained to street views with only a single
facade, but street views in UK and HK often contain multiple facades, leading
to multiple vanishing points detected and unstable rectification results; (ii)
buildings in modern cities may contain facades without obvious line segments,
thereby making it hard to deduce the vanishing points and perform the
rectification. Hence, we report the results only on 300 testing images from
the eTrims, arcDataset, LabelMeFacade, TSG-20, and ZuBuD+ datasets, which
contain only one building facade in most of the images.

Fig. 10 shows typical results produced by our approach and _DeepFacade-
Variant_. From the results shown on the left (1st & 2nd columns), we can see
that both methods produce good results for facades that are orthogonal to
camera view. Yet, _DeepFacade-Variant_ is prone to fail when the improper
homographic matrix inferred by RANSAC (see row 2 for an example), whilst our
approach overcomes the limitation with the refinement made by geometry
constraints. More failure cases for _DeepFacade-Variant_ are presented in 3rd
& 4th columns, where the facades are in perspective view. Input image of row 1
is not rectified properly, hence the interpolated lines point to wrong
directions. In rows 2, windows on up floors have little saliency due to
distortion, thus the results miss floor-level lines on high levels. The
building in row 3 has partial irregular facade textures, such as lintel and
stairs, and the method wrongly recognizes adjacent floors. In the last row,
there are two facades, whilst the rectification cannot be managed by one
single homographic matrix. In all these cases, our approach produces good
predictions.

TABLE III: Quantitative comparison between our approach and _DeepFacade-
Variant_ in terms of accuracy and time efficiency.

| Line-wise | Recti- | Network | Post- | SUM
---|---|---|---|---|---
| accuracy | fication | inference | processing
DeepFacade-Variant | 0.721 | 1.08s | 0.06s | 0.04s | 1.18s
Ours | 0.876 | - | 0.05s | 0.14s | 0.19s

Table III quantitatively compares the results of _DeepFacade-Variant_ and _Our
Full Method_. From the line-wise accuracies, we can see an obvious advancement
of our approach over the competitor. Also, our approach outperforms
_DeepFacade-Variant_ in terms of execution time, in which the total running
time is only 0.19 seconds vs. 1.18 seconds. In _DeepFacade-Variant_ , a major
bottleneck is the heavy time cost of RANSAC rectification, whilst our approach
directly predicts pixelwise segmentation without requiring rectification.

###  VI-E Comparison with Semantic Segmentation

As the pixel-wise floor segmentation can be formulated as an image
segmentation task, we also present comparison with semantic segmentation
methods on the pixel accuracy of the intermediate results. Here we choose
PSPNet [52] and OCR [47] as baselines, since PSPNet is widely used as baseline
for semantic segmentation task, and OCR is one of the state-of-the-art methods
with highest pixel accuracy in some urban semantic segmentation datasets. For
each method, we train the model with both CMP dataset and our augmented
dataset, and test with 600 real images following the same settings as the
ablation study in Sec. VI-C.

Table IV presents quantitative results in terms of pixel-wise accuracy.
Similar to the results of ablation analysis, the low overall pixel accuracy by
the comparison methods trained on CMP dataset indicates the difficulty of
inferring floor-level lines, especially without proper dataset. Our data
augmentation significantly improves the prediction accuracy to an acceptable
level for both segmentation methods, showing well generalizability of the
baseline network architectures. Moreover, our methods with multi-task learning
and height-attention mechanism achieves the best performance, for both the
facades & buildings datasets and GSVs in UK & HK.

TABLE IV: Quantitative comparison between our method and image segmentation baselines in terms of pixel-wise accuracy.  Datasets | Segmentation methods | Lower | Upper | Overall
---|---|---|---|---
Facades & buildings | PSPNet + CMP | 0.258 | 0.059 | 0.197
PSPNet + DataAugm | 0.649 | 0.528 | 0.609
OCR + CMP | 0.379 | 0.254 | 0.330
OCR + DataAugm | 0.610 | 0.492 | 0.570
Ours | 0.676 | 0.586 | 0.644
GSV in UK & HK | PSPNet + CMP | 0.081 | 0.001 | 0.067
PSPNet + DataAugm | 0.369 | 0.205 | 0.321
OCR + CMP | 0.083 | 0.016 | 0.062
OCR + DataAugm | 0.626 | 0.474 | 0.581
Ours | 0.636 | 0.531 | 0.605

###  VI-F Applicability

Fig. 11 shows some example usage scenarios that demonstrates the potential of
our method in enriching AR-related applications. Enabled by the recognized
floor-level lines, we can augment real-world scenes with virtual contents in a
floor-aware manner. First, from the floor-level line results, we can identify
the bound of each facade,  _i.e._ , the lowest and highest floor-level lines
(and their locations), and overlay virtual contents that cover the whole
facade, while skipping the real contents on the ground level (Fig. 11 (top)).
By this means, we may deliver AR contents, without obscuring necessary real
contents around the user on the street.

More importantly, with the floor-level lines, we can place context-aware
information on specific floors. For example, in the bottom-row result on the
2nd column of Fig. 11, we specifically mark the location of the target room
for convenient navigation; and in the middle-row result on the 3rd column of
Fig. 11, we put shopping directory information over the corresponding floors
to aid users to find the user’s target items in the department store. Besides,
we may mark specific apartments in a building that are for sales or for rent
to aid the property agents. On the other hand, we may also employ our approach
to present floor-related data (_e.g._ , floor size, occupancy information,
etc.) on buildings and extend it to estimate the number of floor levels and
infer the building heights, which could be helpful for land usage analysis and
urban planning [33, 42, 48].

Figure 12: Limitations of our current approach: obstacle interference (left),
glass reflection (middle), and facades too close to camera (right).

##  VII Conclusion, Discussion, and Future Work

We presented a new approach for recognizing floor-level lines in street-view
images. As a first attempt to this challenging task, we contribute to (i)
devising a new data augmentation scheme that leverages an existing facade
dataset to efficiently generate data samples for network training; and (ii)
formulating _FloorLevel-Net_ , a new multi-task learning framework that
associates explicit features of facades and implicit floor-level lines and
incorporates a height-attention mechanism to enforce a vertical ordering of
floor-level lines. The pixelwise semantic segmentations by _FloorLevel-Net_
are further refined through a geometry post-processing module for plausible
and consistent reconstruction of floor-level lines. Quantitative and
qualitative comparisons with existing methods demonstrate the effectiveness of
our proposed method. Further, we demonstrate the potential of our approach in
supporting various AR scenarios.

_Discussion._ This work targets at inferring floor-level lines in street-view
images. We proposed the data augmentation scheme and _FloorLevel-Net_ to
overcome the challenges of being lack of _training dataset_ and _suitable
network architecture_. Though the methods are dedicated to the specific task,
we believe the ideas of augmenting dataset and tuning network based on data
characteristics can be feasibly extended to some other applications, where
labeling is expensive and data have certain patterns. For instance, in
autonomous driving, a key requirement is to detect driving lanes and
pedestrians simultaneously. In such scenarios, we can possibly improve the
network performance by carefully modeling the relationship between driving
lane and pedestrian.

_Limitations._ First, our method could be disturbed by obstacles at critical
regions,  _e.g._ , a cab that occludes the ground floor of a building (Fig. 12
(left)) may cause a wrongly-recognized ground floor. Second, our method cannot
handle facades that are reflective (Fig. 12 (middle)). Third, when the camera
is too close to the facades (Fig. 12 (right)), our method cannot get a wide
view of the facades, so it may wrongly recognize the upper and lower bounds of
a floor-level line region as two successive floor-lever lines.

_Future works._ We plan to focus on improving the prediction precision and
efficiency of our approach. First, a possible direction is to work on short
video clips instead of on single images, so that we can obtain temporal
information,  _e.g._ , structure from motion (SfM) features, to improve the
prediction precision. Second, we plan to develop strategies to estimate the
distance from camera to facades to address the last limitation mentioned
above. Third, our current pipeline takes around 0.19 seconds (see Table III),
which is not slow compared with DeepFacade-Variant but insufficient to run in
real-time. At present, the main bottleneck is the post-processing module,
which can be improved through multiprocessing implementations, since we can
simultaneously process line grouping and polyfit. Also, we plan to explore
lightweight networks to further improve the pipeline efficiency.

## Acknowledgment

We thank the street-view images from the Google Street View service. This work
is supported partially by the Research Grants Council of the Hong Kong Special
Administrative Region (Project no. CUHK 14206320) and Guangdong Basic and
Applied Basic Research Foundation (2021A1515011700).

## References

  * [1] Google’s street view static API.  https://developers.google.com/maps/documentation/streetview.  Accessed: 2020-10-01.
  * [2] Tourist sights graz image database.  http://dib.joanneum.at/cape/TSG-20/.  Accessed: 2021-04-12.
  * [3] D. Anguelov, C. Dulong, D. Filip, C. Frueh, S. Lafon, R. Lyon, A. Ogale, L. Vincent, and J. Weaver.  Google street view: Capturing the world at street level.  Computer, 43(6):32–38, 2010.
  * [4] O. Barinova, V. Konushin, A. Yakubenko, K. Lee, H. Lim, and A. Konushin.  Fast automatic single-view 3-D reconstruction of urban scenes.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 100–113, 2008.
  * [5] K. Chaudhury, S. DiVerdi, and S. Ioffe.  Auto-rectification of user photos.  In Proceedings of the IEEE International Conference on Image Processing (ICIP), pages 3479–3483, 2014.
  * [6] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 801–818, 2018.
  * [7] Z. Chen, J. Zhang, and D. Tao.  Recursive context routing for object detection.  International Journal of Computer Vision, 129(1):142–160, 2021\.
  * [8] S. Choi, J. T. Kim, and J. Choo.  Cars can’t fly up in the sky: Improving urban-scene segmentation via height-driven attention networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9373–9383, 2020.
  * [9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.  The Cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3213–3223, 2016.
  * [10] A. Criminisi, I. Reid, and A. Zisserman.  Single view metrology.  International Journal of Computer Vision, 40(2):123–148, 2000.
  * [11] D. Eigen and R. Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2650–2658, 2015.
  * [12] B. Fröhlich, E. Rodner, and J. Denzler.  A fast approach for pixelwise labeling of facade images.  In Proceedings of the International Conference on Pattern Recognition (ICPR), pages 3029–3032, 2010.
  * [13] A. Geiger, P. Lenz, and R. Urtasun.  Are we ready for autonomous driving? The KITTI vision benchmark suite.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3354–3361, 2012.
  * [14] R. Grompone von Gioi, J. Jakubowicz, J.-M. Morel, and G. Randall.  LSD: a Line Segment Detector.  Image Processing On Line, 2:35–55, 2012.
  * [15] A. Gupta, A. A. Efros, and M. Hebert.  Blocks world revisited: Image understanding using qualitative geometry and mechanics.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 482–496, 2010.
  * [16] O. Haines and A. Calway.  Recognising planes in a single image.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):1849–1861, 2015.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [18] D. Hoiem, A. A. Efros, and M. Hebert.  Recovering surface layout from an image.  International Journal of Computer Vision, 75(1):151–172, 2007.
  * [19] J. Hu, L. Shen, and G. Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7132–7141, 2018.
  * [20] Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu.  CCNet: Criss-cross attention for semantic segmentation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 603–612, 2019.
  * [21] F. Korč and W. Förstner.  eTRIMS Image Database for interpreting images of man-made scenes.  Technical Report TR-IGG-P-2009-01, April 2009.
  * [22] J.-T. Lee, H.-U. Kim, C. Lee, and C.-S. Kim.  Semantic line detection and its applications.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3229–3237, 2017.
  * [23] H. Liu, Y. Xu, J. Zhang, J. Zhu, Y. Li, and C. H. S. Hoi.  DeepFacade: A deep learning approach to facade parsing with symmetric loss.  IEEE Transactions on Multimedia, pages 1–1, 2020.
  * [24] H. Liu, J. Zhang, J. Zhu, and S. C. H. Hoi.  DeepFacade: A deep learning approach to facade parsing.  In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 2301–2307, 2017.
  * [25] X. Liu, Y. Zhao, and S. Zhu.  Single-view 3D scene reconstruction and parsing by attribute grammar.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3):710–725, 2018.
  * [26] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431–3440, 2015.
  * [27] A. Martinović, M. Mathias, and L. Van Gool.  ATLAS: A three-layered approach to facade parsing.  International Journal of Computer Vision, 118(1):22–48, 2016.
  * [28] P. Mueller, G. Zeng, P. Wonka, and L. Van Gool.  Image-based procedural modeling of facades.  ACM Trans. Graph. (SIGGRAPH), 26(3):85:1–85:10, 2007.
  * [29] P. Musialski, P. Wonka, D. G. Aliaga, M. Wimmer, L. Van Gool, and W. Purgathofer.  A survey of urban reconstruction.  Computer Graphics Forum (Eurographics), 32(6):146–177, 2013.
  * [30] Z. Ren and Y. J. Lee.  Cross-domain self-supervised multi-task feature learning using synthetic imagery.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 762–771, 2018.
  * [31] G. Schindler, P. Krishnamurthy, R. Lublinerman, Y. Liu, and F. Dellaert.  Detecting and matching repeated patterns for automatic geo-tagging in urban environments.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–7, 2008.
  * [32] H. Shao, T. Svoboda, and L. Van Gool.  Zubud-zurich buildings database for image based recognition.  Computer Vision Lab, Swiss Federal Institute of Technology, Switzerland, Tech. Rep, 260(20):6, 2003.
  * [33] Q. Shen, W. Zeng, Y. Ye, S. Mueller Arisona, S. Schubiger, R. Burkhard, and H. Qu.  StreetVizor: Visual exploration of human-scale urban forms based on street views.  IEEE Transactions on Visualization and Computer Graphics, 24(1):1004 – 1013, 2018.
  * [34] G. Simon, A. Fond, and M.-O. Berger.  A-contrario horizon-first vanishing point detection using second-order grouping laws.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 323–338, 2018.
  * [35] O. Teboul, I. Kokkinos, L. Simon, P. Koutsourakis, and N. Paragios.  Shape grammar parsing via reinforcement learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2273–2280, 2011.
  * [36] O. Teboul, L. Simon, P. Koutsourakis, and N. Paragios.  Segmentation of building facades using procedural shape priors.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3105–3112, 2010.
  * [37] R. Tyleček and R. Šára.  Spatial pattern templates for recognition of objects with regular structure.  In Proceedings of German Conference on Pattern Recognition (GCPR), pages 364–374, 2013.
  * [38] X. Wang, D. Fouhey, and A. Gupta.  Designing deep networks for surface normal estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 539–547, 2015.
  * [39] J. Weissenberg, H. Riemenschneider, M. Prasad, and L. Van Gool.  Is there a procedural logic to architecture?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 185–192, 2013.
  * [40] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon.  CBAM: Convolutional block attention module.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19, 2018.
  * [41] C. Wu, J.-M. Frahm, and M. Pollefeys.  Repetition-based dense single-view reconstruction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3113–3120, 2011.
  * [42] Y. Wu, L. S. Blunden, and A. S. Bahaj.  City-wide building height determination using light detection and ranging data.  Environment and Planning B: Urban Analytics and City Science, 46(9):1741–1755, 2018.
  * [43] Z. Xu, D. Tao, Y. Zhang, J. Wu, and A. C. Tsoi.  Architectural style classification using multinomial latent logistic regression.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 600–615, 2014.
  * [44] C. Yang, T. Han, L. Quan, and C. Tai.  Parsing façade with rank-one approximation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1720–1727, 2012.
  * [45] F. Yang and Z. Zhou.  Recovering 3D planes from a single image via convolutional neural networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 87–103, 2018.
  * [46] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang.  Learning a discriminative feature network for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1857–1866, 2018.
  * [47] Y. Yuan, X. Chen, and J. Wang.  Object-contextual representations for semantic segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 173–190, 2020.
  * [48] W. Zeng and Y. Ye.  VitalVizor: A visual analytics system for studying urban vitality.  IEEE Computer Graphics and Applications, 38:38–53, 09 2018.
  * [49] Z. Zeng, M. Wu, W. Zeng, and C.-W. Fu.  Deep recognition of vanishing-point-constrained building planes in urban street views.  IEEE Transactions on Image Processing, 29:5912–5923, 2020.
  * [50] J. Zhang, Y. Xu, B. Ni, and Z. Duan.  Geometric constrained joint lane segmentation and lane boundary detection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 502–518, 2018.
  * [51] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2881–2890, 2017.
  * [52] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

|  Mengyang Wu received the B.S. degree from University College London. He is
currently a PhD student in Computer Science and Engineering of Chinese
University of Hong Kong. His recent research interests include deep learning
for 3D vision, scene understanding, and outdoor augmented reality.
---|---
|  Wei Zeng is currently an associate professor at Shenzhen Institute of
Advanced Technology, Chinese Academy of Sciences. He received his bachelor’s
degree (2011) and Ph.D. (2015), both in computer science, from Nanyang
Technological University. He served as co-chairs for PacificVis Poster, and
program committee members for various research conferences, including IEEE
Visualization, ChinaVis, IVAPP. His recent research interests include data
visualization and AR/VR, and their applications in smart city.
---|---
|  Chi-Wing Fu is currently an associate professor in the Chinese University
of Hong Kong. He served as the co-chair of SIGGRAPH ASIA 2016’s Technical
Brief and Poster program, associate editor of IEEE Computer Graphics &
Applications and Computer Graphics Forum, panel member in SIGGRAPH 2019
Doctoral Consortium, and program committee members in various research
conferences, including SIGGRAPH Asia Technical Brief, SIGGRAPH Asia Emerging
tech., IEEE visualization, CVPR, IEEE VR, VRST, Pacific Graphics, GMP, etc.
His recent research interests include computation fabrication, point cloud
processing, 3D computer vision, user interaction, and data visualization.
---|---