# Disentangled Latent Transformer for Interpretable Monocular Height
Estimation

Zhitong Xiong, Sining Chen, Yilei Shi, , and Xiao Xiang Zhu  Z. Xiong, S. Chen
and X. X. Zhu are with the Data Science in Earth Observation (SiPEO, formerly:
Signal Processing in Earth Observation), Technical University of Munich (TUM),
80333 Munich, Germany. S. Chen and X. X. Zhu are also with the Remote Sensing
Technology Institute (IMF), German Aerospace Center (DLR), 82234 Weßling,
Germany (e-mails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Y. Shi is with the Chair of Remote Sensing Technology,
Technical University of Munich (TUM), 80333 Munich, Germany. (e-mail:
<EMAIL_ADDRESS>
###### Abstract

Monocular height estimation (MHE) from remote sensing imagery has high
potential in generating 3D city models efficiently for a quick response to
natural disasters. Most existing works pursue higher performance. However,
there is little research exploring the interpretability of MHE networks. In
this paper, we target at exploring how deep neural networks predict height
from a single monocular image. Towards a comprehensive understanding of MHE
networks, we propose to interpret them from multiple levels: 1) Neurons: unit-
level dissection. Exploring the semantic and height selectivity of the learned
internal deep representations; 2) Instances: object-level interpretation.
Studying the effects of different semantic classes, scales and spatial
contexts on height estimation; 3) Attribution: pixel-level analysis.
Understanding which input pixels are important for the height estimation.
Based on the multi-level interpretation, a disentangled latent Transformer
network is proposed towards a more compact, reliable and explainable deep
model for monocular height estimation. Furthermore, a novel unsupervised
semantic segmentation task based on height estimation is first introduced in
this work. Additionally, we also construct a new dataset for joint semantic
segmentation and height estimation. Our work provides novel insights for both
understanding and designing MHE models. The dataset and code are publicly
available at https://github.com/ShadowXZT/DLT-Height-Estimation.pytorch.

###### Index Terms:

Disentangled representation learning, height estimation, interpretable deep
models, remote sensing, Transformer deep networks

##  1 Introduction

Urban disaster monitoring has a close relationship with the lives of residents
living in cities. The geometric information of 3D cities can be useful for
urban planning, damage monitoring, disaster forecasting, and so on. In this
context, obtaining the geometric information efficiently from remote sensing
imagery is essential for a rapid response to time-critical world events, e.g.,
natural hazards and damage assessment. Estimating the geometry from images are
long-standing goals of remote sensing and Earth observation [1]. LiDAR, radar,
stereo photogrammetry require expensive devices and complicated data
processing, while predicting height from a single image is a low-cost and
timely-updated way.

Figure 1: MHE networks learn to recognize different semantic objects (road,
building and tree) and height ranges implicitly. This figure shows the strong
selectivity of Transformer-based MHE networks on both the GTAH dataset and the
real-world DFC 2019 dataset. (Best viewed with zoom in.)

Motivated by the development of monocular depth estimation (MDE) task [2, 3,
4], various methods have been proposed for monocular height estimation. Mou
et. al [5] designed a residual convolutional network for height estimation and
demonstrated its effectiveness on the instance segmentation task. Christie et
al. [6, 7] proposed to estimate the geocentric pose from monocular oblique
images. Since estimating height from monocular images is an ill-posed task,
improving the interpretability and reliability of deep models is the highest
priority for risk-sensitive applications. However, there are few works
focusing on understanding deep networks for MHE.

Figure 2: Visualization of the high correlation between height ranges and
feature maps of MHE networks. (Best viewed with zoom in)

As for interpretable deep models, previous research on image classification
[8, 9] and object detection [10] has been studied. However, MHE is a dense
prediction task involving pixel-wise regression. This makes image-level and
object-level interpretation methods not applicable. The most relevant task to
MHE is MDE. Dijk et al. [11] investigated important visual cues in input
monocular images for depth estimation. Hu et al. [12] tried to find the most
relevant sparse pixels for estimating the depth. You et al. [13] first found
the depth selectivity of some hidden units, which showed promising insights
for interpreting MDE models. However, only focusing on the unit-level or
global explanations neglects the semantic content of input images, which is
insufficient to interpret complex and out-of-distribution scenarios.

Figure 3: The whole pipeline of the proposed multi-level interpretation
framework. Three levels are considered: 1) Neurons: unit-level network
dissection; 2) Instances: object-level interpretation; 3) Attribution
analysis: pixel-level interpretation.

Although MHE shares some similar characteristics with MDE, there are still
several aspects making them quite different. Firstly, height is an inherent
attribute of objects, which should not change under different views. While the
depth of objects highly depend on the camera pose. Secondly, estimating height
from top-view may be far more ambiguous than depth estimation from street-view
because of the severe occlusion and lack of textures. Thirdly, object types,
scales and scene layouts vary greatly in remotely-sensed imagery. These
differences also make it not suitable to apply interpretation methods of MDE
directly to MHE.

Considering the aforementioned issues, in this work, we aim at 1)
understanding deep networks for MHE from multiple levels; 2) designing
reliable and explainable deep models for MHE task. Specifically, we propose to
explain the deep MHE model from the following three levels:

1) Neurons: unit-level network dissection. Exploring the properties of hidden
units to understand what internal representations the MHE model has learned.
This is a fundamental component towards an inherent interpretable machine
learning model. From this perspective, we find that MHE networks learn
disentangled representations to different semantic concepts and height ranges.
As shown in Fig. 1 and Fig. 2, roads, trees and buildings are automatically
recognized, and they are also selective to different height ranges.

2) Instances: object-level interpretation. Based on the understanding of
neurons in MHE networks, we study the effects of changes in semantic objects
on MHE networks. By observing the behavior of MHE networks against the changes
of these factors, we find that semantic classes, scales and contexts are the
main factors influencing the height prediction results.

3) Attribution: pixel-level analysis. Towards a comprehensive understanding of
the MHE model, it is necessary to know which input pixels are responsible for
the height estimation. Based on the understanding of neurons in MHE networks,
we propose to attribute the height prediction locally to its input images.
Additionally, both Transformer and CNN are analyzed and compared using local
attribution analysis.

Considering that there are limited datasets that contain high resolution
height maps and pixel-wise semantic labels, we construct a new dataset, named
Washington DC (WDC) dataset, to foster research on improving both the
interpretability and performance of MHE models. To the best of our knowledge,
this work is the first attempt to analyze how deep networks work on the
monocular height estimation task. Our contributions can be summarized as
follows.

  * (1)

We find that deep neurons learned by MHE models are highly selective to both
height ranges and semantic classes. Based on the high class-selectivity of MHE
networks, we propose a simple yet effective out-of-distribution detection
method for MHE task.

  * (2)

Based on the observed class-selectivity, we conduct object-level experiments
and find that semantic classes, object scales, spatial contexts are main
factors influencing height prediction.

  * (3)

We compare Transformer and CNN-based models on MHE task by the local
attribution analysis. Results reveal that Transformer-based networks have
stronger height and class-selectivity, and can learn more effective contexts
than CNNs.

  * (4)

Based on the multi-level interpretation, we propose a novel disentangled
latent Transformer model for learning disentangled representations towards a
more explainable and efficient MHE model.

  * (5)

A novel unsupervised semantic segmentation task is first introduced in this
work based on the understanding of height estimation networks. Additionally, a
new dataset that contains RGB and nDSM (normalized Digital Surface Models)
modality is constructed for joint semantic segmentation and height estimation.

##  2 Related Work

We review related works in this section from the following three aspects:

###  2.1 Monocular Height Estimation

With the development of deep learning, various methods have been proposed for
MHE. Srivastava et al. [14] proposed to predict height and semantic labels
jointly in a multi-task deep learning framework. Mou et al. [5] designed a
residual CNN for height estimation and demonstrated its effectiveness on
instance segmentation task. Besides, Conditional generative adversarial
network (cGAN) [15] was proposed to frame height estimation as an image
translation task. Kunwar et al. [16] exploited semantic labels as priors to
enhance the performance of height estimation on the large-scale Urban Semantic
3D (US3D) dataset [17]. Xiong et al. [18] designed and constructed a large-
scale benchmark dataset for cross-dataset transfer learning on the height
estimation task, which includes a large-scale synthetic dataset and several
real-world datasets. Swin Transformer [19] was used in their work for
transferable representation learning on MHE task. Monocular height estimation
can be widely used in high-risk Earth observation tasks, while there is no
research that focuses on the interpretability of deep learning models for MHE
task.

###  2.2 Methods for Understanding Monocular Depth Estimation

Recently, monocular depth estimation has been studied extensively. With the
surge of deep learning, extensive methods have been designed to obtain better
performance, including geometry-constrained learning [20], multi-scale [2],
multi-task learning methods [21]. To understand what these MDE networks have
learned, Dijk et al. [11] studied important visual clues used by deep networks
when predicting the depth. They mainly focused on object-level interpretation
of MDE networks. Hu et al. [12] attempted to find only a selected sparse set
of image pixels to estimate depth. A separate network is designed to predict
those sparse pixels. However, these methods neglect the inherent
representations that the model have learned. You et al. [13] first found the
depth selectivity of some hidden units, which showed promising insights for
interpreting MDE models. However, merely focusing on hidden units for model
interpretation ignores the semantic content of input images.

###  2.3 Interpretable and Explainable Deep Neural Networks

Interpretable deep models. For image classification and object detection
tasks, several methods attempted to design inherently interpretable models.
Chen et al. [9] proposed to find prototypical parts and explain the reason for
making final decisions. Towards the interpretability for person re-
identification task, Liao et al. [22] designed a model to make the matching
process of feature maps explicit. Zhang et al. [8] designed interpretable CNNs
by making each filter represent a specific object part. Liang et al. [23]
trained interpretable CNNs by learning class-specific deep filters, namely,
encouraging each filter only to account for few classes. Similarly, You et al.
[13] proposed to improve the depth selectivity by designing specific loss
functions for MDE models.

Explainable deep networks. Many researchers focused on saliency-based and
attribution-based methods [24] for explainable deep networks. They aimed to
highlight which pixels of input images are important for predicting the result
[25, 26, 27]. However, attribution and saliency-based methods are not directly
applicable to dense prediction tasks, including MDE and MHE, since it is not
reasonable to highlight all the pixels to attribute the dense prediction
globally. Gu et al. [28] proposed a local attribution method for interpreting
super-resolution networks. They chose to interpret features instead of pixels
for super-resolution task, which inspires our work on pixel-wise attribution
for MHE networks.

In this paper, our proposed method considers multi-level factors to give a
more comprehensive understanding of the MHE task, which differs from the above
approaches.

Figure 4: Visualization of the average response HR for units with large
height-selectivity on both the DFC 2019 dataset and the GTAH dataset. Figure
5: Visualization of the average responses for units with large class-
selectivity.

##  3 Methodology

The motivation of this work is two-folds: 1) interpreting the behavior of MHE
networks from multiple levels for a better understanding of MHE models; 2)
designing reliable and explainable deep models for MHE task. The whole
pipeline for illustrating the multi-level interpretability framework is
presented in Fig. 3. To understand the behavior of learned hidden units, we
examine the learned knowledge of MHE networks by visualizing the
representation properties of specialized deep filters. Based on the finding
that semantic classes are highly related to height predictions, we then study
the effects of different semantic objects and properties on MHE networks.
Finally, we design a pixel-level local attribution method for interpreting MHE
networks based on the finding in unit-level and object-level interpreting
results.

In this section, we will introduce the proposed multi-level interpretation
framework as well as the proposed disentangled latent Transformer model in
detail.

###  3.1 Unit-level Interpretation of MHE Models

Objects of different semantic types usually have distinct height attributes.
Thus, the geometric information in height maps should have high correlation
with the semantic information [16]. Inspired by this, we choose to examine the
learned interior activations to find human-understandable representations.
From numerous visualizations of deep activations, we find that some units of
the model are highly selective to different semantic classes and height
ranges. As shown in Fig. 1, different semantic objects including road,
building and tree are localized accurately in an implicit manner with only
height supervision. This finding supports the assumption that semantic
information and geometric information are highly correlated.

Moreover, we also find that some neurons are selective to different height
ranges, which shares the same conclusion with [13] for MDE task. As displayed
in Fig. 2, the left columns show masks of different discretized depth ranges,
and the right columns are feature maps of the MHE network. Obviously, we can
see a very high correlation between them.

Figure 6: Out-of-distribution objects (_cow_ and _bottle_) are highlighted in
almost all the feature maps, which means these out-of-distribution objects are
not recognized by the network. In contrast, the added _tree_ is only
highlighted in the tree-selective feature maps. (Best viewed with zoom in)

To quantify the selectivity of the network units, we compute both the class-
selectivity [29] and the height-selectivity for each internal unit. In this
work, we adapt the Swin Transformer [19] as the backbone for height estimation
task. Denote each test sample in the dataset $D$ as
$\mathbf{(x_{i},y_{i},h_{i})}$, where ${\mathbf{x_{i}}}\in\mathbb{R}^{3,H,W}$
is the input image, ${\mathbf{y_{i}}\in\mathbb{R}^{H,W}}$,
$\mathbf{h_{i}}\in\mathbb{R}^{H,W}$ are the ground truth semantic map and
height map. $i\in\\{1,...,N\\}$. We upsample the $k_{th}$ feature map of the
penultimate layer to the image size as $F_{k}(x_{i})\in\mathbb{R}^{H,W}$. The
class-selectivity and height-selectivity can be computed by:

$\small\begin{gathered}\mathit{CR}^{c}_{k}=\frac{\sum_{1}^{N}S(F_{k}(\mathbf{x_{i}})\odot
M^{c}_{i})}{\sum_{1}^{N}S(M^{c}_{i})};\quad\mathit{HR}^{h}_{k}=\frac{\sum_{1}^{N}S(F_{k}(\mathbf{x_{i}})\odot
M^{h}_{i})}{\sum_{1}^{N}S(M^{h}_{i})},\\\
\mathit{CS}_{k}=\frac{|\mathit{CR}_{k}^{max}-\bar{\mathit{CR}}_{k}^{-max}|}{|\mathit{CR}_{k}^{max}+\bar{\mathit{CR}}_{k}^{-max}|};\quad\mathit{HS}_{k}=\frac{|\mathit{HR}_{k}^{max}-\bar{\mathit{HR}}_{k}^{-max}|}{|\mathit{HR}_{k}^{max}+\bar{\mathit{HR}}_{k}^{-max}|},\end{gathered}$
(1)

where $\mathit{CR}^{c}_{k}$ and $\mathit{HR}^{h}_{k}$ are the average response
of the units for different classes and height ranges. $h$ is the index of
discretized height ranges, and $c$ is the index of semantic classes. $M^{c}$
is a binary mask indicating the pixels with semantic class $c$. $M^{h}$ is
also a binary mask indicating the pixels in height range $h$. $S(\cdot)$
denotes the summation operation on all the elements of a matrix. We use
$\odot$ to represent the element-wise multiplication. With the defined average
responses $\mathit{CR}^{c}_{k}$ and $\mathit{HR}^{h}_{k}$, class-selectivity
and height-selectivity for the $k_{th}$ unit $\mathit{CS}_{k}$ and
$\mathit{HS}_{k}$ can be computed.

Figure 7: Visualization of out-of-distribution results. Out-of-distribution
objects are highlighted in the anomaly response maps $R_{ood}$. (Best viewed
in color).

As defined in Eq. 1, the value of $\mathit{CS}_{k}$ and $\mathit{HS}_{k}$
ranges from 0 to 1. Units of larger selectivity values tend to response
preferably to specific semantic classes or height ranges. We have visualized
the feature maps with high height-selectivity values in Fig. 2, which shows
high correlation between feature maps and different height ranges. Besides
that, the quantified height-selectivity is also presented in Fig. 4. For both
the real-world dataset and the synthetic dataset GTAH [18], there exist clear
selectivity over different height ranges.

In Fig. 5, the class-selectivity of the Transformer-based MHE model is
visualized. The class-selectivity of unit 255, 87 and 20 are larger than 0.8,
and they are highly selective to categories _Ground, High Vegetation,_ and
_Building_ , respectively. However, there are no clear selective units for
_Water_ and _Elevated Road_ , since these categories have no clear differences
compared with _Ground_ regarding height changes.

Based on the high class-selectivity in MHE networks, we propose a simple yet
effective out-of-distribution method for MHE task. To better explain this
method, we construct an out-of-distribution sample by adding some irrelevant
objects, such as _cow_ and _bottle_ to the input image. As shown in Fig. 6, we
add a _tree_ (in-distribution) object as a comparison sample. Compared to the
original input, we can see that the out-of-distribution objects (_cow_ and
_bottle_) are highlighted in almost all the feature maps. This means that
these out-of-distribution objects are not recognized by the network. In
contrast, the added _tree_ object is recognized by the MHE model, and is only
highlighted in the tree-selective feature maps.

To quantify this, we propose to compute the variance of pixels in the feature
maps. Given the feature maps $F(\mathbf{x_{i}})\in\mathbb{R}^{K,H,W}$, we
first normalize it along the channels by
$F^{\prime}(\mathbf{x_{i}})=\frac{F(\mathbf{x_{i}})}{\sum_{1}^{K}F^{k}(\mathbf{x_{i}})}$.
Then, we can compute the variance of $F^{\prime}(\mathbf{x_{i}})$ for each
pixel along the channel dimension. Finally, We can define the anomaly response
map $R_{ood}=\bm{I}-var(F^{\prime}(\mathbf{x_{i}}))$ as a measurement for out-
of-distribution detection. Some visualization examples are shown in Fig. 7. It
can be seen that out-of-distribution objects are detected with high response
values.

###  3.2 Instance-level Interpretation of MHE Models

Considering that the learned units are highly selective to different semantic
objects, it is natural to study the behavior of MHE networks by changing the
semantics of the original input image. Thus, to keep the semantic concepts of
original inputs, we choose to modify object instances of the original input
image.

Figure 8: Visualization of the effects of object-level changes on the MHE
networks.

Specifically, inspired by the cognition process of human beings, we explore
the effects of changing the semantic class, scale and shadow state of object
instances. Firstly, we define templates $t^{c}_{s}$ for each semantic category
$c\in\\{\emph{road},\emph{tree},\emph{building}\\}$ and scale
$s\in\\{s_{1},s_{2},s_{3},s_{4},s_{5}\\}$. Suppose the original scale (area)
of the template for each category is $s_{t}$, then
$\\{s_{1}=0.3*st,s_{2}=1.0*st,s_{3}=1.5*st,s_{4}=2.5*st,s_{5}=3.0*st\\}$.

To study the effects of changing semantic categories on height prediction, for
each image $(\mathbf{x_{i}})$ in test dataset $D$, we select an appropriate
patch with semantic class $\\{\emph{road},\emph{tree},\emph{building}\\}$
replaced with the predefined template $t^{c}_{s}$. The experimental results
are shown in the left side of Fig. 8. We can see that the predicted heights
change as expected after replacing objects of different semantic categories.
This indicates that the height prediction results are highly correlated with
the semantic categories of objects.

In order to explore the effects of changing object scales, for each image in
the test set $D$, we first choose an appropriate location $\\{p_{x},p_{y}\\}$
in the original image. Then, for each scale
$s\in\\{s_{1},s_{2},s_{3},s_{4},s_{5}\\}$, we place the predefined template
$t^{c}_{s}$ on the selected location. Since changing the scale of _road_ is
not meaningful, we only conduct experiments on _building_ and _tree_ objects.
The right side of Fig. 8 presents the average height results for each scale.
It can be seen that, in general, the larger the object scale is, the larger
the height value is predicted. However, It is worth mentioning that, if the
scale is too small, the network may not be able to recognize the object.
Additionally, we also explore the effects of different scales of object
shadows. From Fig. 8 we can see that, the height predictions are not clearly
correlated with the scale of shadows.

###  3.3 Pixel-level Attribution Analysis of MHE Models

Since MHE is a dense prediction task, interpreting the MHE networks for all
pixels globally cannot provide meaningful interpretation information. From the
unit-level and object-level analysis, we find that if the semantic class of an
object cannot be recognized in the feature map, then the prediction result
will be unreliable. This indicates that the existence of objects on feature
maps is a fundamental and important factor for height prediction. Thus,
inspired by [28], we design a local attribution analysis method based on
feature existence and path integrated gradient [24] to interpret MHE networks
in a pixel-level.

For an input image $\mathbf{x}$, $F_{h}{\mathbf{x}}$ represents the height
prediction map predicted by MHE networks. We define $D_{px,py}(\mathbf{x})$ to
quantify the existence of one local object as

$D_{px,py}(\mathbf{x})=\sum_{i\in[px,px+n],y\in[py,py+n]}F_{h}(\mathbf{x})_{ij},$

where $n$ is the window size of the selected patch. To mine the important
pixels that account for the existence of one object, we use a whole black
image as the baseline input $\mathbf{x^{\prime}}$. In order to obtain the
attribution map for the selected local patch, we need to integrate gradients
along the gradually changing path from $\mathbf{x^{\prime}}$ to $\mathbf{x}$.
Note that, for the baseline image $\mathbf{x^{\prime}}$, the output of the MHE
network $F_{h}(\mathbf{x^{\prime}})$ is not zero.

Then the $i_{th}$ dimension of the integrated gradients for the local patch IG
can be computed by:

$\text{IG}_{i}(x):=(\mathbf{x}-\mathbf{x^{\prime}})\int_{\alpha=0}^{1}\frac{\partial
D(F_{h}(\mathbf{x^{\prime}}+\alpha(\mathbf{x}-\mathbf{x^{\prime}})))}{\partial\mathbf{x}_{i}}\,d\alpha.$
(2)

In this work, we use the linear path function to define $\alpha$. To be
specific, we interpolate $\alpha$ smoothly from 0 to 1 with $m$ steps, i.e.,
$\alpha=\frac{k}{m}$. In practice, we approximate the computation of the
integral defined in Eq. 2 by the following summation along $m$ steps:

$\text{IG}_{i}(x):=\frac{(\mathbf{x}-\mathbf{x^{\prime}})}{m}\sum_{k=1}^{m}\frac{\partial
D(F_{h}(\mathbf{x^{\prime}}+\frac{k}{m}(\mathbf{x}-\mathbf{x^{\prime}})))}{\partial\mathbf{x}_{i}}\,d\alpha.$
(3)

Empirically, setting $m$ to 100 works well for MHE networks in our experiments
on both the real-world and synthetic datasets.

##  4 Disentangled Latent Transformer

Based on the finding of semantic selectivity, we pave a new perspective to
understand the MHE model from multiple levels: 1) The semantic selectivity
inspires the instance-level explanation and provides the reason why instance
replacement makes sense; 2) The instance-level experiments further show that
detecting the existence of local semantic objects by local attribution
analysis is an effective way for pixel-level interpretation.

To make better use of the semantic selectivity, in this work, we further
propose a disentangled latent Transformer (DLT) networks to explicitly model
the representations for different semantic classes. By disentangling the deep
neurons of MHE models into different semantic groups, a new unsupervised
semantic segmentation method is motivated with the proposed DLT model.
Specifically, DLT exploits a Transformer backbone to better learn the pairwise
relations between different pixels. Next, a latent random variable is
predicted for each semantic class, which can be used to generate semantic
segmentation maps for a specific class.

We observe that different neurons of the final layer have specific response to
different semantic classes, and there are more than one neurons responsible
for a specific class. Thus, to enable the disentanglement representation
learning, we first cluster these neurons into different semantic groups.
Formally, given a pre-trained Transformer-based deep networks with $n$ neurons
at the final layer, the weights $\\{\bm{W_{i}}\in\mathbb{R}^{n_{o}\times
k\times k},i=1,...,n\\}$ of the final layer are clustered into $K$ groups
using K-means. Note that for each pre-trained deep models, we only need to
cluster the neurons once and do not need to cluster the feature maps at each
inference. Based on the predicted cluster indices, we can further obtain $K$
groups of feature maps $\\{\bm{F_{ci}\in\mathbb{R}^{n_{i},H,W}},i=1,...,K\\}$.
Each semantic group contains $n_{i}$ feature maps, and
$\sum_{i=1}^{K}n_{i}=n$.

Feature maps in the same semantic group are with similar semantic responses
with slightly differences. To improve the reliability, we propose to treat the
feature maps within the same semantic group as noisy observations, and model
the true semantic response $\bm{F_{v}\in\mathbb{R}^{K\times H\times W}}$ as a
latent variable remaining to be estimated. In practice, for each semantic
class, we aim to sample the latent semantic responses from a posterior
distribution $p(\bm{F_{v}}\mid x)$ to explicitly model the noise in a Bayesian
probabilistic framework.

As the true posterior is intractable to obtain, we adopt the variational
inference [30] to approximate $p(\bm{F_{v}}\mid x)$ with a simple distribution
$q_{x}(\bm{F_{v}})$. Thus, the optimization goal is to minimize the Kullback-
Leibler (KL) divergence $\mathcal{L}_{KL}=KL(q(\bm{F_{v}})\mid\mid
p(\bm{F_{v}}|x))$. As pointed by [30], maximizing the likelihood distribution
is equivalent to maximizing the variational evidence lower bound (ELBO), which
can be defined as:

$\displaystyle\mathcal{L}_{KL}$
$\displaystyle=\sum_{\bm{F_{v}}}q(\bm{F_{v}}\mid
x)\log\left(\frac{p(\bm{F_{v}},x)}{q(\bm{F_{v}}\mid x)}\right)$ (4)
$\displaystyle=\sum_{\bm{F_{v}}}q(\bm{F_{v}}\mid
x)\log\left(\frac{p(x\mid\bm{F_{v}})p(\bm{F_{v}})}{q(\bm{F_{v}}\mid
x)}\right)$
$\displaystyle=\sum_{\bm{F_{v}}}q_{x}(\bm{F_{v}})\log\left(\frac{p(\bm{F_{v}})}{q_{x}(\bm{F_{v}})}\right)+\sum_{\bm{F_{v}}}q_{x}(\bm{F_{v}})\log(p(x\mid\bm{F_{v}}))$
$\displaystyle=\mathbb{E}_{q_{x}(\bm{F_{v}})}[\text{log}~{}p(\bm{F_{c}}|\bm{F_{v}})]-KL(q_{x}(\bm{F_{v}})||p(\bm{F_{v}})).$

In practice, the first expected log-likelihood term is used to minimize the
feature reconstruction loss. Namely, we aim to reconstruct the original
feature maps using the estimated latent variables $\bm{F_{v}}$. This can also
be viewed as a feature distillation process. Then, the following KL divergence
term is used to minimize the distance between $q_{x}(\bm{F_{v}})$ and the true
posterior $p(\bm{F_{v}}\mid x)$.

To train the whole model in an end-to-end manner using the gradient descent
optimizer, we need to compute the gradients w.r.t. the parameters of
distributions. For the sake of simplicity, we assume that the latent semantic
responses $\bm{F_{v}}$ follow the isotropic Gaussian distribution with mean
$\mu_{i}\in\mathbb{R}^{H,W}$ and variance
${\sigma_{i}}^{2}\in\mathbb{R}^{H,W}$. However, the gradients of parameters
$\\{\mu_{i},{\sigma_{i}}^{2}\\}$ of the approximate distribution, which are
predicted by deep networks, cannot be computed directly. Thus, we utilize the
reparameterization trick to re-write $q_{x}(\bm{F_{v}})$ as
${\bm{F_{vi}}=\mu_{i}+{\sigma_{i}}\varepsilon,i=1,...,K}$, where $\varepsilon$
is the standard normal distribution, i.e., $\varepsilon\sim{\cal
N}(\textbf{0},\textbf{I})$. With the reparameterization trick, the final loss
function can be rewritten as:

$\displaystyle{\cal L}(x^{(i)})$
$\displaystyle=\frac{N}{M}\sum_{i=1}^{M}(\frac{1}{L}\sum_{l=1}^{L}\text{log}~{}p(y^{(i)}|x^{(i)},\mu)$
(5)
$\displaystyle+\sum\text{log}~{}p(\bm{F_{c}}^{(i)}|x^{(i)},\mu+{\sigma}\varepsilon)-KL(q_{x}(\bm{F_{v}}|x^{(i)})||p(\bm{F_{v}}))),$

where $M$ is the size of mini-batch, and $N$ is the total number of samples.
$L$ is the number of sampling process for Monte Carlo estimation. $y_{i}$ is
the ground truth height maps for training the height estimation models.

For the prior distribution, we assume that the latent semantic responses
follow isotropic Gaussian distribution with mean
$\bm{\mu_{i}\in\mathbb{R}^{H,W}}$ and variance
$\bm{{\sigma_{i}}^{2}\in\mathbb{R}^{H,W}}$. In this work, we use the mean of
${\bm{F_{c}}}$ as the prior mean of $\bm{F_{v}}$, and set the variance to be
$\bm{I}$. Then the latent semantic responses $\bm{Fv_{i}}$ can be defined as

$\displaystyle\bm{F_{vi}}\sim\mathcal{N}(\mu_{p}i,{\sigma_{p}i}^{2}),$ (6)
$\displaystyle\mu_{p}i=\frac{1}{ni}\sum{\bm{F_{ci}}},\,\,\sigma_{p}i$
$\displaystyle=\bm{I},$

With the defined notations, the KL divergence loss in Eq. 5 for distribution
approximation can be computed analytically as:

$\small
KL(q_{x}(\bm{F_{v}})||p(\bm{F_{v}}))=-\frac{1}{2}\sum\left[1+\log\sigma_{i}^{2}-\sigma_{i}^{2}-(\mu_{i}-\mu_{p}i)^{2}\right].$
(7)

After the training stage, the predicted mean of $\bm{F_{v}}$ can be used to
generate the semantic segmentation maps for a specific class. In this work, we
simply use OTSU [31] algorithm to binarize $\bm{F_{v}}$ into segmentation
maps. Note that the final segmentation results are obtained from a MHE model
without using any segmentation annotations.

Figure 9: Visualization of the proposed Washington DC dataset. Patches of
rural, forest, water, and urban are shown as examples. For each patch, the
orthophoto, nDSM (height map), and semantic map are displayed.

##  5 Experiments

Figure 10: Transformer-based model can learn disentangled representations for
different semantic objects. While the feature maps of UNet-based model are
somehow entangled. (Best viewed in color and with zoom in)

In this section, we evaluate the proposed multi-level interpretation framework
on different datasets. In addition, we use it to analyze the reason why Swin
Transformer-based MHE model can outperform CNN-based MHE networks.

###  5.1 Datasets

Three datasets are used in this work for the method evaluation. The first one
is a real-world dataset from IEEE GRSS, Data Fusion Contest 2019 (DFC 2019)
[32]. DFC 2019 contains 2,786 images with both the semantic segmentation label
and normalized Digital Surface Model (nDSM). The second dataset is a synthetic
dataset, called GTAH111https://thebenchmarkh.github.io/, which is constructed
using the game engine GTA V [33]. GTAH contains 85,881 images with the
corresponding height map labels. Finally, we also evaluate the proposed method
on our newly constructed WDC dataset in this work.

The proposed WDC dataset is processed based on open data at Open Data DC
222https://opendata.dc.gov/. The dataset covers whole Washington DC, USA. The
aerial orthophoto was collected in 2021, and the nDSM was collected via LiDAR
in 2020. There are 5 classes in the semantic map, namely ground, vegetation,
buildings, water, and roads. Building footprints and roads in shape files are
used to construct the semantic maps. For other classes, we refer to the labels
in LiDAR point clouds. Some examples of the proposed dataset are visualized in
Fig. 9.

WDC dataset contains in total 2,159 patches of aerial orthophotos. Each image
corresponds to a pixel-wise semantic map and a height map. The resolution of
the image patch is 1024$\times$1024, and the GSD is 0.33 m. A variety of
scenes are covered by the dataset, including urban, rural, forest, etc., thus,
rich diversity of classes and heights.

###  5.2 Implementation Details

We implement the proposed method using Pytorch. On the GTAH dataset, we pre-
train the Swin Transformer model with 100 epochs, and fine-tune the proposed
DLT model for 20 epochs. For the U-Net model with ResNet-34 backbone, the
code333https://github.com/pubgeo/monocular-geocentric-pose from [6] is used.
To optimize the Transformer-based models, AdamW [34] is used with an initial
learning rate of 6e-5 for pre-training the deep models. For UNet-based models,
Adam [35] is used as the optimizer with an initial learning rate of 1e-4. For
all the experiments, we set the batch size to 4. More implementation details
of this work are provided in the publicly available code.

We adapt the tiny version of Swin Transformer as the baseline method for MHE
task. To compare with the CNN-based methods, we evaluate the UNet-based height
estimation model with ResNet-34 and ResNet-101 backbones [6].

###  5.3 Results and Comparisons

To evaluate the performance of the proposed methods on MHE, we follow the
previous work [18] and use four metrics for performance comparisons including
Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), Scale-Invariant
RMSE (SI-RMSE) and Multi-scale Gradient Error (MSG). MAE is defined as $\text{
MAE }=1/n*\sum\left|y_{i}-\hat{y}_{i}\right|$, which is used to measure the
mean absolute difference between the predicted values and the reference
values. RMSE is defined as $\text{ RMSE
}=\sqrt{\Sigma\left(y_{i}-\hat{y}_{i}\right)^{2}/n}$, which is a commonly used
measurement for regression task. To care more about the relative relations in
the height maps, we also adopt the SI-RMSE and MSGE as metrics for model
evaluation. SI-RMSE is defined as

$\text{SI-
RMSE}=\frac{1}{n}\sum_{i}R_{i}^{2}-\frac{1}{n^{2}}\left(\sum_{i}R_{i}\right)^{2}.$
(8)

The multi-scale gradient matching error, MSGE, can be formulated as:

$\text{MSGE}=\frac{1}{M}\sum_{k=1}^{K}\sum_{i=1}^{M}\left(\left|\nabla_{x}R_{i}^{k}\right|+\left|\nabla_{y}R_{i}^{k}\right|\right).$
(9)

For the comparison of semantic segmentation, we use the commonly used
Intersection over Union (IoU) and mean Intersection over Union (mIoU) as the
evaluation metrics. In the following sections, we first analyse and compare
the performances of UNet and Swin Transformer models regarding the model
interpretability. Both the unit-level and pixel-level interpretation methods
are analysed. Then, we evaluate and compare the proposed DLT model on the
performances of MHE and unsupervised semantic segmentation. Furthermore, we
also compare the proposed new method with several state-of-the-art
unsupervised segmentation methods.

TABLE I: Class-selectivity and response of different units of Swin Transformer
on the DFC 2019 Dataset

Category | Ground | High Vege. | Building | Water | Selectivity
---|---|---|---|---|---
Unit 87 | 0.0833 | 0.7083 | 0.0485 | 0.1597 | 0.8893
Unit 20 | 0.0433 | 0.2051 | 0.7241 | 0.0273 | 0.8359
Unit 255 | 0.6724 | 0.1473 | 0.0606 | 0.1194 | 0.8302
Unit 371 | 0.6412 | 0.1701 | 0.0443 | 0.1442 | 0.8089

TABLE II: Class-selectivity and response of different units of UNet
(ResNet-34) on the DFC 2019 Dataset

Category | Ground | High Vege. | Building | Water | Selectivity
---|---|---|---|---|---
Unit 17 | 0.7801 | 0.1826 | 0.0069 | 0.0302 | 0.8248
Unit 29 | 0.7233 | 0.1873 | 0.0142 | 0.0751 | 0.8223
Unit 14 | 0.1419 | 0.4519 | 0.2801 | 0.1259 | 0.5635
Unit 16 | 0.1512 | 0.2738 | 0.3993 | 0.1756 | 0.5092

####  5.3.1 Unit-level Analysis and Comparison

In this section, we aim to analyze the reasons why the Transformer-based model
can obtain better results than the UNet-based MHE method from the perspective
of learned deep representations. We compute the class-selectivity for two
models and present the results in Table II and Table II. The last column of
both tables indicates that Transformer-based MHE model has significantly
stronger class-selectivity than UNet-based model. Besides, we also display the
normalized average responses of these units. In general, the units of
Transformer-based model are obviously more selective to different semantic
categories. For a more clear comparison, the feature maps of two models are
visualized in Fig. 10. We can see that the Swin transformer-based model can
learn disentangled representations for different semantic objects: _building,
tree, road_. While the feature maps of UNet-based model are somehow entangled,
i.e, buildings and trees are not well separated.

From the unit-level analysis, we can see the obvious superiority of the
Transformer-based model in disentangled representation learning. When it comes
to the quantitative performance, from the results in Table III we can see that
the Swin Transformer-based MHE model can largely outperform the CNN-based UNet
model. Especially for the GTAH dataset, Transformer-based model can reduce the
SI-RMSE metric by almost 50%.

####  5.3.2 Pixel-level Analysis and Comparison

For a more comprehensive comparison of the CNN and Transformer-based models,
we conduct local attribution analysis towards a pixel-level understanding of
them.

Figure 11: Comparison of the attribution maps for the same object template but
different spatial context. (Best viewed in color and with zoom in) Figure 12:
Comparison of the attribution maps for Transformer-based and UNet-based MHE
networks. We can see that Transformer-based model can learn to exploit more
effective and compact context. (Best viewed in color and with zoom in)

To start with, we first analyze the importance of spatial context for height
estimation task. For a clear presentation, we created an image containing four
buildings with two different scales but the same template, as shown in Fig.
11. The first and second buildings are of the same scale. However, their
height predictions are quite different: 10.8 m vs. 6.9 m. While the ground
truth is 7.2 m. This indicates that spatial context is also an important
factor for MHE task in addition to the semantic class and object scale. From
the attribution map, we can observe that the second building exploits much
large context pixels on the surrounding buildings. Meanwhile, the height
prediction for the third building is also influenced by the second building
and the tree on the right.

TABLE III: Experimental results on the GTAH dataset in the zero-shot transfer
setting. The best and second-best results are in blue and green.

Methods | Height Estimation
---|---
MAE | RMSE | SI-RMSE | MSGE
U-Net (ResNet34) | 4.860 | 6.731 | 39.511 | 3.357
U-Net (ResNet101) | 3.216 | 5.807 | 29.761 | 2.968
Swin Transformer[19] | 2.970 | 4.631 | 21.161 | 2.403
Swin (16 Units) | 3.093 | 4.750 | 22.039 | 2.527
Swin (16 Units)+Distillation | 3.194 | 4.778 | 22.313 | 2.571
Swin (16 Units)+DLT(Ours) | 2.974 | 4.731 | 21.614 | 2.516

Since context information is an important factor for MHE, we visualize the
attribution maps of Transformer-based and UNet-based models to compare their
spatial contexts. As shown in Fig. 12, the upper rows show that the
Transformer-based model tends to use the pixels of the specific object. Thus,
the context is more compact than the UNet-based model. In contrast, the UNet-
based MHE model usually contains many irrelevant pixels, which makes it not
robust in complex scenes. From this perspective, we can conclude that the
self-attention mechanism in Transformer models is helpful for learning
disentangled representation and exploiting more effective spatial context.

TABLE IV: Height estimation results of different methods on the DFC 2019
dataset. The best and second-best results are in blue and green.

Method | Height Estimation
---|---
MAE | RMSE | SI-RMSE | MSGE
U-Net (ResNet-34) | 1.432 | 2.417 | 7.114 | 4.015
U-Net (ResNet-101) | 1.323 | 2.314 | 6.415 | 3.393
Swin Transformer | 1.236 | 2.116 | 5.326 | 3.031
SwinT (16 Units) | 1.288 | 2.206 | 5.674 | 2.975
SwinT (16 Units)+Distillation | 1.414 | 2.240 | 5.947 | 2.959
SwinT (16 Units)+DLT (Ours) | 1.274 | 2.179 | 5.589 | 3.016

TABLE V: Unsupervised semantic segmentation results on the DFC 2019 dataset.
The best results are presented in bold.

Method | Semantic Segmentation
---|---
Building(IoU) | Tree(IoU) | Mean IoU
PiCIE | 0.150 | 0.280 | 0.215
IIC | 0.157 | 0.275 | 0.216
MaskContrast | 0 | 0.0001 | 0.0001
U-Net (16 Units) | 0.092 | 0.077 | 0.0845
SwinT (16 Units) | 0.269 | 0.348 | 0.309
SwinT (16 Units)+Distillation | 0.281 | 0.357 | 0.319
SwinT (16 Units)+DLT (Ours) | 0.306 | 0.389 | 0.348

###  5.4 Disentangled Latent Transformer

In the following sections, we evaluate and compare the performances of our
proposed DLT model with previous state-of-the-art (SOTA) methods on both the
height estimation and unsupervised semantic segmentation tasks.

####  5.4.1 Height Estimation Performance Comparison

TABLE VI: Height estimation results of different methods on the proposed WDC
2019 dataset. The best and second-best results are in blue and green.

Method | Height Estimation
---|---
MAE | RMSE | SI-RMSE | MSGE
U-Net (ResNet-34) | 4.860 | 6.271 | 39.670 | 6.924
U-Net (ResNet-101) | 4.483 | 5.832 | 35.048 | 6.245
Swin Transformer | 4.146 | 5.403 | 31.960 | 5.857
SwinT (16 Units) | 4.412 | 5.734 | 35.608 | 6.136
SwinT (16 Units)+Distillation | 4.462 | 5.753 | 35.948 | 6.357
SwinT (16 Units)+DLT (Ours) | 4.346 | 5.709 | 33.89 | 6.094

TABLE VII: Unsupervised semantic segmentation results on the proposed WDC
dataset. The best results are presented in bold.

Method | Semantic Segmentation
---|---
Building(IoU) | Tree(IoU) | Mean IoU
PiCIE | 0.278 | 0.221 | 0.250
IIC | 0.201 | 0.289 | 0.245
MaskContrast | 0.002 | 0.368 | 0.185
U-Net (16 Units) | 0.113 | 0.127 | 0.120
SwinT (16 Units) | 0.336 | 0.389 | 0.363
SwinT (16 Units)+Distillation | 0.348 | 0.397 | 0.373
SwinT (16 Units)+DLT (Ours) | 0.365 | 0.412 | 0.389

Comparisons on the GTAH Dataset. As we find that more than one neurons are
responsible for the same semantic class, we propose to reduce the number of
neurons in the final layer by removing redundant neurons. Thus, we can make
each neuron in the last layer to be only sensitive to a specific semantic
class or height range. With this neuron compression process, we can obtain a
more compact deep neural network with higher interpretability. As presented in
Table IV, ”SwinT (16 Units)” denotes the model where the units (number of
channels) of the last layer are reduced from 512 to 16.

Figure 13: Unsupervised semantic segmentation results of the proposed “SwinT
(16 Units)+DLT” on the GTAH dataset. The feature maps and segmentation maps of
Unit 8 (building) and Unit 14 (tree) are shown in this figure. (Best viewed
with zoom in)

However, reducing the number of neurons from 512 to 16 significantly limits
the capacity of the model, which also affects the final height estimation
performance of the model. Therefore, the performance of model “SwinT (16
units)” is lower than that of the original model (512 units) on the GTAH
dataset. To cope with this issue, we propose to distill the feature maps of
the original model to the compressed one by encouraging the high feature
similarity between them. However, it can be seen from the experimental results
that the model ”SwinT (16 Units)+Distillation” with a simple feature
distillation does not lead to performance improvement. The reason is that
simple feature distillation limits the expressiveness of the model due to the
noise in the features. The DLT model proposed in this paper, on the other
hand, better captures the noise in the features by explicitly modeling the
features as latent random variable, thus obtaining very similar results to the
original model. The experimental results in the table demonstrate the
effectiveness of the proposed DLT model. We also visualize some unsupervised
segmentation results in Fig. 13.

Figure 14: Unsupervised semantic segmentation results of the proposed “SwinT
(16 Units)+DLT” on the DFC 2019 dataset. The segmentation maps of Unit 3
(building, the left side) and Unit 7 (tree, the right side) are shown in this
figure. Compared with its counterparts, our DLT can obtain clearly better
segmentation results. (Best viewed with zoom in)

Comparisons on the DFC 2019 Dataset. Similar to the GTAH dataset, we also
compare the proposed method with other SOTA methods on the DFC 2019 dataset.
From the results in Table IV, it can be clearly seen that our proposed ”SwinT
(16 Units)+DLT” with only 16 units can achieve competitive height estimation
performance compared with the original Transformer model. This indicates the
effectiveness of the proposed DLT model on the real-world dataset. From the
results, we can also observe that Transformer-based models can obtain better
performance when compared with CNN-based UNet models. This comparison
demonstrates that Transformer-based deep models can be more suitable than CNN-
based models for the pixel-wise dense regression task.

Comparisons on the WDC Dataset. On the proposed WDC dataset, we can also
observe that Transformer-based models have advantages in terms of height
estimation performance. The proposed “SwinT (16 Units)+DLT” obtains similar
performance compared with the “Swin Transformer” model. It shows that the
compressed DLT model can improve the interpretability and reduce the
computational complexity of the original Transformer model, with no loss of
accuracy.

####  5.4.2 Unsupervised Segmentation Results Comparison

We also compare our results with three unsupervised semantic segmentation
networks, including MaskContrast [36], PiCIE [37], and IIC [38]. MaskContrast
uses saliency as the prior knowledge for contrastive learning, and gains SOTA
performance for natural images. However, remote sensing images are quite
different from natural images in terms of contexts and scales. In our
experiments, we find that the saliency-based method fails to distinguish
foreground and background on DFC dataset. Thus, Almost all the image patches
are assigned with only the ground label. PiCIE considers the photometric and
geometric invariance of pixel embeddings. It also suffers from the
difficulties caused by the differences between natural and remote sensing
images. Though IoU socres are high, the predictions do not capture the layouts
of input images. Of these three methods, The mutual information maximization-
based method, IIC, is the only one that has proven to be effective on remote
sensing images. From the visualization, it predicts well in terms of the
layout.

For the unsupervised segmentation task, we only compare the performance on
Building and Tree classes. The reason is that buildings and trees are objects
that are important for urban planning, climate change and disaster monitoring.
Traditional supervised learning-based methods usually require a large amount
annotation data for model training. However, annotating pixel-level semantic
labels is expensive for large-scale or even global-scale Earth observation
applications.

In contrast, our proposed DLT model can obtain the pixel-level semantic
segmentation maps using only the height maps as supervision, which has great
potential for large-scale applications. The results in Table V show that DLT
can outperform existing unsupervised semantic segmentation methods clearly by
a large margin. For example, compared with IIC, DLT can increase the IoU
metric from 0.157 to 0.306. As shown in Table VII, on the WDC Dataset, we can
also observe that compared with MaskContrast [36], PiCIE [37], and IIC [38],
the proposed DLT model can achieve much better segmentation performance. The
aforementioned experimental results have demonstrated the effectiveness of our
proposed DLT model on both height estimation and unsupervised segmentation
tasks. In Fig.14 and Fig. 15, we visualize some qualitative examples of our
DLT method and its counterparts for a clear comparison.

Figure 15: Unsupervised semantic segmentation results of the proposed “SwinT
(16 Units)+DLT” on the WDC dataset. The segmentation maps of Unit 12
(building, upper two rows) and Unit 0 (tree, lower two rows) are shown in this
figure. Compared with its counterparts, our DLT can obtain clearly better
segmentation results. (Best viewed with zoom in)

##  6 Conclusion

In this paper, we explore how deep networks predict height from single images.
To understand the MHE networks comprehensively, three levels of interpretation
are studied: 1) Neurons: unit-level network dissection; 2) Instances: object-
level interpretation; 3) Attribution: pixel-level analysis. From our
experiments, we find that: 1) Deep neurons learned by MHE networks have high
selectivity to both the height ranges and the semantic categories; 2) Based on
the class-selectivity, we propose a simple yet effective out-of-distribution
detection method for MHE; 3) Semantic class, object scale and spatial context
are the main factors influencing the prediction results; 4) Transformer based
networks have stronger height and class selectivity and can use spatial
context more effectively than CNNs. Based on the understandings of MHE models,
a disentangled latent Transformer model is designed towards a more compact and
explainable deep networks for monocular height estimation. The proposed DLT
model has shown superiority on the unsupervised semantic segmentation task.
Furthermore, we also release a new dataset for both semantic segmentation and
height estimation. This work provides novel insights for future works to
better understand and design MHE models.

## References

  * [1] X. X. Zhu, D. Tuia, L. Mou, G.-S. Xia, L. Zhang, F. Xu, and F. Fraundorfer, “Deep learning in remote sensing: A comprehensive review and list of resources,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 5, no. 4, pp. 8–36, 2017.
  * [2] D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” in _International Conference on Neural Information Processing Systems_ , 2014, pp. 2366–2374.
  * [3] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2002–2011.
  * [4] Y. Kuznietsov, J. Stuckler, and B. Leibe, “Semi-supervised deep learning for monocular depth map prediction,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6647–6655.
  * [5] L. Mou and X. X. Zhu, “Im2height: Height estimation from single monocular imagery via fully residual convolutional-deconvolutional network,” _arXiv preprint arXiv:1802.10249_ , 2018.
  * [6] G. Christie, R. R. R. M. Abujder, K. Foster, S. Hagstrom, G. D. Hager, and M. Z. Brown, “Learning geocentric object pose in oblique monocular images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 512–14 520.
  * [7] G. Christie, K. Foster, S. Hagstrom, G. D. Hager, and M. Z. Brown, “Single view geocentric pose in the wild,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1162–1171.
  * [8] Q. Zhang, Y. N. Wu, and S.-C. Zhu, “Interpretable convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8827–8836.
  * [9] C. Chen, O. Li, C. Tao, A. J. Barnett, J. Su, and C. Rudin, “This looks like that: deep learning for interpretable image recognition,” _arXiv preprint arXiv:1806.10574_ , 2018.
  * [10] T. Wu and X. Song, “Towards interpretable object detection by unfolding latent structures,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 6033–6043.
  * [11] T. v. Dijk and G. d. Croon, “How do neural networks see depth in single images?” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2183–2191.
  * [12] J. Hu, Y. Zhang, and T. Okatani, “Visualization of convolutional neural networks for monocular depth estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3869–3878.
  * [13] Z. You, Y.-H. Tsai, W.-C. Chiu, and G. Li, “Towards interpretable deep networks for monocular depth estimation,” _IEEE International Conference on Computer Vision_ , 2021.
  * [14] S. Srivastava, M. Volpi, and D. Tuia, “Joint height estimation and semantic labeling of monocular aerial images with cnns,” in _2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS)_. IEEE, 2017, pp. 5173–5176.
  * [15] P. Ghamisi and N. Yokoya, “Img2dsm: Height simulation from single imagery using conditional generative adversarial net,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 15, no. 5, pp. 794–798, 2018.
  * [16] S. Kunwar, “U-net ensemble for semantic and height estimation using coarse-map initialization,” in _IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium_. IEEE, 2019, pp. 4959–4962.
  * [17] M. Bosch, K. Foster, G. Christie, S. Wang, G. D. Hager, and M. Brown, “Semantic stereo for incidental satellite images,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 1524–1532.
  * [18] Z. Xiong, W. Huang, J. Hu, Y. Shi, Q. Wang, and X. X. Zhu, “The benchmark: Transferable representation learning for monocular height estimation,” 2021.
  * [19] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” _arXiv preprint arXiv:2103.14030_ , 2021.
  * [20] W. Yin, Y. Liu, C. Shen, and Y. Yan, “Enforcing geometric constraints of virtual normal for depth prediction,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5684–5693.
  * [21] S. Zhu, G. Brazil, and X. Liu, “The edge of depth: Explicit constraints between segmentation and depth,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 116–13 125.
  * [22] S. Liao and L. Shao, “Interpretable and generalizable person re-identification with query-adaptive convolution and temporal lifting,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_. Springer, 2020, pp. 456–474.
  * [23] H. Liang, Z. Ouyang, Y. Zeng, H. Su, Z. He, S.-T. Xia, J. Zhu, and B. Zhang, “Training interpretable convolutional neural networks by differentiating class-specific filters,” in _European Conference on Computer Vision_. Springer, 2020, pp. 622–638.
  * [24] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 3319–3328.
  * [25] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [26] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in _European conference on computer vision_. Springer, 2014, pp. 818–833.
  * [27] Y. Wang, H. Su, B. Zhang, and X. Hu, “Learning reliable visual saliency for model explanations,” _IEEE Transactions on Multimedia_ , vol. 22, no. 7, pp. 1796–1807, 2019.
  * [28] J. Gu and C. Dong, “Interpreting super-resolution networks with local attribution maps,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9199–9208.
  * [29] A. S. Morcos, D. G. T. Barrett, N. C. Rabinowitz, and M. Botvinick, “On the importance of single directions for generalization,” in _International Conference on Learning Representations_ , 2018.
  * [30] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [31] N. Otsu, “A threshold selection method from gray-level histograms,” _IEEE transactions on systems, man, and cybernetics_ , vol. 9, no. 1, pp. 62–66, 1979.
  * [32] B. Le Saux, N. Yokoya, R. Hänsch, and M. Brown, “Data fusion contest 2019 (dfc2019),” 2019. [Online]. Available: https://dx.doi.org/10.21227/c6tm-vw12
  * [33] Rockstar, “Grand theft auto v,” www.rockstargames.com, 2021.
  * [34] I. Loshchilov and F. Hutter, “Fixing weight decay regularization in adam,” 2018\.
  * [35] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [36] W. Van Gansbeke, S. Vandenhende, S. Georgoulis, and L. Van Gool, “Unsupervised semantic segmentation by contrasting object mask proposals,” in _International Conference on Computer Vision_ , 2021.
  * [37] J. H. Cho, U. Mall, K. Bala, and B. Hariharan, “Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering,” in _CVPR_ , 2021.
  * [38] X. Ji, J. F. Henriques, and A. Vedaldi, “Invariant information clustering for unsupervised image classification and segmentation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 9865–9874.

|  Zhitong Xiong received the Ph.D. degree in computer science and technology
from the Northwestern Polytechnical University, Xi’an, China, in 2021. He is
currently a Postdoc with the Data Science in Earth Observation, Technical
University of Munich (TUM), Munich, Germany. His research interests include
computer vision, machine learning and remote sensing.
---|---
|  Sining Chen received the master’s degree in Earth-oriented Space Science
and Technology (ESPACE) from Technical University of Munich (TUM), Germany. He
is currently a Ph.D. student with the Remote Sensing Technology Institute,
German Aerospace Center (DLR) and Data Science in Earth Observation, Technical
University of Munich (TUM), Germany. His research interests include deep
learning and building reconstruction.
---|---
|  Yilei Shi (M’18) received his Diploma (Dipl.-Ing.) degree in Mechanical
Engineering, his Doctorate (Dr.-Ing.) degree in Engineering from Technical
University of Munich (TUM), Germany. In April and May 2019, he was a guest
scientist with the department of applied mathematics and theoretical physics,
University of Cambridge, United Kingdom. He is currently a senior scientist
with the Chair of Remote Sensing Technology, Technical University of Munich.
---|---
|  Xiaoxiang Zhu . (S’10-M’12-SM’14-F’20) received the M.Sc., Dr.Ing., and
Habilitation degrees in signal processing from the Technical University of
Munich (TUM), Munich, Germany, in 2008, 2011, and 2013, respectively. She is
currently the Professor for Data Science in Earth Observation at Technical
University of Munich (TUM) and the Head of the Department “EO Data Science” at
the Remote Sensing Technology Institute, German Aerospace Center (DLR). Since
2019, Zhu is a co-coordinator of the Munich Data Science Research School
(www.mu-ds.de). Since 2019 She also heads the Helmholtz Artificial
Intelligence – Research Field “Aeronautics, Space and Transport”. Since May
2020, she is the director of the international future AI lab ”AI4EO –
Artificial Intelligence for Earth Observation: Reasoning, Uncertainties,
Ethics and Beyond”, Munich, Germany. Since October 2020, she also serves as a
co-director of the Munich Data Science Institute (MDSI), TUM. Prof. Zhu was a
guest scientist or visiting professor at the Italian National Research Council
(CNR-IREA), Naples, Italy, Fudan University, Shanghai, China, the University
of Tokyo, Tokyo, Japan and University of California, Los Angeles, United
States in 2009, 2014, 2015 and 2016, respectively. She is currently a visiting
AI professor at ESA’s Phi-lab. Her main research interests are remote sensing
and Earth observation, signal processing, machine learning and data science,
with a special application focus on global urban mapping.
---|---