Pano Popups: Indoor 3D Reconstruction with a Plane-Aware Network

Marc Eder

Univeristy of North Carolina at Chapel Hill

Chapel Hill, NC

Pierre Moulon

Zillow Group

Seattle, WA

Li Guan

Wormpex AI Research

Seattle, WA

In this work we present a method to train a plane-aware convolutional neural network for dense depth and surface normal estimation as well as plane boundaries from a single indoor image. Using our proposed loss function, our network outperforms existing methods for single-view, indoor, omnidirectional depth estimation and provides an initial benchmark for surface normal prediction from images. Our improvements are due to the use of a novel plane-aware loss that leverages principal curvature as an indicator of planar boundaries. We also show that including geodesic coordinate maps as network priors provides a significant boost in surface normal prediction accuracy. Finally, we demonstrate how we can combine our network's outputs to generate high quality 3D “pop-up" models of indoor scenes.

§ INTRODUCTION

Omnidirectional imaging is currently experiencing a surge in popularity, thanks to the advent of interactive panorama photo sharing on social media platforms, the rise of small, affordable cameras like the Ricoh Theta and Samsung Gear360, and the host of potential applications that arise from capturing wide field of view (FoV) in a single frame. At the same time, deep learning has never been a more useful tool for solving computer vision tasks from object recognition to 3D reconstruction. In order to fully utilize this rising form of media, we must extend existing deep learning methods to the omnidirectional domain. Unfortunately, this is not necessarily a trivial task.

Due to the radically different camera models, deep networks trained on perspective images do not transfer well to omnidirectional images. Omnidirectional images replace the concept of the image plane with that of the image sphere. Yet because we require a 2D planar representation of the image, omnidirectional cameras typically provide outputs as $180^{\circ} \times 360^{\circ}$ FoV equirectangular projections. This representation of the spherical image, while compact, suffers from significant horizontal distortion, especially near the poles.

While there have been a number of efforts to handle the difficulties of equirectangular projections  [1, 2, 3, 5, 25, 26], we are interested in exploring their possible uses. There is excitement over the range of applications of omnidirectional imaging from head-mounted displays to medical scopes to autonomous vehicles. In this paper, we target indoor scene modeling.

Perspective image methods are impeded by a small FoV that is more likely to be limited by featureless, homogeneous regions in an indoor scene. With the larger FoV in images, these homogeneous regions can be reasoned about in the larger context of the scene. Our goal is to predict the dense depth and surface normals for a piecewise-planar reconstruction of the scene. This objective differs from much of the existing work that uses omnidirectional images for indoor 3D modeling. Those, such as RoomNet [14] and LayoutNet [30], aim to generate a simple model of the scene by leveraging a Manhattan World constraint to estimate the dominant planes. That type of model is useful for determining the shapes of rooms and floor-plans of buildings, but not for modeling the objects that comprise the captured scene. While we, too, are essentially estimating planes in the scene, we aim for a more fine-grained model in order to better capture these important details. To this end, we relax the Manhattan constraint to a simple planar one. That is, we assume only that our scene is piecewise-planar.

We use a convolutional neural network (CNN) to predict depth and surface normal estimates per pixel as well as a map of the plane boundaries in the image. We enforce the planar assumption by using a plane-aware loss function that modifies each pixel's contribution to the learning based on its principal curvature. Using our network outputs, we then generate high quality 3D planar models of the scene as seen in Figure <ref>.

We summarize our contributions in this paper as follows:

* We propose a plane-aware cost function to estimate depth, surface normals, and plane boundaries from a single image.

* We demonstrate that the inclusion of geodesic coordinate maps as extra inputs to the network improves surface normal prediction from omnidirectional images.

* We qualitatively show that our network can be used to generate a 3D planar model from a single image.

§ RELATED WORK

§.§ Single-view estimation

There is a significant body of existing research on the task of monocular depth estimation from perspective images. One of the first papers to report success in this task was from Saxena [21], who use a Markov Random Field to infer depth from a blend of local and global image features. With the advent of practical deep learning, more recent methods have focused on applying CNNs to estimate depth. Eigen [7] present a CNN for depth estimation that uses multi-scale predictions to provide coarse and fine supervision for the depth predictions.  Eigen [6] built on that work to simultaneously generate surface normal predictions and semantic labels as well. Dharmasiri [4] follow a similar network design but replace semantic label prediction with principal curvature prediction. Our network architecture has some commonalities with the aforementioned, primarily in our use of multi-scale predictions and similar prediction modalities. However, our goal is more aligned with that of Qi [19] who propose a method for enforcing geometric consistency in the network outputs. In that work, the authors use the depth predictions to refine normal predictions and vice versa. In our case, we use a plane-aware loss to make our network predictions geometrically consistent. Our objective is also somewhat similar to that of Liu [16] who predict a planar segmentation of the scene. However, they rely on a separate plane classification branch in their network and are limited to a fixed number of planes. We use a parametric definition of a plane derived from the principal curvature map and are thus unlimited in the number of planes we can predict.

There have been other recent works in monocular depth estimation that, while interesting and useful, are not currently feasible for our task. Godard [10] use stereo image pairs to train a model for monocular depth estimation using an image reconstruction loss. In our case, we only have access to monocular images. Li and Snavely [15] train a network on a dataset built from large-scale, unordered image collections. Alas, there is not yet such a repository for omnidirectional images.

§.§ Omnidirectional images

The primary distinction between our work and those presented above is the mode of our input data. Most research in monocular depth estimation has relied thus far on perspective image projections. We instead operate on equirectangular image projections, which image a spherical capture oo a plane. This representation carries high levels of distortion. There is an active branch of research in developing solutions to account for these factors. Su and Grauman [25] propose a transfer learning approach to train networks to operate on equirectangular projections. Using an existing perspective-projection-trained network as the target, they train an equirectangular network with a learnable adaptive convolutional kernel to match the outputs. Tateno [26] present a distortion-aware convolutional kernel that convolves over the sampling grid transformed by a distortion function. In this way, the network can be trained on perspective images and still perform effectively on spherical projections. Coors [3] independently derive the same operation and show that it can be highly effective for object detection on images. Both methods train on perspective images and evaluate on spherical projections. Another promising method is the spherical convolution derived by Cohen [1] [2]. Spherical convolutions address the nuances of spherical projections by filtering rotations of the feature maps rather than translations. Most recently, Eder and Frahm [5] demonstrate that resampling spherical images to a subdivided icosahedron substantially improves the performance of CNNs trained on spherical data.  In our work we do not directly address the problem of specialized convolutions. Rather, we explore the application of omnidirectional image inference for the task of indoor 3D modeling. Our work is most similar to that of Zioulis [29] who estimate depth directly from omnidirectional images.

There is also a growing body of work using panorama images to generate indoor scene layouts. Xu [28] fuse object detection and 3D geometry estimation use Bayesian inference to generate 3D room layouts from a single image. Rather than dividing the problem into sub-tasks, Lee [14] use an end-to-end CNN to generate a 3D room layout from a single perspective image. Zhou [30] improve this technique by incorporating vanishing point alignment and prediction additional layout elements to their model. All of the aforementioned layout generation models assume a Manhattan World in their predictions. While this may be useful for common room shapes, it is too simple a prior for general indoor scene modeling. Our work focuses on a more complete indoor 3D model, so we relax this Manhattan constraint to a planar one.

§ PLANE-AWARE ESTIMATION

We present a CNN that estimates dense depth and surface normal predictions as well as a planar boundary map from a single image. To learn depth and normal prediction, we supervise training with ground truth values. Observing that a non-zero principal curvature indicates the presence of a planar boundary, we supervise training for the planar boundary map using the $L_2$ norm of the principal curvature.

§.§ Network architectures

We analyze our plane-aware loss function using a network based on the RectNet architecture used by Zioulis [30]. Our network uses the same encoder-decoder structure with rectangular filter banks on the input layers, but with two decoder branches: one for depth predictions and one for joint surface normal and plane boundary map prediction. We also include skip connections from encoder to decoder layers as in U-Net from Ronneberger [20], as we observe it improves performance. Our network takes a five-channel input: an RGB equirectangular projection and the associated geodesic map containing latitude and longitude coordinates for each pixel. This design is based on the observation that distortion in equirectangular projections is location dependent. Given that these images are indexed by their geodesic coordinates, given in latitude and longitude, we provide the network with location information in the form of a geodesic coordinate map of the image. We find that this provides a significant boost in performance for surface normal prediction in particular and discuss it in more detail in Section <ref>. Figure <ref> provides a detailed overview of our network.

§.§ Training

Recall our premise that each scene is piecewise-planar. This assumption provides a few constraints. First, each scene should be segmented by some web of edges that define the boundaries between each plane. Second, each planar region should have a constant depth gradient and all pixels within should have the same surface normal. Furthermore, the principal curvature, which is effectively the second derivative of depth, should be zero. Lastly, the depth and normal predictions within a planar region should satisfy the plane equation $n^TX + d = 0$, where $n$ is the normal, $X$ is the 3D point, and $d$ is the plane's distance from the origin.

We enforce these constraints through a multi-scale, multi-task loss function. We compute individual losses over the depth, surface normals, and plane boundary map predictions as well as a loss over the plane distance prediction for each pixel, denoted as $L_{z}$, $L_{n}$, $L_{C}$, and $L_{D}$, respectively. This last term is computed as a function of both the depth and normal predictions, which encourages planar consistency. Each of the losses is also weighted using a plane-aware function $P(x)$. For the depth, curvature, and plane distance losses, we use the reverse Huber, or BerHu, loss proposed by Laina [13]. This loss is given as

\begin{equation}
B(x) =
\begin{cases}
|x| & |x| \leq T \\
\frac{x^2 + T^2}{2T} & |x| > T
\end{cases}
\label{eq:berhu}
\end{equation}

where we adjust $T$ on a per-batch basis to be 20% of the max per-batch error as in [13]. Our plane-aware function $P(x)$ weights the impact of each pixel $x$ to the loss by the $L_2$ norm of its ground truth principal curvature, $c^*$:

\begin{equation}\label{eq:penalty}
P(x) = x e^{-||c^*||}
\end{equation}

As curvature is zero on a planar surface, this term gives full weight to all pixels that lie on planes. However, pixels that fall along sharp plane boundaries and thus have higher curvatures will have their contribution to the loss down-weighted. This is similar to the texture-edge-aware loss weighting used by Godard [10], except that we use the curvature values instead of intensity gradients. Our formulation makes more sense for our task, given that we are interested in planar boundaries rather than texture ones.

Each component of the loss is given below. The subscript $i$ denotes the $i$-th pixel in the image; $z_i$ is depth, $n_i$ is normal, and $c_i$ is curvature.

\begin{equation}\label{eq:depthloss}
L_{z} = \frac{1}{|\mathcal{M}|}
\sum_{i \in \mathcal{M}}
P\left(B\left(z_{i} - z_{i}^{*}\right)\right)
\end{equation}

\begin{equation}\label{eq:normalloss}
L_{n} = \frac{1}{|\mathcal{M}|}
\sum_{i \in \mathcal{M}}
P\left(-{n_i}^T n_i^* \right)
\end{equation}

\begin{equation}\label{eq:curvloss}
L_{C} = \frac{1}{|\mathcal{M}|}
\sum_{i \in \mathcal{M}}
P\left(B\left(||c_{i}||_2 - ||c_{i}^{*}||_2\right) + \eta||c_i||_1\right)
\end{equation}

\begin{equation}\label{eq:planeloss}
L_{D} = \frac{1}{|\mathcal{M}|}
\sum_{i \in \mathcal{M}}
P\left(B\left(n_i^T X_{z_i} - {n_i^*}^T X_{z_i^*}\right)\right)
\end{equation}

where $\mathcal{M}$ is the relevant output map and the asterisks denote ground truth values. In Equation (<ref>), $X_{z_i} = z_i \hat{b}$ where $\hat{b}$ is the directional unit vector from the camera center to pixel $i$ on the sphere, i.e. $X_{z_i}$ is the back-projected 3D point.

It is worth noting that other single-view depth estimation papers typically include an $L_2$ penalty on the gradient of the depth or disparity prediction to account for homogeneous regions where depth may be ambiguous [9, 29]. However, this term is known to lead to over-smoothing, especially for surfaces that are not fronto-planar to the camera. In the case of images, where depth is defined as the distance from a 3D point to the camera center (rather than to the image plane), this gradient penalty would encourage the prediction of a circular scene wherein each point is locally fronto-planar to the camera. Thus, we do not penalize the depth gradient at all. In the planar boundary map prediction, however, we do include an $L_1$ penalty to encourage sparsity in the edge predictions.

Our total loss is thus the sum of all of these terms at two scales weighted by some hyper-parameters $\alpha$, $\beta$, $\gamma$, and $\zeta$:

\begin{equation}
L_{total} = \sum_{s\in\{0,1\}} \alpha_s L_{z} + \beta_s L_{\hat{n}} + \gamma_s L_{C} + \zeta_s L_{D}
\label{eq:totalloss}
\end{equation}

We empirically set the hyper-parameters to balance the contribution of each component loss. In our reported results, $\alpha_{0} = 0.3$, $\alpha_{1} = 0.6$,  $\beta_{0} = 0.1$, $\beta_{1} = 0.4$,  $\gamma_{0} = 0$, $\gamma_{1} = 0.3$,  $\zeta_{0} = 0.3$, and $\zeta_{1} = 0.6$. The $L_1$ penalty coefficient in Equation (<ref>) is always $\eta = 0.1$. Nonetheless, we observed that small changes to these hyper-parameters have negligible effects on the network training. Note that we do not use any loss for planar boundary map prediction for the down-scaled prediction ($\gamma_{0}$) as we observed that it made no impact in the final plane boundary map.  We train the network for $20$ epochs with a batch size of $10$ and use the Adam optimizer [12] with an initial learning rate of $2e^{-4}$ decayed by half every $3$ epochs.

§ EVALUATION

In this section we evaluate our proposed plane-aware depth and normal estimation. First, we demonstrate the benefit of our plane-aware loss through comparison to a baseline, the loss used by Zioulis [29], as well as in a series of ablation experiments. Second, we demonstrate the importance of predicting surface normals rather than relying on derived normals from predicted depth. We then examine the effect of including coordinate priors as inputs to the network. Finally, we qualitatively show how we can leverage the predicted plane boundary map to create 3D reconstructions in Section <ref>.

§.§ Dataset

We train and evaluate our method using the Scene Understanding and Modeling (SUMO) dataset [27], a collection of 58,631 computer generated omnidirectional images of indoor scenes derived from SunCG [23]. As released, the SUMO dataset contains RGB-D cube map images with a cube face dimension of $1024$ pixels. To prepare this data for our experiments, we resample the cube maps to $256 \times 512$ pixel equirectangular images using bilinear interpolation for color information and nearest-neighbor interpolation for depth. For the purposes of surface normal and principal curvature prediction, we augment the dataset with normal and curvature maps for each image as well. We derive the ground truth normal maps from the provided images by first resampling them to the vertices of icosahedral triangular mesh as in [5], scaling each vertex by the ground truth depth, computing the surface normal for each face, and rendering the normal maps back into an equirectangular projection. For the ground truth planar boundary maps, we use the $L_2$ norm of the principal curvature. The curvature maps are derived as in [24] using the eigenvalues of the $2\times2$ matrix given by:

$$\mathbf{II} =
\left[\begin{array}{cc}
du & dv
\end{array}\right]
\left[\begin{array}{cc}
A & B \\
B & C
\end{array}\right]
\left[\begin{array}{c}
du \\
\end{array}\right]

where $u$ and $v$ are vectors that, with the surface normal $\hat{n}$, form an orthonormal basis at a given point $(x,y)$. $A$, $B$, and $C$ are defined by the derivatives of the the surface normal at that point:

$$ A = -\frac{\delta \hat{n}(x,y)}{dx} \cdot u,  B = -\frac{\delta \hat{n}(x,y)}{dy} \cdot u,  C = -\frac{\delta \hat{n}(x,y)}{dx} \cdot v$$

§.§ Depth estimation

We evaluate the depth estimation task using the standard set of metrics defined in Eigen [7], shown in Table <ref>. Because depth estimates are subject to the arbitrary scale of the training distribution, we use the median scaling technique given by [29] to normalize the depth distributions during evaluation. The numbers we report are based on pixels whose ground truth depth falls within the range $\left[ 0, T_{depth} \right]$. We set $T_{depth}$ to be 4.375 standard deviations above the mean of the training set, deriving this value from an analysis of the evaluation threshold used by Zioulis [29]. To evaluation our proposed loss, we compare to network training under the loss used by Zioulis [29] as a baseline. This loss is simply an $L2$ minimization with a gradient penalty at two scales, as given by Equation (<ref>):

\begin{equation}\label{eq:omnidepthloss}
L_{baseline} = \frac{1}{|\mathcal{M}|}\sum_{s \in \{0,1\}}\sum_{i \in M}\alpha_s||z_ - z_i^*||_2^2 + \beta_s||\nabla z_||_2^2
\end{equation}

The results in Table <ref> show that our loss formulation outperforms the baseline. We note that the training on synthetic images leads to a high performance for the baseline as well, so we also look to a qualitative analysis to reinforce the effect of our plane-aware formulation. Figure <ref> shows a selection of network outputs comparing our loss to the baseline. Observe the finer-grained depth estimate of lounge chair in the center of row (1) and the shelving and counters in rows (2) and (3). We find that training with our proposed plane-aware loss results in sharper details in the resulting depth maps. We posit that this effect is due to extra supervision provided by the ground truth curvature penalty, which limits smoothing on geometric edges.

We perform an ablation study on elements of our loss function, also listed in Table <ref>. Among other things, these results demonstrate that our improvement is not simply due to the use of the BerHu loss. We see a moderate impact from both the planar-consistency regularizer as well as the curvature penalty. Interestingly, we found that removing the associated curvature prediction task altogether neither affected the depth or normal prediction accuracy. However, we keep it in the network as it plays a key role in generating the 3D reconstructions, discussed in Section <ref>.

§.§ Surface normal estimation

For surface normal estimates, we examine pixels that fall within the same valid ground truth depth range. We evaluate the average angular error per pixel as well as the percentage of pixels whose angular error falls within a threshold of the ground truth. Table <ref> shows that our loss formulation is useful for improving surface normal prediction. As a baseline we use the surface normals derived from the depth predictions. These results indicate that derived normals are no replacement for an independent surface normal prediction. Our predicted normals are much less susceptible to noisy depth values than their derived counterparts. Figure <ref> shows a qualitative comparison of our predicted results compared to the derived normals. When the depth estimation is fairly accurate, the derived normals are only slightly noisier than the prediction, as in row (1). However, in cases where the depth predictions are not as high quality, the predicted normals are often still very good, while the derived normals degrade significantly, as in rows (2) and (3). This effect is why we rely on the indepdendent surface normal prediction branch when generating a 3D reconstruction.

§.§ Geodesic map inputs

We also delve deeper into the impact of the latitude and longitude map priors in the network. Fixing all other aspects of the network, we evaluate the performance of our network on the SUMO dataset with and without the geodesic map channels. Consistent with our expectations, the results in the bottom block of Table <ref> suggest that the geodesic map inputs have a positive impact in surface normal estimation. We surmise that the geodesic map helps the network disambiguate the orientation of the surface normal. It is notable that without the geodesic map, we see errors occur at the peak point of barreling on planes in the equirectangular projection as in the top-left image in Figure <ref>. Interestingly, longitude provides the most important information, which aligns with what we observe in Figure <ref>: predictions changing abruptly along the rows.

Because the equirectangular grid is indexed by spherical coordinates rather than a Cartesian grid, the distance between adjacent pixels is row-dependent as well. Adjacent pixels nearer to the top and bottom of the image actually lie closer together on the sphere than adjacent pixels near the middle of the image do. This sampling scheme is problematic for CNNs because the convolution operation's translation equivariance inherently assumes an even sampling. Somehow the network needs to learn to map the geodesic sampling to a Cartesian one. Our experiments suggest that including the geodesic maps as extra input channels is a useful way to pass this information to the network. These findings line up with the results of Liu [17] who show that incorporating pixel location information can help a network learn some degree of translation dependence, which is what we also need to achieve.

§ 3D PLANAR MODEL RECONSTRUCTION

An important consequence of our planarity assumption is that the network provides all of the information necessary to detect and segment planes in the input images. By defining these planes, we can generate “pop-up" models from a single image, as proposed by Hoiem [11]. Indoor omnidirectional images are uniquely suited to produce these types of reconstructions as they are capable of capturing entire rooms in a single image.

To generate these reconstructions, we first isolate the sharpest edges in the planar boundary map using Otsu thresholding [18] and then identify each connected component in the resulting segmentation. An example of the result of this plane segmentation is shown in Figure <ref>. Thanks to the quality of our plane boundary predictions, this segmentation process requires no threshold tuning. To turn this segmentation into a 3D planar model, we first compute the median normal within each segmented plane. Then, we estimate the distance parameter of the plane equation in each segment using a 1-parameter RANSAC [8] with a final least-squares refinement over the inliers. Lastly, we project each pixel onto its associated plane. The model is finally “popped-up" in 3D by back-projecting the point cloud according to these new depths. We mesh the points by resampling to the vertices of a icosahedral triangular grid and scaling the vertices according to the adjusted depths, resulting in the models shown in Figure <ref>.

Reiterating the importance of surface normal prediction, we found incorporating normal information to be vital to our RANSAC routine. Estimating planes solely from the depth estimates gives a much noisier reconstruction. Furthermore, we observe that having plane information allows us to produce higher quality 3D models than those generated from depth estimates alone. Figure <ref> compares our method, which leverages depth, normals, and boundary information, to the baseline network, which only estimates depth. Where the latter model suffers from smoothed edges, ours is able to produce sharp plane boundaries.

The significant drawback of monocular depth estimation is that the lack of any regularization over the estimates leads to fairly noisy predictions. This stands in contrast to stereo methods (and even pseudo-stereo methods like Godard [10]) in which a second image can be used to ensure consistency in the depth map. However, with our planar assumption, we can resolve some of the depth ambiguity while staying purely monocular. Moreover, the planar constraint removes the dependence on texture to recover depth.  Although making assumptions about the scene may be impractical for specific tasks like autonomous vehicle depth estimation [22], Figure <ref> demonstrates that a simple planarity assumption can be leveraged with great effect for indoor 3D modeling.

§ CONCLUSION

We have presented a CNN capable of predicting depth, surface normals, and planar boundaries from a single indoor image. Using a novel plane-aware loss function, we have achieved state-of-the-art results for these tasks. We have also demonstrated that the inclusion of a geodesic map can improve surface normal estimates for omnidrectional images. Lastly, we have shown that our network provides all the information necessary to produce a 3D planar model of the scene. Looking ahead, we see an emerging opportunity to utilize this type of all-in-one prediction from omnidirectional images to bootstrap indoor 3D reconstruction.

[1]
T. Cohen, M. Geiger, J. Köhler, and M. Welling.
Convolutional networks for spherical signals.
arXiv preprint arXiv:1709.04893, 2017.

[2]
T. S. Cohen, M. Geiger, J. Köhler, and M. Welling.
Spherical cnns.
arXiv preprint arXiv:1801.10130, 2018.

[3]
B. Coors, A. P. Condurache, and A. Geiger.
Spherenet: Learning spherical representations for detection and
classification in omnidirectional images.
In Proceedings of the European Conference on Computer Vision
(ECCV), pages 518–533, 2018.

[4]
T. Dharmasiri, A. Spek, and T. Drummond.
Joint prediction of depths, normals and surface curvature from rgb
images using cnns.
In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ
International Conference on, pages 1505–1512. IEEE, 2017.

[5]
M. Eder and J.-M. Frahm.
Convolutions on spherical images.
arXiv preprint arXiv:1905.08409, 2019.

[6]
D. Eigen and R. Fergus.
Predicting depth, surface normals and semantic labels with a common
multi-scale convolutional architecture.
In Proceedings of the IEEE International Conference on Computer
Vision, pages 2650–2658, 2015.

[7]
D. Eigen, C. Puhrsch, and R. Fergus.
Depth map prediction from a single image using a multi-scale deep
In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q.
Weinberger, editors, Advances in Neural Information Processing Systems
27, pages 2366–2374. Curran Associates, Inc., 2014.

[8]
M. A. Fischler and R. C. Bolles.
Random sample consensus: a paradigm for model fitting with
applications to image analysis and automated cartography.
Communications of the ACM, 24(6):381–395, 1981.

[9]
R. Garg, V. K. BG, G. Carneiro, and I. Reid.
Unsupervised cnn for single view depth estimation: Geometry to the
In European Conference on Computer Vision, pages 740–756.
Springer, 2016.

[10]
C. Godard, O. Mac Aodha, and G. J. Brostow.
Unsupervised monocular depth estimation with left-right consistency.
In CVPR, volume 2, page 7, 2017.

[11]
D. Hoiem, A. A. Efros, and M. Hebert.
Automatic photo pop-up.
In ACM transactions on graphics (TOG), volume 24, pages
577–584. ACM, 2005.

[12]
D. P. Kingma and J. Ba.
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014.

[13]
I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab.
Deeper depth prediction with fully convolutional residual networks.
In 3D Vision (3DV), 2016 Fourth International Conference on,
pages 239–248. IEEE, 2016.

[14]
C.-Y. Lee, V. Badrinarayanan, T. Malisiewicz, and A. Rabinovich.
Roomnet: End-to-end room layout estimation.
In Computer Vision (ICCV), 2017 IEEE International Conference
on, pages 4875–4884. IEEE, 2017.

[15]
Z. Li and N. Snavely.
Megadepth: Learning single-view depth prediction from internet
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 2041–2050, 2018.

[16]
C. Liu, J. Yang, D. Ceylan, E. Yumer, and Y. Furukawa.
Planenet: Piece-wise planar reconstruction from a single rgb image.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 2579–2588, 2018.

[17]
R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and
J. Yosinski.
An intriguing failing of convolutional neural networks and the
coordconv solution.
arXiv preprint arXiv:1807.03247, 2018.

[18]
N. Otsu.
A threshold selection method from gray-level histograms.
IEEE transactions on systems, man, and cybernetics,
9(1):62–66, 1979.

[19]
X. Qi, R. Liao, Z. Liu, R. Urtasun, and J. Jia.
Geonet: Geometric neural network for joint depth and surface normal
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 283–291, 2018.

[20]
O. Ronneberger, P. Fischer, and T. Brox.
U-net: Convolutional networks for biomedical image segmentation.
In International Conference on Medical image computing and
computer-assisted intervention, pages 234–241. Springer, 2015.

[21]
A. Saxena, S. H. Chung, and A. Y. Ng.
Learning depth from single monocular images.
In Advances in neural information processing systems, pages
1161–1168, 2006.

[22]
N. Smolyanskiy, A. Kamenev, and S. Birchfield.
On the importance of stereo for accurate depth estimation: An
efficient semi-supervised deep neural network approach.
arXiv preprint arXiv:1803.09719, 2018.

[23]
S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser.
Semantic scene completion from a single depth image.
IEEE Conference on Computer Vision and Pattern Recognition,

[24]
A. Spek, W. H. Li, and T. Drummond.
A fast method for computing principal curvatures from range images.
arXiv preprint arXiv:1707.00385, 2017.

[25]
Y.-C. Su and K. Grauman.
Learning spherical convolution for fast features from 360 imagery.
In Advances in Neural Information Processing Systems, pages
529–539, 2017.

[26]
K. Tateno, N. Navab, and F. Tombari.
Distortion-aware convolutional filters for dense prediction in
panoramic images.
In Proceedings of the European Conference on Computer Vision
(ECCV), pages 707–722, 2018.

[27]
L. Tchapmi and D. Huber.
The sumo challenge.

[28]
J. Xu, B. Stenger, T. Kerola, and T. Tung.
Pano2cad: Room layout from a single panorama image.
In Applications of Computer Vision (WACV), 2017 IEEE Winter
Conference on, pages 354–362. IEEE, 2017.

[29]
N. Zioulis, A. Karakottas, D. Zarpalas, and P. Daras.
Omnidepth: Dense depth estimation for indoors spherical panoramas.
arXiv preprint arXiv:1807.09620, 2018.

[30]
C. Zou, A. Colburn, Q. Shan, and D. Hoiem.
Layoutnet: Reconstructing the 3d room layout from a single rgb image.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 2051–2059, 2018.

§ EXTENDED RESULTS

In this section, we provide a further qualitative review of our work. Figure <ref> shows more examples of our network's depth estimates compared to our baseline. Figure <ref> provides more cases to justify the prediction of normals independently from depth. Finally, Figure <ref> shows more comparisons of popup reconstructions along with examples of the plane boundary predictions and segmentations.


# Spontaneous Formation of Exceptional Points at the Onset of Magnetism

L. Crippa  Institut für Theoretische Physik und Astrophysik and Würzburg-
Dresden Cluster of Excellence ct.qmat, Universität Würzburg, 97074 Würzburg,
Germany  G. Sangiovanni  Institut für Theoretische Physik und Astrophysik and
Würzburg-Dresden Cluster of Excellence ct.qmat, Universität Würzburg, 97074
Würzburg, Germany  J. C. Budich  Institute of Theoretical Physics, Technische
Universität Dresden and and Würzburg-Dresden Cluster of Excellence ct.qmat,
01062 Dresden, Germany

###### Abstract

We reveal how symmetry-protected nodal points in topological semimetals may be
promoted to pairs of generically stable exceptional points (EPs) by symmetry-
breaking fluctuations at the onset of long-range order. This intriguing
interplay between non-Hermitian (NH) topology and spontaneous symmetry
breaking is exemplified by a magnetic NH Weyl phase spontaneously emerging at
the surface of a strongly correlated three-dimensional topological insulator,
when entering the ferromagnetic regime from a high-temperature paramagnetic
phase. Here, electronic excitations with opposite spin acquire significantly
different lifetimes, thus giving rise to an anti-Hermitian structure in spin
that is incompatible with the chiral spin texture of the nodal surface states,
and hence facilitates the spontaneous formation of EPs. We present numerical
evidence of this phenomenon by solving a microscopic multi-band Hubbard model
non-perturbatively in the framework of dynamical mean-field theory.

In complex quantum many-body systems, physical properties are largely
determined by correlation functions defining quasi-particles and their
fundamental symmetries, rather than by elementary constituents such as bare
electrons [1]. The lifetime of quasi-particles, effectively modeled as an
imaginary damping part of their energy, has played a crucial role in the
literature for decades, e.g. for the understanding of Fermi- and Luttinger-
liquids [2, 3, 4]. More recently, with the advent of non-Hermitian (NH)
topological phases [5, 6], numerous physical phenomena have been discovered
that go conceptually beyond the simple damping of eigenmodes of an underlying
Hermitian system [7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 6, 21,
22], prominently including exceptional points (EPs) as the generic NH
counterpart of nodes in topological semimetals [23, 24, 25, 26, 27, 28, 29,
30, 31]. For such points to manifest, it is crucial that the NH self-energy
$\Sigma$, describing the dissipation of a mode from its interaction with other
degrees of freedom, acquires a matrix structure that is incompatible (non-
commuting) with the free eigenmodes of the system [32, 6, 33]. To this end,
various systems have been considered that explicitly break a symmetry between
some degrees of freedom such as spin or orbital, e.g. by a matrix structure in
their interactions that is inherited by $\Sigma$ [34, 35].

Figure 1: Formation of surface EPs from ferromagnetic fluctuations in a
topological insulator setup with slab-geometry. In the noninteracting case, a
paramagnetic surface semimetal with Dirac dispersion is found around high-
symmetry points in the Brillouin Zone (left). Adding correlations, in a range
of intermediate temperatures the onset of ferromagnetism with small
$z$-magnetization and sizable magnetic fluctuations stabilizes a non-Hermitian
Weyl phase (right). There, the quasi-particle dispersion exhibits a pair of
exceptional points along the $k_{z}$ direction, from which a characteristic
square root-dispersion emanates, and which are connected by a flat non-
Hermitian Fermi arc.

In this work, we reveal a profound interplay between spontaneous symmetry
breaking and NH topology: close to an ordering transition, symmetry breaking
fluctuations may _spontaneously_ give rise to a non-commuting anti-Hermitian
structure, thus inducing topologically stable NH phenomena despite symmetric
microscopic couplings. As a case in point, we present numerical evidence that
the symmetry-protected nodal surface states of a three-dimensional (3D)
topological insulator (TI) can be promoted to a topologically robust magnetic
NH Weyl phase featuring EPs (see Fig. 1 for an illustration). This occurs at
intermediate temperature, where magnetic fluctuations induce a sizable anti-
Hermitian spin dependence, in between a bulk ferromagnetic low-temperature
phase (strong Hermitian spin dependence) and a paramagnetic high temperature
phase (no spin dependence). While the ferromagnetic ground-state may be
understood at Hartree-Fock level, the predicted NH Weyl phase with its
exceptional degeneracies clearly eludes a mean-field description.

To reveal this phenomenon, we solve a multi-orbital Hubbard model for a 3D TI
employing dynamical mean-field theory (DMFT) [36, 37, 38]. In this framework,
the ferromagnetic susceptibility at fractional filling as well as local
dynamical fluctuations giving rise to anti-Hermitian self-energy contributions
are well captured. We extract an effective quasi-particle Hamiltonian, compute
and analyze the phase diagram of the system, and characterize the magnetic NH
Weyl phase. As a spectroscopic hallmark, we find an increased spectral weight
at the Fermi level, signaling an exceptional metallic dispersion rather than
the Dirac semimetallic one of the noninteracting system. We argue that our
present findings exemplify a generic route towards NH topological phases,
where the crucial ingredients are (i) symmetry-breaking fluctuations at the
onset of long-range order, and (ii) the vicinity to a symmetry-protected nodal
phase in the underlying free system.

Exceptional points from magnetic fluctuations. – We consider a Hermitian
quantum-many body system of electrons in which correlations give rise to non-
Hermitian effects at the single-particle level, namely via the imaginary
scattering rates entering the self-energy $\Sigma$. Such NH quasi-particle
properties are conveniently encoded in the effective Hamiltonian, defined as
$H_{\mathrm{eff}}=H(\mathbf{k})+\Sigma(\mathbf{k},\omega)$ [32, 33]. We will
be mainly interested in the physics around the Fermi level, i.e. at $\omega=0$
in our notation. Here, a NH contribution to the self-energy
$\Sigma(\mathbf{k},0)$ arises at finite temperature as a finite lifetime of
the single-particle excitations [1, 32, 39].

In the absence of spontaneous symmetry breaking, the self-energy will possess
the symmetries that are not explicitly broken by the electron-electron
interaction. In the symmetric regime both spin species have the same lifetime,
thus excluding the formation of genuine NH phenomena such as EPs in spin-
space. In contrast, in a long-range ordered phase such as a ferromagnet, a
nontrivial $\Sigma$-matrix structure in spin may spontaneously emerge and, as
we demonstrate in this work, stabilize EPs. We consider the case where, due to
a spontaneous magnetization along the $\hat{z}$ direction, the self-energy
matrix can be expressed in spin space as a linear combination
$\delta\sigma_{0}+\gamma\sigma_{z}$, with $\delta$, $\gamma$ complex due to
finite temperature and correlation strength. While $\delta$ amounts to a
constant (complex) energy shift, a finite $\gamma$ can generate exceptional
points along the $k_{z}$ direction when added to a linearized Dirac
Hamiltonian of the type $\mathbf{d}\cdot\overrightarrow{\sigma}$, where
$\mathbf{d}\propto[k_{z},0,k_{x}]$ [40, 6]. Generalizing from this minimal
example, magnetic fluctuations with a strong in-plane component are capable of
inducing exceptional points in 2D Dirac systems, as will be shown in the
following within a microscopic lattice model.

Symmetry protected nodal Hamiltonian– One of the two crucial ingredients to
the formation of EPs through ferromagnetism is a symmetry-protected 2D Dirac
dispersion. To achieve it, we start from the 3D Bernevig-Hughes-Zhang (BHZ)
model, described in reciprocal space by the Bloch Hamiltonian [41, 42]

$\begin{split}H_{\mathrm{BHZ}}(\mathbf{k})=&\mathcal{M}(\mathbf{k})\sigma_{0}\otimes\tau_{z}+\lambda(\mathrm{sin}k_{x}\sigma_{z}\otimes\tau_{x}\\\
&+\mathrm{sin}k_{y}\sigma_{0}\otimes\tau_{y}+\mathrm{sin}k_{z}\sigma_{x}\otimes\tau_{x})\end{split}$
(1)

where
$\mathcal{M}(\mathbf{k})=M-\epsilon(\cos{k_{x}}+\cos{k_{y}}+\cos{k_{z}})$ is
the scalar expression for the diagonal hopping term and $M$ is a constant
local splitting between two opposite-parity orbitals. The Pauli matrices
$\sigma_{\\{0,x,y,z\\}}$ and $\tau_{\\{0,x,y,z\\}}$ act on the spin and
orbital subspaces, respectively. We choose $\epsilon=1$ as the unit of energy,
and set the spin-orbit coupling strength to $\lambda=0.3\epsilon$. In the
following, we consider the case $M=0$, where the two orbitals are equally
occupied: differently from the 2D BHZ model, the system is still insulating
for this parameter set, and realizes a Weak Topological Insulator (WTI) phase
[43] protected by time-reversal symmetry. We study a finite slab geometry of
the model, finite along the the spatial $y$-direction: in this setup, the
Hamiltonian of the BHZ slab [41] will show gapless surface states crossing at
the high-symmetry points $X=[\pi,0]$ and $Z=[0,\pi]$. In the proximity of such
points, the dispersion is linear in $k_{x}$ and $k_{z}$, and coupled to the
Dirac matrices $\sigma_{z}\otimes\tau_{x}$ and $\sigma_{x}\otimes\tau_{x}$
respectively. In particular, in the spin subspace it is of the characteristic
Dirac form $\mathbf{d}\cdot\overrightarrow{\sigma}$, where $\mathbf{d}=[\pm
k_{z},0,\mp k_{x}]$. We note that the surface states of the WTI are neither
protected against TRS-breaking perturbations, nor against weak disorder which
entails a cell-doubling and projects the surface Dirac points on top of each
other [44].

From the form of the bulk Hamiltonian, it is easy to see that a NH
perturbation of the type $i\zeta\sigma_{z}\otimes\tau_{x}$ (for real $\zeta$)
in spin-orbital space guarantees the presence of exceptional points.
Explicitly, the Hamiltonian in the spin subspace becomes non-diagonalizable if
and only if $k_{x}=0$ and $k_{z}=\pm\zeta$, thereby realizing two
$\mathrm{EPs}$ around the $X$ and $Z$ high symmetry points along the $k_{z}$
direction. We can further simplify the form of such a NH perturbation term in
a slab geometry that is thick enough that the exponentially decaying surface
states do not hybridize significantly [41, 42]. Under these conditions, the
projected Hamiltonian of the surface states is of the general form
$H_{\mathrm{surf}}=\lambda\langle\psi_{0}|\tau_{x}|\psi_{0}\rangle(\sigma_{x}k_{z}+\sigma_{z}k_{x})$,
where $\psi_{0}$ is the 2-component spin up/down part of the TRS-related
4-component surface eigenstates $\Psi_{\uparrow}=[\psi_{0},0]$ and
$\Psi_{\downarrow}=[0,\psi_{0}]$ [45]. The orbital subspace structure is now
``embedded" in a finite scalar coefficient, and a simple imaginary
ferromagnetic term of the form $\sigma_{z}\otimes\tau_{0}$ is equally
effective in generating EPs.

Figure 2: (color online) Magnetization $m_{z}$ as a function of $T$ for fixed
filling $n=0.39,U=13,U^{\prime}=7,J=6$. The yellow shaded region represents
the area where the magnetization is low enough to allow for EPs to appear at
the Fermi energy. At larger $T$ (grey shaded), the system is in a paramagnetic
phase. At smaller $T$ (white region), the system is ferromagnetic, but the
difference in the real part of the spin-dependent self-energies is so high as
to surmount the topological bandgap.

Interaction-mediated ferromagnetism– The second ingredient necessary to
stabilize EPs is here represented by magnetic fluctuations. In strongly
correlated electron systems, it is well established that Hund's coupling can
stabilize ferromagnetic phases in a wide range of fractional fillings for
multi-orbital models [46, 47]. However, the above BHZ slab model does not
belong to this class of systems, as the surface Dirac point is pinned at half-
filling. There, for large enough $U$, an antiferromagnetic phase is expected
to develop instead [48, 49, 50]. To identify candidate systems that are
susceptible to ferromagnetic order, we note that numerous compounds in both 2D
and 3D exist that exhibit Dirac or Weyl physics away from half-filling, e.g.
in the kagome and pyrochlore lattice family [51, 52, 53, 54]. As a minimal
model for driving the Dirac points away from half-filling, we amend the 3D BHZ
Hamiltonian of Eq. (1) by adding a third orbital with a trivial cosine
dispersion. We refer to this additional orbital with the label $1$, while the
two BHZ orbitals are $2$ and $3$, respectively. In this simple realization,
orbital $1$ is coupled to the others only through the many-body interaction
term $H_{\mathrm{int}}$. The full Bloch Hamiltonian for the system then
becomes

$H(\mathbf{k})=\mathcal{M}_{1}(\mathbf{k})\sigma_{0}\oplus
H_{BHZ}(\mathbf{k}),$ (2)

where $\mathcal{M}_{1}(\mathbf{k})$ is defined analogously to
$\mathcal{M}(\mathbf{k})$ (see Eq.(1)) with parameters $M_{1}=-3.0\epsilon$
and $\epsilon_{1}=0.1\epsilon$, so as to be far from the Fermi level. To
account for electronic interactions, we choose the full Hubbard-Kanamori
potential [55, 56], which in real space can be written as

$\begin{split}&H_{\mathrm{int}}=U\sum_{il}n_{il\uparrow}n_{il\downarrow}+\sum_{i,l<l^{{}^{\prime}},\sigma\sigma^{{}^{\prime}}}(U^{\prime}-J\delta_{\sigma\sigma^{{}^{\prime}}})n_{il\sigma}n_{il^{{}^{\prime}}\sigma^{{}^{\prime}}}\\\
&-J\sum_{il\neq
l^{{}^{\prime}}}(c^{\dagger}_{il\uparrow}{c}_{il\downarrow}c^{\dagger}_{il^{{}^{\prime}}\downarrow}c_{il^{{}^{\prime}}\uparrow})+J\sum_{il\neq
l^{{}^{\prime}}}(c^{\dagger}_{il\uparrow}{c}^{\dagger}_{il\downarrow}c_{il^{{}^{\prime}}\downarrow}c_{il^{{}^{\prime}}\uparrow})\end{split}$
(3)

where $i$ represents the site and $l,l^{{}^{\prime}}$ the orbitals. The first
two terms refer to the density-density electron-electron interaction and
Hund's coupling, while the last two terms describe spin-flip and pair-hopping
processes.

This represents the most general two-body interaction form for a 3-orbital
setup. In particular, we choose the Hund's coupling $J$, of crucial importance
for the insurgence of ferromagnetism, to be rather large. The specific set of
interaction parameters used in the following is thus
$(U,U^{\prime},J)=(13.0,7.0,6.0)$ in units of $\epsilon$.

This strongly correlated model is treated numerically making use of DMFT in
its real-space formulation, solved by continuous time quantum Monte Carlo
(CTQMC) [38, 57, 58, 59, 60]. Here, every layer of the system in the finite
spatial direction is mapped to an inequivalent impurity model coupled to a
bath. This approximation captures all local quantum fluctuations of the
interacting lattice problem and the resulting self-energy is purely local.
Notwithstanding the absence of momentum dependent and inter-layer correlations
beyond mean field, real-space DMFT is expected to be well justified for our
analysis: as recently shown with numerical and analytical cluster methods on
similar models [61], nonlocal correlation effects become relevant only when
the contribution from the local self-energy is very small. Such regime is
however far from the case of our ferromagnetic setup for which we can conclude
that non-local self-energy components amount at most to modest perturbations,
against which the predicted EPs are globally robust. The slab we consider has
20 layers stacked along $\hat{y}$: at this thickness, the surface states are
sufficiently decoupled from each other and the self-energy of the internal
layers settles to a uniform bulk value, consistent with a ``thick slab''
geometry. This we have also confirmed by further increasing the number of
layers results, finding only minimal quantitative changes.

To stabilize the ferromagnetic phase, the value of the chemical potential
$\mu$ is adjusted to obtain an occupation between 0.3 and 0.4, i.e.
sufficiently away from half filling [47]. The temperature $T$ is of crucial
importance both for the magnetic and the topological phase: on one hand, too
high a value of temperature will destroy long-range order and yield a
paramagnetic state, where $\mathrm{Im}\Sigma$ becomes spin-independent. On the
other hand, if $T$ is too low the real (Hermitian) magnetization will be so
large as to completely close the topological bandgap at the chemical
potential, thus rendering the system a bulk ferromagnetic metal. The NH Weyl
phase featuring EPs, then, resides in an intermediate temperature regime,
where sizable magnetic fluctuations coexist with a moderate magnetization at
the onset of ferromagnetism.

In Fig. 2 we show the site-resolved magnetization curve for our three-band
model (see Eqs. (2-3)) for fixed filling $n=0.39$. For the range of
temperatures highlighted by the yellow area, coinciding with the smearing of
the phase transition due to finite-size effects [62], exceptional points exist
that are located precisely at, or very near to the Fermi energy. For higher
values of temperature T, the system is paramagnetic, while for lower T it is a
ferromagnetic bulk metal. It is interesting to note the magnetization profile
reversal happening between surface and bulk around $T\approx 0.085$: for
higher temperatures, the magnetization of the surface decreases more rapidly
than that of the bulk approaching the critical point. This effect, related to
the spin fluctuations in each layer of the slab at high temperature, is in
agreement both with both experiments [63, 64] and theoretical simulations
[65], but is completely unrelated to the EP physics. Indeed, it is sufficient
for the magnetization to be finite and small enough throughout the system to
stabilize them.

Hallmarks of the EPs– We now analyze a series of properties that make the
exceptional character of the surface metallic phase manifest. A first glimpse
of the EP physics comes by analyzing the site-resolved spectral function
$A(\omega)$ on the real frequency axis, shown in Fig. 3. The overall
occupation of the model is around $n\approx 0.39$, well within the stable
ferromagnetic phase.

Figure 3: (color online) Site-resolved spectral function on the real frequency
axis, for different orbitals (color) and spin (full/dashed line). The
parameters of the model are $(U,U^{\prime},J,T,\mu)=(13.0,7.0,6.0,0.13,2.00)$.
Top and bottom panels refer to the slab edge and bulk respectively. The
auxiliary orbital $1$ (green) does not contribute spectral weight at the Fermi
level, i.e. the physics is dominated by the EP and Fermi arc in the protected
surface-state region. In the top inset, an averaged spectral weight for
orbitals 2 and 3, resolved for spin and site, is plotted around the Fermi
level, comparing the surface layer (solid lines) and a bulk layer (dashed
lines).

The data are obtained from the DMFT Matsubara local Green's function through
maximum entropy analytical continuation [66]. Comparing the density profiles
of orbitals $2$ and $3$ at the surface (top panel) and in the bulk (bottom
panel) the presence of additional spectral weight in the first case is clearly
visible. At the Fermi level, $A(\omega)$ features a finite difference between
surface and bulk, as shown by the inset. This suggests that the Fermi surface
of the interacting system has measure grater than 0, i.e. it doesn't consist
simply of a discrete set of points, as is the case for the noninteracting
semimetal. Furthermore, Fig. 3 shows that the weight distribution of the
auxiliary orbital $1$ is divided into two distinct regions: a broad peak below
the Fermi level, in correspondence with the non-interacting band, and an
incoherent dome at higher energy. Orbital 1 does not therefore contribute to
the spectral weight at the Fermi level, and the physics of the exceptional
points fully characterizes the behavior of the system at the given filling.

Figure 4: (color online) Effective Hamiltonian and self-energies for
parameters $(U,U^{\prime},J,T,\mu)=(13.0,7.0,6.0,0.13,2.00)$ a) Real part of
the eigenvalue dispersion of the effective Hamiltonian near the Fermi level.
The bands are plotted along the $k_{z}$ direction for the $k_{x}$ value at
which the EPs sit. The two EPs have positive (solid circle) and negative
(dotted circle) vorticity $\nu$, respectively. Between the EPs, the real
energy dispersion of the surface bands is zero along a line (non-Hermitian
Fermi arc) whose shape is shown in the inset. b-c) Real and imaginary parts of
the orbital-diagonal self-energy of orbital $2$ on the Matsubara axis (orbital
$3$ is analogous). Color encodes spin, and the multiple lines refer to the
different planes of the slab. d) Imaginary part of the eigenvalue dispersion
of the effective Hamiltonian. Between the EPs, the energy dispersion of the
surface bands is purely imaginary.

The most evident confirmation of the presence of EPs comes by studying the
complex eigenvalue dispersion of the effective Hamiltonian at zero frequency
$H_{\mathrm{eff}}(\mathbf{k})=H(\mathbf{k})+\Sigma(\mathbf{k},\omega=0)$ [67,
32, 28]. Fig. 4a shows the real energy dispersion of $H_{\mathrm{eff}}$ along
a momentum direction parallel to $k_{z}$ in the two-dimensional Brillouin
Zone. We only plot the bands possessing orbital character $2$ and $3$ in an
energy window near the Fermi level, where the relevant physics is
concentrated. As a consequence of the spin-dependent self-energy (a diagonal
component of which is plotted in Fig. 4b and c as an example), distinctive NH
topological features emerge in the effective bandstructure: the surface bands,
which traverse the topological bandgap, acquire a characteristic square-root
dispersion [6, 32], and touch at the two exceptional points, highlighted by
the black circles. In the momentum region between the EPs, a zero-energy
dispersion is found: this is the NH Fermi arc, responsible for the increased
spectral weight of the system at the Fermi level. Here, the surface state
eigenvalues of $H_{\mathrm{eff}}$ are purely imaginary (see Fig. 4d). To
confirm the topological protection of the EPs, we calculate the associated
invariant known as the vorticity $\nu$: this quantity, unique to the non-
Hermitian realm, is defined in the space of complex quasi-particle energies
rather than the manifold of eigenvectors, and takes the form of a winding
number of the energy spectrum [28, 34]. For our finite-size system, it can be
readily obtained from the dispersion of the surface bands via the expression

$\nu=\dfrac{1}{2\pi}\oint
d^{2}\mathbf{k}\mathbf{\nabla}_{\mathbf{k}}\mathrm{arg}[\Delta E(\mathbf{k})]$
(4)

where $\Delta E(\mathbf{k})$ is the complex energy difference at the given
$k$-point and the integral is done along a path encircling the EP. By
numerically performing the integration, we can confirm the topological
protection of the left and right EP-candidates in Fig. 4, which possess $\pm
1/2$ vorticity respectively. We can further assess the robustness of the EPs
via a final observation: while a self-energy proportional to $i\sigma_{z}$
splits the Hermitian Dirac cone into two EPs that are connected by a Fermi arc
along $k_{z}$, the real and orbital off-diagonal parts of $\Sigma$ generally
do not favor this mechanism. However, such terms only manifest in a shift of
the EPs along the $k_{x}$ direction 111See Supplemental Material at [URL will
be inserted by publisher] for further details on the surface state
Hamiltonian, effective band dispersion and full characterization of the
topological gap. and in a curvature of the Fermi arcs connecting them (see the
inset of Fig. 4a), highlighting the topological robustness of EPs.

Conclusion– We have demonstrated how symmetry-protected surface Dirac cones
can be promoted to pairs of generically stable exceptional points by means of
symmetry-breaking fluctuations. As a case in point, we have discussed an
extended weak topological insulator model, the nodal surface states of which
split into EPs as a consequence of ferromagnetic fluctuations. Ferromagnetic
ordering has been facilitated by augmenting the model by a third band so as to
drive the Dirac points away from half filling. The topological protection of
the EPs strengthens the gapless surface phase of the system, which in the non-
interacting limit is susceptible to both time-reversal symmetry breaking and
weak disorder. Furthermore, the free semi-metallic spectral density at the
surface is enhanced to a metallic one due to the presence of Fermi arcs
connecting the EPs.

Conceptually different from previous predictions of EPs, our setup does not
require any ad-hoc assumption on the orbital- or spin-dependence of the
interaction parameters, nor does it build on coupling the system to
environmental degrees of freedom. Instead, the nontrivial anti-Hermitian
structure inducing the topologically stable exceptional points inherently
emerges from spontaneous symmetry breaking.

Generalizations of our case study to other ordering transitions are readily
conceivable. When spontaneous symmetry breaking selects the quantum number of
some degree of freedom, a nontrivial matrix structure in this degree of
freedom appears in correlation functions with respect to the symmetry broken
state. There, fluctuations will generically induce a sizable anti-Hermitian
contribution close to the transition that may stabilize EPs.

Acknowledgments – This work is funded by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) through Project-ID 258499086-SFB 1170,
Project-ID 247310070-SFB 1143 (Subproject A06), the Würzburg-Dresden Cluster
of Excellence on Complexity and Topology in Quantum Matter –ct.qmat Project-ID
390858490-EXC 2147, and RE1469/13-1. The authors gratefully acknowledge the
Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this
project by providing computing time on the GCS Supercomputer SuperMUC at
Leibniz Supercomputing Centre (www.lrz.de).

## References

  * Abrikosov and Khalatnikov [1959] A. A. Abrikosov and I. M. Khalatnikov, Reports on Progress in Physics 22, 329 (1959).
  * Tomonaga [1950] S. Tomonaga, Progress of Theoretical Physics 5, 544 (1950).
  * Luttinger and Ward [1960] J. M. Luttinger and J. C. Ward, Phys. Rev. 118, 1417 (1960).
  * Luttinger [1963] J. M. Luttinger, Journal of Mathematical Physics 4, 1154 (1963).
  * Ashida  _et al._ [2020] Y. Ashida, Z. Gong, and M. Ueda, Advances in Physics 69, 249 (2020).
  * Bergholtz  _et al._ [2021] E. J. Bergholtz, J. C. Budich, and F. K. Kunst, Rev. Mod. Phys. 93, 015005 (2021).
  * Rudner and Levitov [2009] M. S. Rudner and L. S. Levitov, Phys. Rev. Lett. 102, 065703 (2009).
  * Zeuner  _et al._ [2015] J. M. Zeuner, M. C. Rechtsman, Y. Plotnik, Y. Lumer, S. Nolte, M. S. Rudner, M. Segev, and A. Szameit, Phys. Rev. Lett. 115, 040402 (2015).
  * Lee [2016] T. E. Lee, Phys. Rev. Lett. 116, 133903 (2016).
  * Zhou  _et al._ [2018] H. Zhou, C. Peng, Y. Yoon, C. W. Hsu, K. A. Nelson, L. Fu, J. D. Joannopoulos, M. Soljačić, and B. Zhen, Science 359, 1009 (2018).
  * Lieu [2018] S. Lieu, Phys. Rev. B 97, 045106 (2018).
  * Gong  _et al._ [2018] Z. Gong, Y. Ashida, K. Kawabata, K. Takasan, S. Higashikawa, and M. Ueda, Phys. Rev. X 8, 031079 (2018).
  * Kunst  _et al._ [2018] F. K. Kunst, E. Edvardsson, J. C. Budich, and E. J. Bergholtz, Phys. Rev. Lett. 121, 026808 (2018).
  * Yao and Wang [2018] S. Yao and Z. Wang, Phys. Rev. Lett. 121, 086803 (2018).
  * Yao  _et al._ [2018] S. Yao, F. Song, and Z. Wang, Phys. Rev. Lett. 121, 136802 (2018).
  * Nakagawa  _et al._ [2018] M. Nakagawa, N. Kawakami, and M. Ueda, Phys. Rev. Lett. 121, 203001 (2018).
  * Yuce [2020] C. Yuce, Annals of Physics 415, 168098 (2020).
  * Xiao  _et al._ [2020] L. Xiao, T. Deng, and K. e. a. Wang, Nature Physics 16, 761 (2020).
  * Budich and Bergholtz [2020] J. C. Budich and E. J. Bergholtz, Phys. Rev. Lett. 125, 180403 (2020).
  * Weidemann  _et al._ [2020] S. Weidemann, M. Kremer, T. Helbig, T. Hofmann, A. Stegmaier, M. Greiter, R. Thomale, and A. Szameit, Science 368, 311 (2020).
  * Okuma and Sato [2021] N. Okuma and M. Sato, Phys. Rev. Lett. 126, 176601 (2021).
  * Yoshida [2021] T. Yoshida, Phys. Rev. B 103, 125145 (2021).
  * Kato [1966] T. Kato,  _Perturbation Theory for Linear Operators_, Classics in Mathematics (Springer Berlin Heidelberg, 1966).
  * Berry [1994] M. Berry, Current Science 67, 220 (1994).
  * Heiss and Harney [2001] W. D. Heiss and H. L. Harney, The European Physical Journal D - Atomic, Molecular, Optical and Plasma Physics 17, 149 (2001).
  * Heiss [2012] W. D. Heiss, Journal of Physics A: Mathematical and Theoretical 45, 444016 (2012).
  * Heiss [2016] D. Heiss, Nature Physics 12, 823 (2016).
  * Shen  _et al._ [2018] H. Shen, B. Zhen, and L. Fu, Phys. Rev. Lett. 120, 146402 (2018).
  * Yang  _et al._ [2021] Z. Yang, A. P. Schnyder, J. Hu, and C.-K. Chiu, Phys. Rev. Lett. 126, 086401 (2021).
  * Peters  _et al._ [2021] R. Peters, K. Kimura, Y. Michishita, T. Yoshida, and N. Kawakami, Phys. Rev. B 104, 235153 (2021).
  * Stålhammar and Bergholtz [2021] M. Stålhammar and E. J. Bergholtz, Phys. Rev. B 104, L201104 (2021).
  * Kozii and Fu [2017] V. Kozii and L. Fu, Non-hermitian topological theory of finite-lifetime quasiparticles: Prediction of bulk fermi arc due to exceptional point (2017), arXiv:1708.05841 [cond-mat.mes-hall] .
  * Rausch  _et al._ [2021] R. Rausch, R. Peters, and T. Yoshida, New Journal of Physics 23, 013011 (2021).
  * Yoshida  _et al._ [2020] T. Yoshida, R. Peters, N. Kawakami, and Y. Hatsugai, Progress of Theoretical and Experimental Physics 2020, 10.1093/ptep/ptaa059 (2020).
  * Crippa  _et al._ [2021a] L. Crippa, J. C. Budich, and G. Sangiovanni, Phys. Rev. B 104, L121109 (2021a).
  * Georges and Kotliar [1992] A. Georges and G. Kotliar, Phys. Rev. B 45, 6479 (1992).
  * Müller-Hartmann [1989] E. Müller-Hartmann, Zeitschrift für Physik B Condensed Matter 74, 507 (1989).
  * Georges  _et al._ [1996] A. Georges, G. Kotliar, W. Krauth, and M. J. Rozenberg, Rev. Mod. Phys. 68, 13 (1996).
  * Yoshida  _et al._ [2018] T. Yoshida, R. Peters, and N. Kawakami, Phys. Rev. B 98, 035141 (2018).
  * Bergholtz and Budich [2019] E. J. Bergholtz and J. C. Budich, Phys. Rev. Research 1, 012003 (2019).
  * Zhang  _et al._ [2009] H. Zhang, C.-X. Liu, X.-L. Qi, X. Dai, Z. Fang, and S.-C. Zhang, Nature Physics 5, 438 (2009).
  * Liu  _et al._ [2010] C.-X. Liu, X.-L. Qi, H. Zhang, X. Dai, Z. Fang, and S.-C. Zhang, Phys. Rev. B 82, 045122 (2010).
  * Fu  _et al._ [2007] L. Fu, C. L. Kane, and E. J. Mele, Phys. Rev. Lett. 98, 106803 (2007).
  * Fu and Kane [2007] L. Fu and C. L. Kane, Phys. Rev. B 76, 045302 (2007).
  * Note [1] See Supplemental Material at [URL will be inserted by publisher] for further details on the surface state Hamiltonian, effective band dispersion and full characterization of the topological gap.
  * Vollhardt  _et al._ [1999] D. Vollhardt, N. Blümer, K. Held, M. Kollar, J. Schlipf, M. Ulmke, and J. Wahle, in  _Advances in Solid State Physics 38_, edited by B. Kramer (Springer Berlin Heidelberg, Berlin, Heidelberg, 1999) pp. 383–396.
  * Held, K. and Vollhardt, D. [1998] Held, K. and Vollhardt, D., Eur. Phys. J. B 5, 473 (1998).
  * Kubo and Uchinami [1975] K. Kubo and M. Uchinami, Progress of Theoretical Physics 54, 1289 (1975).
  * Yoshida  _et al._ [2013] T. Yoshida, R. Peters, S. Fujimoto, and N. Kawakami, Phys. Rev. B 87, 085134 (2013).
  * Amaricci  _et al._ [2016] A. Amaricci, J. C. Budich, M. Capone, B. Trauzettel, and G. Sangiovanni, Phys. Rev. B 93, 235112 (2016).
  * Guo and Franz [2009] H.-M. Guo and M. Franz, Phys. Rev. Lett. 103, 206805 (2009).
  * Berke  _et al._ [2018] C. Berke, P. Michetti, and C. Timm, New Journal of Physics 20, 043057 (2018).
  * Lim  _et al._ [2020] L.-K. Lim, J.-N. Fuchs, F. Piéchon, and G. Montambaux, Phys. Rev. B 101, 045131 (2020).
  * Li  _et al._ [2021] M. Li, Q. Wang, G. Wang, Z. Yuan, W. Song, R. Lou, Z. Liu, Y. Huang, Z. Liu, H. Lei, Z. Yin, and S. Wang, Nature Communications 12, 3129 (2021).
  * Kanamori [1963] J. Kanamori, Progress of Theoretical Physics 30, 275 (1963).
  * Georges  _et al._ [2013] A. Georges, L. d. Medici, and J. Mravlje, Annual Review of Condensed Matter Physics 4, 137 (2013).
  * Gull  _et al._ [2011] E. Gull, A. J. Millis, A. I. Lichtenstein, A. N. Rubtsov, M. Troyer, and P. Werner, Rev. Mod. Phys. 83, 349 (2011).
  * Potthoff and Nolting [1999] M. Potthoff and W. Nolting, Phys. Rev. B 60, 7834 (1999).
  * Schüler  _et al._ [2017] M. Schüler, S. Barthel, T. Wehling, M. Karolak, A. Valli, and G. Sangiovanni, The European Physical Journal Special Topics 226, 2615 (2017).
  * Wallerberger  _et al._ [2019] M. Wallerberger, A. Hausoel, P. Gunacker, A. Kowalski, N. Parragh, F. Goth, K. Held, and G. Sangiovanni, Computer Physics Communications 235, 388 (2019).
  * Crippa  _et al._ [2021b] L. Crippa, A. Amaricci, S. Adler, G. Sangiovanni, and M. Capone, Phys. Rev. B 104, 235117 (2021b).
  * Binder [1987] K. Binder, Ferroelectrics 73, 43 (1987).
  * Liebermann  _et al._ [1969] L. N. Liebermann, D. R. Fredkin, and H. B. Shore, Phys. Rev. Lett. 22, 539 (1969).
  * Walker  _et al._ [1984] J. C. Walker, R. Droste, G. Stern, and J. Tyson, Journal of Applied Physics 55, 2500 (1984).
  * Ma  _et al._ [2009] P.-W. Ma, C. Woo, and S. Dudarev, Philosophical Magazine 89, 2921 (2009).
  * Kaufmann and Held [2021] J. Kaufmann and K. Held, ana_cont: Python package for analytic continuation (2021).
  * Wang and Zhang [2012] Z. Wang and S.-C. Zhang, Phys. Rev. X 2, 031008 (2012).