X-ray irradiation of three planets around Hyades star K2-136

1,2]Jorge Fernandez Fernandez*
1,2]Peter J. Wheatley

J. Fernandez et al.

[1]Centre for Exoplanets and Habitability, University of Warwick, Gibbet Hill Road, Coventry CV4 7AL, UK
[2]Department of Physics, University of Warwick, Gibbet Hill Road, Coventry CV4 7AL, UK

We study the X-ray irradiation and likely photoevaporation of the three planets around the star K2-136. These are the Earth-sized K2-136 b, the mini-Neptune K2-136 c, and the super-Earth K2-136 d.  observations of the star indicate an X-ray luminosity of $(1.18\pm0.1)\times10^{28}$ erg s$^{-1}$ in the range 0.15 to 2.4 keV, resulting in an activity of $L_X/L_{\rm bol}=(1.80\pm0.68)\times10^{-5}$. The evaporation past of the planets were modelled using the XUV stellar tracks by <cit.>, the energy limited mass loss formulation <cit.>, and the thermal evolution formulation by <cit.>. Our results suggest that planets b and d are most probably purely rocky and have been stripped of their envelopes. On the other hand, planet c likely still has an envelope consisting of 0.8% of the total mass, with its core being relatively large (2.3 $R_{\rm E}$).

J. Fernandez Fernandez, and
P. J. Wheatley (2021),
The photoevaporation of three planets around Hyades star K2-136, Journal, Vol.

§ INTRODUCTION

The Kepler mission has uncovered a surprisingly large population of planets between Earth and Neptune in size. Their radii have been observed to follow a bimodal distribution <cit.>, with twin peaks at 1.3 and 2.4 $R_{\rm E}$, and a gap at 1.8 $R_{\rm E}$. Whilst the first group is populated by dense rocky planets, the second one can be explained by the additional presence of H/He gaseous envelopes that can double a planet's radius and yet comprise less than a percent of its mass. This phenomenon can be observed as a valley on the radius-period parameter space in Figure <ref>. There is evidence that points to stellar high-energy radiation being the main driver for this behaviour <cit.>, although other channels such as core-powered mass loss have also been suggested  <cit.>.

The X-ray and extreme ultraviolet radiation (together XUV) that stars emit is absorbed by the upper layers of the atmospheres of close-in exoplanets. This heats the upper atmosphere driving a hydrodynamic wind that overflows the Roche lobe and escapes the planet. Gradual mass loss sculpts the observed planet populations, particularly during the first 100 Myr when the XUV emission is the strongest, stripping the envelopes from planets that are too close or too light. There is also evidence that EUV radiation exerts significant influence on envelopes on much-longer Gyr timescales <cit.>.

Observations, however, only provide a snapshot of a planet's current evolutionary state. One can attempt to model its current physical properties but its history can be hard to constrain due to uncertainty in the stellar rotation and hence X-ray luminosity history. A straightforward solution to break the degeneracy is to study multiplanetary systems: where all the planets in the system share the same the XUV irradiation history.  Their current states can then be used to constrain each other's pasts, especially if the planets are found on both sides of the evaporation valley <cit.>.

§ THE TARGET

K2-136 is a K-type star located $59$ parsecs away (as determined by Gaia EDR3). It is a member of the Hyades cluster, making it around 600 Myr old <cit.>, although more recent estimates go as high as 800 Myr <cit.>. In this work, we adopt an age of $700\pm100$ Myr. The star also hosts a three-planet system: the Earth-sized K2-136 b, the mini-Neptune K2-136 c, and the super-Earth K2-136 d, which are all situated within 0.2 AU of their host star <cit.>. <cit.> also reports the existence of a candidate stellar-mass companion, a M7/8V dwarf, at a projected separation of 40 AU, although follow-up observations are still needed to confirm the stars are gravitationally bound to each other.

§.§ Planetary system

The three discovery papers of the planetary system <cit.> agree within $1\sigma$ in their estimates for the periods and radii of the three planets. These are $R_b=1.05\pm0.16$ R$_{\rm E}$ and $P_b=7.975$ days for planet b, $R_c=3.14\pm0.36$ R$_{\rm E}$ and $P_c=17.307$ days for planet c, and $R_d=1.55\pm0.24$ R$_{\rm E}$ and $P_d=25.575$ days for planet d. All periods have uncertainties of less than a percent. These are put in context with the evaporation valley on Figure <ref>. It is interesting how the middle planet (c) is the one that remains above the valley whereas the one furthest away is not. This points to planet c having a much heavier core than d in order to avoid having its envelope stripped.

The planet masses have also been recently measured by <cit.>; with a direct estimate for the mini-Neptune ($15.9\pm2.4$ M$_{\rm E}$) and upper limits for the other two ($<2.67$ and $<6.47$ M$_{\rm E}$ for b and d, respectively). We make use of the mass-radius relationships by <cit.> to obtain mass estimates for b and d, and to put planet c in context (Figure <ref>).
The small radii of planets b and d place them below the evaporation valley, probably with purely solid cores with no envelope. Their masses are estimated to be $1.07^{+0.67}_{-0.46}$ M$_{\rm E}$ and $4.09^{+2.63}_{-1.80}$ M$_{\rm E}$ respectively. In contrast, the mass of planet c indicates a large core, and its location above the valley points to the presence of an envelope. As its envelope likely contributes less than a percent of its mass, we fit its mass to the rocky relation and obtain an estimate for the core size of $R_{c,core} = 2.30\pm0.13$ R$_{\rm E}$, hence its envelope would comprise 25% of the total radius.

Radius against orbital period plot of the exoplanet population from <cit.> alongside their fit for the evaporation valley (solid black line, uncertainties as two dotted lines). The three K2-136 planets are plotted in red (b), blue (c), and green (d).

<cit.> mass-radius relations for rocky (red) and volatile-rich (blue) planets, alongside the three K2-136 planets.

§.§ The Hyades

The Hyades is a well studied nearby open cluster, with several extensive X-ray studies <cit.> as well as rotational surveys <cit.>. <cit.> compiled 1100 Hyades members from several studies, both from the core of the cluster and its recently discovered tidal tails. About 130 of these cluster members have measurements for both the rotation periods and the X-ray luminosity. The data presents a tight rotation-spectral type relation for F, G, and K-dwarfs (Figure <ref>, left panel), with M-dwarfs being much more scattered. According to <cit.>, M-dwarfs retain their initial distribution of rotation rates for longer before settling into a single mass-dependent value, joining the tight relation of F,G, and K-type stars. K2-136 fits well on the tight relation. The stray K-type stars at faster rotation rates below the tight relation are likely close binaries (and indeed we see they are offset from the main sequence in an H-R diagram).

Left: rotation period of Hyades members versus colour index (Gaia blue and red magnitudes, BP–RP). Right: X-ray luminosity against Rossby number for Hyades members. The rotation-activity relation by <cit.> is plotted in green and the relation for Hyades members  by <cit.> in magenta. Data have been taken from <cit.>. K2-136 is plotted as a black star in both panels.

§ OBSERVATIONS

We observed the target with the  telescope in November 2018 for 43 kiloseconds. Data reduction was performed on the EPIC-pn data, which has an operative energy range of 0.15 to 12 keV. The spectrum was modelled with a TBABS model <cit.> to account for interstellar absorption as well as an APEC model <cit.> for the emission spectrum from a collisionally-ionized diffuse plasma. Solar abundances by <cit.> were used. The hydrogen column density was set to $n_H = 10^{18}$ cm$^{-2}$, as estimated by <cit.> for the Hyades. Furthermore, the ROSAT-range flux was estimated by extrapolating the model down to 0.1 keV, and the EUV flux from the relations by <cit.>.

§ DATA ANALYSIS

Since the source is only detected up to 2.4 keV, only the range 0.15 to 2.4 keV was used for model fitting. Due to the low number of counts per bin, we use the C statistic as the fit parameter <cit.>. We find that a three-temperature APEC model works best for our data, with temperatures $0.14^{+0.05}_{-0.06}$, $0.29^{+0.17}_{-0.06}$, and $0.97^{+0.17}_{-0.14}$ keV. The spectrum is plotted in Figure <ref>.

Integrating the model yields a flux of $F_X = 2.83\pm0.25 \times 10^{-14} $ erg cm$^{-2}$ s$^{-1}$, resulting in a luminosity of $1.18 \pm 0.1 \times 10^{28}$ erg s$^{-1}$ in the range 0.15 to 2.4 keV. Extrapolating down to an energy of 0.1 keV yields a ROSAT-range luminosity of $1.32\pm0.12\times10^{28}$ erg s$^{-1}$.

Some of this X-ray flux is likely to arise from the close-by M-dwarf companion. We estimate this from the measured 2MASS fluxes of the candidate companion, which <cit.> find to be $J = 14.1\pm0.1$, $H = 13.47\pm0.04$, and $Ks= 13.03\pm0.03$. Interpolating with the stellar quantity sequences by <cit.>, we estimate its bolometric luminosity to be $L_{\rm bol} = 2.00\pm0.35\times10^{30}$ erg s$^{-1}$, about $5\times10^{-4}$ times lower than that of K2-136. Since <cit.> finds that stars tend to be saturated at an X-ray activity of $log_{10} R_X = -3.13\pm0.08$, and we assume that such a low mass star remains X-ray saturated even after 800 Myr <cit.>, we can estimate its X-ray luminosity to be $L_X \approx 1.48\pm0.38\times10^{27}$ erg s$^{-1}$, about 10% of the flux from K2-136 measured in this work, which is comparable to the uncertainty in our measurement.

Finally, we estimate the EUV flux using the relations by <cit.> and thus obtain a value of $L_{\rm EUV} = 2.38\pm0.17\times10^{28}$ erg s$^{-1}$ in the range 0.0136 to 0.15 keV. This results in a combined XUV luminosity of $L_{\rm XUV}=3.7\pm0.2\times10^{28}$ erg s$^{-1}$, three times stronger than X-rays alone.

Plotting the star on the rotation-activity relation with its cluster siblings (Figure <ref>, right panel) indicates good agreement with the other Hyades K-dwarfs. It is also worth noting the large scatter in activity around the fit, up to one order of magnitude. It is yet to be explained whether this originates from random variability or intrinsic tracks for each star.

X-ray spectrum of K2-136 as observed with the  telescope. The model fitted (blue) is described in Sections <ref> and <ref>.

§ EVOLUTION MODELLING

In order to explore the pasts of the planetary envelopes, three ingredients are necessary: the XUV emission track of the host star as it spins down, a formulation for the mass loss that updates the envelope mass on each time step, and a description of the planetary interior that relates the envelope mass fraction to its size, vital for recalculating the envelope radius after mass is lost on each time step.

§.§ Stellar tracks

The XUV emission of a star is tied to its Rossby number, the ratio between rotation period and convective turnover time <cit.>. Stars spin down with age, which results in a decrease of the star's emission with time, eventually diminishing its capability to evaporate atmospheres past a few Gyr.

We make use of the physically motivated X-ray tracks proposed by <cit.>, which model the radiative core and convective envelope of F, G, and K-type stars as two independently rotating solid components. Each component has certain angular momentum affected by torques of different origins (stellar wind, momentum exchange, and PMS spin-up), which can be used to predict the star's rotation and XUV dependence on age. These tracks are plotted in Figure <ref> (left panel) for a star of the mass of K2-136 ($0.7$ M$_{\odot}$) and a distribution of initial rotation rates. K2-136 seems to fit well with the average rotation track (50th percentile).

Stellar tracks from the model by <cit.>. Left: possible rotation tracks for a 0.7 $M_{\odot}$ star given a range of initial rotation rates; 5th percentile (blue), 50th percentile (red), and 95th percentile (green). The track that best fits K2-136 is the solid black line, with the uncertainties set by the two dotted black lines. Right: X-ray activity $R_X = L_X / L_{\rm bol}$ (black line). The X-ray (red) and EUV (blue) tracks are plotted separately as well. K2-136 is plotted as a magenta star.

§.§ Thermal evolution

We adopt the envelope structure model described by <cit.>, which models the cooling and contraction of adiabatic envelopes assuming a hot-start model. The planet's energy budget is tracked by taking into account heat from different sources, such as stellar radiation, radioactive decay from heavy elements, and thermal inertia from the isothermal rocky core. Although they begin their models at 1 Myr, the hot-start assumption might result in over-inflated radii (>10 R$_{\rm E}$) for low-mass planets at early ages. Despite that, these envelopes cool down and contract quickly and should be insensitive to the choice of initial entropy by the age of 10 Myr. Computed model tables and polynomial fits are provided by <cit.>, which only run from 100 Myr to 10 Gyr, however. Regardless, we choose a lower age limit of 10 Myr in our simulations as most of the mass loss occurs before 100 Myr.

§.§ Mass loss

The photoevaporation is modelled using the energy-limited approach <cit.>. This balances the incident energy flux from the star against the energy necessary to displace matter from the planet’s upper atmosphere to its Roche lobe. The mass loss rate is given by

\begin{equation}
\dot{M} = \frac{\beta^2 \eta \pi F_{\rm XUV} R^3_p}{G K M_p},
\label{eqn:massloss}
\end{equation}

where $M_p$ and $R_p$ are the mass and radius of the planet, respectively, $F_{\rm XUV}$ is the incident combined X-ray and EUV flux into the planet, $\beta$ is the ratio between the radii at which the atmosphere becomes optically thick for XUV wavelengths and for visible wavelengths ($R_{\text{\rm XUV}}/R_p$), $\eta$ is the energy efficiency of the process, and $K$ accounts for the potential energy difference between the surface and the height of the Roche-lobe (see <cit.> for more details). This approach is attractively simple, although it does mask more complex physics behind the $\eta$ and $\beta$ parameters.

Generally, constant canonical values for these two parameters are used, but there more sophisticated formulations as well. For instance, <cit.> perform hydrodynamic simulations to derive scaling laws for the efficiency and the XUV radius based on both the gravitational potential of a planet and its irradiation level. In this work, we adopt a canonical efficiency of $\eta = 15\%$ and a XUV radius ratio $\beta = 1$ <cit.>, and we compare with the formulation by <cit.> for $\beta$.

§.§ Simulation results

Planets c and d were de-evolved from their current states back to 10 Myr. This assumes that planet d just finished losing its now-missing envelope, which sets an upper limit for the total mass lost. Using an efficiency of $\eta=15$% and a XUV radius ratio of $\beta=1$, at 10 Myr we obtain envelope mass fraction of 0.83% for planet c, expanding from 3.14 to $\sim$3.8 R$_{\rm E}$, and 0.12% for planet d, inflating its radius from 1.55 to $\sim$2.15 R$_{\rm E}$.

We also evolved forwards two synthetic tracks for a planet with a core identical to planet b and envelope mass fractions equal to those of planets c and d at 10 Myr (as computed above). These tracks are evolved from the age of 10 Myr up to 700 Myr. The results show that, whichever envelope planet b had, it was likely lost within 5 to 10 Myr. This is expected, as b is the smallest and closest planet, thus more susceptible to photoevaporation. Planet b is therefore less useful in constraining the evaporation pasts of the system. Our evolutionary tracks are shown in Figure <ref> (top panel).

We also compare these results with tracks generated using an XUV radius that follows the relation by <cit.> (Figure <ref>, bottom panel). In both cases, an efficiency of 15% is used. We note a mass loss enhancement 2 to 5 times greater. This results in planets c & d having a remarkably similar envelope mass fraction of about $\sim$0.9% at 10 Myr.

Top: planet radius against age. Planets c (green line) and d  (blue) were de-evolved from their current age of 700 Myr. Using their envelope mass fractions at 10 Myr, planet b was evolved forward to 700 Myr, with initial fractions of $M_{env}/M_p=0.83\%$ (red dotted line) and $M_{env}/M_p=0.12\%$ (magenta dashed line). The envelope loss timescales for planet b were estimated to be $\sim$2 and $\sim$4 Myr. Bottom: comparison between an extended radius of $\beta = 1$ and the formulation by <cit.>, ran using the tracks for planets c (green) and d (blue). An efficiency of $\eta=15\%$ was used.

§ DISCUSSION & CONCLUSION

Using  measurements, we estimate the X-ray luminosity of K-type star K2-136 to be $1.18 \pm 0.1 \times 10^{28}$ erg s$^{-1}$ in the range 0.15 to 2.4 keV. The star, with an age of $700\pm100$ Myr, is about 50% X-ray fainter than the luminosity predicted by <cit.> in their stellar XUV tracks (see Figure <ref>). The discrepancy is not unexpected, however, given the relatively large scatter of X-ray luminosities in relation to age <cit.>.

We modelled the evaporation pasts of the three planets in the system using the stellar tracks by <cit.>, the energy-limited mass loss formulation <cit.>, and the thermal evolution model by  <cit.>. We find that planets b and d are most likely rocky, given their current density and size, and planet c possesses a gaseous envelope that consist of 0.8% of its total mass. Using the extended radius formulation by <cit.>, we obtain similar initial envelope mass fractions for planets c & d of 0.9% at 10 Myr. <cit.> finds the distribution of envelope mass fractions to be strongly peaked at 4% for their sample of exoplanets, which might be attained by evolving our planets further back in time to 1 Myr with a alternate thermal evolution formulation.

Further mass loss might be attained by taking into account additional mass loss sources, such as stellar wind and Coronal Mass Ejections (CMEs). The degree to which these phenomena influence the atmospheres of mini-Neptunes is still under study. <cit.> found it to be several times lower than thermal mass loss rates. Fast CMEs, however, could have a greater influence on planets that are extremely close-in (<0.1 AU), or lack a significant magnetosphere <cit.>.
The planetary evolution code used in this work is available on GitHub at <https://github.com/jorgefz/photoevolver>

, Grevesse
, Sauval

Asplund, M.
, Grevesse, N.
, Sauval, AJ.
Scott, P.

Brandt, TD.
Huang, CX.

Cash, W.

Ciardi, DR.
, Crossfield, IJM.
, Feinstein, AD.
et al.

Collura, A.
, Maggio, A.
, Micela, G.
, Sciortino, S.
, Harnden, J., F. R.
Rosner, R.

David, TJ.
Hillenbrand, LA.

Delorme, P.
, Collier Cameron, A.
, Hebb, L.
et al.

Douglas, ST.
, Curtis, JL.
, Agüeros, MA.
, Cargile, PA.
, Brewer, JM.
, Meibom, S.
Jansen, T.

Erkaev, NV.
, Kulikov, YN.
, Lammer, H.
, Selsis, F.
, Langmayr, D.
, Jaritz, GF.
Biernat, HK.

, Robrade
, Schneider

Freund, S.
, Robrade, J.
, Schneider, PC.
Schmitt, JHMM.

Fulton, BJ.
, Petigura, EA.
, Howard, AW.
et al.

Gupta, A.
Schlichting, HE.

, Davis

Jackson, AP.
, Davis, TA.
Wheatley, PJ.

, Bartel

Johnstone, CP.
, Bartel, M.
Güdel, M.

arXiv e-printsarXiv:2009.07695.

King, GW.
Wheatley, PJ.

King, GW.
, Wheatley, PJ.
, Salz, M.
et al.

Kislyakova, KG.
, Johnstone, CP.
, Odert, P.
et al.

Lammer, H.
, Odert, P.
, Leitzinger, M.
et al.

Lecavelier Des Etangs

Lecavelier Des Etangs

Lecavelier Des Etangs, A.

Livingston, JH.
, Dai, F.
, Hirano, T.
et al.

Lopez, ED.
Fortney, JJ.

, Fortney

Lopez, ED.
, Fortney, JJ.
Miller, N.

Mann, AW.
, Vanderburg, A.
, Rizzuto, AC.
et al.

, Dressing
, Buchhave
The Harps-N Collaboration

Mayo, A.
, Dressing, C.
, Buchhave, L.
The Harps-N Collaboration.

The First Mass Measurement for a Sub-Neptune in an Open
Cluster The First Mass Measurement for a Sub-Neptune in an Open
American Astronomical Society Meeting Abstracts
American Astronomical Society Meeting Abstracts  53,  117.04.

Micela, G.
, Sciortino, S.
, Vaiana, GS.
, Schmitt, JHMM.
, Stern, RA.
, Harnden, J., F. R.
Rosner, R.

, Bouchy

Otegi, JF.
, Bouchy, F.
Helled, R.

Owen, JE.

Annual Review of Earth and Planetary

Campos Estrada

Campos Estrada

Owen, JE.
Campos Estrada, B.

Owen, JE.
Wu, Y.

Pecaut, MJ.
Mamajek, EE.

Perryman, MAC.
, Brown, AGA.
, Lebreton, Y.
et al.

Redfield, S.
Linsky, JL.

Rogers, JG.
Owen, JE.

, Schneider
, Czesla

Salz, M.
, Schneider, PC.
, Czesla, S.
Schmitt, JHMM.

, Brickhouse
, Liedahl

Smith, RK.
, Brickhouse, NS.
, Liedahl, DA.
Raymond, JC.

, Zolcinski
, Antiochos

Stern, RA.
, Zolcinski, MC.
, Antiochos, SK.
Underwood, JH.

Van Eylen
Van Eylen

Van Eylen, V.
, Agentoft, C.
, Lundkvist, MS.
et al.

, Donahue

Watson, AJ.
, Donahue, TM.
Walker, JCG.

, Allen

Wilms, J.
, Allen, A.
McCray, R.

, Drake
, Mamajek

Wright, NJ.
, Drake, JJ.
, Mamajek, EE.
Henry, GW.

, Newton
, Williams
, Drake

Wright, NJ.
, Newton, ER.
, Williams, PKG.
, Drake, JJ.
Yadav, RK.


#  _PyTorch, Explain!_
A Python Library for Logic Explained Networks

Pietro Barbiero
Department of Computer Science
University of Cambridge
Cambridge, UK
<EMAIL_ADDRESS>
Gabriele Ciravegna
Department of Computer Science
Universita’ di Siena
Siena, IT
<EMAIL_ADDRESS>
Dobrik Georgiev
Department of Computer Science
University of Cambridge
Cambridge, UK
<EMAIL_ADDRESS>
Franscesco Giannini
Department of Computer Science
Universita’ di Siena
Siena, IT
<EMAIL_ADDRESS>

###### Abstract

PyTorch, Explain! is a Python module integrating a variety of state-of-the-art
approaches to provide logic explanations from neural networks. This package
focuses on bringing these methods to non-specialists. It has minimal
dependencies and it is distributed under the Apache 2.0 licence allowing both
academic and commercial use. Source code and documentation can be downloaded
from the github repository: https://github.com/pietrobarbiero/pytorch_explain.

_Keywords_ Python $\cdot$ logic $\cdot$ deep learning $\cdot$ XAI

##  1 Introduction

The lack of transparency in the decision process of some machine learning
models, such as neural networks, limits their application in many safety-
critical domains (Chander et al.,, 2018). Employing black-box111In the context
of this paper, a black-box classifier is any classifier that cannot provide
human understandable explanations about its decisions models may be
unacceptable in contexts such as industry, medicine or courts, where the
potential economical or ethical repercussions are calling for lawmakers to
discourage from a reckless application of non-interpretable models (EUGDPR,,
2017; Law,, 10; Goddard,, 2017; Gunning,, 2017). As a consequence, research in
Explainable Artificial Intelligence (XAI) has become strategic and has been
massively encouraged, leading to the development of a variety of techniques
that aim at explaining black-box models (Das and Rad,, 2020; Brundage et al.,,
2020) or at developing effective interpretable models (Carvalho et al.,, 2019;
Rudin,, 2019).

The need for high-quality human-friendly explanations is one of the main
reasons why concept-based explanations are receiving ever-growing
consideration. Explanations are given in terms of human-understandable symbols
(the concepts) rather than raw features such as pixels or characters (Kim et
al.,, 2018; Ghorbani et al.,, 2019; Koh et al.,, 2020). As a consequence, they
seem more suitable to serve many strategic human purposes such as decision
making tasks. For instance, a concept-based explanation may describe a high-
level category through its attributes as in “a human has hands and a head”.
While collecting high-quality evidences for an explanation is a common feature
of concept-based techniques, there are very few approaches formulating
hypothesis and even less providing synthetic descriptions whose validity can
be quantitatively assessed (Das and Rad,, 2020). A possible solution is to
rely on a formal language that is very expressive and closely related to
natural language and reasoning, such as First-Order Logic (FOL). A FOL
explanation can be considered a special kind of a concept-based explanation,
where the description is given in terms of logical predicates, connectives and
quantifiers, such as “$\forall x:\
\textit{is\\_human}(x)\rightarrow\textit{has\\_hands}(x)\wedge\textit{has\\_head}(x)$”.
However, FOL formulas generally express much more complex relationships among
the concepts involved in a certain explanation. Compared to other concept-
based techniques, logic-based explanations provide many key advantages, that
we briefly described in what follows. An explanation reported in FOL is a
rigorous and unambiguous statement (clarity). This formal clarity may serve
cognitive-behavioral purposes such as engendering trust, aiding bias
identification, or taking actions/decisions. For instance, dropping
quantifiers and variables for simplicity, the formula “snow $\wedge$ tree
$\leftrightarrow$ wolf” may easily outline the presence of a bias in the
collection of training data. Different logic-based explanations can be
combined to describe groups of observations or global phenomena (modularity).
For instance, for an image showing only the face of a person, an explanation
could be “(nose $\wedge$ lips) $\rightarrow$ human”, while for another image
showing a person from behind a valid explanation could be “(feet $\wedge$ hair
$\wedge$ ears) $\rightarrow$ human”. The two local explanations can be
combined into “(nose $\wedge$ lips) $\vee$ (feet $\wedge$ hair $\wedge$ ears)
$\rightarrow$ human”. The quality of logic-based explanations can be
quantitatively measured to check their correctness and completeness
(measurability). For instance, once the explanation “(nose $\wedge$ lips)
$\vee$ (feet $\wedge$ hair $\wedge$ ears)” is extracted for the class human,
this logic formula can be applied on a test set to check its generality in
terms of quantitative metrics like accuracy, fidelity and consistency.
Finally, FOL-based explanations can be rewritten in different equivalent forms
such as in _Disjunctive Normal Form_ (DNF) and _Conjunctive Normal Form_ (CNF)
(simplifiability). Further, techniques such as the Quine–McCluskey algorithm
can be used to compact and simplify logic explanations (McColl,, 1878; Quine,,
1952; McCluskey,, 1956). For instance, the explanation “(person $\wedge$ nose)
$\vee$ ($\neg$person $\wedge$ nose)” can be easily simplified in “nose”.

This work presents a Python library for XAI enabling neural networks to solve
and explain a categorical learning problem integrating elements from deep
learning and logic. Differently from vanilla neural architectures, these
models can be directly interpreted by means of a set of FOL formulas. In order
to implement such a property, such models require their inputs to represent
the activation scores of human-understandable concepts. Then, specifically
designed learning objectives allow them to make predictions in a way that is
well suited for providing FOL-based explanations that involve the input
concepts. In order to reach this goal, LENs exploit parsimony criteria aimed
at keeping their structure simple as described in recent works (Koh et al.,,
2020; Ciravegna et al., 2020b, ; Ciravegna et al., 2020a, ).

##  2 Background

Classification is the problem of identifying a set of categories $y\in
Y\subset[0,1]^{r}$ an observation $x\in X\subset\mathbb{R}^{d}$ belongs to. A
standard neural network is a black-box model $f:X\mapsto Y$ predicting for any
sample $x\in X$ the corresponding class membership $\hat{y}\in Y$. In case the
input features $x$ are not easily interpretable (low-level feature, image
pixels) concept-based classifiers have been introduced to predict class
memberships $Y$ from human-understandable categories (a.k.a. concepts)
$C\subset[0,1]^{k}$: $f:C\mapsto Y$ to improve the understanding of black
boxes and their decision process. Concepts can either correspond to the
predictions of a classifier (i.e. $g:X\mapsto C$) (Koh et al.,, 2020) or
simply to a re-scaling of the inputs space from the unbounded $\mathbb{R}^{d}$
to the unit interval $[0,1]^{k}$ such that input features can be treated as
logic predicates. Concept-based classifiers improve human understanding as
their input and output spaces consists of interpretable symbols.

Recent work on concept-based neural networks has led to the development of
models like the $\psi$ network (Ciravegna et al., 2020a, ) or the entropy-
based network Barbiero et al., (2021) i.e., concept-based classifiers
explaining their own decision process. These models are designed to compute
concise logic formulas representing _how_ the network combines input concepts
in order to arrive to a prediction.

This library implements the core learning criteria and methods allowing a
customized implementation of neural models providing logic explanations.

##  3 Application Programming Interfaces (APIs)

The code library is designed with intuitive APIs requiring only a few lines of
code to train and get explanations from deep neural networks. The library
currently provides the APIs for three explainable neural architectures: $\psi$
networks described in Ciravegna et al., 2020b , multi-layer neural networks
with rectified linear units, and entropy-based networks Barbiero et al.,
(2021).

The library supports a fine-grained customization of the neural networks as
shown in the following code example 1. The architecture of the model and the
training loop is defined by means of standard PyTorch APIs (Paszke et al.,,
2019). The entropy_logic_loss allows the network to get rid of less relevant
input concepts and to generate concise explanations for each prediction
(Ciravegna et al., 2020b, ). The explain_class method allows the extraction of
logic formulas from the trained model. Once extracted, formulas can be tested
on an unseen set of test samples using the test_explanation method providing
the accuracy of the logic formula. Explanations will be logic formulas in
disjunctive normal form. In the example below, the explanation will be
$y=1\leftrightarrow(x_{1}\wedge\neg x_{2})\vee(x_{2}\wedge\neg x_{1})$
corresponding to $y=1\leftrightarrow x_{1}\oplus x_{2}$.

⬇

1import torch

2from torch.nn.functional import one_hot

3import torch_explain as te

4from torch_explain.nn.functional import entropy_logic_loss

5from torch_explain.logic.nn import entropy

6from torch_explain.logic.metrics import test_explanation, complexity

7

8# train data

9x_train = torch.tensor([

10 [0, 0],

11 [0, 1],

12 [1, 0],

13 [1, 1],

14], dtype=torch.float)

15y_train = torch.tensor([0, 1, 1, 0], dtype=torch.float).unsqueeze(1)

16

17# instantiate an "entropy-based network"

18layers = [

19 te.nn.EntropyLinear(x_train.shape[1], 10, n_classes=2),

20 torch.nn.LeakyReLU(),

21 torch.nn.Linear(10, 4),

22 torch.nn.LeakyReLU(),

23 torch.nn.Linear(4, 1),

24]

25model = torch.nn.Sequential(*layers)

26

27# fit the model

28optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)

29loss_form = torch.nn.CrossEntropyLoss()

30model.train()

31for epoch in range(1001):

32 optimizer.zero_grad()

33 y_pred = model(x_train).squeeze(-1)

34 loss = loss_form(y_pred, y_train) + 0.00001 * entropy_logic_loss(model)

35 loss.backward()

36 optimizer.step()

37

38# get first-order logic explanations for a specific

39y1h = one_hot(y_train.squeeze().long())

40explanation = entropy.explain_class(model, x_train)

41

42# compute explanation accuracy and complexity

43accuracy, preds = test_explanation(explanation, x_train, y1h,
target_class=1)

44explanation_complexity = complexity(explanation)

Listing 1: Example on how to use _PyTorch, Explain!_ APIs.

##  4 Software and documentation availability

In order to make state-of-the-art approaches accessible to the whole
community, we released this library as a Python package on PyPI:
https://pypi.org/project/torch-explain/. An extensive documentation on methods
is available on read the docs222https://pytorch-
explain.readthedocs.io/en/latest/. and unit tests results on
TravisCI333https://travis-ci.org/pietrobarbiero/pytorch_explain/.. The Python
code and the scripts used for benchmarking against state-of-the-art white-box
models, including parameter values and documentation, is freely available
under Apache 2.0 Public License from a GitHub
repository444https://github.com/pietrobarbiero/logic_explainer_networks..

##  5 Conclusion

###  5.1 Limitations and future research directions

The extraction of a first-order logic explanation requires symbolic input and
output spaces. This constraint is the main limitation of our framework, as
narrows the range of applications down to symbolic I/O problems. In some
contexts, such as computer vision, the use of LENs may require additional
annotations and attribute labels to get a consistent symbolic layer of
concepts. However, recent work on automatic concept extraction may partially
solve this issue leading to more cost-effective concept annotations (Ghorbani
et al.,, 2019; Kazhdan et al.,, 2020).

The improvement of LENs models is an open research area. The efficiency and
the classification performances of fully interpretable LENs, i.e. $\psi$
network, is still quite limited due to the extreme pruning strategy.

###  5.2 Broader impact

Current legislation in US and Europe binds AI to provide explanations
especially when the economical, ethical, or financial impact is significant
(EUGDPR,, 2017; Law,, 10). This work contributes to a lawful and safer
adoption of some of the most powerful AI technologies allowing deep neural
networks to have a greater impact on society. Extracting first-order logic
explanations from deep neural networks enables satisficing (Simon,, 1956)
knowledge distillation while achieving performances comparable with the state
of the art. The formal language of logic provides clear and synthetic
explanations, suitable for laypeople, managers, and in general for decision
makers outside the AI research field.

Thanks to their explainable nature, LENs can be effectively used to understand
the behavior of an existing algorithm, to reverse engineer products, to find
vulnerabilities, or to improve system design. From a scientific perspective,
formal knowledge distillation from state-of-the-art networks may enable
scientific discoveries or confirmation of existing theories.

## Acknowledgments and Disclosure of Funding

We thank Stefano Melacci, Pietro Lió, and Marco Gori for useful feedback and
suggestions.

This work was partially supported by the European Union’s Horizon 2020
research and innovation programme under grant agreement No 848077.

## References

  * Barbiero et al., (2021) Barbiero, P., Ciravegna, G., Giannini, F., Lió, P., Gori, M., and Melacci, S. (2021).  Entropy-based logic explanations of neural networks.  arXiv preprint arXiv:2106.06804.
  * Brundage et al., (2020) Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020).  Toward trustworthy ai development: mechanisms for supporting verifiable claims.  arXiv preprint arXiv:2004.07213.
  * Carvalho et al., (2019) Carvalho, D. V., Pereira, E. M., and Cardoso, J. S. (2019).  Machine learning interpretability: A survey on methods and metrics.  Electronics, 8(8):832.
  * Chander et al., (2018) Chander, A., Srinivasan, R., Chelian, S., Wang, J., and Uchino, K. (2018).  Working with beliefs: Ai transparency in the enterprise.  In IUI Workshops.
  * (5) Ciravegna, G., Giannini, F., Gori, M., Maggini, M., and Melacci, S. (2020a).  Human-driven fol explanations of deep learning.  In Twenty-Ninth International Joint Conference on Artificial Intelligence and Seventeenth Pacific Rim International Conference on Artificial Intelligence $\\{$IJCAI-PRICAI-20$\\}$, pages 2234–2240. International Joint Conferences on Artificial Intelligence Organization.
  * (6) Ciravegna, G., Giannini, F., Melacci, S., Maggini, M., and Gori, M. (2020b).  A constraint-based approach to learning and explanation.  In AAAI, pages 3658–3665.
  * Das and Rad, (2020) Das, A. and Rad, P. (2020).  Opportunities and challenges in explainable artificial intelligence (xai): A survey.  arXiv preprint arXiv:2006.11371.
  * EUGDPR, (2017) EUGDPR (2017).  Gdpr. general data protection regulation.
  * Ghorbani et al., (2019) Ghorbani, A., Wexler, J., Zou, J., and Kim, B. (2019).  Towards automatic concept-based explanations.  arXiv preprint arXiv:1902.03129.
  * Goddard, (2017) Goddard, M. (2017).  The eu general data protection regulation (gdpr): European regulation that has a global impact.  International Journal of Market Research, 59(6):703–705.
  * Gunning, (2017) Gunning, D. (2017).  Explainable artificial intelligence (xai).  Defense Advanced Research Projects Agency (DARPA), nd Web, 2(2).
  * Kazhdan et al., (2020) Kazhdan, D., Dimanov, B., Jamnik, M., Liò, P., and Weller, A. (2020).  Now you see me (cme): Concept-based model extraction.  arXiv preprint arXiv:2010.13233.
  * Kim et al., (2018) Kim, B., Gilmer, J., Wattenberg, M., and Viégas, F. (2018).  Tcav: Relative concept importance testing with linear concept activation vectors.
  * Koh et al., (2020) Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. (2020).  Concept bottleneck models.  In International Conference on Machine Learning, pages 5338–5348. PMLR.
  * Law, (10) Law, P. A. (10).  Code of federal regulations.  Wash.: Gov. print. off.
  * McCluskey, (1956) McCluskey, E. J. (1956).  Minimization of boolean functions.  The Bell System Technical Journal, 35(6):1417–1444.
  * McColl, (1878) McColl, H. (1878).  The calculus of equivalent statements (third paper).  Proceedings of the London Mathematical Society, 1(1):16–28.
  * Paszke et al., (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019).  Pytorch: An imperative style, high-performance deep learning library.  arXiv preprint arXiv:1912.01703.
  * Quine, (1952) Quine, W. V. (1952).  The problem of simplifying truth functions.  The American mathematical monthly, 59(8):521–531.
  * Rudin, (2019) Rudin, C. (2019).  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence, 1(5):206–215.
  * Simon, (1956) Simon, H. A. (1956).  Rational choice and the structure of the environment.  Psychological review, 63(2):129.


# Bayesian Deep Neural Networks for Supervised Learning of Single-View Depth

Javier Rodríguez-Puigvert, Rubén Martínez-Cantín, Javier Civera The authors
are with I3A, Universidad de Zaragoza, Spain<EMAIL_ADDRESS>
###### Abstract

Uncertainty quantification is essential for robotic perception, as
overconfident or point estimators can lead to collisions and damages to the
environment and the robot. In this paper, we evaluate scalable approaches to
uncertainty quantification in single-view supervised depth learning,
specifically MC dropout and deep ensembles. For MC dropout, in particular, we
explore the effect of the dropout at different levels in the architecture.

We show that adding dropout in _all_ layers of the encoder brings better
results than other variations found in the literature. This configuration
performs similarly to deep ensembles with a much lower memory footprint, which
is relevant for applications. Finally, we explore the use of depth uncertainty
for pseudo-RGBD ICP and demonstrate its potential to estimate accurate two-
view relative motion with the real scale.

##  I Introduction

The quantification of the uncertainty is critical in robotics, in order to
implement systems that are robust and reliable in real-world applications.
Point estimators, which dominate the landscape of multi-view [5, 20] and
single-view [7, 10] scene reconstruction, do not typically compute the full
distribution but just the maximum-likelihood state. Higher-level decision
blocks have no means to judge how accurate these estimates are, and hence its
use for safe planning might be questionable. Uncertainty is often present in
the formulation of model-based estimators (e.g., [1]), but much less in the
training of deep learning models. Furthermore, learning-based approaches tend
to overfit on standard datasets, which might lead us to assume a reasonable
general performance while they are strongly biased. In such cases, their
outputs should not be trusted in real-world applications, for which we would
need generalizable and self-aware models. Bayesian learning is one of the
approaches that can address such application challenges.

In this work, we evaluate two sources of uncertainty –epistemic and aleatoric–
in supervised depth learning, with the aim to determine the quality of the
model predictions and hence its potential for robotic applications. In neural
networks, uncertainty can stem from the input data or the network weights. For
the latter, scalable approaches to Bayesian deep learning have shown to be
effective to model uncertainty. Fig. 1 shows the output of our Bayesian
framework for supervised single-view depth learning. Notice first the small
depth error, but most importantly, how such error is mostly coherent with the
predicted uncertainty. Observe also how the aleatoric uncertainty is in this
case bigger, but the epistemic one is still relevant for an accurate
quantification.

Figure 1: Bayesian single-view depth prediction for a SceneNet image. Notice
in the middle row the small depth error, and how the total uncertainty models
it accurately. Notice also in the bottom row how epistemic and aleatoric
sources are both significant and relevant for uncertainty quantification.

The contributions of this paper are: first, we provide a unified framework and
a thorough evaluation of scalable uncertainty estimation methods, namely Monte
Carlo (MC) dropout and deep ensembles, for supervised single-view depth
learning with deep convolutional networks. Secondly, we propose to apply MC
dropout in the encoder, contrary to recent works [11, 23] that apply it in the
decoder. We demonstrate in our evaluations that, in the particular task of
supervised depth learning, the dropout in the encoder achieves a better
performance than the dropout in the decoder.

Such a result has relevant practical implications, as MC dropout has a lower
memory footprint than ensembles. Finally, we also provide results in pseudo-
RGBD ICP, a potential application for our single-view depth uncertainty
models. In our experiments, we demonstrate that our uncertainty estimates are
reasonably well calibrated and has significant potential to provide accurate
and scaled motion estimates from monocular views.

##  II Background and Related Work

###  II-A Structure Estimation and Learning from Images

Reconstructing a 3D scene from visual data has been addressed from a wide
variety of perspectives, the more typical being based on multiple images
either alone [20, 5] or fused with other sensors (e.g., visual-inertial setups
[25]). However, multi-view and visual-inertial pipelines present two
limitations: they require sufficiently textured scenes to find
correspondences, and also sufficient motion for observability. Single-view
depth learning can help with these two issues, although it is significantly
more challenging due to its ill-posed nature.

Following the seminal work of [28] on single-view depth, Eigen et al. [4] were
the first ones that used depth supervision to train a deep network for such
task. Many works followed with different contributions: Laina et al. [15]
proposed deeper fully convolutional models. Fu et al. [7] proposed a spacing-
increasing depth discretization that learns depth from an ordinal regression
perspective. Dijk and de Croon [3] evaluated a self-trained networks to
investigate which visual cues they use. From their conclusions, depth networks
favor vertical positions and disregard obstacles in their apparent size.
Similarly, our work contributes to understanding the behavior of depth
networks from a Bayesian perspective.

Several works have proposed self-supervised approaches, using photometric
reprojection losses between stereo or multiple views [9, 34]. However, self-
supervised approaches still underperform compared to supervised ones. For this
reason, and given the current abundance of RGB-D data and the huge growth and
potential of such sensors, we focus on supervised methods in this work.

###  II-B Bayesian Deep Learning

Bayesian deep learning combines the strengths of deep neural architectures
with the uncertainty quantification of probabilistic (Bayesian) learning and
inference methods. Regarding uncertainty, we must differentiate between what
the model does not know and what is missing from the input data. Accordingly,
the uncertainty sources can be classified into two: aleatoric and epistemic.
Aleatoric (also referred to as statistical) uncertainty, refers to the
variations caused by the realization of different experiments with stochastic
components. In our models, it encodes the variability in the different inputs
from the test data and hence cannot be reduced by increasing the amount of
training data. Some models assume that the aleatoric uncertainty is
homoscedastic; that is, it is independent of the input data. In this work, we
train the network to predict the uncertainty for each input datum resulting in
a heteroscedastic uncertainty model [14, 2].

Epistemic (also known as systematic) uncertainty, represents the lack of
knowledge of a trained model. This type of uncertainty is deeply related to
the training data and the model ability to generalize. For example, epistemic
uncertainty is high for out-of-distribution data or extrapolation in regions
where training data was scarce. In Bayesian deep learning, the epistemic
uncertainty can be estimated from the uncertainty in the model parameters,
assuming that the model architecture is correct. In this case, epistemic
uncertainty can be obtained using a prior distribution $p(\mathbf{\theta})$
over the neural network parameters $\mathbf{\theta}$ and computing its
posterior distribution $p(\theta|\mathcal{D})$ given a dataset $\mathcal{D}$
using Bayes rule:
$p(\theta|\mathcal{D})=\frac{p(\mathcal{D}|\theta)p(\theta)}{p(\mathcal{D})}$.
In general, this equation is intractable for state-of-the-art deep
architectures, but there are several approaches to tackle this problem that we
describe below.

Variational inference (VI). VI proposes the use of a tractable approximation
$q(\theta)$ to the posterior distribution $p(\theta|\mathcal{D})$. Mean-field
variational inference assumes an isotropic Gaussian distribution for
$q(\theta)\sim\mathcal{N}(\theta|\mu,\mathbf{I}\sigma)$. The parameters of the
approximate distribution $q(\theta)$ are optimized by minimizing the KL-
divergence between the approximate distribution and the true posterior
$D(q||p)$. Mean-field variational inference with Gaussian approximation
suffers from the soap-bubble effect, reducing the predictive performance as
most samples fall in a ring. The Radial Bayesian Neural Networks [6] avoid
that effect, but the distribution is biased towards the center resulting in
uncertainty underestimation. Furthermore, VI methods are very sensitive to
calibration and configuration. A natural-gradient VI method [22] was
introduced to improve the robustness of the optimization. However, it requires
strong approximations of the Hessian, resulting in lower performance.

Monte Carlo (MC) Dropout. MC dropout can be used to approximate the posterior
distribution, as proposed in [8]. It can be considered as a specific case of
VI, where the variational distribution includes a set of binary random
variables that represent the corresponding unit to be turned off or dropped.
The approximation makes the computation more tractable and robust. MC dropout
is able to approximate multimodal distributions. However, the epistemic
distribution on the weight-space only has discrete support. [14] presented a
framework to combine both aleatoric and epistemic uncertainty, where MC
dropout is used to obtain epistemic uncertainty, while the function mapping
the aleatoric uncertainty is learned from the input data.

Deep Ensembles. Deep ensembles [16] involves training the same architecture
many times optimizing some MAP loss, but starting from different random
initialization of its parameters. Therefore, deep ensembles are not truly a
Bayesian approach as the samples are distributed according to the different
local optima. Conversely, these models in an ensemble perform reasonably well,
even considering the small number of random samples considered in practice, as
all of the models are optimized and have a high likelihood. Therefore, deep
ensembles can be considered an approximate Bayesian model average, although,
in practice, they can also be used as a rough posterior approximation.
Contrary to MC dropout, where the model weights are shared between samples, in
the case of deep ensembles, each _sample_ is trained independently. Therefore
the number of model parameters required grows linearly with the number of
samples. Furthermore, deep ensembles also result in a distribution with
discrete support on the weight-space.

###  II-C Bayesian Deep Learning in Computer Vision

Evaluating uncertainty correctly is still in open discussion, as it is task-
related. Mukhoti and Yal [19] evaluated MC dropout for semantic segmentation
and designed the metrics for such case. Similarly, Gustafsson et al. [11]
designed a framework to explore uncertainty metrics for semantic segmentation
and depth completion, using MC dropout and deep ensembles.

Ilg et al. [13] compare different strategies and techniques for quantifying
the uncertainty of optical flow. They also introduce a multi-hypothesis
network based on a winner-takes-all loss function that penalizes the best
hypothesis result. However, disentangling aleatoric and epistemic uncertainty
could be an arduous task in multi-hypothesis approaches. The network that
merge all hypothesis contains its own epistemic uncertainty that is not taking
into account. Nevertheless, they show to be competitive in comparison to deep
ensemble and MC dropout. For depth estimation, Yang et al. [32] proposes a
multinomial distribution to learn uncertainty based on discretizing the depth
space.

For our case of single-view depth regression, Poggi et al. [23] evaluated the
uncertainty in self-supervised networks, which leverage photometric
consistency between views. They observed that depth accuracy is improved by
uncertainty estimation along the training paradigms. Our work complements the
ones mentioned in this section by evaluating uncertainty quantification in a
supervised regression setting.

##  III Bayesian Single-View Depth Learning from Supervised Data

As described in Section II-B, MC dropout and deep ensembles provide a sample
representation of the posterior distribution over the network parameters.
Here, we introduce a unified formulation to analyze the posterior and
predictive distribution for these sample representations. For brevity, we have
particularized our framework for depth perception applications, although it
could be extended to other tasks.

###  III-A Architecture and Loss

We adapt a U-Net [26] encoder-decoder architecture as in [10, 23]. Our encoder
is a Resnet18 [12] pre-trained in ImageNet [27]. Table I summarizes our
decoder architecture.

Depth Decoder
---
layer | # filters | inputs | activation
upconv5 | 256 | econv5 | ELU
iconv5 | 256 | ↑upconv5, econv4 | ELU
upconv4 | 128 | iconv5 | ELU
iconv4 | 128 | ↑upconv4, econv3 | ELU
depth_unc4 | 2 | iconv4 | -
upconv3 | 64 | iconv4 | ELU
iconv3 | 64 | ↑upconv4, econv3 | ELU
depth_unc3 | 2 | iconv3 | -
upconv2 | 32 | iconv3 | ELU
iconv2 | 32 | ↑upconv2, econv1 | ELU
depth_unc2 | 2 | iconv2 | -
upconv1 | 16 | iconv2 | ELU
iconv1 | 16 | ↑upconv1 | ELU
depth_unc1 | 2 | iconv1 | -
TABLE I: Decoder architecture. Kernels are always $3\times 3$ with stride $1$.
↑ stands for $2\times 2$ nearest-neighbor upsampling.

Our training data
$\mathcal{D}=\\{\\{\mathcal{I}_{1},d_{1}\\},\ldots,\\{\mathcal{I}_{N},d_{N}\\}\\}$
is composed by $N$ supervised pairs, each pair $i\in\\{1,\ldots,N\\}$
containing a RGB image $\mathcal{I}_{i}\in\\{0,\ldots,255\\}^{w\times h\times
3}$ and its ground truth depth $d_{i}\in\mathbb{R}^{w\times h}_{>0}$. For a
single input image, the network $f_{\theta}(\mathcal{I})$ outputs two
channels: per-pixel depth $\widehat{d}(\mathcal{I})$ and uncertainty
$\sigma_{d}(\mathcal{I})$. The later corresponds to aleatoric uncertainty,
which can also be interpreted as heteroscedastic observation noise. We
incorporate both output channels in a single loss per image by using a
standard Laplace log-likelihood [14]:

$\pazocal{L}(\theta)=\frac{1}{w\cdot
h}\sum_{j\in\Omega}\frac{||d_{j}-\widehat{d}(\mathcal{I}_{j})||}{\sigma_{d}(\mathcal{I}_{j})}+\log\sigma_{d}(\mathcal{I}_{j})$
(1)

where $j\in\Omega$ is the pixel index in the image domain $\Omega$.

For deep ensembles, the loss function is evaluated independently for each
sample model as they are trained separately, resulting in $M$ sets of
parameters $\\{\theta_{m}\\}_{m=1}^{M}$. Although the sample models are not
drawn from the posterior distribution, it still can be considered an
approximation in practice. Deep ensembles are especially suitable for our
problem as we need to maintain the number of samples small to keep it
tractable. Therefore, it is important that we are not wasting valuable
resources in low probability models that might reduce the overall performance.

In the case of MC dropout, the loss function can be used for approximate
variational inference on the posterior distribution of the weights by training
with dropout after every layer. The actual Monte Carlo phase is done by also
performing random dropout at test time to sample from the variational
distribution computed during training [14]. This sampling at test time results
again in a set of $\\{\theta_{m}\\}_{m=1}^{M}$ different parameters. However,
note that this time they are all generated from the same trained model,
resulting in a much lower memory and computational footprint compared to deep
ensembles or other variational methods.

In practice, we found that adding dropout at every layer reduced the
predictive performance considerably for our application, which is consistent
with previous results [19]. Therefore, in section IV-C, we study different
configurations of dropout and compare their quality both in terms of depth
error and uncertainty quantification.

###  III-B Bayesian prediction of sample-based deep networks

The predictive distribution for a pixel depth can be computed by integrating
over the model parameters. We use the same strategy for MC dropout and deep
ensembles, as they both use sample representations of the model parameters:

$\begin{split}p(d|\mathcal{I},\mathcal{D})&=\int
p(d|\mathcal{I},\theta)p(\theta|\mathcal{D})d\theta\\\
&\approx\sum_{m=0}^{M}p(d|\mathcal{I},\theta_{m})\end{split}$ (2)

As our architecture generates a Gaussian prediction for the pixel depth
$\mathcal{N}(\widehat{d},\sigma^{2}_{d})$, the sample-based output is a
mixture of Gaussians that can be approximated by a single Gaussian. In
particular, for the $\\{\theta_{m}\\}_{m=1}^{M}$ model samples (MC Dropout) or
models (ensembles) with respective outputs $\widehat{d}(m)$ and
$\sigma_{d}(m)$, we approximate the total predictive distribution per pixel as
a Gaussian
$p(d|\mathcal{I},\mathcal{D})\approx\mathcal{N}(\widehat{d}_{t},\sigma^{2}_{t})$
with:

$\begin{split}\widehat{d}_{t}&=\frac{1}{M}\sum_{m=1}^{M}\widehat{d}_{m}\\\
\sigma^{2}_{t}&=\underbrace{\frac{1}{M}\sum_{m=1}^{M}\left(\widehat{d}_{t}-\widehat{d}(m)\right)^{2}}_{\text{epistemic}}+\underbrace{\frac{1}{M}\sum_{m=1}^{M}\sigma^{2}_{d}(m)}_{\text{aleatoric}}\end{split}$
(3)

In the experiments, we will show that identifying and quantifying the
epistemic from the aleatoric uncertainty will be fundamental to finding the
uncertainty source and improving the quality of the model and the predictions.

##  IV Experiments

Figure 2: Variations of MC Dropout in our experiments.  Model | M | Abs Rel | Sq Rel | RMSE | RMSE Log | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$ | AUCE | AUSE
---|---|---|---|---|---|---|---|---|---|---
MCD CEnc p=0.3 | 64 | 0.1231 | 0.1222 | 0.6396 | 0.1982 | 0.8719 | 0.9635 | 0.9850 | 0.4940 | 0.0985
MCD CEnc p=0.5 | 64 | 0.1243 | 0.1228 | 0.6484 | 0.2023 | 0.8675 | 0.9615 | 0.9842 | 0.4833 | 0.0968
MCD FEnc p=0.3 | 64 | 0.1291 | 0.1326 | 0.6688 | 0.2047 | 0.8591 | 0.9591 | 0.9834 | 0.4954 | 0.1210
MCD FEnc p=0.5 | 64 | 0.1320 | 0.1363 | 0.6633 | 0.2062 | 0.8583 | 0.9586 | 0.9833 | 0.4869 | 0.1229
MCD LEnc p=0.5 | 64 | 0.1244 | 0.1245 | 0.6582 | 0.2010 | 0.8658 | 0.9607 | 0.9846 | 0.4955 | 0.1219
MCD LEnc p=0.3 | 64 | 0.1244 | 0.1248 | 0.6769 | 0.2000 | 0.8629 | 0.9595 | 0.9848 | 0.4955 | 0.1327
MCD CDec p=0.3 | 64 | 0.1316 | 0.1322 | 0.6781 | 0.2044 | 0.8567 | 0.9597 | 0.9841 | 0.4917 | 0.1268
MCD CDec p=0.5 | 64 | 0.1369 | 0.1378 | 0.6988 | 0.2080 | 0.8494 | 0.9579 | 0.9835 | 0.4899 | 0.1323
MCD FDec p=0.5 | 64 | 0.1263 | 0.1252 | 0.6690 | 0.2035 | 0.8604 | 0.9593 | 0.9841 | 0.4872 | 0.1353
MCD FDec p=0.3 | 64 | 0.1264 | 0.1263 | 0.6622 | 0.2016 | 0.8641 | 0.9604 | 0.9842 | 0.4956 | 0.1304
MCD LDec p=0.5 | 64 | 0.1336 | 0.1371 | 0.6866 | 0.2065 | 0.8568 | 0.9588 | 0.9833 | 0.4933 | 0.1241
MCD LDec p=0.3 | 64 | 0.1293 | 0.1303 | 0.6782 | 0.2042 | 0.8589 | 0.9593 | 0.9837 | 0.4942 | 0.1285
Deep ensembles | 18 | 0.1283 | 0.1244 | 0.6529 | 0.1993 | 0.8617 | 0.9613 | 0.9850 | 0.4823 | 0.0838
TABLE II: Depth and uncertainty metrics for several variations of MC Dropout
and Deep Ensembles in SceneNet RGB-Depth. Best results are boldfaced, second
best ones are underlined. Figure 3: Comparison of MC dropout variations and
deep ensembles for different numbers of forward passes $M$. Left: RMSE.
Center: AUSE. Right: AUCE. The higher $M$ is, the better the performance, but
with slight improvements for $M>18$.

###  IV-A Datasets

SceneNet RGB-D dataset [18] contains photorealistic sequences of synthetic
indoor scenes from general camera trajectories, along with their ground truth.
Our models are trained over $210,000$ synthetic images of $700$ scenes and
tested on $90,000$ images of $300$ different scenes. We chose this dataset as
it provides a wide variety of viewpoints and scenes, challenging occlusions
and different lighting conditions, which are relevant for the network
generalization.

NYU Depth V2 [21] consists of $120,000$ RGB-D images in $464$ indoor scenes.
For training, we use $36,253$ images of $249$ scenes as proposed by Lapdepth
[30]. We test our models in the official split of $654$ images [4].

###  IV-B Metrics

We use the depth error metrics that are standard in literature: Absolute
Relative difference, Square Relative difference, RMSE, RMSE log and
$\delta<1.25^{i}$ with $i\in\\{1,2,3\\}$ (see their definitions in [4]). For
the pseudo-RGBD Bayesian ICP in Section IV-D, we report the translational and
rotational RMSE. For uncertainty, we use the Area Under the Calibration Error
curve (AUCE) and Area Under the Sparsification Error curve (AUSE). Following
Gustafsson et al. [11], we define $100$ prediction intervals of confidence
level $p\in\left[0,1\right]$ and use the cumulative density function of our
output distribution (Gaussian). For each confidence level, we expect for a
perfect calibration that the ratio of the prediction interval covering the
true target $\widehat{p}$ to be identical to the confidence level $p$. AUCE is
defined as the area between the absolute error curve with respect to a perfect
calibration $|p-\hat{p}|$. As a second metric for the uncertainty quality, we
use AUSE in terms of RMSE. Introduced by Ilg et al. [13], it is a relative
measure of uncertainty. This metric compares the ordering of the per-pixel
uncertainties against the order of the per-pixel depth errors. The ordering
should be similar for a well-calibrated uncertainty, as uncertain predictions
will tend to have larger errors.

###  IV-C Bayesian Single-View Depth

We evaluate several variations of MC dropout and deep ensembles. Specifically,
for MC dropout, we report depth and uncertainty metrics for dropouts at
different layers and with $p=0.3$ and $p=0.5$, $p$ being the probability of an
element to be zeroed.

MC Dropout. We consider seven variations (see Fig. 2 for a summary plot): In
the decoder, dropout after every convolutional layer, (MC Dropout Complete
Decoder), the first two convolutional layers (MC Dropout First Decoder), and
the last convolution layer of the decoder (MC Dropout Last Decoder). And, in
the encoder, dropout after every convolutional layer (MC Dropout Complete
Encoder), the first convolutional layer (MC Dropout First Encoder), and after
the last convolution layer (MC Dropout Last Encoder). Finally, in NYU RGB-D v2
we also evaluate the effect of the dropout in all layers of the architecture
(MC Dropout Complete Encoder-Complete Decoder).

Deep Ensembles. We examine uncertainty and depth by averaging an ensemble
composed of a variable number of networks. We initialise the network weights
with different seeds from a normal distribution $\mathcal{N}(0,10^{-2})$.

Results. Table II shows the depth and uncertainty metrics for the MC dropout
variations and deep ensembles on SceneNet RGB-D. We observe that the
uncertainty metrics (AUCE and AUSE) for deep ensembles outperforms all
variants of MC dropout. This is due to the fact that deep ensembles are
optimized to be close to a minimum. However, the depth error metrics are
consistently better for MCD CEnc, which also show the second best uncertainty
metrics.

The performance of the different MC dropout models varies significantly, which
is a novel result of our analysis. We found that introducing dropout in all
layers of the encoder (MCD CEnc) improves the results with respect to applying
it to a few layers of the encoder (MCD FEnc, LEnc) or to the decoder (MCD
FDec, LDec and CDec). Again, this result is of relevance as MC dropout is
commonly applied in the decoder for depth estimation [11, 23]. Our rationale
for this result is as follows: we believe that applying MC dropout only in the
decoder makes the network learn deterministic representations (the encoder is
deterministic). In contrast, applying MC dropout in the image encoder allows
us to learn probabilistic representations modeling uncertainty in the feature
space. This seems to be a more appropriate choice for Bayesian image
processing.

Our experiments show that there is small variations in the MC samples when we
solely apply dropout close to the code, as done in MCD FDec or MCD Lenc. This
results in worse calibrated models and also less satisfactory depth
estimations. We also observed that applying MC dropout after the first and
before the last layers of the network leads to poor performance (see MCD LDec,
MCD FEnc).

Comparing the two dropout probabilities, $p=0.3$ and $p=0.5$, we observe that
the depth prediction is usually better for $p=0.3$ but the uncertainty is
better calibrated for $p=0.5$, as it introduces greater variability. One or
the other should be preferred depending on the application.

MC dropout has a much smaller memory footprint than deep ensembles and our
results indicate that the MCD CEnc performs similarly to ensembles, so it
could be relevant option in certain applications. Specifically, only $56$ Mb
are required for our MC dropout models, while the memory for deep ensembles
grows linearly with the number of samples (for $M=18$, around $1$ Gb). The run
time grows linearly with the number of samples in both cases (around 3–4 ms
per forward pass). All data stems from a NVIDIA GeForce RTX 3090 GPU.

(a) AUCE Deep Ensembles

(b) AUCE MCD CEnc p=0.5

(c) AUCE MCD CDec p = 0.3

(d) AUSE Deep Ensembles M=18

(e) AUSE MCD CEnc p=0.5 M=64

(f) AUSE MCD CDec p=0.3 M=64

Figure 4: Calibration curves (AUCE and AUSE) for MC dropout and deep
ensembles.

Fig. 3 shows the evolution of the metrics with the number of forward passes
$M$. We can take $M=1$ as the baseline, since there is no epistemic
contribution. In general, predictions improve as $M$ increases. However, such
improvements are hardly noticeable for $M>18$. Interestingly, for MCD CDec,
the epistemic uncertainty results in a worse uncertainty calibration in
comparison to a single forward pass.

Fig. 4 shows the calibration error curves and sparsification error curves from
where AUSE and AUCE were extracted. In these figures it can be seen that all
models are overconfident, and the similarity between the sparsification error
curves that leads to similar AUSE values.

Figure 5: Depth and uncertainty predictions in a SceneNet RBG-D image. Figure 6: Predictions for three out-of-distribution images, showing high uncertainty for unfamiliar objects and textures. Top row: the aleatoric uncertainty is large in depth discontinuities and object boundaries, and the epistemic one concentrates in unknown patterns (the dog and the door). Middle row: aleatoric and epistemic uncertainties are both large due to large differences in appearance with respect to the training data. Bottom row: prediction in NYU Depth V2 image by an ensemble trained in SceneNet. The uncertainty is only very high for the human, not present in the training data. Model | M | Abs Rel | Sq Rel | RMSE | RMSE Log | $\delta<1.25$ | $\delta<1.25^{2}$ | $\delta<1.25^{3}$ | AUCE | AUSE
---|---|---|---|---|---|---|---|---|---|---
Deep Ensembles | 24 | 0.1431 | 0.1052 | 0.5842 | 0.1973 | 0.8157 | 0.9596 | 0.9894 | 0.1302 | 0.1588
MCD CENC-CDEC p=0.3 | 32 | 0.1560 | 0.1250 | 0.6332 | 0.2155 | 0.7867 | 0.9454 | 0.9849 | 0.1453 | 0.1703
MCD CDec p=0.3 | 32 | 0.1574 | 0.1286 | 0.6307 | 0.2160 | 0.7850 | 0.9460 | 0.9851 | 0.1382 | 0.1770
MCD CEnc p=0.3 | 32 | 0.1495 | 0.1173 | 0.6180 | 0.2090 | 0.8006 | 0.9491 | 0.9858 | 0.1436 | 0.1653
MCD CENC-CDEC p=0.5 | 32 | 0.1642 | 0.1385 | 0.6592 | 0.2264 | 0.7671 | 0.9385 | 0.9824 | 0.1356 | 0.1668
MCD CEnc p=0.5 | 32 | 0.1525 | 0.1220 | 0.6239 | 0.2127 | 0.7942 | 0.9467 | 0.9848 | 0.1377 | 0.1720
MCD CDec p=0.5 | 32 | 0.1578 | 0.1291 | 0.6326 | 0.2164 | 0.7861 | 0.9457 | 0.9843 | 0.1286 | 0.1716
TABLE III: Depth and uncertainty metrics for several variations of MC Dropout
and Deep Ensembles in NYU Depth V2. Best results are boldfaced, second best
ones are underlined. Figure 7:  Comparison of MCD CDec, CEnc and Deep
Ensembles for forward passes $M$. Left: RMSE. Center: AUSE. Right: AUCE. The
higher $M$ is, the better the performance, but with slight improvements for
$M>18$.

Table III shows the metrics for depth and uncertainty on NYU Depth v2. We
evaluate three MC dropout variations: MCD CEnc, MCD CDec, and MCD CEnc-CDec.
In this dataset, deep ensembles show the best results in terms of uncertainty
and depth metrics. As in SceneNet, applying dropout in the complete encoder
(MCD CEnc) shows better results than applying it in the decoder (MCD CDec) or
in the whole network (MCD CEnc-CDec). In this last case, the worse performance
is caused by a too strong regularization effect. It is worth mentioning that
the network is unable to recover the object boundaries and that the aleatoric
uncertainty becomes diffuse for the high dropout rate case $p=0.5$. This
effect becomes more noticeable when the decoder contains dropout layers. Fig.
8 shows results for one sample image. Observe that the areas with higher error
correspond with the areas with higher uncertainty.

For all methods, increasing the number $M$ of forward passes improves the
performance in uncertainty and depth (see Fig. 7). We do not see a noticeable
improvement, though, for $M>18$ for deep ensembles and $M>32$ for MC Dropout.

Figure 8:  Depth and uncertainty results in NYU Depth v2 for MC dropout
variations and deep ensembles. Top Row: input image and ground truth depth.
First column: predicted depth. Second column: predicted uncertainty. Third
column: depth error. Colors are equalized per image for better visualization.

Aleatoric uncertainty appears mainly in depth discontinuities, at object
edges, and regions with sharp contrast in lighting (see Fig. 5). As additional
examples, Fig. 6 shows highly uncertain depth predictions for three out-of-
distribution images. It indicates high uncertainty values for unfamiliar
objects not seen during training, like the dog and the door in the background
of the first picture, the unrealistic patterns of the painting “Bedroom in
Arles” by Van Gogh in the second one, and the person in the third one.

###  IV-D Bayesian Pseudo-RGBD ICP

In this section, we evaluate the application of Bayesian depth neural networks
for two-view relative motion. Relative motion can be directly computed from
deep neural networks where the inductive bias can be used to estimate absolute
values even for uncalibrated cameras and blurry or poorly illuminated images.
But geometric methods are more precise in a multi-view setting with a known
baseline for scale disambiguation and sensible features can be tracked between
multiple views [33]. In this work, we present a mixed approach where we use a
two-view geometric method (ICP) to accurately compute the relative motion
based on the depth prediction from neural networks. Thus, our approach is able
to work with poor quality images and provide an unambiguous 3D transformation.

Our proposal leverages the depth predicted by a network to augment monocular
images into what we call pseudo-RGBD views, and then aligns them using
Iterative Closest Point (ICP). Similar ideas were proposed recently in [31,
17]. Differently from us, they rely on Structure from Motion [29] or visual
SLAM [20] to estimate the motion from the pseudo-RGBD views. Also, we use
depth uncertainty for a more informed point cloud alignment, specifically
excluding highly uncertain points from ICP.

Figure 9: Left: Single-view reconstruction. Right: Same reconstruction, with the $10\%$ most uncertain points plotted in red. These points corresponds to spurious or high error points that will degrade the performance of ICP. Percentile | .30 | .50 | .75 | .90 | .95 | .99 | 1.00
---|---|---|---|---|---|---|---
RMSE t [m] | 0.238 | 0.216 | 0.188 | 0.182 | 0.179 | 0.190 | 0.190
RMSE r [∘] | 1.992 | 1.911 | 1.936 | 1.847 | 1.888 | 1.912 | 1.891
TABLE IV: ICP errors for percentiles .30, .50, .75, .90, .95, .99, 1.00. Best
are boldfaced, second best underlined.

Our experimental setup is as follows. We selected $1408$ random image pairs,
separated by at least $4$ frames, from SceneNet RGB-D. We excluded pairs with
large areas without ground-truth depth (e.g., windows) and small overlap
(rotations larger than $60^{\circ}$). We kept the image pairs for which there
is sufficient evidence that ICP converged.

For each pair, we back-projected the estimated depth distributions into a
point clouds and applied ICP to the following percentiles of the most certain
points according to our estimation: $.30$, $.50$, $.75$, $.90$, $.95$, $.99$,
and $1.00$ (the percentile $1.00$ corresponds to the full point clouds). We
used our deep ensemble model, as it showed the best uncertainty calibration in
Table II.

Fig. 9 illustrates our hypothesis with an example. The left point cloud is the
original one, and the right one corresponds to the percentile $.90$. The
points highlighted in red represent the $10\%$ most uncertain points according
to our uncertainty estimation, and clearly correspond to highly erroneous
ones, as they lie on depth discontinuities. Removing such points from ICP will
improve its accuracy.

Table IV shows the translational and rotational errors for all evaluated
pairs. As motivated before, removing the most uncertain points (corresponding
in well calibrated models to the most errouneous ones) reduces the estimation
errors. The best results are for percentiles $.90$ and $.95$ ($1.00$
corresponds to the original point cloud). When the percentage of points
removed is higher the error grows. This effect becomes obvious for percentiles
$.30$ and $.50$, for which $70\%$ and $50\%$ of the points with the highest
uncertainty were removed respectively. In these cases, the number of points
removed is too large and the estimation becomes less accurate.

##  V Conclusions

In this paper, we evaluated MC dropout and deep ensembles as scalable Bayesian
approaches to uncertainty quantification for single-view supervised depth
learning. We demonstrate empirically that using MC dropout in the encoder
outperforms other variations used in the literature, which is a result of
practical relevance. The placement of dropout in the architecture indeed has a
significant effect in the estimation of depth and uncertainty. We believe that
this finding would be also relevant for works based on noise layers like [24].
As a second conclusion of our analysis, deep ensembles have the best
calibrated uncertainty estimations. However, applying dropout in the encoder
performs only slightly worse than deep ensembles. As MC dropout needs much
less memory than deep ensembles, it may be considered the Bayesian approach
with more potential for applications.

In our experimental results, we also show the application of Bayesian depth
networks to pseudo-RGBD ICP, with the result that relative transformation can
be improved by excluding the points with higher uncertainties. For future
investigation into this topic, we would like to address the overconfidence
that we observed in our depth prediction networks and research a better
representation of uncertainty. Furthermore, it would be interesting to
investigate how to benefit from the posterior computation to overcome the
domain changes from synthetic to real data and between different scenes.

## References

  * [1] T. D. Barfoot.  State estimation for robotics.  Cambridge University Press, 2017.
  * [2] A. Der Kiureghian and O. Ditlevsen.  Aleatory or epistemic? does it matter?  Structural Safety, 31:105–112, 03 2009.
  * [3] T. v. Dijk and G. de Croon.  How do neural networks see depth in single images?  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2183–2191, 2019.
  * [4] D. Eigen, C. Puhrsch, and R. Fergus.  Depth map prediction from a single image using a multi-scale deep network.  arXiv preprint arXiv:1406.2283, 2014.
  * [5] J. Engel, V. Koltun, and D. Cremers.  Direct sparse odometry.  IEEE transactions on pattern analysis and machine intelligence, 40(3):611–625, 2017.
  * [6] S. Farquhar, M. Osborne, and Y. Gal.  Radial bayesian neural networks: Beyond discrete support in large-scale bayesian deep learning.  Proceedings of the 23rtd International Conference on Artificial Intelligence and Statistics, 2020.
  * [7] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao.  Deep ordinal regression network for monocular depth estimation.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 2002–2011, 2018.
  * [8] Y. Gal and Z. Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  * [9] C. Godard, O. Mac Aodha, and G. J. Brostow.  Unsupervised monocular depth estimation with left-right consistency.  Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-Janua:6602–6611, 2017.
  * [10] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow.  Digging into self-supervised monocular depth estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3828–3838, 2019.
  * [11] F. K. Gustafsson, M. Danelljan, and T. B. Schon.  Evaluating scalable bayesian deep learning methods for robust computer vision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 318–319, 2020.
  * [12] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [13] E. Ilg, Ö. Çiçek, S. Galesso, A. Klein, O. Makansi, F. Hutter, and T. Brox.  Uncertainty estimates and multi-hypotheses networks for optical flow.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11211 LNCS:677–693, 2018.
  * [14] A. Kendall and Y. Gal.  What uncertainties do we need in Bayesian deep learning for computer vision?  Advances in Neural Information Processing Systems, 2017-Decem(Nips):5575–5585, 2017.
  * [15] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab.  Deeper depth prediction with fully convolutional residual networks.  In 2016 Fourth international conference on 3D vision (3DV), pages 239–248, 2016.
  * [16] B. Lakshminarayanan, A. Pritzel, and C. Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  Advances in Neural Information Processing Systems, pages 6403–6414, 2017.
  * [17] X. Luo, J.-B. Huang, R. Szeliski, K. Matzen, and J. Kopf.  Consistent video depth estimation.  ACM Transactions on Graphics (TOG), 39(4):71–1, 2020.
  * [18] J. McCormac, A. Handa, S. Leutenegger, and A. J. Davison.  Scenenet rgb-d: 5m photorealistic images of synthetic indoor trajectories with ground truth.  arXiv preprint arXiv:1612.05079, 2016.
  * [19] J. Mukhoti and Y. Gal.  Evaluating bayesian deep learning methods for semantic segmentation.  arXiv preprint arXiv:1811.12709, 2018.
  * [20] R. Mur-Artal and J. D. Tardós.  Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras.  IEEE Transactions on Robotics, 33(5):1255–1262, 2017.
  * [21] P. K. Nathan Silberman, Derek Hoiem and R. Fergus.  Indoor segmentation and support inference from rgbd images.  In ECCV, 2012.
  * [22] K. Osawa, S. Swaroop, A. Jain, R. Eschenhagen, R. E. Turner, R. Yokota, and M. E. Khan.  Practical deep learning with bayesian principles.  arXiv preprint arXiv:1906.02506, 2019.
  * [23] M. Poggi, F. Aleotti, F. Tosi, and S. Mattoccia.  On the uncertainty of self-supervised monocular depth estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3227–3237, 2020.
  * [24] J. Postels, F. Ferroni, H. Coskun, N. Navab, and F. Tombari.  Sampling-free epistemic uncertainty estimation using approximated variance propagation.  CoRR, abs/1908.00598, 2019.
  * [25] T. Qin, P. Li, and S. Shen.  Vins-mono: A robust and versatile monocular visual-inertial state estimator.  IEEE Transactions on Robotics, 34(4):1004–1020, 2018.
  * [26] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [27] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [28] A. Saxena, M. Sun, and A. Y. Ng.  Make3d: Learning 3d scene structure from a single still image.  IEEE transactions on pattern analysis and machine intelligence, 31(5):824–840, 2008.
  * [29] J. L. Schonberger and J.-M. Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016.
  * [30] M. Song, S. Lim, and W. Kim.  Monocular depth estimation using laplacian pyramid-based depth residuals.  IEEE Transactions on Circuits and Systems for Video Technology, 31(11):4381–4393, 2021.
  * [31] L. Tiwari, P. Ji, Q.-H. Tran, B. Zhuang, S. Anand, and M. Chandraker.  Pseudo RGB-D for self-improving monocular SLAM and depth prediction.  In European Conference on Computer Vision, pages 437–455. Springer, 2020.
  * [32] G. Yang, P. Hu, and D. Ramanan.  Inferring distributions over depth from a single image.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6090–6096. IEEE, 2019.
  * [33] Q. Zhou, T. Sattler, M. Pollefeys, and L. Leal-Taixe.  To learn or not to learn: Visual localization from essential matrices.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3319–3326. IEEE, 2020.
  * [34] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe.  Unsupervised learning of depth and ego-motion from video.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1851–1858, 2017.