# Shortcuts to thermodynamic quasistaticity

Artur Soriani<EMAIL_ADDRESS>Eduardo Miranda  Gleb Wataghin
Institute of Physics, University of Campinas, Campinas, São Paulo 13083–950,
Brazil  Sebastian Deffner  Department of Physics, University of Maryland,
Baltimore County, Baltimore, Maryland 21250, USA  Marcus V. S. Bonança  Gleb
Wataghin Institute of Physics, University of Campinas, Campinas, São Paulo
13083–950, Brazil

###### Abstract

The operation of near-term quantum technologies requires the development of
feasible, implementable, and robust strategies of controlling complex many
body systems. To this end, a variety of techniques, so-called “shortcuts to
adiabaticty”, have been developed. Many of these shortcuts have already been
demonstrated to be powerful and implementable in distinct scenarios. Yet, it
is often also desirable to have additional, approximate strategies available,
that are applicable to a large class of systems. In this work, we hence take
inspiration from thermodynamics and propose to focus on the macrostate, rather
than the microstate. Adiabatic dynamics can then be identified as such
processes that preserve the equation of state, and systematic corrections are
obtained from adiabatic perturbation theory. We demonstrate this approach by
improving upon fast quasiadiabatic driving, and by applying the method to the
quantum Ising chain in the transverse field.

The word “adiabatic” is derived from the Greek _adiabatos_ , which means
literally “impassable”. In thermodynamics, an adiabatic constraint is a “wall”
that is impassable to heat, and thus an adiabatic process is a thermodynamic
state transformation during which no heat is exchanged [1]. However, the
notion of adiabaticity has found a much broader application in Hamiltonian
dynamics [2]. In classical mechanics, an “adiabatic invariant” is any quantity
that remains constant under the Hamiltonian equations of motion, given
infinitely slow variations of the Hamiltonian [2].

This insight led Born to the formulation of the quantum adiabatic theorem [3],
which states that, during infinitely slow variation of the Hamiltonian, no
transitions between energy levels occur. Obviously, such adiabatic processes
are highly desirable in quantum technological applications. Recent years have
seen tremendous research efforts in facilitating such excitation-free
processes with finite time driving. Under the umbrella of _shortcuts to
adiabaticity_ (STA) [4, 5] a large variety of techniques has been developed,
of which counterdiabatic driving [6, 7, 8, 9, 10, 11], invariant based inverse
engineering protocols [12, 13, 14, 15, 16], and the fast-forward technique
[17, 18, 19, 20, 21, 22, 23] have arguably received the most attention, with
applications in vastly different physical scenarios. For instance,
counterdiabatic driving is particularly well-suited to optimally control the
dynamics of cold ion traps [24, 25]. However, implementing STA in more complex
quantum system can become rather involved [26, 27, 28, 29, 30, 31]. Thus, it
appears very desirable to find alternative and approximate schemes, that may
provide more universally applicable control strategies. This has already led
to the development of “resource friendly” control strategies [32, 33, 34, 35,
36, 37], that provide alternative means to suppress excitations arising from
populating energetically high-lying microstates.

One of the main causes for the complexity of finding realistically useful STA
rests in the fact that, to a certain degree, all methods originate in
circumventing the quantum adiabatic theorem [3]. Hence, the focus is on
preserving the occupation probabilities of the energy eigenstates, i.e.,
microstates [1]. However, in most experimental settings quantum states cannot
be easily measured; rather, _thermodynamic observables_ are monitored.
Therefore, _thermodynamic control_ has been suggested as a possible way to
construct approximate STA [38], see Ref. [39] for a recent perspective.
However, thermodynamic control methods are usually applied with a focus on
lowering the energetic cost of a given thermodynamic process [40, 41, 42, 43,
44].

In the present letter, we change the paradigm of this approach by proposing
genuine _shortcuts to thermodynamic quasistaticity_. To this end, we fully
accept the thermodynamic mind set, namely, we seek STA that preserve the
_adiabatic macrostate_ and not the occupations of microscopic energy
eigenstates of a quantum system. Hence, we demand that the macrostate of a
driven system (approximately) fulfills an instantaneous equation of state.
Such a control strategy is constructed by exploiting _adiabatic perturbation
theory_ [45], which has recently proven powerful in assessing nonequilibrium
excitations in driven quantum Ising chains [46, 47]. To demonstrate the
versatility of the approach, we benchmark our results against other STA, in
particular, against _fast quasiadiabatic driving_ [48, 49, 50, 51], which is
closest in spirit to our approach.

## Preliminaries

We start by establishing notions and notations. Consider a quantum system
described by a Hamiltonian
$H(\lambda)=\sum_{n}E_{n}(\lambda)\ket{n(\lambda)}\bra{n(\lambda)}$, where
$E_{n}(\lambda)$ and $\ket{n(\lambda)}$ are parametric, nondegenerate
eigenvalues and eigenstates, respectively. Moreover, $\lambda$ is an external
control parameter, such as the volume of a gas container or a magnetic field.
In the following, we will be interested in thermodynamic state transformations
that are driven by varying $\lambda=\lambda(t)$ (also called a protocol),
between times $t_{i}$ and $t_{f}$, taking the external parameter from an
initial value $\lambda_{i}$ to a final value $\lambda_{f}$. Moreover, we
assume that the quantum system is thermally insulated, and therefore, its time
evolution is unitary. Note that unitary dynamics are necessarily
thermodynamically adiabatic in the traditional sense, since no heat is
exchanged. Thus, unless otherwise stated, “adiabatic” means “quasistatic”
henceforth.

We further assume that the system is initially prepared in a quantum state
that is diagonal in the energy eigenbasis,
$\rho_{i}=\sum_{n}p_{n}\ket{n_{i}}\bra{n_{i}}$, where the subscript $i$ means
that a given quantity is evaluated at $t_{i}$, and
$\ket{n_{i}}=\ket{n(\lambda_{i})}$. The time-dependent state is then
determined by the von Neumann equation,
$i\hbar\,\dot{\rho}(t)=[H(\lambda),\rho(t)]$, and we denote derivatives with
respect to time by a dot.

It is worth emphasizing that, even if the initial state, $\rho_{i}$, is chosen
to be an equilibrium state, $\rho(t>t_{i})$ may be arbitrarily far from
equilibrium. Given an initially canonical state
[$\rho_{i}\propto\exp{\left(-\beta H_{i}\right)}$], even an infinitely slow
process will generally not keep the system in canonical equilibrium. This is
because the quasistatic evolution preserves the statistical weights in the
initial Hamiltonian. However, in the present analysis, our main focus is also
not the microstate, but rather the thermodynamic macrostate.

In (quantum) thermodynamics, a macrostate is fully characterized by its state
variables [1, 52], which fulfill an equation of state (EOS). At any instant,
the EOS can be obtained by calculating the equilibrium average of the
generalized force, $F(\lambda)$, which is given by [1]

$F(\lambda)=-\frac{\partial H(\lambda)}{\partial\lambda},$ (1)

and $\Lambda\equiv\text{tr}\left\\{\rho F\right\\}$ is the state variable
conjugate to $\lambda$. For any driven process, and writing the time-dependent
quantum state as $\rho(t)=\sum_{n}p_{n}\ket{\psi_{n}(t)}\bra{\psi_{n}(t)}$,
the corresponding average generalized force reads

$\Lambda(t)=\sum_{n}p_{n}\braket{\psi_{n}(t)}{F(\lambda)}{\psi_{n}(t)}.$ (2)

Here, $\ket{\psi_{n}(t)}$ is a solution of the corresponding Schrödinger
equation.

## Thermodynamic state transformations

Before we analyze the more general out of equilibrium situation, we inspect
Eq. (2) in the adiabatic limit $\tau\to\infty$. The adiabatic theorem dictates
that, if the evolution is slow enough, the solution to Schrödinger’s equation
can be written as [53]

$\ket{\psi_{n}^{(0)}(t)}=e^{i\phi_{n}(t)}\ket{n(\lambda)},$ (3)

where the superscript $(0)$ denotes the adiabatic limit and $\phi_{n}(t)$ is
the usual adiabatic phase (dynamic plus geometric). In this case, Eq. (2)
simplifies to

$\Lambda^{(0)}=\sum_{n}p_{n}F_{nn}(\lambda),$ (4)

where $F_{mn}(\lambda)=\braket{m(\lambda)}{F(\lambda)}{n(\lambda)}$. Notice
the lack of explicit time dependence in Eq. (4): this is the conventional EOS.
For infinitely slow variations of $\lambda$, Eq. (4) describes the evolution
of the macroscopic state in any mechanically adiabatic (and thermodynamically
adiabatic) process, i.e., for a thermodynamic state transformation.

## Beyond the adiabatic limit

Using adiabatic perturbation theory (APT), whose details we leave for the
Supplemental material [54], we can systematically compute finite-time
corrections to the EOS (4). Using Eqs. (1)–(3) of the Supplemental Material
[54] in Eq. (2) and keeping terms up to $\mathcal{O}(\tau^{-1})$, the first-
order correction becomes

$\begin{split}\Lambda^{(1)}(t)&=\underset{m\neq
n}{\sum_{m,n}}p_{n}\Re\left\\{2C_{mn}^{(1)}(t)F_{mn}^{*}(\lambda)\right\\}\\\
&=2\hbar\dot{\lambda}_{i}\underset{m\neq
n}{\sum_{m,n}}p_{n}\Im\left\\{F_{mn,i}\frac{e^{i\phi_{mn}(t)}}{E_{mn,i}^{2}}F_{mn}^{*}(\lambda)\right\\},\end{split}$
(5)

where we used the fact that the product of $F_{mn}^{*}(\lambda)$ and the first
term of Eq. (2) of the Supplemental Material [54] is purely imaginary. We
immediately observe that the first-order correction to the EOS is directly
proportional to the time derivative of the external parameter _at the
beginning of the process_. Hence, for all protocols with
$\dot{\lambda}_{i}=0$, the EOS is preserved up to $\mathcal{O}(\tau^{-2})$ in
any sufficiently slow process. We stress that this conclusion is independent
of the Hamiltonian considered, only depending on the validity of APT. Thus, we
have unveiled a universal design principle for optimal control strategies
applicable in any gapped quantum system, simple as well as complex.

Strategies where the time derivatives of the protocols vanish at the end
points of the evolution have already been discussed as ways to guarantee
adiabaticity in the microstate [55, 56, 57, 58]. However, we emphasize that
the first-order result for the macrostate only depends on the initial
derivative, and not the final derivative. This still leaves a lot of freedom
in finding “optimal” and experimentally implementable protocols. Thus, it
should be obvious that even better results can be achieved by complementing
our macroscopic strategy with microscopic methods.

(a)  (b)

Figure 1:  Magnetization of the TI chain in the entirely paramagnetic process
with $N=100$ starting at zero temperature. The results were numerically
obtained from the exact time-dependent dynamics. (a) State diagram of the TI
chain for an adiabatic (quasistatic) evolution (EOS), the LIN, the FQA, and
the FQ2 protocols for $J\tau=3$, starting from the top right corner. The inset
shows the time dependence of each protocol. (b) Excess magnetization
$\mu_{\mathrm{ex}}=\mu-\mu^{(0)}$ at the end of the process vs process
duration.

## Fast quasiadiabatic driving

One strategy to ensure APT convergence is the application of fast
quasiadiabatic (FQA) protocols [48, 49, 50, 51] and related approaches [5]. If
there is only one relevant energy gap $E_{mn}(\lambda)$ in the quantum system,
FQA provides a protocol $\lambda(t)$ for which first-order APT transitions
between eigenstates $m$ and $n$ are equally likely at any instant. This
protocol is the solution to a first order differential equation [48, 49, 50,
51]

$\hbar\left|\frac{\dot{\lambda}(t)F_{mn}(\lambda)}{E_{mn}^{2}(\lambda)}\right|=c_{1},$
(6)

where $c_{1}$ is a constant that, together with the integration constant, is
uniquely defined by the boundary conditions $\lambda(t_{i})=\lambda_{i}$ and
$\lambda(t_{f})=\lambda_{f}$. For a generic protocol, microscopic adiabaticity
is secured if the left-hand side of Eq. (6) is much smaller than unity for any
$t$, the quantitative adiabatic condition [59, 60] [Eq. (4) of the
Supplemental Material [54]]. The boundary conditions always lead to
$c_{1}\propto\tau^{-1}$, which means that the FQA protocol still requires
large enough $\tau$ for the adiabatic condition to be fulfilled. FQA’s
advantage is that it _naturally slows down_ where $E_{mn}(\lambda)$ is small
[see Eq. (6)], and thus, it may reach the adiabatic condition and make APT
converge for a smaller $\tau$, when compared to a generic protocol.

Curiously, FQA is limited to suppressing first-order transitions. The authors
of Ref. [51] remark that considering transitions of higher-than-one order APT
is not possible, since the associated differential equation would not have
enough constants to satisfy the boundary conditions on $\lambda$ and its
derivatives. For example, demanding the second-order APT transition
probabilities to be uniform along the process gives a second-order
differential equation,

$\hbar^{2}\left|\frac{1}{E_{mn}(\lambda)}\frac{d}{dt}\left(\frac{\dot{\lambda}(t)F_{mn}(\lambda)}{E_{mn}^{2}(\lambda)}\right)\right|=c_{2},$
(7)

which was obtained from Eq. (6) with the proper substitution to second order
coefficients, discussed in the Supplemental Material [54]. The three available
constants ($c_{2}$ plus two integration constants) in the solution of Eq. (7)
are insufficient to satisfy the four boundary conditions — two on $\lambda$
(same as FQA) and two on $\dot{\lambda}$, which are necessary to make the
second-order APT correction be the relevant correction.

Above, we have seen that, from the macroscopic dynamics, Eq. (5), optimal
driving protocols obey $\dot{\lambda}=0$ at the beginning (and not at the
end). This additional condition permits us to uniquely solve Eq. (7), if we
impose the same boundary conditions as the FQA method _plus_
$\dot{\lambda}(t_{i})=0$, which leads to $c_{2}\propto\tau^{-2}$. We will be
referring to this strategy as FQ2, and as we will see shortly, FQ2 clearly
outperforms FQA. We once again bring attention to the fact that making
$\dot{\lambda}(t_{i})=0$ gives null first order APT correction for the EOS of
_any gapped system_. Equations (6) and (7), which do depend on the system
through its eigenspectrum, are primarily used to guarantee early APT validity
and can be applied even when the Hamiltonian is only numerically
diagonalizable. In fact, at low temperature, knowledge of only a few
eigenlevels may be necessary, since only transitions between the lowest energy
eigenstates are relevant (see Fig. 2 of the Supplemental Material [54]).

## Illustrative example: quantum Ising chain

We now apply the above developed strategy to control a thermodynamically
relevant, exactly solvable system: the transverse field Ising model (TI) [61,
62]. The Hamiltonian reads

$H_{\mathrm{TI}}(\Gamma)=-\frac{1}{2}\left(J\sum_{j=1}^{N}\sigma^{z}_{j}\sigma^{z}_{j+1}+\Gamma\sum_{j=1}^{N}\sigma^{x}_{j}\right),$
(8)

where $J$ is the coupling constant, $\Gamma$ is the external magnetic field
and $\sigma_{j}^{x,z}$ are standard Pauli matrices for each spin $j$ (with
periodic boundary conditions). In the thermodynamic limit $N\to\infty$, this
system displays a quantum critical point (QCP) at $\Gamma=J$, where the energy
gap between ground and first excited states vanishes. For simplicity, we
assume $N$ to be even and that the system is initially prepared in its ground
state. The force is $F_{\mathrm{TI}}=\sum_{j=1}^{N}\sigma^{x}_{j}/2$, while
the nonequilibrium magnetization per spin reads

$\mu(t)=\frac{1}{2N}\sum_{j=1}^{N}\left\langle\sigma^{x}_{j}\right\rangle(t).$
(9)

In any finite time process, the magnetization can be separated into an
adiabatic contribution $\mu^{(0)}$ and an excess contribution
$\mu_{\mathrm{ex}}$. Details for how to calculate the nonequilibrium average
in Eq. (9) can be found in the Supplemental Material [54].

(a)  (b)

Figure 2:  Magnetization of the TI chain in the QCP crossing process starting
at zero temperature with $N=100$. The results of both panels were numerically
obtained from the exact time-dependent dynamics. (a) State diagram of the TI
chain for an adiabatic (quasistatic) evolution (EOS), the LIN, the UQA, and
the UQ2 protocols for $J\tau=50$, starting from the top right corner. The
inset shows the time dependence of each protocol. (b) Excess magnetization
$\mu_{\mathrm{ex}}=\mu-\mu^{(0)}$ at the end of the process vs process
duration.

First, we consider a process keeping the chain entirely in its paramagnetic
phase ($\Gamma>J$) and starting at zero temperature, i.e., with the chain
initially prepared in its ground state. We solve FQA and FQ2 for the smallest
gap of the system and compare them to a naive linear protocol (LIN) — the
results for a chain of finite size are shown in Fig. 1. In Fig. 1a, we show
$\mu$ of Eq. (9) vs $\Gamma$ in a process that approaches, but does not cross,
the QCP. The inset contains the time-dependence of each protocol, where it can
be seen that both FQA and FQ2 adapt to the system’s spectrum, but FQ2 does so
while still keeping null first derivative at the start. FQA has a very high
first derivative at the initial time, and this ultimately makes its evolution
have notable oscillations around the EOS. On the other hand, LIN follows the
EOS closely, up until a point where the gap gets too small, and it ends up
breaking adiabaticity. Finally, FQ2 follows the EOS right until the end, which
is a consequence of its compromise to attain adiabaticity while zeroing the
first order correction to the EOS. In Fig. 1b, we depict the excess
magnetization $\mu_{\mathrm{ex}}$ at $t_{f}$ as a function of $\tau$. It is
clear that FQ2 outperforms FQA for a generic $\tau$, even if FQ2 first crosses
the “adiabatic” $\mu_{\mathrm{ex}}=0$ line for a marginally bigger $\tau$ than
FQA.

As a second case, we consider the crossing of the QCP, from the paramagnetic
phase to the ferromagnetic phase. In a finite size chain, the gap at the QCP
is small but nonzero, which makes adiabaticity difficult but possible to
achieve. In this scenario, the smallness of the energy gap forces the FQA
protocol to slow down dramatically around the QCP and, consequently, to speed
up around the end points. This speed-up is detrimental in the ferromagnetic
phase of the TI chain, where the gap of many other sub-levels are comparable
to the gap of the lowest sub-level (see Fig. 1 of the Supplemental Material
[54]). Other energy differences can be taken into account when building FQA
protocols (see Ref. [63]), but the associated differential equation is not
exactly solvable and hardly numerically solvable when traversing the QCP.
Thus, to circumvent this issue, we apply a similar strategy known as uniform
quasi-adiabatic (UQA) [64] to the lowest sub-level of the TI chain. It is the
solution to Eq. (6) with the substitution
$F_{mn}(\lambda)\to\partial_{\lambda}E_{mn}(\lambda)$ [5], motivated by the
Kibble-Zurek mechanism of second-order quantum phase transitions. Thus, we
define a UQ2 protocol as the solution of Eq. (7) with the aforementioned
substitution, and we compare it to LIN and UQA in Fig. 2. Figure 2a is the
equivalent of Fig. 1a, but with a considerably larger process duration, which
evidences the difficulty of crossing the QCP while maintaining adiabaticity
(in the mechanical sense). The inset once again shows the time-dependence of
each strategy, and it is clear that both UQA and UQ2 slow down around the QCP.
The conclusion is the same as in the paramagnetic process: UQ2 follows the EOS
more closely. Furthermore, as can be seen in Fig. 2b, UQ2 gives final
$\mu_{\mathrm{ex}}=0$ for a significantly smaller $\tau$ than the other two
protocols, which is a consequence of its final first derivative also being
null at the end point (see inset of Fig. 2a).

## Concluding remarks

Controlling complex many body quantum systems is an involved task. While some
strategies have been successfully employed in platforms with great
technological promise, such as counterdiabatic driving in ion traps [24, 25],
more universally applicable paradigms appear desirable. To this end, we have
proposed to take inspiration from the mother of all control theories —
thermodynamics. Rather than aiming to control the microstate, we have
suggested controlling the macrostate and identifying protocols that preserve
the equation of state. This approach is somewhat akin to invariant based
strategies [13, 5], on which we comment in the Supplemental Material [54],
where we study thermodynamic shortcuts for the driven harmonic oscillator [65,
66, 67]. However, our approach significantly goes beyond existing methods,
since using adiabatic perturbation theory, finite-time corrections can be
systematically computed, which gives systematic conditions for the optimal
driving protocols. The utility of the approach has been demonstrated by
improving upon fast quasiadiabatic driving, and its applicability has been
demonstrated for the driven Ising chain.

The analyses of state diagrams demonstrate the difference between microscopic
adiabaticity and macroscopic adiabaticity. More specifically, strategies that
are better suited for parametric following of microstates (eigenstates) are
not necessarily better for parametric following of macrostates (state
variables). It is also worth noting that a notion of relaxation time seems to
be absent, which is perhaps expected in isolated systems where relaxation to
some sort of equilibrium is not guaranteed. Nonetheless, there is still the
notion of a timescale with which the driving rate must be compared, related to
the energy gap between eigenstates. Last, it is interesting to see that, even
though it is possible to stay close to the equation of state in finite time
driving, such possibility a does not lead to thermodynamic reversibility. In
other words, applying the same “optimal” protocol in the reverse process does
not give the same curve in the state diagram as in the forward process and, in
fact, the FQ2 strategy we devised to better follow the equation of state does
not provide protocols with time-reversal symmetry.

Finally, we note that the present paper fills the gap in a hierarchy of
strategies developed for securing adiabaticity in finite time. First, there
are standard shortcuts to adiabaticity, where one seeks to follow the
parametric eigenstates of the system. Second, we have the thermodynamic
shortcuts introduced in the present letter, which follow the equation of
state. Third, we have the methods from thermodynamic control, where the focus
is on making sure that the energetic cost of a certain manipulation of the
system is as close as possible to the cost in a quasistatic process. It is
expected that the further down you go in the hierarchy, the less information
is needed to determine the associated optimal driving protocol.

###### Acknowledgements.

A. S. and M. V. S. B. thank the National Council for Scientific and
Technological Development — CNPq under grant 140549/2018-8 and FAEPEX (Fundo
de Apoio ao Ensino, à Pesquisa e à Extensão)(Brazil)(Grant 2146-22). M. V. S.
B. also acknowledges the financial support of FAPESP (Fundação de Amparo à
Pesquisa do Estado de São Paulo) (Brazil)(Grant 2020/02170-4). E.M. also
thanks the support of CNPq through Grant No. 309584/2021-3 and Capes through
Grant No. 0899/2018. S.D. acknowledges support from the U.S. National Science
Foundation under Grant No. DMR-2010127.

## References

  * Callen [1985] H. B. Callen,  _Thermodynamics and an introduction to thermostatistics_ (Wiley, New York, USA, 1985).
  * Goldstein [1980] H. Goldstein,  _Classical Mechanics_ (Addison-Wesley, New York, USA, 1980).
  * Born and Fock [1928] M. Born and V. Fock, Z. Physik 51, 165 (1928).
  * Torrontegui  _et al._ [2013] E. Torrontegui, S. Ibáñez, S. Martínez-Garaot, M. Modugno, A. del Campo, D. Guéry-Odelin, A. Ruschhaupt, X. Chen, and J. G. Muga, Adv. At. Mol. Opt. Phys. 62, 117 (2013).
  * Guéry-Odelin  _et al._ [2019] D. Guéry-Odelin, A. Ruschhaupt, A. Kiely, E. Torrontegui, S. Martínez-Garaot, and J. G. Muga, Rev. Mod. Phys. 91, 045001 (2019).
  * Demirplak and Rice [2003] M. Demirplak and S. A. Rice, J. Chem. Phys. A 107, 9937 (2003).
  * Demirplak and Rice [2005] M. Demirplak and S. A. Rice, J. Phys. Chem. B 109, 6838 (2005).
  * Berry [2009] M. Berry, J. Phys. A: Math. Theor. 42, 365303 (2009).
  * Deffner  _et al._ [2014] S. Deffner, C. Jarzynski, and A. del Campo, Phys. Rev. X 4, 021013 (2014).
  * Iram  _et al._ [2021] S. Iram, E. Dolson, J. Chiel, J. Pelesko, N. Krishnan, Ö. Güngör, B. Kuznets-Speck, S. Deffner, E. Ilker, J. G. Scott, and M. Hinczewski, Nat. Phys. 17, 135 (2021).
  * Ilker  _et al._ [2022] E. Ilker, O. Güngör, B. Kuznets-Speck, J. Chiel, S. Deffner, and M. Hinczewski, Phys. Rev. X 12, 021048 (2022).
  * Lewis Jr. and Riesenfeld [1969] H. R. Lewis Jr. and W. B. Riesenfeld, J. Math. Phys. 10, 1458 (1969).
  * Chen  _et al._ [2010] X. Chen, A. Ruschhaupt, S. Schmidt, A. del Campo, D. Guéry-Odelin, and J. G. Muga, Phys. Rev. Lett. 104, 063002 (2010).
  * Torrontegui  _et al._ [2014] E. Torrontegui, S. Martínez-Garaot, and J. G. Muga, Phys. Rev. A 89, 043408 (2014).
  * Kiely  _et al._ [2015] A. Kiely, J. P. L. McGuinness, J. G. Muga, and A. Ruschhaupt, J. Phys. B: At. Mol. Opt. Phys. 48, 075503 (2015).
  * Levy  _et al._ [2018] A. Levy, A. Kiely, J. G. Muga, R. Kosloff, and E. Torrontegui, New J. Phys. 20, 025006 (2018).
  * Masuda and Nakamura [2010] S. Masuda and K. Nakamura, Proc. R. Soc. A 466, 1135 (2010).
  * Masuda and Nakamura [2011] S. Masuda and K. Nakamura, Phys. Rev. A 84, 043434 (2011).
  * Torrontegui  _et al._ [2012] E. Torrontegui, X. Chen, M. Modugno, A. Ruschhaupt, D. Guéry-Odelin, and J. G. Muga, Phys. Rev. A 85, 033605 (2012).
  * Masuda and Rice [2015] S. Masuda and S. A. Rice, J. Phys. Chem. A 119, 3479 (2015).
  * Deffner [2015] S. Deffner, New J. Phys. 18, 012001 (2015).
  * Jarzynski  _et al._ [2017] C. Jarzynski, S. Deffner, A. Patra, and Y. Subaşı, Phys. Rev. E 95, 032122 (2017).
  * Myers and Deffner [2021] N. M. Myers and S. Deffner, PRX Quantum 2, 040312 (2021).
  * An  _et al._ [2016] S. An, D. Lv, A. del Campo, and K. Kim, Nature Communications 7, 12999 (2016).
  * Funo  _et al._ [2017] K. Funo, J.-N. Zhang, C. Chatou, K. Kim, M. Ueda, and A. del Campo, Phys. Rev. Lett. 118, 100602 (2017).
  * del Campo  _et al._ [2012] A. del Campo, M. M. Rams, and W. H. Zurek, Phys. Rev. Lett. 109, 115703 (2012).
  * Campbell  _et al._ [2015] S. Campbell, G. De Chiara, M. Paternostro, G. M. Palma, and R. Fazio, Phys. Rev. Lett. 114, 177206 (2015).
  * Balasubramanian  _et al._ [2018] S. Balasubramanian, S. Han, B. T. Yoshimura, and J. K. Freericks, Phys. Rev. A 97, 022313 (2018).
  * Cohn  _et al._ [2018] J. Cohn, A. Safavi-Naini, R. J. Lewis-Swan, J. G. Bohnet, M. Gärttner, K. A. Gilmore, J. E. Jordan, A. M. Rey, J. J. Bollinger, and J. K. Freericks, New J. Phys. 20, 055013 (2018).
  * Ness  _et al._ [2018] G. Ness, C. Shkedrov, Y. Florshaim, and Y. Sagi, New J. Phys. 20, 095002 (2018).
  * Carolan  _et al._ [2022] E. Carolan, A. Kiely, and S. Campbell, Phys. Rev. A 105, 012605 (2022).
  * Sels and Polkovnikov [2017] D. Sels and A. Polkovnikov, Proceedings of the National Academy of Sciences 114, E3909 (2017).
  * Claeys  _et al._ [2019] P. W. Claeys, M. Pandey, D. Sels, and A. Polkovnikov, Phys. Rev. Lett. 123, 090602 (2019).
  * Passarelli  _et al._ [2020] G. Passarelli, V. Cataudella, R. Fazio, and P. Lucignano, Phys. Rev. Research 2, 013283 (2020).
  * Passarelli  _et al._ [2022] G. Passarelli, R. Fazio, and P. Lucignano, Phys. Rev. A 105, 022618 (2022).
  * Hegade  _et al._ [2021] N. N. Hegade, P. Chandarana, K. Paul, X. Chen, F. Albarrán-Arriagada, and E. Solano, Portfolio optimization with digitized-counterdiabatic quantum algorithms (2021), arXiv:2112.08347 .
  * Mbeng and Lechner [2022] G. B. Mbeng and W. Lechner, Rotated ansatz for approximate counterdiabatic driving (2022), arXiv:2207.03553 .
  * Martínez  _et al._ [2016] I. A. Martínez, A. Petrosyan, D. Guéry-Odelin, E. Trizac, and S. Ciliberto, Nat. Phys. 12, 843 (2016).
  * Deffner and Bonança [2020] S. Deffner and M. V. S. Bonança, EPL (Europhysics Letters) 131, 20001 (2020).
  * Li  _et al._ [2017] G. Li, H. T. Quan, and Z. C. Tu, Phys. Rev. E 96, 012144 (2017).
  * Chen  _et al._ [2019] J.-F. Chen, C.-P. Sun, and H. Dong, Phys. Rev. E 100, 032144 (2019).
  * Pancotti  _et al._ [2020] N. Pancotti, M. Scandi, M. T. Mitchison, and M. Perarnau-Llobet, Phys. Rev. X 10, 031015 (2020).
  * Li  _et al._ [2022] G. Li, J.-F. Chen, C. P. Sun, and H. Dong, Phys. Rev. Lett. 128, 230603 (2022).
  * Frim and DeWeese [2022] A. G. Frim and M. R. DeWeese, Phys. Rev. E 105, L052103 (2022).
  * Rigolin  _et al._ [2008] G. Rigolin, G. Ortiz, and V. H. Ponce, Phys. Rev. A 78, 052508 (2008).
  * Soriani  _et al._ [2022a] A. Soriani, P. Nazé, M. V. S. Bonança, B. Gardas, and S. Deffner, Phys. Rev. A 105, 042423 (2022a).
  * Soriani  _et al._ [2022b] A. Soriani, P. Nazé, M. V. S. Bonança, B. Gardas, and S. Deffner, Phys. Rev. A 105, 052442 (2022b).
  * Kastberg  _et al._ [1995] A. Kastberg, W. D. Phillips, S. L. Rolston, R. J. C. Spreeuw, and P. S. Jessen, Phys. Rev. Lett. 74, 1542 (1995).
  * Bowler  _et al._ [2012] R. Bowler, J. Gaebler, Y. Lin, T. R. Tan, D. Hanneke, J. D. Jost, J. P. Home, D. Leibfried, and D. J. Wineland, Phys. Rev. Lett. 109, 080502 (2012).
  * Martínez-Garaot  _et al._ [2013] S. Martínez-Garaot, E. Torrontegui, X. Chen, M. Modugno, D. Guéry-Odelin, S.-Y. Tseng, and J. G. Muga, Phys. Rev. Lett. 111, 213001 (2013).
  * Martínez-Garaot  _et al._ [2015] S. Martínez-Garaot, A. Ruschhaupt, J. Gillet, T. Busch, and J. G. Muga, Phys. Rev. A 92, 043406 (2015).
  * Deffner and Campbell [2019] S. Deffner and S. Campbell,  _Quantum Thermodynamics_ (Morgan and Claypool, Bristol, 2019).
  * Messiah [1962] A. Messiah,  _Quantum mechanics: volume II_ (North-Holland Publishing Company Amsterdam, 1962).
  * [54] See supplemental material for a review of adiabatic perturbation theory, details of the ising chain, and comparisons of our method with invariant-based inverse engineering for the driven harmonic oscillator.
  * Jansen  _et al._ [2007] S. Jansen, M.-B. Ruskai, and R. Seiler, Journal of Mathematical Physics 48, 102111 (2007).
  * Morita and Nishimori [2008] S. Morita and H. Nishimori, J. Math. Phys. 49, 125210 (2008).
  * Rezakhani  _et al._ [2010] A. T. Rezakhani, A. K. Pimachev, and D. A. Lidar, Phys. Rev. A 82, 052305 (2010).
  * Campos Venuti and Lidar [2018] L. Campos Venuti and D. A. Lidar, Phys. Rev. A 98, 022315 (2018).
  * Tong  _et al._ [2005] D. M. Tong, K. Singh, L. C. Kwek, and C. H. Oh, Phys. Rev. Lett. 95, 110407 (2005).
  * Tong [2010] D. M. Tong, Phys. Rev. Lett. 104, 120401 (2010).
  * Pfeuty [1970] P. Pfeuty, Ann. Phys. (N.Y.) 57, 79 (1970).
  * Dziarmaga [2005] J. Dziarmaga, Phys. Rev. Lett. 95, 245701 (2005).
  * Soriani  _et al._ [2022c] A. Soriani, E. Miranda, and M. V. S. Bonança, Failure of the geometric approach prediction of excess work scaling for open and isolated quantum systems (2022c), arXiv:2206.07105 .
  * Quan and Zurek [2010] H. T. Quan and W. H. Zurek, New J. Phys. 12, 093025 (2010).
  * Husimi [1953] K. Husimi, Prog. Theor. Phys. 9, 381 (1953).
  * Deffner and Lutz [2008] S. Deffner and E. Lutz, Phys. Rev. E 77, 021128 (2008).
  * Deffner  _et al._ [2010] S. Deffner, O. Abah, and E. Lutz, Chem. Phys. 375, 200 (2010).


# DeFeat-Net: General Monocular Depth via Simultaneous Unsupervised
Representation Learning

Jaime Spencer, Richard Bowden, Simon Hadfield
Centre for Vision, Speech and Signal Processing (CVSSP)
University of Surrey
{jaime.spencer, r.bowden<EMAIL_ADDRESS>
###### Abstract

In the current monocular depth research, the dominant approach is to employ
unsupervised training on large datasets, driven by warped photometric
consistency. Such approaches lack robustness and are unable to generalize to
challenging domains such as nighttime scenes or adverse weather conditions
where assumptions about photometric consistency break down.

We propose DeFeat-Net (Depth & Feature network), an approach to simultaneously
learn a cross-domain dense feature representation, alongside a robust depth-
estimation framework based on warped feature consistency. The resulting
feature representation is learned in an unsupervised manner with no explicit
ground-truth correspondences required.

We show that within a single domain, our technique is comparable to both the
current state of the art in monocular depth estimation and supervised feature
representation learning. However, by simultaneously learning features, depth
and motion, our technique is able to generalize to challenging domains,
allowing DeFeat-Net to outperform the current state-of-the-art with around 10%
reduction in all error measures on more challenging sequences such as
nighttime driving.

##  1 Introduction

Recently there have been many advances in computer vision tasks related to
autonomous vehicles, including monocular depth estimation [22, 83, 73] and
feature learning [13, 61, 65]. However, as shown in Figure 1, these approaches
tend to fail in the most complex scenarios, namely adverse weather and
nighttime conditions.

In the case of depth estimation, this is usually due to the assumption of
photometric consistency, which starts to break down in dimly-lit environments.
Feature learning can overcome such strong photometric assumptions, but these
approaches tend to require ground truth pixel-wise correspondences and
obtaining this ground truth in cross-seasonal situations is non-trivial.
Inconsistencies between GPS measurements and drift from Visual Odometry (VO)
makes automatic pointcloud alignment highly inaccurate and manual annotation
is costly and time-consuming.

Figure 1: Left: Challenging lighting conditions during nighttime driving.
Right: A catastrophic failure during depth map estimation for a current state-
of-the-art monocular depth estimation framework, after being trained
specifically for this scenario.

We make the observation that depth estimation and feature representation are
inherently complementary. The process of estimating the depth for a scene also
allows for the computation of ground-truth feature matches between any views
of the scene. Meanwhile robust feature spaces are necessary in order to create
reliable depth-estimation systems with invariance to lighting and appearance
change.

Despite this relationship, all existing approaches tackle these challenges
independently. Instead, we propose DeFeat-Net, a system that is capable of
jointly learning depth from a single image in addition to a dense feature
representation of the world and ego-motion between consecutive frames. What’s
more, this is achieved in an entirely self-supervised fashion, requiring no
ground truth other than a monocular stream of images.

We show how the proposed framework can use the existing relationships between
these tasks to complement each other and boost performance in complex
environments. As has become commonplace [23], the predicted depth and ego-
motion can be used to generate a correspondence map between consecutive
images, allowing for the use of photometric error based losses. However, these
correspondences can also be used as positive examples in relative metric
learning losses [65]. In turn, the learnt features can provide a more robust
loss in cases where photometric errors fail, _i.e_. nighttime conditions.

The remainder of the paper provides a more detailed description of the
proposed DeFeat-Net framework in the context of previous work. We extensively
show the benefits of our joint optimization approach, evaluating on a wide
variety of datasets. Finally, we discuss the current state-of-the-art and
opportunities for future work. The contributions of this paper can be
summarized as:

  1. 1.

We introduce a framework capable of jointly and simultaneously learning
monocular depth, dense feature representations and vehicle ego-motion.

  2. 2.

This is achieved entirely self-supervised, eliminating the need for costly and
unreliable ground truth data collection.

  3. 3.

We show how the system provides robust depth and invariant features in all
weather and lighting conditions, establishing new state-of-the-art
performance.

##  2 Related Work

Here we review some of the most relevant previous work, namely in depth
estimation and feature learning.

###  2.1 Depth Estimation

Traditionally, depth estimation relied on finding correspondences between
every pixel in pairs of images. However, if the images have been stereo
rectified, the problem can be reduced to a search for the best match along a
single row in the target image, known as disparity estimation. Initial methods
for disparity estimation relied on hand-crafted matching techniques based on
the Sum of Squared Differences (SSD), smoothness and energy minimization.

Supervised. Ladickỳ [33] and Žbontar [79] showed how learning the matching
function can drastically improve the performance of these systems. Mayer _et
al_. [46] instead proposed DispNet, a Fully Convolutional Network (FCN) [40]
capable of directly predicting the disparity map between two images, which was
further extended by [50]. Kendall _et al_. [30] introduced GC-Net, where the
disparity is processed as a matching cost-volume in a 3D convolutional
network. PSMNet [9] and GA-Net [81] extended these cost-volume networks by
introducing Spatial Pooling Pyramid (SPP) features and Local/Semi-Global
aggregation layers, respectively.

Estimating depth from a single image seemed like an impossible task without
these disparity and perspective cues. However, Saxena [58] showed how it is
possible to approximate the geometry of the world based on superpixel
segmentation. Each superpixel’s 3D position and orientation is estimated using
a trained linear model and an MRF. Liu _et al_. [38, 39] improve on this
method by instead learning these models using a CNN, while Ladickỳ _et al_.
[34] incorporate semantic information as an alternative cue.

Eigen _et al_. [14, 15] introduced the first methods for monocular depth
regression using end-to-end deep learning by using a scale-invariant loss.
Laina [35] and Cao [7] instead treated the task of monocular estimation as a
classification problem and introduced a more robust loss function. Meanwhile,
Ummenhofer _et al_. [66] introduced DeMoN, jointly training monocular depth
and egomotion in order to perform Structure-from-Motion (SfM). In this paper
we go one step further, jointly learning depth, egomotion and the feature
space used to support them.

Unsupervised - Stereo Training. In order to circumvent the need for costly
ground truth training data, an increasing number of approaches have been
proposed using photometric warp errors as a substitute. For instance,
DeepStereo [17] synthesizes novel views using raw pixels from arbitrary nearby
views. Deep3D [74] also performs novel view synthesis, but restricts this to
stereo pairs and introduces a novel image reconstruction loss. Garg [18] and
Godard [23] greatly improved the performance of these methods by introducing
an additional autoencoder and left-right consistency losses, respectively.
UnDeepVO [37] additionally learns monocular VO between consecutive frames by
aligning the predicted depth pointclouds and enforcing consistency between
both stereo streams. More recently, there have been several approaches making
use of GANs [1, 53]. Most notably, [62] uses GANs to perform day-night
translation and provide an additional consistency to improve performance in
nighttime conditions. However, the lack of any explicit feature learning makes
it challenging to generalize across domains.

Unsupervised - Monocular Training. In order to learn unsupervised monocular
depth without stereo information, it is necessary to learn a surrogate task
that allows for the use of photometric warp losses. Zhou _et al_. [82, 83]
introduced some of the first methods to make use of VO estimation to warp the
previous and next frames to reconstruct the target view. Zhan [80] later
extended this by additionally incorporating a feature based warp loss. Babu
_et al_. [3, 44] proposed an unsupervised version of DeMoN [66]. Other
published methods are based upon video processing with RNNs [69] and LSTMs
[51] or additionally predicting scene motion [67] or optical flow [29, 70,
78].

The current state-of-the-art has been pushed by methods that incorporate
additional constraints [68] such as temporal [45], semantic [10], edge &
normal [75, 76], cross-task [84] and cycle [52, 73] consistencies. Godard _et
al_. [22] expanded on these methods by incorporating information from the
previous frame and using the minimum reprojection error in order to deal with
occlusions. They also introduce an automasking process which removes
stationary pixels in the target frame. However, they still compute photometric
losses in the original RGB colourspace, making it challenging to learn across
domains.

###  2.2 Feature Learning

Hand-Crafted. Initial approaches to feature description typically relied on
heuristics based on intensity gradients in the image. Since these were
computationally expensive, it became necessary to introduce methods capable of
finding interesting points in the image, _i.e_. keypoints. Some of the most
well-know methods include SIFT [41] and its variant RootSIFT [2], based on a
Difference of Gaussians and Non-Maxima Suppression (NMS) for keypoint
detection and HOG descriptors.

Research then focused on improving the speed of these systems. Such is the
case with SURF [5], BRIEF [6] and BRISK [36]. ORB features [56] improved the
accuracy, robustness and speed of BRIEF [6] and are still widely used.

Sparse Learning. Initial feature learning methods made use of decision trees
[55], convex optimization [63] and evolutionary algorithms [31, 32] in order
to improve detection reliability and discriminative power. Intelligent Cost
functions [24] took this a step further, by using Gaussian Processes to learn
appropriate cost functions for optical/scene flow.

Since the widespread use of deep learning, several methods have been proposed
to learn feature detection and/or description. Balntas _et al_. [4] introduced
a method for learning feature descriptors using in-triplet hard negative
mining. LIFT [77] proposes a sequential pipeline consisting of keypoint
detection, orientation estimation and feature description, each performed by a
separate network. LF-Net [49] builds on this idea, jointly generating dense
score and orientation maps without requiring human supervision.

On the other hand, several approaches make use of networks with shared encoder
parameters in order to simultaneously learn feature detection and description.
Georgakis _et al_. [20] learn 3D interest points using a shared Fast R-CNN
[21] encoder. Meanwhile, DeTone introduced SuperPoint [12] where neither
decoder has trainable parameters, improving the overall speed and
computational cost. More recently, D2-Net [13] proposed a describe-then-detect
approach where the network produces a dense feature map, from which keypoints
are detected using NMS.

Dense Learning. Even though SuperPoint [12] and D2-Net [13] produce dense
feature maps, they still focus on the detection of interest points and don’t
use their features in a dense manner. Weerasekera _et al_. [72] learn dense
features in the context of SLAM by minimizing multi-view matching cost-
volumes, whereas [60] use generative feature learning with scene completion as
an auxiliary task to perform visual localisation.

The Universal Correspondence Network [11] uses optical correspondences to
create a pixel-wise version of the contrastive loss. Schmidt [59] instead
propose semi-supervised training with correspondences obtained from
KinectFusion [47] and DynamicFusion [48] models. Fathy [16] and Spencer [65]
extended the pixel-wise contrastive loss to multiple scale features through a
coarse-to-fine network and spatial negative mining, respectively. On the other
hand, SDC-Net [61] focuses on the design of the network architecture,
increasing the receptive field through stacked dilated convolution, and apply
the learnt features to optical flow estimation.

In this work we attempt to unify state-of-the-art feature learning with
monocular depth and odometry estimation. This is done in such a way that the
pixel-wise correspondences from monocular depth estimation can support dense
feature learning in the absence of ground-truth labels. Meanwhile, computing
match-costs in the learned feature space greatly improves the robustness of
the depth estimation in challenging cross-domain scenarios.

##  3 Methodology

Figure 2: Overview of _DeFeat-Net_ which combines complementary networks to
simultaneously solve for feature representation, depth and ego-motion. The
introduction of feature warping improves the robustness in complex scenarios.

The main objective of DeFeat-Net is to jointly learn monocular depth and dense
features in order to provide more robust estimates in adverse weather
conditions. By leveraging the synergy between both tasks we are able to do
this in a fully self-supervised manner, requiring only a monocular stream of
images. Furthermore, as a byproduct of the training losses, the system
additionally learns to predict VO between consecutive frames.

Figure 2 shows an overview of DeFeat-Net. Each training sample is composed of
a target frame $I_{t}$ and a set of support frames $I_{t+k}$, where
$k\in\\{-1,1\\}$. Using the predicted depth for $I_{t}$ and the predicted
transforms to $I_{t+k}$ we can obtain a series of correspondences between
these images, which in turn can be used in the photometric warp and pixel-wise
contrastive losses. The code and pre-trained models for this technique will be
available at https://github.com/jspenmar/DeFeat-Net.

###  3.1 Networks

DispNet. Given a single input image, $I_{t}$, its corresponding depth map is
obtained through

$D_{t}=\frac{1}{a\;\Phi_{D}(I_{t})+b},$ (1)

where $a$ and $b$ scale the final depth to the range $[0.1,100]$. $\Phi_{D}$
represents the disparity estimation network, formed by a ResNet [25] encoder
and decoder with skip connections. This decoder also produces intermediate
disparity maps at each stage, resulting in four different scales.

PoseNet. Similarly, the pose prediction network $\Phi_{P}$ consists of a
multi-image ResNet encoder, followed by a 4-layer convolutional decoder.
Formally,

$P_{t\rightarrow t+k}=\Phi_{P}(I_{t},I_{t+k}),$ (2)

where $P_{t\rightarrow t+k}$ is the predicted transform between the cameras at
times $t$ and $t+k$. As in [22, 68] the predicted pose is composed of a
rotation in axis-angle representation and a translation vector, scaled by a
factor of 0.001.

FeatNet. The final network produces a dense $n$-dimensional feature map of the
given input image, $\Phi_{F}:\mathbb{N}^{H\times W\times
3}\mapsto\mathbb{R}^{H\times W\times n}$. As such, we define the corresponding
L2-normalized feature map as

$F=\left|\left|\Phi_{F}(I)\right|\right|.$ (3)

In this case, $\Phi_{F}$ is composed of a residual block encoder-decoder with
skip connections, where the final encoder stage is made up of an SPP [9] with
four scales.

###  3.2 Correspondence Module

Using the predicted $D_{t}$ and $P_{t\rightarrow t+k}$ we can obtain a set of
pixel-wise correspondences between the target frame and each of the support
frames. Given a 2D point in the image $p$ and its homogeneous coordinates
$\dot{p}$ we can obtain its corresponding location $q$ in the 3D world through

$q=\pi^{-1}(\dot{p})=K_{t}^{-1}\dot{p}\;D_{t}(p),$ (4)

where $\pi^{-1}$ is the backprojection function, $K_{t}$ is the camera’s
intrinsics and $D_{t}(p)$ the depth value at the 2D pixel location estimated
using (1).

We can then compute the corresponding point $c_{t\rightarrow t+k}$ by
projecting the resulting 3D point onto a new image with

$c_{t\rightarrow t+k}(p)=\pi(\dot{q})=K_{t}P_{t\rightarrow t+k}\dot{q},$ (5)

where $P_{t\rightarrow t+k}$ is the transform to the new coordinate frame,
_i.e_. the next or previous camera position from (2). Therefore, the final
correspondences map is defined as

$C_{t\rightarrow t+k}=\left\\{c_{t\rightarrow t+k}(p):\forall p\right\\}.$ (6)

These correspondences can now be used in order to determine the sampling
locations for the photometric warp loss and the positive matches in a pixel-
wise contrastive loss to learn an appropriate feature space.

###  3.3 Losses

Once again, it is worth noting that DeFeat-Net is entirely self-supervised. As
such, the only ground truth inputs required are the orginal images and the
camera’s intrinsics.

Pixel-wise Contrastive. In order to train $\Phi_{F}$, we make use of the well
established pixel-wise contrastive loss [11, 59, 65]. Given two feature
vectors from the dense feature maps, $f_{1}=F_{1}(p_{1})$ and
$f_{2}=F_{2}(p_{2})$, the contrastive loss is defined as

$l(y,f_{1},f_{2})=\begin{cases}\frac{1}{2}(d)^{2}&\text{if }y=1\\\
\frac{1}{2}\\{\max(0,m-d)\\}^{2}&\text{if }y=0\\\
0&otherwise\end{cases}\vspace{-0.2cm}$ (7)

with $y$ as the label indicating if the pair is a correspondence,
$d=\left|\left|f_{1}-f_{2}\right|\right|$ and $m$ the target margin between
negative pairs. In this case, the set of positive correspondences is given by
$C_{t\rightarrow t+k}$. Meanwhile, the negative examples are generated using
one of the spatial negative mining techniques from [65].

From both sets, a label mask $Y$ is created indicating if each possible pair
of pixels is a positive, negative or should be ignored. As such, the final
loss is defined as

$L_{C}=\sum_{p_{1}}\sum_{p_{2}}l(Y(p_{1},p_{2}),F_{t}(p_{1}),F_{t+k}(p_{2})).$
(8)

This loss serves to drive the learning of a dense feature space which enables
matching regardless of weather and seasonal appearance variations.

Photometric and Feature Warp. We also use the correspondences in a
differentiable bilinear sampler [28] in order to generate the warped support
frames and feature maps

$I_{t+k\rightarrow t}=I_{t+k}\langle C_{t\rightarrow t+k}\rangle$ (9)
$F_{t+k\rightarrow t}=F_{t+k}\langle C_{t\rightarrow t+k}\rangle$ (10)

where $\langle\rangle$ is the sampling operator. The final warp losses are a
weighted combination of SSIM [71] and L1, defined by

$\Psi(I_{1},I_{2})=\alpha\,\frac{1\\!-\\!SSIM(I_{1},I_{2})}{2}+(1\\!-\\!\alpha)\,\left|\left|I_{1}\\!-\\!I_{2}\right|\right|$
(11) $L_{P}=\Psi(I_{t},I_{t+k\rightarrow t}),$ (12)
$L_{F}=\Psi(F_{t},F_{t+k\rightarrow t}),$ (13)

The photometric loss $L_{P}$ serves primarily to support the early stages of
training when the feature space is still being learned.

Smoothness. As an additional regularizing constraint, we incorporate a
smoothness loss [27]. This enforces local smoothness in the predicted depths
proportional to the strength of the edge in the original image, $\partial
I_{t}$. This is defined as

$L_{S}=\frac{\lambda}{N}\sum_{p}\left|\partial
D_{t}(p)\right|e^{-\left|\left|\partial I_{t}(p)\right|\right|},$ (14)

where $\lambda$ is a scaling factor typically set to 0.001. This loss is
designed to avoid smoothing over edges by reducing the weighting in areas of
strong intensity gradients.

###  3.4 Masking & Filtering

Some of the more recent improvements in monocular depth estimation have arisen
from explicit edge-case handling [22]. This includes occlusion filtering and
the masking of stationary pixels. We apply these automatic procedures to the
correspondences used to train both the depth and dense features.

Minimum Reprojection. As the camera capturing the monocular stream moves
throughout the scene, various elements will become occluded and disoccluded.
In terms of a photometric error based loss, this means that some of the
correspondences generated by the system will be invalid. However, when
multiple consecutive frames are being used, _i.e_. $k\in\\{-1,1\\}$, different
occlusions occur in each image.

By making the assumption that the photometric error will be greater in the
case where an occlusion is present, we can filter these out by simply
propagating the correspondence with the minimum error. This is defined as

$C_{t\rightarrow t+k}\\!=\\!\begin{cases}c_{t\rightarrow
t-1}&\\!\\!\text{where }\Psi(I_{t},I_{t\rightarrow
t-1})\\!<\\!\Psi(I_{t},I_{t\rightarrow t+1})\\\ c_{t\rightarrow
t+1}&\\!\\!\text{otherwise}\end{cases}$ (15)

Automasking. Due to the nature of the training method and implicit depth
priors (_i.e_. regions further away change less) stationary frames or moving
objects can cause holes of infinite depth in the predicted depth maps. An
automasking procedure is used to remove these stationary pixels from
contributing to the loss,

$\mu=\left[\min_{k}\Psi(I_{t},I_{t+k})<\min_{k}\Psi(I_{t},I_{t+k\rightarrow
t})\right],$ (16)

where $\mu$ is the resulting mask indicating if a correspondence is valid or
not and $[]$ is the Iverson bracket. In other words, pixels that exhibit lower
photometric error to the unwarped frame than to the warped frame are masked
from the cost function.

Method | Abs-Rel | Sq-Rel | RMSE | RMSE-log | A1 | A2 | A3
---|---|---|---|---|---|---|---
LEGO [75] | 0.162 | 1.352 | 6.276 | 0.252 | - | - | -
Ranjan [54] | 0.148 | 1.149 | 5.464 | 0.226 | 0.815 | 0.935 | 0.973
EPC++ [42] | 0.141 | 1.029 | 5.350 | 0.216 | 0.816 | 0.941 | 0.976
Struct2depth ‘(M)’ [8] | 0.141 | 1.026 | 5.291 | 0.215 | 0.816 | 0.945 | 0.979
Monodepth V2 [22] | 0.123 | 0.944 | 5.061 | 0.197 | 0.866 | 0.957 | 0.980
DeFeat | 0.126 | 0.925 | 5.035 | 0.200 | 0.862 | 0.954 | 0.980
Table 1: Monocular depth evaluation on the KITTI dataset Method | $\mu_{+}$ | Global $\mu_{-}$ | Global AUC | Local $\mu_{-}$ | Local AUC
---|---|---|---|---|---
ORB [56] | N/A | N/A | 85.83 | N/A | 84.06
ResNet [26] | 8.5117 | 25.9872 | 94.77 | 11.1335 | 68.26
ResNet-L2 | 0.341 | 1.0391 | 99.25 | 0.4371 | 71.80
VGG [64] | 4.0077 | 12.6543 | 92.94 | 5.9088 | 70.03
VGG-L2 | 0.3905 | 1.2235 | 99.57 | 0.565 | 77.06
SAND-G [65] | 0.093 | 0.746 | 99.73 | 0.266 | 87.06
SAND-L | 0.156 | 0.592 | 98.88 | 0.505 | 94.34
SAND-GL | 0.183 | 0.996 | 99.28 | 0.642 | 93.34
DeFeat | 0.105 | 1.113 | 99.10 | 0.294 | 83.64
Table 2: Learned feature evaluation on the KITTI dataset

##  4 Results

Each subsystem in DeFeat-Net follows a U-Net structure with a ResNet18 encoder
pretrained on ImageNet, followed by a 7 layer convolutional decoder similar to
[23]. The code and pre-trained models will be available at
https://github.com/jspenmar/DeFeat-Net. In all our experiments, the warp loss
parameter is set to $\alpha=0.85$ as per [28].

On the KITTI dataset [19] we follow the Eigen-Zhou evaluation protocol of [23,
83]. This dataset split provides 39,810 training images and 4,424 validation
images. These images are all from a single domain (sunny daytime driving).

We also make use of the RobotCar Seasons dataset [57]. This is a curated
subset of the larger RobotCar dataset [43], containing 49 sequences. The
dataset was specifically chosen to cover a wide variety of seasons and weather
conditions, leading to greater diversity in appearance than KITTI.

Unlike the KITTI dataset, which provides sparse ground-truth depth from LiDAR,
RobotCar Seasons does not include any depth ground truth. Our proposed
technique is unsupervised, and can still be trained on this varied dataset,
but the lack of ground truth makes quantitative evaluation on RobotCar Seasons
impossible. To resolve this, we returned to the original RobotCar dataset and
manualy created a validation dataset comprising of 12,000 images with their
corresponding ground-truth LiDAR depth maps, split evenly across day and night
driving scenarios.

Test domain | Method | Abs-Rel | Sq-Rel | RMSE | RMSE-log | A1 | A2 | A3
---|---|---|---|---|---|---|---|---
Day | Monodepth V2 [22] | 0.271 | 3.438 | 9.268 | 0.329 | 0.600 | 0.840 | 0.932
Day | DeFeat | 0.265 | 3.129 | 8.954 | 0.323 | 0.597 | 0.843 | 0.935
Night | Monodepth V2 [22] | 0.367 | 4.512 | 9.270 | 0.412 | 0.561 | 0.790 | 0.888
Night | DeFeat | 0.335 | 4.339 | 9.111 | 0.389 | 0.603 | 0.828 | 0.914
Table 3: Monocular depth evaluation on the RobotCar dataset |  |  |
---|---|---|---
|  |  |
Figure 3: Top: input images from the RobotCar dataset. Middle: estimated depth
maps from Monodepth V2 [22]. Bottom: estimated depth maps from DeFeat-Net.

Figure 4: Feature space visualizations for DeFeat-Net trained on the single-
domain KITTI dataset (centre) and multi-domain RobotCar Seasons dataset
(right).

###  4.1 Single Domain Evaluation

We first evaluate our approach on the KITTI dataset, which covers only a
single domain. For evaluation of depth accuracy, we use the standard KITTI
evaluation metrics, namely the absolute relative depth error (ABS_REL), the
relative square error (SQ_REL) and the root mean square error (RMSE). For
these measures, a lower number is better. We also include the inlier ratio
measures (A1, A2 and A3) of [23] which measure the fraction of relative depth
errors within 25%, $25^{2}$% and $25^{3}$% of the ground truth. For these
measures, a larger fraction is better.

To evaluate the quality of the learned feature representations, we follow the
protocol of [65]. We compute the average distance in the feature space for the
positive pairs from the ground-truth ($\mu_{+}$), and the negative pairs
($\mu_{-}$). Naturally a smaller distance between positive pairs, and a larger
distance between negative pairs, is best. We also compute the Area Under the
Curve (AUC) which can be interpreted as the probability that a randomly chosen
negative sample will have a larger distance than the corresponding positive
ground truth match. Therefore, higher numbers are better. Following [65] all
three errors are split into both local (within 25 pixels) and global
measurements.

The results of the depth evaluation are shown in Table 1 and the feature
evaluation is shown in Table 2. We can see that in this single-domain
scenario, the performance of our technique is competitive with MonodepthV2 and
clearly outperforms most other state-of-the-art techniques for monocular depth
estimation. The results for [22] were obtained by training a network using the
code provided by the authors.

Regarding the features, L2 denotes the L2-normalized versions, whereas G, L &
GL represent the different negative mining variants from [65]. We can also see
that despite being unsupervised, our learned feature space is competitive with
contemporary supervised feature learning techniques and greatly outperforms
pretrained features when evaluating locally. It is interesting, however, to
note that the simple act of L2-normalizing can improve the global performance
of the pretrained features.

Our feature space tends to perform better in the Global evaluation metrics
than the local ones. This is unsurprising as the negative samples for the
contrastive loss in (7) are obtained globally across the entire image.

###  4.2 Multi-Domain Evaluation

However, performance in the more challenging RobotCar Seasons dataset
demonstrates the real strength of jointly learning both depth and feature
representations. RobotCar Seasons covers multiple domains, where traditional
photometric based monocular depth algorithms struggle and where a lack of
cross-domain ground-truth has historically made feature learning a challenge.
For this evaluation, we select the best competing approach from Table 1
(MonodepthV2) and retrain both it and DeFeat-Net on the RobotCar Seasons
dataset. All techniques are trained from scratch.

The results are shown in Table 3 and example depth map comparisons are shown
in Figure 3. We can see that in this more challenging task, the proposed
approach outperforms the previous state of the art technique across all error
measures. While for the daytime scenario, the improvements are modest, on the
nighttime data there is a significant improvement with around 10% reduction in
all error measures.

We believe that the main reason behind this difference is that in well-lit
conditions, the photometric loss is already a good supervision signal. In this
case, incorporating the feature learning adds to the complexity of the task.
However, nighttime scenarios make photometric matching less discriminative,
leading to weaker supervision. Feature learning provides the much needed
invariance and robustness to the loss, leading to the significant increase in
performance.

It is interesting to note that the proposed approach is especially robust with
regards to the number of estimated outliers. The A1, A2 and A3 error measures
are fairly consistent between the day and night scenarios for the proposed
technique. This indicates that even in areas of uncertain depth (due to under-
exposure and over-saturation), the proposed technique fails gracefully rather
than producing catastrophically incorrect estimates.

Since previous state-of-the-art representations cannot be trained
unsupervised, and RobotCar Seasons does not provide any ground-truth depth, it
is not possible to repeat the feature comparison from Table 2 in the multi-
domain scenario. Instead Figure 4 compares qualitative examples of the learned
feature spaces. For these visualizations, we find the linear projection that
best shows the correlation between the feature map and the images and map it
to the RGB color cube. This dimensionality reduction removes a significant
amount of discriminative power from the descriptors, but allows for some form
of visualization.

In all cases, the feature descriptors can clearly distinguish scene structures
such as the road. It is interesting to note that a significant degree of
context has been encoded in the features, and they are capable of easily
distinguishing a patch in the middle of the road, from one on the left or
right, and from a patch of similarly colored pavement. The feature maps
trained on the single domain KITTI dataset can sometimes display more contrast
than those trained on RobotCar Seasons. Although this implies a greater degree
of discrimination between different image regions, this is likely because the
latter representation can cover a much broader range of appearances from other
domains. Regarding, the nighttime features, it is interesting that those
trained on a single domain seem to exhibit strange behaviour around external
light sources such as the lampposts, traffic lights and headlights. This is
likely due to the bias in the training data, with overall brighter image
content.

Dataset | Method | Abs-Rel | Sq-Rel | RMSE | RMSE-log | A1 | A2 | A3
---|---|---|---|---|---|---|---|---
KITTI | DeFeat (no feat) | 0.123 | 0.948 | 5.130 | 0.197 | 0.863 | 0.956 | 0.980
KITTI | DeFeat | 0.126 | 0.925 | 5.035 | 0.200 | 0.862 | 0.954 | 0.980
RobotCar Day | DeFeat (no feat) | 0.274 | 3.885 | 8.953 | 0.335 | 0.640 | 0.853 | 0.934
RobotCar Day | DeFeat | 0.265 | 3.129 | 8.954 | 0.323 | 0.597 | 0.843 | 0.935
RobotCar Night | DeFeat (no feat) | 0.748 | 13.502 | 8.956 | 0.657 | 0.393 | 0.624 | 0.759
RobotCar Night | DeFeat | 0.335 | 4.339 | 9.111 | 0.389 | 0.603 | 0.828 | 0.914
Table 4: Performance with and without concurrent feature learning, on each
dataset

###  4.3 Ablation

Finally, for each dataset we explore the benefits of concurrent feature
learning, by re-training with the FeatNet subsystem disabled. As shown in
Table 4, the removal of the concurrent feature learning from our technique
causes a small and inconsistent change on the KITTI and RobotCar Day data.
However, on the RobotCar Night data, our full approach drastically outperforms
the version which does not learn a specialist matching representation. For
many error measures, the performance doubles in these challenging scenarios,
and the reduction in outliers causes a three-fold reduction in the Sq-Rel
error.

These findings reinforce the observation that the frequently used photometric
warping loss is insufficient for estimating depth in challenging real-world
domains.

##  5 Conclusions & Future Work

This paper proposed DeFeat-Net, a unified framework for learning robust
monocular depth estimation and dense feature representations. Unlike previous
techniques, the system is able to function over a wide range of appearance
domains, and can perform feature representation learning with no explicit
ground truth. This idea of co-training an unsupervised feature representations
has potential applications in many areas of computer vision beyond monocular
depth estimation.

The main limitation of the current approach is that there is no way to enforce
feature consistency across seasons. Although depth estimation and feature
matching work robustly within any given season, it is currently unclear
weather feature matching between different seasons is possible. It would be
interesting in the future to explore cross-domain consistency as an additional
training constraint. However, this will necessitate the collection of new
datasets with cross seasonal alignments.

### Acknowledgements

This work was partially funded by the EPSRC under grant agreements
(EP/R512217/1, EP/S016317/1 and EP/S035761/1). We would also like to thank
NVIDIA Corporation for their Titan Xp GPU grant.

## References

  * [1] Filippo Aleotti, Fabio Tosi, Matteo Poggi, and Stefano Mattoccia.  Generative adversarial networks for unsupervised monocular depth prediction.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 11129 LNCS, pages 337–354, 2019.
  * [2] Relja Arandjelovic.  Three Things Everyone Should Know to Improve Object Retrieval.  In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2911–2918, 2012.
  * [3] Madhu Babu, Swagat Kumar, Anima Majumder, and Kaushik Das.  UnDEMoN 2.0: Improved Depth and Ego Motion Estimation through Deep Image Sampling.  arXiv preprint, nov 2018.
  * [4] Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk.  Learning local feature descriptors with triplets and shallow convolutional neural networks.  In British Machine Vision Conference 2016, BMVC 2016, volume 2016-Septe, pages 119.1–119.11, 2016.
  * [5] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  SURF: Speeded up robust features.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 3951 LNCS, pages 404–417, 2006.
  * [6] Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua.  BRIEF: Binary robust independent elementary features.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 6314 LNCS, pages 778–792, 2010.
  * [7] Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen.  Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks.  IEEE Transactions on Circuits and Systems for Video Technology, 28(11):3174–3182, nov 2016.
  * [8] Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova.  Depth Prediction without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos.  Proceedings of the AAAI Conference on Artificial Intelligence, 33:8001–8008, jul 2019.
  * [9] Jia-Ren Chang and Yong-Sheng Chen.  Pyramid Stereo Matching Network.  CVPR, 2018.
  * [10] Po-Yi Chen, Alexander H. Liu, Yen-Cheng Liu, and Yu-Chiang Frank Wang.  Towards Scene Understanding: Unsupervised Monocular Depth Estimation With Semantic-Aware Representation.  In CVPR, pages 2619–2627, 2019.
  * [11] Christopher B. Choy, Jun Young Gwak, Silvio Savarese, and Manmohan Chandraker.  Universal correspondence network.  In Advances in Neural Information Processing Systems, pages 2414–2422. Neural information processing systems foundation, 2016.
  * [12] Daniel Detone, Tomasz Malisiewicz, and Andrew Rabinovich.  SuperPoint: Self-supervised interest point detection and description.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, volume 2018-June, pages 337–349, dec 2018.
  * [13] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-Net: A Trainable CNN for Joint Detection and Description of Local Features.  CVPR, may 2019.
  * [14] David Eigen and Rob Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In Proceedings of the IEEE International Conference on Computer Vision, volume 2015 Inter, pages 2650–2658, nov 2015.
  * [15] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In Advances in Neural Information Processing Systems, volume 3, pages 2366–2374, 2014.
  * [16] Mohammed E Fathy, Quoc-Huy Tran, M. Zeeshan Zia, Paul Vernaza, and Manmohan Chandraker.  Hierarchical Metric Learning and Matching for 2D and 3D Geometric Correspondences.  ECCV, 2018.
  * [17] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely.  Deep stereo: Learning to predict new views from the world’s imagery.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-Decem, pages 5515–5524, jun 2016.
  * [18] Ravi Garg, B. G. Vijay Kumar, Gustavo Carneiro, and Ian Reid.  Unsupervised CNN for single view depth estimation: Geometry to the rescue.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9912 LNCS, pages 740–756, mar 2016.
  * [19] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite.  Conference on Computer Vision and Pattern Recognition (CVPR), 2012\.
  * [20] Georgios Georgakis, Srikrishna Karanam, Ziyan Wu, Jan Ernst, and Jana Kosecka.  End-to-End Learning of Keypoint Detector and Descriptor for Pose Invariant 3D Matching.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1965–1973, 2018.
  * [21] Ross Girshick.  Fast R-CNN.  In Proceedings of the IEEE International Conference on Computer Vision, volume 2015 Inter, pages 1440–1448, 2015.
  * [22] Clément Godard, Oisin Mac Aodha, and Gabriel Brostow.  Digging Into Self-Supervised Monocular Depth Estimation.  ICCV, jun 2019.
  * [23] Clément Godard, Oisin Mac Aodha, and Gabriel J. Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, volume 2017-Janua, pages 6602–6611, sep 2017\.
  * [24] Simon Hadfield and Richard Bowden.  Scene flow estimation using intelligent cost functions.  In BMVC 2014 - Proceedings of the British Machine Vision Conference 2014, 2014.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-Decem, pages 770–778. IEEE Computer Society, dec 2016.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-Decem, pages 770–778, 2016\.
  * [27] Philipp Heise, Sebastian Klose, Brian Jensen, and Alois Knoll.  PM-Huber: PatchMatch with Huber Regularization for Stereo Matching.  ICCV, 2013.
  * [28] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial Transformer Networks.  NIPS, jun 2015.
  * [29] Joel Janai, Fatma Güney, Anurag Ranjan, Michael Black, and Andreas Geiger.  Unsupervised Learning of Multi-Frame Optical Flow with Occlusions.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 11220 LNCS, pages 713–731, 2018.
  * [30] Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry.  End-to-End Learning of Geometry and Context for Deep Stereo Regression.  In Proceedings of the IEEE International Conference on Computer Vision, volume 2017-Octob, pages 66–75, mar 2017.
  * [31] Tomáš Krajník, Pablo Cristóforis, Keerthy Kusumam, Peer Neubert, and Tom Duckett.  Image features for visual teach-and-repeat navigation in changing environments.  Robotics and Autonomous Systems, 88:127–141, feb 2017.
  * [32] Tomáš Krajník, Pablo Cristóforis, Matías Nitsche, Keerthy Kusumam, and Tom Duckett.  Image features and seasons revisited.  In 2015 European Conference on Mobile Robots, ECMR 2015 - Proceedings, pages 1–7. IEEE, sep 2015.
  * [33] Lubor Ladický, Christian Häne, and Marc Pollefeys.  Learning the Matching Function.  arXiv preprint, feb 2015.
  * [34] Lubor Ladický, Jianbo Shi, and Marc Pollefeys.  Pulling things out of perspective.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 89–96, 2014.
  * [35] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In Proceedings - 2016 4th International Conference on 3D Vision, 3DV 2016, pages 239–248, jun 2016.
  * [36] Stefan Leutenegger, Margarita Chli, and Roland Y. Siegwart.  BRISK: Binary Robust invariant scalable keypoints.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2548–2555, 2011.
  * [37] Ruihao Li, Sen Wang, Zhiqiang Long, and Dongbing Gu.  UnDeepVO: Monocular Visual Odometry Through Unsupervised Deep Learning.  In Proceedings - IEEE International Conference on Robotics and Automation, pages 7286–7291, sep 2018.
  * [38] Fayao Liu, Chunhua Shen, and Guosheng Lin.  Deep convolutional neural fields for depth estimation from a single image.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 07-12-June, pages 5162–5170, nov 2015.
  * [39] Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid.  Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(10):2024–2039, feb 2016.
  * [40] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 07-12-June, pages 3431–3440, 2015.
  * [41] David G Lowe.  Distinctive Image Features from Scale-Invariant Keypoints.  International Journal of Computer Vision, 60(2):91–110, 2004.
  * [42] Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ramkant Nevatia, and Alan Yuille.  Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, jul 2019.
  * [43] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman.  1 Year , 1000km : The Oxford RobotCar Dataset.  IJRR, 3(December 2015), 2015.
  * [44] V. Madhu Babu, Kaushik Das, Anima Majumdar, and Swagat Kumar.  UnDEMoN: Unsupervised Deep Network for Depth and Ego-Motion Estimation.  In IEEE International Conference on Intelligent Robots and Systems, pages 1082–1088, aug 2018.
  * [45] Reza Mahjourian, Martin Wicke, and Anelia Angelova.  Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 5667–5675, feb 2018.
  * [46] Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox.  A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-Decem, pages 4040–4048, 2016.
  * [47] Richard A Newcombe.  KinectFusion : Real-Time Dense Surface Mapping and Tracking.  Technical report, 2013.
  * [48] Richard A Newcombe, Dieter Fox, and Steven M Seitz.  DynamicFusion: Reconstruction and Tracking of Non-rigid Scenes in Real-Time.  Computer Vision and Pattern Recognition (CVPR), pages 343–352.
  * [49] Yuki Ono, Pascal Fua, Eduard Trulls, and Kwang Moo Yi.  LF-Net: Learning local features from images.  In Advances in Neural Information Processing Systems, volume 2018-Decem, pages 6234–6244, 2018.
  * [50] Jiahao Pang, Wenxiu Sun, Jimmy SJ. Ren, Chengxi Yang, and Qiong Yan.  Cascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching.  In Proceedings - 2017 IEEE International Conference on Computer Vision Workshops, ICCVW 2017, volume 2018-Janua, pages 878–886, aug 2018.
  * [51] Viorica Patraucean, Ankur Handa, and Roberto Cipolla.  Spatio-temporal video autoencoder with differentiable memory.  ICLR, nov 2016.
  * [52] Andrea Pilzer, Stéphane Lathuilière, Nicu Sebe, and Elisa Ricci.  Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation.  CVPR, pages 9768–9777, 2019.
  * [53] Andrea Pilzer, Dan Xu, Mihai Puscas, Elisa Ricci, and Nicu Sebe.  Unsupervised adversarial depth estimation using cycled generative networks.  In Proceedings - 2018 International Conference on 3D Vision, 3DV 2018, pages 587–595, jul 2018.
  * [54] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J. Black.  Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation.  CVPR, pages 12240–12249, 2019.
  * [55] Edward Rosten and Tom Drummond.  Machine learning for high-speed corner detection.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 3951 LNCS, pages 430–443, 2006.
  * [56] Ethan Rublee and Gary Bradski.  ORB: an efﬁcient alternative to SIFT or SURF.  ICCV, 2011.
  * [57] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, Fredrik Kahl, and Tomas Pajdla.  Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions.  CVPR, 2018.
  * [58] Ashutosh Saxena, Min Sun, and Andrew Y. Ng.  Make3D: Learning 3D scene structure from a single still image.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5):824–840, may 2009.
  * [59] Tanner Schmidt, Richard Newcombe, and Dieter Fox.  Self-Supervised Visual Descriptor Learning for Dense Correspondence.  IEEE Robotics and Automation Letters, 2(2):420–427, 2017.
  * [60] Johannes L. Schonberger, Marc Pollefeys, Andreas Geiger, and Torsten Sattler.  Semantic Visual Localization.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 6896–6906, 2018.
  * [61] René Schuster, Oliver Wasenmüller, Christian Unger, and Didier Stricker.  SDC - Stacked Dilated Convolution: A Unified Descriptor Network for Dense Matching Tasks.  CVPR, 2019.
  * [62] Aashish Sharma, Robby T. Tan, and Loong-Fah Cheong.  Depth Estimation in Nighttime using Stereo-Consistent Cyclic Translations.  arXiv preprint, sep 2019.
  * [63] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Descriptor learning using convex optimisation.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 7572 LNCS, pages 243–256, 2012.
  * [64] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 2015.
  * [65] Jaime Spencer, Richard Bowden, and Simon Hadfield.  Scale-Adaptive Neural Dense Features : Learning via Hierarchical Context Aggregation.  CVPR, 2019.
  * [66] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox.  DeMoN: Depth and motion network for learning monocular stereo.  In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, volume 2017-Janua, pages 5622–5631, 2017.
  * [67] Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, and Katerina Fragkiadaki.  SfM-Net: Learning of Structure and Motion from Video.  arXiv preprint, apr 2017.
  * [68] Chaoyang Wang, Jose Miguel Buenaposada, Rui Zhu, and Simon Lucey.  Learning Depth from Monocular Videos Using Direct Methods.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 2022–2030, nov 2018.
  * [69] Rui Wang, Stephen M. Pizer, and Jan-Michael Frahm.  Recurrent Neural Network for (Un-)supervised Learning of Monocular VideoVisual Odometry and Depth.  CVPR, pages 5555–5564, 2019.
  * [70] Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu.  UnOS: Unified Unsupervised Optical-flow and Stereo-depth Estimation by Watching Videos.  Technical report, 2019.
  * [71] Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P Simoncelli.  Image quality assessment: From error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [72] Chamara Saroj Weerasekera, Ravi Garg, Yasir Latif, and Ian Reid.  Learning Deeply Supervised Good Features to Match for Dense Monocular Reconstruction.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 11365 LNCS, pages 609–624. Springer Verlag, 2019.
  * [73] Alex Wong, Byung-Woo Hong, and Stefano Soatto.  Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Prediction.  CVPR, pages 5644–5653, 2019.
  * [74] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Deep3D: Fully automatic 2D-to-3D video conversion with deep convolutional neural networks.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9908 LNCS, pages 842–857, apr 2016.
  * [75] Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, and Ram Nevatia.  LEGO: Learning Edge with Geometry all at Once by Watching Videos.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 225–234, mar 2018.
  * [76] Zhenheng Yang, Peng Wang, Wei Xu, Liang Zhao, and Ramakant Nevatia.  Unsupervised learning of geometry from videos with edge-aware depth-normal consistency.  In 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pages 7493–7500, nov 2018.
  * [77] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua.  LIFT: Learned invariant feature transform.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9910 LNCS, pages 467–483, 2016.
  * [78] Zhichao Yin and Jianping Shi.  GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1983–1992. IEEE Computer Society, dec 2018.
  * [79] Jure Žbontar and Yann Lecun.  Stereo matching by training a convolutional neural network to compare image patches.  Journal of Machine Learning Research, 17, oct 2016.
  * [80] Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid.  Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction.  CVPR, 2018.
  * [81] Feihu Zhang, Victor Prisacariu, Ruigang Yang, and Philip H. S. Torr.  GA-Net: Guided Aggregation Net for End-to-end Stereo Matching.  CVPR, apr 2019.
  * [82] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe.  Unsupervised learning of depth and ego-motion from video.  In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, volume 2017-Janua, pages 6612–6621, 2017.
  * [83] Tinghui Zhou, Philipp Krähenbühl, Mathieu Aubry, Qixing Huang, and Alexei A. Efros.  Learning dense correspondence via 3D-guided cycle consistency.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-Decem, pages 117–126, apr 2016.
  * [84] Yuliang Zou, Zelun Luo, and Jia Bin Huang.  DF-Net: Unsupervised Joint Learning of Depth and Flow Using Cross-Task Consistency.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 11209 LNCS, pages 38–55, sep 2018.

  *[VO]: Visual Odometry
  *[SSD]: Sum of Squared Differences
  *[FCN]: Fully Convolutional Network
  *[SPP]: Spatial Pooling Pyramid
  *[SfM]: Structure-from-Motion
  *[NMS]: Non-Maxima Suppression