# Rashba and Dresselhaus effects in doped methylammonium lead halide
perovskite MAPbI3

Megha Arya<EMAIL_ADDRESS>Preeti Bhumla  Sajjan Sheoran  Suraj  S
Bhattacharya<EMAIL_ADDRESS>[

###### Abstract

Inorganic-organic lead halide perovskites, particularly methylammonium lead
halide (MAPbI3) perovskite, is perceived to be a promising material for
optoelectronics and spintronics. However, lead toxicity and instability under
air and moisture restrict its practical uses. Hence, it is essential to reduce
lead extent by substituting appropriate alternatives and making a hybrid
perovskite. We substitute Sn and Ge in cubic MAPbI3 and compare various
properties of hybrid perovskites by employing state-of-the-art first-
principles-based methodologies, viz., density functional theory (DFT) with
semilocal and hybrid functional (HSE06) and generalized gradient approximation
(PBE) combined with spin-orbit coupling (SOC). We mainly study the Rashba-
Dresselhaus (RD) effect which occurs here due to two major mechanisms breaking
inversion symmetry i.e., static and dynamic, and the presence of heavy
elements contributing to significant SOC. We find non-negligible spin-
splitting effects in the conduction band minimum (CBm) and valence band
maximum (VBM) for hybrid perovskites. For a deeper understanding of the
observed spin-splitting, the spin textures are analyzed and Rashba
coefficients are calculated. We find that Dresselhaus effect comes into
picture in substituted hybrids in addition to the usual Rashba effect observed
in pristine compound. Further, a significant value of Rashba coefficient was
observed in substituted hybrid perovskites. We also observe that the strength
of Rashba spin-splitting can be substantially tuned on application of uniaxial
strain ($\pm 5\%$). More interestingly, we notice that some of the hybrids are
mechanically stable and ductile. Hence these hybrid perovskites can prove to
be potent for perovskite-based spintronic applications.

###### keywords:

DFT, Rashba, Dresselhaus, symmetry, spin-orbit coupling, spin texture, hybrid
perovskites

+91-11-2659 1359 Indian Institute of Technology Delhi] Department of Physics,
Indian Institute of Technology Delhi, New Delhi, India

In recent years, inorganic-organic lead halide perovskite compounds,
particularly methylammonium lead halide perovskite (MAPbI3), have attracted a
great attention from the scientific community. They have the potential of
revolutionising the field of optoelectronics and spintronics owing to their
exotic properties like long diffusion length, appropriate band gaps, high
carrier mobility, high absorption coefficient and quite cheap cost of
manufacturing.1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 In comparison to the first
experimental work, where the power conversion efficiency (PCE) of perovskite
solar cells was only 3.8$\%$4, intense efforts in past years have led to an
impressive increase of PCE to over 25$\%$.13, 14 In addition to this, because
of the presence of heavy element Pb, there is a large spin-orbit coupling
(SOC) which has an influence on the band structure of these lead halide
perovskites.15, 16 Fascinatingly, SOC in association with inversion asymmetry
induces various exotic phenomena like topological surface states17, persistent
spin textures18, 19 and Rashba-Dresselhaus (RD) effects20, 21, 22, 23, 24. In
case of crystals where center of inversion is not present, there is a finite
gradient of potential and hence a Lorentz transformed magnetic field acts on
relativistically moving electron. So at those k points which are not invariant
under time reversal, the degenerate bands split into states of opposite spin,
lifting Kramer’s degeneracy. The orientation of spin is decided by spin orbit
field which depends on momentum. Dresselhaus25 studied that for crystals
without a centre of inversion i.e. bulk inversion asymmetry (BIA), SOC lifts
spin degeneracy of the electronic bulk states and the effect was originally
reported in zinc-blende. Rashba26 studied spin-orbit effects in two-
dimensional crystals in an asymmetric potential which is referred to as
structural inversion asymmetry (SIA) and the effect was originally reported in
wurtzite. The differentiator between both of these effects is the origin of
the non-centrosymmetry. RD effect has great applications in spintronics27, 28
and thus lead halide perovskites are seen as promising materials for
applications in the field29, 30, 31, 32, 33.

In the bulk of MAPbI3, the inversion symmetry is broken by two major
mechanisms : static and dynamic. The static one is in which PbI framework
which is octahedron gets distorted due to small and long bond lengths34. The
dynamic one is in which PbI framework is fixed but inversion asymmetry comes
into picture due to rotation of MA ions35. Dynamic structural fluctuations can
even take place because of phonon modes or interaction of MA ions with the PbI
framework36, 37, 38. Also, the system has strong SOC due to presence of heavy
element iodine in addition to lead35. However, Pb has toxic nature and it has
hampered MAPbI3 from its practical uses. Additionally, under air and moisture
exposure, Pb is intrinsically unstable and it ultimately leads to oxidation
from Pb2+ to Pb4+, reducing the performance of devices based on MAPbI339, 40.
Thus, it is important to reduce the Pb extent in the compound by substituting
appropriate alternatives and making a hybrid perovskite. It should be noted
that Pb-vacancies are not stable at all41.

Just like other perovskites, MAPbI3 crystals exhibit multiple phases : cubic
with a space group Pm3-bar m above 327 K42, tetragonal with two debatable
space groups i.e centrosymmetric I4/mcm (point group 4/mmm) and non-
centrosymmetric I4cm (point group 4mm) between 162 and 327 K42, 43, 44, 34,
and orthorhombic with a space group Pnma below 162 K43. However, in case of
solar cell applications, just the tetragonal room temperature phase and the
cubic high-temperature phase are of relevance to us. In our present work, we
studied the cubic phase of MAPbI3 using state-of- the-art density functional
theory (DFT)45, 46 and we intend to reduce Pb extent by substituting Sn and Ge
in the perovskite and parallelly maintain spin splitting effects.

Vienna ab Initio Simulation Package (VASP)47, 48 is used to perform DFT
calculations. Projector augmented wave (PAW) pseudopotential method49, 50, as
implemented in VASP, is used to describe all ion-electron interactions in
elemental constituents. The exchange-correlation functionals used for DFT
calculations are Perdew-Burke-Ernzerhof (PBE)51 functional with SOC and the
non-local Heyd–Scuseria–Ernzerhof (HSE06)52 functional with SOC. We have
increased the size of supercell of the structure modelled till the state of
single defect is completely localized (2×2×2), having 96 atoms (MA8Pb8I24)
with periodic boundary conditions. During optimizing the structures, the total
energy difference between two ionic relaxation steps is kept lesser than
0.0001 eV and the tolerance on forces between two steps is kept 0.01 eV/ Å. A
4×4×4 Monkhorst k-mesh size is made use of for optimization. For doing energy
calculations, the k-mesh is converged and fixed at 4 × 4 × 4 k-mesh size. The
plane wave energy cutoff is kept 520 eV for all calculative purposes.

In a 2×2×2 supercell, there are eight Pb atoms. So calculations are done until
all the eight Pb atoms are replaced one by one by Sn/Ge. Firstly, formation
energies are calculated using PBE+SOC and also using HSE06+SOC. It is known
that HSE06+SOC is more accurate for the calculations as it determines position
of energy bands more precisely yet it is interesting to verify whether PBE
works well in the context of thermodynamic stability. Then band structures,
band gaps, electronic atom-projected partial density of states (pDOS), spin
textures and Rashba coefficients are calculated using PBE+SOC. PyProcar53 is
used to plot band structures and calculate the constant energy contour plots
of spin textures. For all these other calculations, in view of computational
cost, we have used PBE+SOC only because in present work, we are more
interested to know the trends we get on successive substitutions in comparison
to pristine structure.

As stated above, there are two major factors breaking inversion symmetry i.e.,
distortion of PbI framework and rotation of MA ions. We quantified the
contribution of each factor by doing all calculations under (i) complete
relaxation of structures and also under (ii) selective dynamics where PbI
framework was fixed (see Figure 1 a, b). So complete relaxed structures
include effects due to both factors and structures relaxed under selective
dynamics include effects due to MA ion rotation only. Hence effects due to PbI
framework distortion only can be seen as a difference of both calculations.
PbI framework is non-centrosymmetric in the fully relaxed structures and
centrosymmetric in the MA-relaxed structures, which are however overall non-
centrosymmetric due to the random MA orientations.

Also two additional properties i.e. the change in Rashba parameters on
application of axial strain and mechanical properties are calculated using
PBE+SOC for completely relaxed structures.

Figure 1: Optimized crystal structures of MAPbI3 in the cubic phase (a) when
all atoms are allowed to relax and (b) when only MA ions are allowed to relax.
(c) Formation energies for completely relaxed pristine and Sn substituted
structures calculted using PBE+SOC. (d) Formation energies for MA relaxed
pristine and Sn substituted structures calculted using HSE06+SOC. (e)
Formation energies for completely relaxed pristine and Ge substituted
structures calculted using PBE+SOC. (f) Formation energies for MA relaxed
pristine and Ge substituted structures calculted using HSE06+SOC.

We have calculated formation energies of all structures to find out whether
they are thermodynamically stable or not. To calculate formation energy of
MAPb1-XSnXI3 ( in case of 2 × 2 × 2 supercell i.e. MA8Pb8-XSnXI24), we
considered total energies of precursor materials i.e. PbI2, MAI and SnI2 and
subtracted them from the energy of original material.

$\displaystyle
E_{f}(x)=E(MA_{8}Pb_{8-X}Sn_{X}I_{24})-8E(MAI)-xE(SnI_{2})-(8-x)E(PbI_{2})$
(1)

The coefficients of all terms are taken such that the number of MA, Pb, Sn and
I atoms are balanced stoichiometrically. Figure 1 c, d, e, f can be referred
to compare formation energies of pristine and Sn substituted hybrid structures
using different functionals and relaxation mechanisms (for exact values, see
section I in Supporting Information).

Similar to Sn, to calculate formation energy of MAPb1-XGeXI3 ( in case of 2 ×
2 × 2 supercell i.e. MA8Pb8-XGeXI24), we considered total energies of
precursor materials i.e. PbI2, MAI and GeI2 and subtracted them from the
energy of original material.

$\displaystyle
E_{f}(x)=E(MA_{8}Pb_{8-X}Ge_{X}I_{24})-8E(MAI)-xE(GeI_{2})-(8-x)E(PbI_{2})$
(2)

Here also, the coefficients of all terms are taken such that the number of MA,
Pb, Ge and I atoms are balanced stoichiometrically. Figure 1 g, h, i, j can be
referred to compare formation energies of pristine and Ge substituted hybrid
structures using different functionals and relaxation mechanisms (for exact
values, see section I in Supporting Information).

The pristine structure is found to be unstable using PBE+SOC, which can’t be
the case, so it tells that due to incorrect positions of valance band maxima
and conduction band minima, thermodynamic stability will be totally wrong
using PBE+SOC. The negative values of formation energies using HSE+SOC for all
structures indicate that they are thermodynamically stable. Besides this, the
fully relaxed structures are more stable as compared to the corresponding only
MA relaxed structures in each case. The contribution of PbI framework
distortion towards formation energy can be analysed as a difference of
corresponding formation energies of completely relaxed and only MA relaxed
structures in each case.

Then, we have plotted band structures inside 1st Brillouin zone, taking into
consideration the high symmetry path. Firstly, we have performed non-spin-
polarized calculations for the pristine structure . After that, we have
considered SOC in the calculation of electronic band structures for pristine
and substituted structures. All calculations were performed using both
relaxation mechanisms as mentioned before. See section II in Supporting
Information for figures. Splitting of bands can be clearly observed at the
high symmetry point G in all the structures after considering SOC. This
removes the spin degeneracy of both the valence and conduction bands, leading
to the formation of distinct parabolic band minima shifted from the high-
symmetry point G of the brillouin zone. The amount of splitting is different
in directions M-G and G-R. The bandstructures of the fully relaxed structures
exhibit a large splitting of the band edges. By contrast, in the MA-relaxed
structures, the splitting is considerably reduced in all cases. Also
comparison of bandstructures of pristine and doped structures showed that the
bandgap is decreasing in Sn substituted structures (see figure 2 a, b). On Sn
substitution, because of strong s-p and p-p couplings, there is a downward
shift in lower part of conduction band. So this indicates that the optical
absorption coefficient is high and this is a good indicator for solar cell
use. On the other hand, comparison of bandstructures of pristine and doped
structures showed that the bandgap is increasing in Ge substituted
structures(see figure 2 c, d).

Figure 2: (a) Band gaps for completely relaxed pristine and Sn substituted
structures. (b) Band gaps for MA relaxed pristine and Sn substituted
structures. (c) Band gaps for completely relaxed pristine and Ge substituted
structures. (d) Band gaps for MA relaxed pristine and Ge substituted
structures.

Then we have calculated projected density of states (pDOS) using PBE+SOC. Here
also, firstly, we have performed non-spin-polarized calculations for the
pristine structure for both relaxation mechanisms. After that, we have
considered SOC in the calculation of electronic band structures for pristine
and substituted structures for both relaxation mechanisms. See section III in
Supporting Information for figures. For all the structures, the major
contribution to DOS comes from p-orbital of I. Also s-orbital of I contributes
to DOS. The contribution from s and p orbitals of Pb decreases and the
contribution from s and p orbitals of Sn/Ge increases as number of substituted
atoms increase.

The nature of band splitting due to Rashba and Dresselhaus effects is similar.
However, by projection of spin-orientation in Fourier space, called as spin
texture, we can come to know about the type of splitting. We plotted spin
textures by calculation of expectation values of spin operators Si (i = x, y,
z), which is given as,

$\displaystyle\langle
S_{i}\rangle=\langle\Psi_{k}\mid\sigma_{i}\mid\Psi_{k}\rangle$ (3)

where $\sigma_{i}$ are the Pauli matrices, and $\Psi_{k}$ is the spinor
eigenfunction obtained from noncollinear spin calculations.

Figure 3: (a) Spin textures for completely relaxed configurations (a)
MA8Pb8I24 (b) MA8Pb7SnI24 (c) MA8Pb6Sn2I24 (d) MA8Pb5Sn3I24 (e) MA8Pb4Sn4I24
(f) MA8Pb3Sn5I24 (g) MA8Pb2Sn6I24 (h) MA8PbSn7I24 (i) MA8Sn8I24. Figure 4: (a)
Spin textures for MA relaxed configurations (a) MA8Pb8I24 (b) MA8Pb7SnI24 (c)
MA8Pb6Sn2I24 (d) MA8Pb5Sn3I24 (e) MA8Pb4Sn4I24 (f) MA8Pb3Sn5I24 (g)
MA8Pb2Sn6I24 (h) MA8PbSn7I24 (i) MA8Sn8I24.

Figure 3, 4, 5, 6 depict spin textures for completely relaxed and MA relaxed
pristine and Sn/Ge hybrid perovskites. The spin textures for pristine and
doped structures have similar nature for lesser number of Sn/Ge atoms
substituted in the supercell. Here, only the spin component Sz is dominant in
these structures. This indicates that the major splitting is Rashba splitting
for lower substitutions. However for higher substitutions, Dresselhaus effect
also comes into picture as evident from spin textures. The impact of PbI
framework distortion can be clearly seen on the spin textures.

Figure 5: (a) Spin textures for completely relaxed configurations (a)
MA8Pb7GeI24 (b) MA8Pb6Ge2I24 (c) MA8Pb5Ge3I24 (d) MA8Pb4Ge4I24 (e)
MA8Pb3Ge5I24 (f) MA8Pb2Ge6I24 (g) MA8PbGe7I24.

Figure 6: (a) Spin textures for MA relaxed configurations (a) MA8Pb7GeI24 (b)
MA8Pb6Ge2I24 (c) MA8Pb5Ge3I24 (d) MA8Pb4Ge4I24 (e) MA8Pb3Ge5I24 (f)
MA8Pb2Ge6I24 (g) MA8PbGe7I24 (h) MA8Ge8I24.

The values of Rashba coefficient ($\alpha$) are calculated using the formula 2
ER / k0 where ER is Rashba spin-splitting energy and k0 is offset momentum as
shown in schematic representation in figure 5 a. Figure 5 b, c depict the
value of Rashba coefficients for lower conduction bands in completely relaxed
and MA relaxed structure respectively for pristine and Sn substituted
structures. Figure 5 d, e depict the value of Rashba coefficients for lower
conduction bands in completely relaxed and MA relaxed structure respectively
for pristine and Ge substituted structures. There are two contradictory
factors which decide value of Rashba Coefficient : structural inversion
asymmetry and effect of less/more heavy elements present. So value depends on
net effect of both. The Rashba coefficient calculated in doped structures in
both directions ( X to G and G to M ) has a significant value for completely
relaxed structures. Our MA-relaxed structures exhibit a small value of Rashba
Coefficient. Hence PbI framework distortion plays a significant role in spin
splitting effects (for exact values, see section IV in Supporting
Information).

Figure 7: (a) Electron dispersion relation of a normal conduction band
depicting a double spin-degenarate band with one minima. (b) Electron
dispersion relation of band under subjection to Rashba splitting. (c) Rashba
coefficients for lower conduction bands in completely relaxed pristine and Sn
substituted structures. (d) Rashba coefficients for lower conduction bands in
MA relaxed pristine and Sn substituted structures. (e) Rashba coefficients for
lower conduction bands in completely relaxed pristine and Ge substituted
structures. (f) Rashba coefficients for lower conduction bands in MA relaxed
pristine and Ge substituted structures.

Rashba coefficients are also calculated for completely relaxed structures on
which axial strain is applied. There are three axis along which strain can be
applied in cubic phase. The electric field component in unstrained
configuration is different along different axis for each configuration. We
applied strain along ‘a’, ‘b’ and ‘c’ axis for pristine structure and only
along ‘c’ axis for all the substituted ones. The lattice vector a / b / c is
changed from -$5\%$ to +$5\%$ . Here “+” indicates tensile strain and and “-”
indicates compressive strain. Then Rashba coefficients are calculated for each
case. See Section V in Supporting Information for details. We observed
variation in Rashba parameters on application of strain which indicates that
the material is tunable for applications in spintronics.

To study the mechanical properties, the elastic constants of the structures
were calculated. We have three independent elastic constants C11, C12 and C44
for the studied structures. By using the values of these elastic constants, we
calculated properties like bulk moduli (B), shear moduli (G), Pugh’s ratio
(B/G) and Poisson’s ratio ($\nu$) . These can be calculated by using the
following equations54, 55:

$\displaystyle B=B_{V}=B_{R}=(C_{11}+2C_{12})/3$ (4) $\displaystyle
G_{V}=(C_{11}-C_{12}+3C_{44})/5$ (5) $\displaystyle
G_{R}=5(C_{11}-C_{12})C_{44}/4C_{44}+3(C_{11}-C_{12})$ (6) $\displaystyle
G=(G_{V}+G_{R})/2$ (7) $\displaystyle\nu=(3B-2G)/(6B+2G)$ (8)

In case of cubic phase, following Born’s stability criteria56 can be used to
evaluate the mechanical stability :

$\displaystyle C_{11}>0,C_{44}>0,C_{11}>|C_{12}|,(C_{11}+2C_{12})>0$ (9)

In order to distinguish ductile materials from brittle ones57, 58, we used
these two famous parameters (Pugh’s ratio and Poisson’s ratio). Pugh’s ratio
and Poisson’s ratio have 1.75 and 0.26, respectively as the critical values
i.e. if Pugh’s ratio (Poisson’s ratio) is greater than 1.75 (0.26), the
material is ductile and if it is less than 1.75 (0.26), the material is
brittle.

We found that the elastic constants for MA8Pb8I24, MA8Pb6Sn2I24, MA8Pb5Sn3I24,
MA8Pb4Sn4I24, MA8Pb2Sn6I24, MA8Pb6Ge2I24, MA8Pb5Ge3I24 and MA8Pb2Ge6I24
satisfy the Born’s stability criteria, implying that these materials are
mechanically stable. Further among these materials, it was found that only
MA8Pb8I24, MA8Pb4Sn4I24, MA8Pb2Sn6I24, MA8Pb4Ge2I24 and MA8Pb2Ge3I24 are
ductile. The data for mechanically stable materials is as shown in Section VI
of Supporting Information.

In conclusion, we have performed relativistic first-principles density
functional theory calculations to study Rashba and Dresselhaus effects in
cubic phase of MAPbI3 with Pm3-bar m space group symmetry when it is
substituted with tin and germanium. All the structures when substituted with
different number of tin and germanium atoms in our supercell were found to be
thermodynamically stable. Due to significant amount of SOC, spin splitting
effects are observed around the k-point G. In Sn doped structures, bandgap was
found to be lesser when compared to pristine structure which is an indicator
of high absorption coefficient. On the other hand, in Ge doped structures,
bandgap was found to be more when compared to pristine structure The spin
textures indicated the Rashba-type of spin splitting. Also it was observed
that in addition to Rashba effect, Dresselhaus effect comes into picture for
higher number of Sn or Ge atoms substituted in pristine supercell. A
significant value of Rashba coefficient was observed in substituted
structures. Also strain was applied and values of Rashba coefficients were
found which indicated that our devices can be tuned for various applications.
Moreover, some of the doped systems are found to become ductile materials.

MAPbI3 is already a promising material for spintronics, optoelectronics and
spin-orbitronics because of its unique combination of useful properties which
are difficult to be found even in the best semiconductors of single crystals,
inspite of being made from solution processing methods. Now, replacing lead
with tin or germanium in devices based on MAPbI3 can prove to be a game
changer. It reduces toxicity and unstability (under air and moisture)
encountered due to lead and parallely gives us the desired RD effects.

M.A. acknowledges High Performance Computing (HPC) facility at IIT Delhi for
computational resources. P.B. acknowledges UGC, India, for the senior research
fellowship [grant no. 1392/(CSIR-UGC NET JUNE 2018)]. S.S. acknowledges CSIR,
India, for the senior research fellowship [grant no. 09/086(1432)/2019-EMR-I].
Suraj acknowledges HPC facility at IIT Delhi for computational resources. S.B.
acknowledges financial support from SERB under a core research grant (Grant
No. CRG/2019/000647) to set up his HPC facility “Veena” at IIT Delhi for
computational resources.

See supplementary material for the details of formation energies, band
structures, band gaps, pDOS (projected density of states), mechanical
properties, Rashba parameters and their variation on application of axial
strain on perovskites.

## References

  * Snaith 2013 Snaith, H. J. Perovskites: the emergence of a new era for low-cost, high-efficiency solar cells. _The journal of physical chemistry letters_ 2013, _4_ , 3623–3630
  * Green et al. 2014 Green, M. A.; Ho-Baillie, A.; Snaith, H. J. The emergence of perovskite solar cells. _Nature photonics_ 2014, _8_ , 506–514
  * Grätzel 2014 Grätzel, M. The light and shade of perovskite solar cells. _Nature materials_ 2014, _13_ , 838–842
  * Kojima et al. 2009 Kojima, A.; Teshima, K.; Shirai, Y.; Miyasaka, T. Organometal halide perovskites as visible-light sensitizers for photovoltaic cells. _Journal of the american chemical society_ 2009, _131_ , 6050–6051
  * Lee et al. 2012 Lee, M. M.; Teuscher, J.; Miyasaka, T.; Murakami, T. N.; Snaith, H. J. Efficient hybrid solar cells based on meso-superstructured organometal halide perovskites. _Science_ 2012, _338_ , 643–647
  * Xing et al. 2013 Xing, G.; Mathews, N.; Sun, S.; Lim, S. S.; Lam, Y. M.; Grätzel, M.; Mhaisalkar, S.; Sum, T. C. Long-range balanced electron-and hole-transport lengths in organic-inorganic CH3NH3PbI3. _Science_ 2013, _342_ , 344–347
  * Manser et al. 2016 Manser, J. S.; Christians, J. A.; Kamat, P. V. Intriguing optoelectronic properties of metal halide perovskites. _Chemical reviews_ 2016, _116_ , 12956–13008
  * Swarnkar et al. 2016 Swarnkar, A.; Marshall, A. R.; Sanehira, E. M.; Chernomordik, B. D.; Moore, D. T.; Christians, J. A.; Chakrabarti, T.; Luther, J. M. Quantum dot–induced phase stabilization of $\alpha$-CsPbI3 perovskite for high-efficiency photovoltaics. _Science_ 2016, _354_ , 92–95
  * Stranks and Snaith 2015 Stranks, S. D.; Snaith, H. J. Metal-halide perovskites for photovoltaic and light-emitting devices. _Nature nanotechnology_ 2015, _10_ , 391–402
  * Nam et al. 2018 Nam, J. K.; Chun, D. H.; Rhee, R. J. K.; Lee, J. H.; Park, J. H. Methodologies toward Efficient and Stable Cesium Lead Halide Perovskite-Based Solar Cells. _Advanced Science_ 2018, _5_ , 1800509
  * Tombe et al. 2017 Tombe, S.; Adam, G.; Heilbrunner, H.; Apaydin, D. H.; Ulbricht, C.; Sariciftci, N. S.; Arendse, C. J.; Iwuoha, E.; Scharber, M. C. Optical and electronic properties of mixed halide (X= I, Cl, Br) methylammonium lead perovskite solar cells. _Journal of Materials Chemistry C_ 2017, _5_ , 1714–1723
  * Burschka et al. 2013 Burschka, J.; Pellet, N.; Moon, S.-J.; Humphry-Baker, R.; Gao, P.; Nazeeruddin, M. K.; Grätzel, M. Sequential deposition as a route to high-performance perovskite-sensitized solar cells. _Nature_ 2013, _499_ , 316–319
  * 13 National Renewable Energy Laboratory (NREL) Best Research-Cell Efficiency Chart. https://www.nrel.gov/pv/cell- efficiency.html, accessed: February 20, 2022
  * Sahli et al. 2018 Sahli, F.; Werner, J.; Kamino, B. A.; Bräuninger, M.; Monnard, R.; Paviet-Salomon, B.; Barraud, L.; Ding, L.; Diaz Leon, J. J.; Sacchetto, D. et al. Fully textured monolithic perovskite/silicon tandem solar cells with 25.2% power conversion efficiency. _Nature materials_ 2018, _17_ , 820–826
  * Even et al. 2012 Even, J.; Pedesseau, L.; Dupertuis, M.-A.; Jancu, J.-M.; Katan, C. Electronic model for self-assembled hybrid organic/perovskite semiconductors: Reverse band edge electronic states ordering and spin-orbit coupling. _Physical Review B_ 2012, _86_ , 205301
  * Even et al. 2013 Even, J.; Pedesseau, L.; Jancu, J.-M.; Katan, C. Importance of spin–orbit coupling in hybrid organic/inorganic perovskites for photovoltaic applications. _The Journal of Physical Chemistry Letters_ 2013, _4_ , 2999–3005
  * Schindler et al. 2018 Schindler, F.; Wang, Z.; Vergniory, M. G.; Cook, A. M.; Murani, A.; Sengupta, S.; Kasumov, A. Y.; Deblock, R.; Jeon, S.; Drozdov, I. et al. Higher-order topology in bismuth. _Nature physics_ 2018, _14_ , 918–924
  * Tao and Tsymbal 2018 Tao, L.; Tsymbal, E. Y. Persistent spin texture enforced by symmetry. _Nature communications_ 2018, _9_ , 1–7
  * Zhao et al. 2020 Zhao, H. J.; Nakamura, H.; Arras, R.; Paillard, C.; Chen, P.; Gosteau, J.; Li, X.; Yang, Y.; Bellaiche, L. Purely cubic spin splittings with persistent spin textures. _Physical Review Letters_ 2020, _125_ , 216405
  * Sandoval et al. 2012 Sandoval, M. T.; da Silva, A. F.; e Silva, E. d. A.; La Rocca, G. Rashba and Dresselhaus spin-orbit interaction strength in GaAs/GaAlAs heterojunctions. _Physics Procedia_ 2012, _28_ , 95–98
  * Ponet and Artyukhin 2018 Ponet, L.; Artyukhin, S. First-principles theory of giant Rashba-like spin splitting in bulk GeTe. _Physical Review B_ 2018, _98_ , 174102
  * Di Sante et al. 2016 Di Sante, D.; Barone, P.; Stroppa, A.; Garrity, K. F.; Vanderbilt, D.; Picozzi, S. Intertwined Rashba, Dirac, and Weyl fermions in hexagonal hyperferroelectrics. _Physical review letters_ 2016, _117_ , 076401
  * Yamauchi et al. 2015 Yamauchi, K.; Barone, P.; Shishidou, T.; Oguchi, T.; Picozzi, S. Coupling ferroelectricity with spin-valley physics in oxide-based heterostructures. _Physical review letters_ 2015, _115_ , 037602
  * Tao et al. 2017 Tao, L.; Paudel, T. R.; Kovalev, A. A.; Tsymbal, E. Y. Reversible spin texture in ferroelectric Hf O 2. _Physical Review B_ 2017, _95_ , 245141
  * Dresselhaus 1955 Dresselhaus, G. Spin-orbit coupling effects in zinc blende structures. _Physical Review_ 1955, _100_ , 580
  * Rashba 1960 Rashba, E. Properties of semiconductors with an extremum loop. I. Cyclotron and combinational resonance in a magnetic field perpendicular to the plane of the loop. _Sov. Phys.-Solid State_ 1960, _2_ , 1109
  * Di Sante et al. 2013 Di Sante, D.; Barone, P.; Bertacco, R.; Picozzi, S. Electric control of the giant Rashba effect in bulk GeTe. _Advanced materials_ 2013, _25_ , 509–513
  * Plekhanov et al. 2014 Plekhanov, E.; Barone, P.; Di Sante, D.; Picozzi, S. Engineering relativistic effects in ferroelectric SnTe. _Physical Review B_ 2014, _90_ , 161108
  * Kim et al. 2014 Kim, M.; Im, J.; Freeman, A. J.; Ihm, J.; Jin, H. Switchable S= 1/2 and J= 1/2 Rashba bands in ferroelectric halide perovskites. _Proceedings of the National Academy of Sciences_ 2014, _111_ , 6900–6904
  * Kutes et al. 2014 Kutes, Y.; Ye, L.; Zhou, Y.; Pang, S.; Huey, B. D.; Padture, N. P. Direct observation of ferroelectric domains in solution-processed CH3NH3PbI3 perovskite thin films. _The journal of physical chemistry letters_ 2014, _5_ , 3335–3339
  * Giovanni et al. 2015 Giovanni, D.; Ma, H.; Chua, J.; Grätzel, M.; Ramesh, R.; Mhaisalkar, S.; Mathews, N.; Sum, T. C. Highly spin-polarized carrier dynamics and ultralarge photoinduced magnetization in CH3NH3PbI3 perovskite thin films. _Nano letters_ 2015, _15_ , 1553–1558
  * Kepenekian and Even 2017 Kepenekian, M.; Even, J. Rashba and Dresselhaus couplings in halide perovskites: Accomplishments and opportunities for spintronics and spin–orbitronics. _The Journal of Physical Chemistry Letters_ 2017, _8_ , 3362–3370
  * Rinaldi et al. 2018 Rinaldi, C.; Varotto, S.; Asa, M.; Sławińska, J.; Fujii, J.; Vinai, G.; Cecchi, S.; Di Sante, D.; Calarco, R.; Vobornik, I. et al. Ferroelectric control of the spin texture in GeTe. _Nano letters_ 2018, _18_ , 2751–2758
  * Rakita et al. 2017 Rakita, Y.; Bar-Elli, O.; Meirzadeh, E.; Kaslasi, H.; Peleg, Y.; Hodes, G.; Lubomirsky, I.; Oron, D.; Ehre, D.; Cahen, D. Tetragonal CH3NH3PbI3 is ferroelectric. _Proceedings of the National Academy of Sciences_ 2017, _114_ , E5504–E5512
  * Frohna et al. 2018 Frohna, K.; Deshpande, T.; Harter, J.; Peng, W.; Barker, B. A.; Neaton, J. B.; Louie, S. G.; Bakr, O. M.; Hsieh, D.; Bernardi, M. Inversion symmetry and bulk Rashba effect in methylammonium lead iodide perovskite single crystals. _Nature communications_ 2018, _9_ , 1–9
  * Etienne et al. 2016 Etienne, T.; Mosconi, E.; De Angelis, F. Dynamical origin of the Rashba effect in organohalide lead perovskites: A key to suppressed carrier recombination in perovskite solar cells? _The journal of physical chemistry letters_ 2016, _7_ , 1638–1645
  * Motta et al. 2015 Motta, C.; El-Mellouhi, F.; Kais, S.; Tabet, N.; Alharbi, F.; Sanvito, S. Revealing the role of organic cations in hybrid halide perovskite CH3NH3PbI3. _Nature communications_ 2015, _6_ , 1–7
  * Wu et al. 2017 Wu, X.; Tan, L. Z.; Shen, X.; Hu, T.; Miyata, K.; Trinh, M. T.; Li, R.; Coffee, R.; Liu, S.; Egger, D. A. et al. Light-induced picosecond rotational disordering of the inorganic sublattice in hybrid perovskites. _Science advances_ 2017, _3_ , e1602388
  * Abdelmageed et al. 2018 Abdelmageed, G.; Mackeen, C.; Hellier, K.; Jewell, L.; Seymour, L.; Tingwald, M.; Bridges, F.; Zhang, J. Z.; Carter, S. Effect of temperature on light induced degradation in methylammonium lead iodide perovskite thin films and solar cells. _Solar Energy Materials and Solar Cells_ 2018, _174_ , 566–571
  * Sun et al. 2016 Sun, P.-P.; Li, Q.-S.; Feng, S.; Li, Z.-S. Mixed Ge/Pb perovskite light absorbers with an ascendant efficiency explored from theoretical view. _Physical Chemistry Chemical Physics_ 2016, _18_ , 14408–14418
  * Basera et al. 2020 Basera, P.; Kumar, M.; Saini, S.; Bhattacharya, S. Reducing lead toxicity in the methylammonium lead halide MAPbI 3: Why Sn substitution should be preferred to Pb vacancy for optimum solar cell efficiency. _Physical Review B_ 2020, _101_ , 054108
  * Kawamura et al. 2002 Kawamura, Y.; Mashiyama, H.; Hasebe, K. Structural study on cubic-tetragonal transition of CH 3 NH 3 PbI 3. _Journal of the Physical Society of Japan_ 2002, _71_ , 1694–1697
  * Weller et al. 2015 Weller, M. T.; Weber, O. J.; Henry, P. F.; Di Pumpo, A. M.; Hansen, T. C. Complete structure and cation orientation in the perovskite photovoltaic methylammonium lead iodide between 100 and 352 K. _Chemical communications_ 2015, _51_ , 4180–4183
  * Stoumpos et al. 2013 Stoumpos, C. C.; Malliakas, C. D.; Kanatzidis, M. G. Semiconducting tin and lead iodide perovskites with organic cations: phase transitions, high mobilities, and near-infrared photoluminescent properties. _Inorganic chemistry_ 2013, _52_ , 9019–9038
  * Hohenberg and Kohn 1964 Hohenberg, P.; Kohn, W. Inhomogeneous electron gas. _Physical review_ 1964, _136_ , B864
  * Kohn and Sham 1965 Kohn, W.; Sham, L. J. Self-consistent equations including exchange and correlation effects. _Physical review_ 1965, _140_ , A1133
  * Kresse and Hafner 1993 Kresse, G.; Hafner, J. Ab initio molecular dynamics for liquid metals. _Phys. Rev. B_ 1993, _47_ , 558–561
  * Kresse and Furthmüller 1996 Kresse, G.; Furthmüller, J. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. _Phys. Rev. B_ 1996, _54_ , 11169–11186
  * Blöchl 1994 Blöchl, P. E. Projector augmented-wave method. _Physical review B_ 1994, _50_ , 17953
  * Kresse and Joubert 1999 Kresse, G.; Joubert, D. From ultrasoft pseudopotentials to the projector augmented-wave method. _Physical review b_ 1999, _59_ , 1758
  * Perdew et al. 1996 Perdew, J. P.; Burke, K.; Ernzerhof, M. Generalized gradient approximation made simple. _Physical review letters_ 1996, _77_ , 3865
  * Heyd et al. 2003 Heyd, J.; Scuseria, G. E.; Ernzerhof, M. Hybrid functionals based on a screened Coulomb potential. _The Journal of chemical physics_ 2003, _118_ , 8207–8215
  * Herath et al. 2020 Herath, U.; Tavadze, P.; He, X.; Bousquet, E.; Singh, S.; Muñoz, F.; Romero, A. H. PyProcar: A Python library for electronic structure pre/post-processing. _Computer Physics Communications_ 2020, _251_ , 107080
  * Hill 1952 Hill, R. The elastic behaviour of a crystalline aggregate. _Proceedings of the Physical Society. Section A_ 1952, _65_ , 349
  * Roknuzzaman et al. 2017 Roknuzzaman, M.; Ostrikov, K. K.; Wang, H.; Du, A.; Tesfamichael, T. Towards lead-free perovskite photovoltaics and optoelectronics by ab-initio simulations. _Scientific reports_ 2017, _7_ , 14025
  * Born 1940 Born, M. On the stability of crystal lattices. I. Mathematical Proceedings of the Cambridge Philosophical Society. 1940; pp 160–172
  * Pugh 1954 Pugh, S. XCII. Relations between the elastic moduli and the plastic properties of polycrystalline pure metals. _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ 1954, _45_ , 823–843
  * Hadi et al. 2017 Hadi, M.; Roknuzzaman, M.; Chroneos, A.; Naqib, S.; Islam, A.; Vovk, R.; Ostrikov, K. Elastic and thermodynamic properties of new (Zr3- xTix) AlC2 MAX-phase solid solutions. _Computational Materials Science_ 2017, _137_ , 318–326


# DIAL: Deep Interactive and Active Learning for Semantic Segmentation in
Remote Sensing

Gaston Lenczner, Adrien Chan-Hon-Tong, Bertrand Le Saux, Nicola Luminari, and
Guy Le Besnerais Gaston Lenczner, Adrien Chan-Hon-Tong, and Guy Le Besnerais
are with ONERA/DTIS, Université Paris-Saclay, FR-91123 Palaiseau, FranceGaston
Lenczner and Nicola Luminari are with Alteia, FR-31400 Toulouse, France
(email: gaston.lenczner@alteia.com)Bertrand Le Saux is with ESA / ESRIN
$\Phi$-lab, IT-00044 Frascati (Roma), Italy (email<EMAIL_ADDRESS>
###### Abstract

We propose in this article to build up a collaboration between a deep neural
network and a human in the loop to swiftly obtain accurate segmentation maps
of remote sensing images. In a nutshell, the agent iteratively interacts with
the network to correct its initially flawed predictions. Concretely, these
interactions are annotations representing the semantic labels. Our
methodological contribution is twofold. First, we propose two interactive
learning schemes to integrate user inputs into deep neural networks. The first
one concatenates the annotations with the other network’s inputs. The second
one uses the annotations as a sparse ground-truth to retrain the network.
Second, we propose an active learning strategy to guide the user towards the
most relevant areas to annotate. To this purpose, we compare different state-
of-the-art acquisition functions to evaluate the neural network uncertainty
such as ConfidNet, entropy or ODIN. Through experiments on three remote
sensing datasets, we show the effectiveness of the proposed methods. Notably,
we show that active learning based on uncertainty estimation enables to
quickly lead the user towards mistakes and that it is thus relevant to guide
the user interventions.

###### Index Terms:

Deep Learning, Interactive segmentation, Active learning, Semantic
Segmentation, Earth Observation

##  I Introduction

###  I-A Context

Semantic segmentation, the task of classifying an image at the pixel level, is
extremely important in remote sensing and is addressed with deep neural
networks for a variety of applications such as land-cover mapping [1], change
detection [2] or farmland monitoring [3]. This task is intrinsically complex
and, while deep neural networks can be very effective, they are still prone to
failure. Indeed, even on academic benchmarks [4, 5], current state-of-the-art
methods often require specific architectures and fine-tuning to obtain high
performances but still imperfect results. Moreover, it often gets even more
tedious on ”real-life” datasets due to different factors such as domain
adaptation between train and test data inherent to remote sensing data
(different weather, geographical areas, types of sensors, cloud shadows, etc)
or the difficulty to have access to large training annotated datasets for
every specific business application, even though lots of efforts are made by
the community in this direction [6, 7]. Hence, the uncertainty about the
quality of the results of neural networks often makes their deployment
complicated. Human intervention may then be necessary. Precisely, we are
thinking of two scenarios representative of real situations. First, the
Refinement use-case, when the user aims to fix errors within a single dataset
and thus to improve the performances of the model on the test data. Second,
the Domain Adaptation use-case, when the user wants to fix on a new dataset
the errors of a model pre-trained on a previous dataset.

A possible way to address these problems comes with Interactive Learning (IL)
[8, 9]. This consists in adding a human in the loop to work in synergy with a
learning algorithm to train it, fine-tune it or adapt it to user inputs.
Compared to classically supervised algorithms, IL algorithms must also
interface smoothly with the human user. This constraint is particularly
challenging with deep neural networks due to their typical high number of
parameters and long training time. One step further than IL, Active Learning
[10] (AL) searches in pools of unlabeled data for examples which are the more
able to lead the model to a better classification. These examples, defined as
queries, are then labeled by the user and incorporated in the training. This
thus aims to find the optimal training dataset for the algorithm. To this
purpose, active learning methods define acquisition functions to estimate
either the model uncertainty associated to the samples [11] or their
representativeness of the dataset [12].

Figure 1: Visual representation of DIAL encompassing Deep Image Segmentation
with Continual Adaptation (DISCA) and Active Learning. The framework starts
with an initial prediction that the annotator can annotate with new points
(e.g. to fix errors). Three algorithmic mechanisms cooperate to improve the
segmentation map: Annotations as Channels (AC) processes jointly image and
annotations without retraining, DISCA additionally retrains the model for
better adaptation, and DIAL also proposes most informative patches to speed up
the interactions. Best viewed in color.

In this article, we explore IL and AL for semantic segmentation. Indeed, as
presented in Figure 1, we propose the Deep Interactive and Active Learning
(DIAL) framework to interactively refine semantic segmentation maps initially
output by a pre-trained neural network. First, it relies on two complementary
IL schemes to integrate information provided by a user in deep learning
algorithms for semantic segmentation. In a nutshell, the first module uses
these annotations to modify at test time the inputs of the network pre-trained
to process them while the second one uses them for retraining to modify the
weights of the network. Second, we integrate active learning within our
framework and propose to guide the user interventions towards relevant areas
to annotate. This additional guidance relies on different uncertainty measures
that we compare with respect to our framework. These measures can be simple-
yet-effective such as entropy [13] or come from the current state-of-the-art
literature such as ODIN [14] or ConfidNet [15]. We extensively evaluate our
framework both in the Refinement and in the Domain Adaptation use-cases to
well apprehend its potential. We notably show that the first IL module is more
suited to correct spatially small mistakes while the second one is more suited
for larger ones and that the active process improves the performance compared
to the unguided one.

In summary, our major contributions are as follows:

  1. 1.

We propose a general framework for interactive multi-class semantic
segmentation in remote sensing.

  2. 2.

We show that active learning for area selection allows to speed-up the
improvement of the segmentation and reduces the number of required
interactions for a given quality.

  3. 3.

We compare different state-of-the-art methods to estimate the algorithm
uncertainty within the DIAL framework sketched in Figure 1, and show that the
model confidence evaluated by ConfidNet and surprisingly the entropy are the
most effective, the latter being also the faster one.

  4. 4.

We show these techniques consistently improve the quality of a segmentation
map, with the greater gain for the domain adaption use-case where it allows to
compensate for the lack of training data in the target domain.

This article extends previous work presented in an international conference
[16]. It deepens the interactive learning experiments and combines the
interactive and active modules within a single framework.

###  I-B Scenario

We assume the following context for the rest of the paper. A user needs to
quickly and accurately semantically segment Earth observation images for one
of the two aforementioned use-cases: the Refinement one and the Domain
Adaptation one. This user has also access to another annotated database which,
depending on the use-case, may or may not belong to the same domain as the
targeted images. For the sake of simplicity, the annotated label space must be
the same as the targeted one.

We propose to first train a neural network on the annotated database. Then,
the user can use this neural network to make predictions on the target images.
If the segmentation result is not accurate enough for the user’s requirements,
they can interact with the network to refine its predictions. These user
interactions come in the form of clicked points on the mislabeled areas and
represent their corresponding labels, as chosen by the user. Finally, we
propose to also guide the user to the most relevant areas of the images using
uncertainty estimations relying on different statistical measures.

We have developed a QGIS111http://qgis.osgeo.org plugin available with the
code to allow potential users to experience the proposed framework. However,
to conduct a large scale evaluation, we have also simulated the user behavior
to automatically generate interactions. Hence, in the rest of this article, we
refer both to the synthetic operator and to the potential human user as the
agent.

When the agent is simulated to automatically generate the annotations, it
samples them in the mistake areas using a comparison between the ground-truth
map and the prediction map. It thus necessarily requires a partial access to
the ground-truth maps.

To summarize, our framework combines the three following criteria:

  * •

Semantic segmentation: The neural network is able to provide accurate semantic
segmentation maps without guidance.

  * •

Interactive Learning: The neural network can also refine these segmentation
maps using the annotations to efficiently fix its mistakes.

  * •

Active Learning:  It estimates the neural network uncertainty to guide the
user towards queries.

###  I-C Related Work

####  I-C1 Interactivity in remote sensing

Interactive interpretation of remote sensing data has a long history,
partially due to the lack of reference data for training in that field.
Interactivity has been processed by various techniques to enhance data mining
tools with relevance feedback capability : Bayesian modeling of sample
distributions was at the core of VisiMine [17], and Support Vector Machines
(SVMs) were used in [18]. More recently, boosting and random forests [19] have
been the method of choice due to the possibility to train quickly in an
incremental manner [9, 20, 21]. Precisely, ALCD [21] trains a random forest on
user annotations. With respect to these works, our approach applies deep
learning for interactive remote sensing, which is challenging due to the long
training time inherent to deep neural networks. Finally, active learning has
recently been applied to deep learning in remote sensing [22, 23] to
interactively update the models. [23] addresses detection of extremely rare
objects (e.g. animal detection in aerial images) while [22] deals with rare
and varied change detection. On our side, we apply active learning to deep
learning in the context of segmentation maps refinement.

####  I-C2 Interactive Segmentation

Interactive Segmentation intends to interactively segment an image into
foreground and background pixels with user annotations. It was initially
addressed using graph-cut based methods [24] and now mostly by deep neural
networks which take as inputs a concatenation of the RGB image and user
annotations [25]. In [26], the authors use the annotations as sparse ground
truth maps to interactively adapt the neural network to a specific object.
Multi-class interactive segmentation broadens interactive segmentation to
correct multi-class segmentation maps. [27] proposed a neural network which
takes as input a concatenation of the image and the extreme points of each
instance in the scene and then lets a user correct the proposed multi-class
segmentation using scribbles. We do not assume such extreme point map
availability as it is extremely costly to acquire in a remote sensing image
with potentially many objects.

####  I-C3 Weakly supervised segmentation

When labels are scarce, training usually boils down to learning the most out
of the available labels while leveraging unlabeled data to learn a better
inner representation as support. To address weakly supervised semantic
segmentation in remote sensing, [28] uses image-level labels while [29]
focuses on domain adaptation using bounding boxes in the target space. Close
to semantic segmentation, [30] addresses change detection in a weakly
supervised setting. Semantic segmentation with point supervision was first
proposed by What’s the Point [31] (WTP) which trains a model from scratch
using cross entropy loss on the point labels. Recently and closely related to
our work, [32] proposed FESTA for weak semantic segmentation in remote
sensing. It mainly consists in a regularization to train a neural network
from-scratch using notably point labels. In our paper, we start from a neural
network pre-trained with full supervision instead of starting from scratch. In
our retraining component, we also design a regularization suited to our use-
cases.

####  I-C4 Active Learning

Active Learning aims at optimizing the training process of a learning
algorithm through an iterative collaboration with a human oracle [10]. It
makes the algorithm choose from a pool of unlabeled data which ones would be
the most relevant to improve itself. Then, the oracle provides the asked
labels and the algorithm can learn from it. As it defines how to select the
data samples to annotate, the acquisition function is the key differentiating
component of these methods. These acquisition functions usually rely on an
uncertainty, a representativeness or a diversity score computed directly with
the model to select the most relevant samples. Uncertainty methods can rely on
simple criteria like entropy [13] or disagreement between ensemble models [33]
to estimate the model’s prediction confidence. As uncertainty-based methods do
not aim to be representative of the dataset, they can select very similar
examples. To address this issue, representativeness-based methods aim to
select the samples in order to form a subset as representative as possible of
the entire dataset. Addressing this as a core-set approach, [12] solves it
like the K-center problem using the L2 distance between the activations of the
final fully-connected layer of the CNN. Finally, often relying on clustering
[34, 35], diversity-based active learning aims to select samples that are as
diverse (i.e. different) as possible to reduce the redundancy among the
selected samples. In the past decade, active learning has been deeply explored
in remote sensing to train algorithms for animal detection [36, 23], image
classification [37, 38], image segmentation [39] and recently for change
detection [22]. We borrow from active learning techniques to smoothly help the
agent to guide our interactive neural network. We focus on uncertainty
measures which fit our use-case better than representativeness ones since we
aim to easily spot wrong predictions and not to increase our training set.

####  I-C5 Uncertainty in deep neural networks

Uncertainty quantification, or confidence estimation, is a long-standing
problem in machine learning and has many applications such as out-of-
distribution (OoD) samples detection [14], the decision to trust the model or
to defer to a human expertise in fields like healthcare or the detection of
new classes in class-incremental learning [40]. Notably, it can also be used
in active learning to determine which samples should be sent to the oracle for
annotation. Many methods to estimate the uncertainty in deep neural networks
have been recently proposed, and they often fall into one of these four
categories.

##### Softmax probabilities

The first category of methods uses the probabilities from the softmax output
space of the neural networks. Indeed, [41] propose a simple yet strong
baseline using the maximum class probability as an uncertainty estimation and
apply to outliers detection. However, it is now well-established that softmax
probabilities are prone to different issues such as poor calibration [42] and
not fit to differentiate in- from out-of-distribution samples [41]. To
overcome these issues, [14] propose ODIN to detect outliers with a tempered
softmax and with adversarial inputs to better distinguish inliers from
outliers. Similarly, [43] perturbs their inputs but instead uses the
representation space before the softmax layer and the Mahanabolis distance to
do the split.

##### Model ensembling

Due to its intuitive concept and ease of implementation, another popular class
of methods estimate the confidence associated to a sample by measuring the
disagreement of different models. This model ensembling can either be explicit
and use different models [44] or implicit to be less computationally greedy
with one stochastic model using dropout [45] (MC Dropout) or batch
normalization [22]. However, all these methods inherently require several
forward propagation and are thus relatively slow, making them not engaging for
interactive interpretation.

##### Auxiliary models

Other recent approaches design an auxiliary model to learn the uncertainty of
the downstream model. While [46] mostly focuses on OoD detection, [15]
addresses failure prediction and proposes ConfidNet, a neural network to
predict if the prediction from the downstream network is accurate or not.
These methods do not require to retrain the downstream network and can thus be
easily plugged into any pre-trained architecture. However, they are
computationally heavy and require a new training phase for each new task and
model. In remote sensing, [47] successfully applies the ConfidNet method for
land cover segmentation.

##### Customized loss

Finally, some works design a specific loss to learn the uncertainty directly
during training. For instance, [48] trains a model to predict the loss
associated to a prediction and [49] proposes a loss which regularizes the
class probabilities to better estimate uncertainty. These methods are
computationally efficient and model agnostic but require a full training from
scratch and can’t be plugged in a pre-trained model.

We compare different methods from these categories in the present work to
optimally guide the agent toward relevant areas to annotate. We focus on the
three first categories since an adapted loss is less tailored for our use-case
as it would require training new models from scratch.

##  II DIAL: Deep Interactive and Active Learning

DIAL encompasses different interactive learning modules and an active learning
module to interactively guide deep neural networks to refine segmentation maps
and we now delve in the details of these components, which are also
illustrated in Figure 1.

###  II-A Interactive learning components

The three following interactive learning mechanisms form the Deep Image
Segmentation with Continual Adaptation (DISCA) module:

##### Annotations as Channels (AC)

The neural network takes as input a concatenation of the RGB image and agent
annotations, extending ideas of DIOS [25] to multi-class segmentation. At test
time, these annotations are initially provided by the agent in the form of
clicked points, then encoded using distance transforms into a $N$-dimensional
tensor where $N$ is the cardinal of the label space. During the initial
supervised training phase, since the neural network needs to learn how to use
clicked points as guidance to enhance its initial predictions, we simply
provide points randomly sampled from the ground truth to the network. Image-
only inputs are also sampled to train segmentation in a standard way and
ensure that the network proposes accurate initial segmentation maps.

##### Retraining on annotations

Since AC only modifies the network’s inputs and not its parameters, the
information provided by the annotations does not improve the predictions
globally in the image. Inspired by WTP [31], we propose to bypass this
locality constraint by retraining the network with a few back-propagation
cycles per annotation. Hence, we use the annotations as a sparse ground-truth
to interactively retrain the network using a cross entropy loss on these
annotated pixels. We note f to represent the neural network parameterized by
$\mathbf{\theta}$ and x its inputs.

##### Regularization

As only a few pixels are annotated among the millions that usually compose a
remote sensing image, the ground-truth maps resulting from the interactions
are extremely sparse. In order to deal with this problem and avoid over-
fitting, we follow ideas from [26, 30] by using the initial prediction
$\mathbf{p_{0}}=f(\mathbf{x},\theta_{0})$ for regularization. Precisely we add
a $L1$-loss term using the original prediction as reference in order to
prevent the model from making a prediction too different from the initial one.
Therefore, our loss during the interactive learning process is defined as
follows:

$\displaystyle\mathcal{L}(\mathbf{x},\mathbf{c},\mathbf{p_{0}};\mathbf{\theta})$
$\displaystyle=\frac{\mathbf{1}_{[\mathbf{c\neq-1}]}}{\|\mathbf{1}_{[\mathbf{c\neq-1}]}\|_{1}}\left\\{-\sum\limits_{i=1}^{N}\mathbf{c}_{i}\log\left(\mathbf{f}_{i}(\mathbf{x};\mathbf{\theta})\right)\right\\}$
(1)
$\displaystyle+\lambda\|\mathbf{f}(\mathbf{x};\mathbf{\theta})-\mathbf{p_{0}}\|_{1}$

where 1 represents the indicator function and c the sparse annotated pixels.
In details, c takes its values in {-1, 0, 1}. For the pixels annotated as
belonging to class $i$, $\mathbf{c}_{i}=1$ and $\mathbf{c}_{j}=0$ for all
$j\neq i$. For the unannotated pixels, $\mathbf{c}_{i}=-1$ for all $i$ in
$\\{1,\dots,N\\}$. $\|\mathbf{1}_{[\mathbf{c\neq-1}]}\|_{1}$ weights the loss
with respect to the number of annotated pixels. Finally, the positive
parameter $\lambda$ balances the influence of user annotations with respect to
the recall towards the initial prediction. Its tuning will be considered in
Section III-C.

The two last mechanisms enable the continual learning potential of DISCA and
avoid catastrophic forgetting. During the interactive training phase, the AC
mechanism is randomly disabled: the annotations are then removed from the
inputs. This avoids over-fitting on the annotation channels.

###  II-B Active learning component

Since remote sensing images can be extremely large, DIAL also incorporates an
active learning strategy to swiftly guide the agent towards queries
representing the most meaningful areas of the image to annotate. With this
aim, we compare different state-of-the-art acquisition functions which
estimate the algorithm uncertainty to find the most suited to our usage
scenario and interaction set-ups described below.

####  II-B1 Formalization

To formalize the problem, we note f to represent the neural network
parameterized by $\mathbf{\theta}$, x an input image, y its associated label
map, a the user annotations and g the annotation encoding function. Our goal
is then to find the optimal annotations $\mathbf{a^{\star}}$ minimizing the
following problem:

$\displaystyle\mathbf{a^{\star}}=\operatorname*{\arg\\!\min}_{\mathbf{a}}\sum\limits_{j\in
I}\left(1-\delta^{u^{j}}_{\mathbf{y}^{j}}\right)$ (2) $\displaystyle\text{with
}u^{j}=\operatorname*{\arg\\!\max}_{c\in\llbracket
0,N\rrbracket}\mathbf{f^{\textit{j}}_{\theta\textit{,c}}}\left(\mathbf{x}\oplus\mathbf{g(a,x,f_{\theta})}\right)$

where $\oplus$ represents the concatenation operation, $\delta$ the Kronecker
operator, $N$ the cardinal of the label space and $I$ the pixels set. The
problem values range from 0 when all pixels are well classified to $card(I)$
when all pixels are misclassified.

####  II-B2 Methodology

We propose the following query strategy to benefit from DIAL on a given image.
The image is divided into a grid of N patches. The patches are annotated
consecutively but the order in which they are annotated depends on the
uncertainty measure. We have also studied a pixel-based query strategy in
Appendix B.

####  II-B3 Acquisition functions

We now present the different acquisition functions that we compare to guide
the agent.

##### Entropy

We compute the entropy per pixel at the softmax output:
$\mathcal{U}=-\sum_{c}y_{c}\times\log(f_{c}(x;\theta))$. As showed by [41],
even though the softmax probabilities of a neural network are poorly
calibrated, they can still provide a strong baseline to guide the user.

##### MC Dropout

MC Dropout [45] introduces stochasticity in the prediction by enabling dropout
regularization at inference time. This allows to obtain an implicit model
ensembling. In practice, we add dropout layers in the neural network
architecture and then make multiple forward passes through the network to
create as many softmax vectors. We then compute the variance of these
predictions to measure their disagreement and use it as the uncertainty
measure.

##### ConfidNet

As proposed by [15], we train a small auxiliary network to learn to estimate
the confidence value of the downstream network using its last layers as
inputs. It is constituted of one transposed convolutional layer and four
$3\times 3$ convolutional layers of respectively 32, 120, 64, 32 and 1 output
layers. A final sigmoid layer provides the confidence score.

##### ODIN

Following [14] which primarily developed this method for outlier detection, we
slightly disturb the image input with an adversarial-like attack aiming to
enforce the predicted probabilities of the softmax output towards the
predicted classes and add a temperature term in the softmax layer. Then, the
adversarial examples are feed-forwarded into the network and we use the
softmax output maximum class probability as a confidence measure. Formally, we
disturb the input with the following perturbation
$\mathbf{x}=\mathbf{x}+\varepsilon\Delta_{\mathbf{x}}\mathcal{L}(f_{\theta}(x),\hat{y})$
where $\mathcal{L}$ represents the cross-entropy loss, $f_{\theta}(x)$ the
predicted probabilities from the softmax output and $\hat{y}$ the predicted
class.

####  II-B4 Computational cost

Therefore, these approaches have different inference costs inherent to their
underlying structure. Indeed, entropy is virtually cost-free since it computes
a simple operation directly on the neural network output. On the contrary, MC
Dropout is particularly expensive since it requires computing multiple
predictions. Despite the extra prediction, ConfidNet is only slightly more
expensive than entropy thanks to the small size of the auxiliary network.
Finally, ODIN falls between ConfidNet and MC Dropout due to the creation and
inference of the adversarial sample.

##  III Experiments

###  III-A Experimental setup

Ground-truth

Initial output with annotations

AC output

DISCA output

Figure 2: Visual comparison of the two approaches on examples from AIRS (rows
1, 3 & 4) & ISPRS (row 2). Row 4 is a zoomed version of row 3. In rows 1, 3 &
4, building labels and predictions are in cyan. In row 2, impervious surface
labels are transparent and the associated annotations are in red, buildings
are in yellow, low vegetation in green, high vegetation in cyan and clutter in
magenta.

####  III-A1 The datasets

We experiment on three semantic segmentation remote sensing datasets: the
INRIA Aerial Image Labelling dataset [4] composed of two classes (buildings
and not buildings) and covering more than 800 $\textrm{km}^{2}$ in different
cities at a 30 cm resolution, the Aerial Imagery for Roof Segmentation (AIRS)
dataset [50] composed of the same two classes and covering 457
$\textrm{km}^{2}$ in New-Zealand at a 7.5 cm resolution and the ISPRS Potsdam
dataset [51] composed of 6 classes (impervious surface, buildings, low
vegetation, tree, car and clutter) covering 3 $\textrm{km}^{2}$ on Potsdam at
a 5 cm resolution. The initial training sets are divided into a smaller
training set and a validation set with a ratio 80%-20%. This allows to
synthesize the annotations required to automatically evaluate the framework.
The images are tiled into patches of size $512\times 512$ with an overlap of
size 128 to be processed.

####  III-A2 The hyper-parameters

We use a neural network based on the LinkNet [52] architecture but our
approach is agnostic from the neural network backbone.

Except in the annotation encoding study, the annotations are encoded into the
neural network channels inputs using distance transform.

For DISCA, during the interactive learning phase, we optimize the weights
using 10 stochastic gradient descent passes with a learning rate of $2e^{-6}$
and minimize the loss defined in Eq. 1 with the regularization parameter
$\lambda$ set to 1.

####  III-A3 Active learning set-up

For ODIN, we set the perturbation parameter $\varepsilon$ to $\frac{1}{255}$
and the temperature term to 100.

For MC Dropout, we add a dropout layer between each encoder and decoder block
of our architecture, set the dropout rate to 0.1 and compute the variance over
5 different inferences.

The ConfidNet auxiliary network is trained for 10 epochs per dataset with Adam
optimizer.

To automatically evaluate the active learning component, we split the test
images into $512\times 512$ patches, sample one annotation per patch and then
make a new prediction on this patch using AC-only and DISCA. With DISCA, we
retrain the network sequentially with each patch. We study whether the
annotation order can be optimized. The annotations are generated inside the
spatially-largest mistakes of the patches. We compute the uncertainty globally
in the images and then compute an uncertainty score per patch by averaging the
uncertainty across all the pixels of the patch. We compare the uncertainty-
ordered sequences to a randomly-drawn one that constitutes the baseline.

TABLE I: Mean IoU after 50 annotated patches with random and active learning (entropy) orders. For 50 patches on Figures 3 & 4, one recovers results from this Table. |  Initial |  Rand. patches | AL patches
---|---|---|---
| LinkNet | AC |  DISCA | AC | DISCA
ISPRS | 70.7  | 71.8 |  71.3  | 73.1 | 73.3
INRIA | 85.4  | 86.3 |  86.2  | 86.6 | 86.4
AIRS | 88  | 88.7 |  89.4  | 91.1 | 92
TABLE II: Mean inference time on a $512\times 512$ patch | Initial | AC | DISCA
---|---|---|---
time (s) | 0.01 | 0.01 | 0.11

(a) ISPRS

(b) INRIA

(c) AIRS

(d) AIRS $\to$ ISPRS

Figure 3: IoU evolution with respect to the number of annotated patches with
AC (one annot. per patch). This compares the different uncertainty measures to
select the patch-to-annotate.

(a) ISPRS

(b) INRIA

(c) AIRS

(d) AIRS $\to$ ISPRS

Figure 4: IoU evolution with respect to the number of annotated patches with DISCA (one annot. per patch). This compares the different uncertainty measures to select the patch-to-annotate. TABLE III: Mean time for prediction with uncertainty computation on $6000\times 6000$ images | Random | Entropy | MC Dropout | ODIN | ConfidNet
---|---|---|---|---|---
time (s) | 9.9 | 10.1 | 22 | 15.8 | 12.6

###  III-B Performances and understanding of DIAL mechanisms

As we can observe on Figure 2 and on Table I where a 50 clicks budget has been
set, both AC and DISCA successfully enhance the outputs initially proposed by
the neural network. DISCA reaches better improvement than AC-only on AIRS and
ISPRS: in Table I, AC’s mean gain with active learning is of 2.2% while
DISCA’s one is of 2.5%. Visually, this translates into correction of areas as
a whole in a single click, like the buildings or the parking lot of Figure 2.
The slight superiority of not retraining the network with DISCA on INRIA is
probably due to a better noise robustness. Indeed, INRIA labeling is based on
land register and is thus not as conforms with the signal than AIRS or ISPRS.
The overall superiority of DISCA has to be also moderated by the inference
time of the two algorithms. Indeed, as shown in Table II, DISCA is more than
$10\times$ slower than AC-only due to its retraining component.

####  III-B1 Active learning with AC

As we can see on Figure 3, the random order leads to an improvement linear
w.r.t. number of processed patches. All active learning schemes speed up the
gain in performances by targeting the more uncertain areas. This is
particularly noticeable on the AIRS dataset where 50 annotations are enough to
reach 75% of the final improvement. This behavior is probably due to the
dataset itself. Indeed, since it covers a lot of rural areas, many images only
contain few buildings and the uncertainty measures then allow to quickly show
the areas of interest to the user.

Regarding the different uncertainty measures, ODIN is consistently the worst
one. Indeed, it is only slightly better than the random order and, contrary to
the other methods, its performance is almost linear on the AIRS dataset. This
behavior might be explained by the method original purpose. Indeed, while the
other methods aim to estimate the model uncertainty, ODIN aims to detect
outliers. Though these tasks are related, it appears here that model errors
are not due to this type of issue in the image area. Moreover, Table III shows
that ODIN and MCDropout considerably slow the prediction process (resp. by
factors 1.5 and 2) compared to entropy (factor 1) and ConfidNet (factor 1.2).

ConfidNet and entropy consistently obtain the best performances, with a slight
advantage for the former on AIRS and the domain adaptation use-case. However,
ConfidNet is also a bit slower and less flexible since it requires to train an
additional network for each dataset. Eventually, entropy offers an excellent
trade-off between high accuracy performances and fast computation, as it is
only slightly slower than a random pick.

####  III-B2 Active learning with DISCA

Since DISCA slightly modifies the neural network parameters, we recompute the
entire prediction and uncertainty after each processed patch. Since MC Dropout
and ODIN proved to be relatively slow and less performing with AC, we only
compare entropy and ConfidNet in this set-up. As we can observe on Figure 4,
results are more complex to interpret than with AC.

On ISPRS, the different methods are all a bit unstable, which is probably
explained by the different improvements for the multiple classes of this
dataset. However, both uncertainty methods still perform better than the
random strategy and the strategy relying on ConfidNet enables a gain up to 5%
compared to 4% for the random one. On INRIA, both uncertainty strategies
outperform the random one for the first 60 patches but end up being caught up,
probably stuck in a local minimum. It is noteworthy that ConfidNet ends up
outperforming entropy on these two datasets by a larger margin than with AC.
On AIRS and in the domain adaptation situation, the behaviors are similar to
the ones obtained with AC, with noticeably higher performances. Indeed, the
gain are around 20% and 5% with DISCA while they were around 10% and 4% with
AC respectively on the domain adaptation situation and the AIRS dataset.

Hence, these results confirm the benefits of a guidance towards relevant
patches relying on uncertainty measures. ConfidNet is on average the best
method to this aim. However, the faster, simpler and only slightly under-
performing entropy is a very good alternative for successfully recognizing the
most relevant areas to annotate.

###  III-C Ablation study and comparison with state-of-the-art

####  III-C1 Active learning

TABLE IV: Performances in terms of mean IoU before and after the interactive processes with only 10 annotations per image, w.r.t. corresponding complexity. |  Initial |  Rand. patches |  AL patches | Whole image
---|---|---|---|---
|  |  $\mathcal{O}(n_{\mathrm{annots}}*d^{2}_{\mathrm{patch}})$ |  $\mathcal{O}(n_{\mathrm{annots}}*d^{2}_{\mathrm{patch}})$ |  $\mathcal{O}(n_{\mathrm{annots}}*d^{2}_{\mathrm{image}})$
| LinkNet | AC |  DISCA | AC |  DISCA | AC | DISCA
ISPRS | 70.7  | 71 |  68.7  | 71.8 |  71.9  | 71.7 | 72.4
INRIA | 85.4  | 85.5 |  85.5  | 86 |  86  | 86.4 | 86.5
AIRS | 88  | 88.2 |  88.8  | 89.6 |  90.7 | 89.8 | 90.2

As shown in the previous section, an active learning patch order leads to
better agent annotations than a random patch order with both AC and DISCA. We
compare it here to a theoretical upper bound of AC and DISCA: the agent
generates each click at the center of the largest spatial error on the whole
image, which would be optimal in terms of potential improvement but at the
cost of a whole image search. As we can observe in Table IV, this leads, with
10 annotations with AC/DISCA, to an average 1.3/1.7% improvement over the
three datasets against a 1.1/1.5% improvement with the active learning
strategy. However, this slight superiority is mitigated by the complexity to
find the annotations. Indeed, in the whole image case, the agent has to browse
through $3.6\times 10^{7}$ pixels for each click in a $6000\times 6000$ image
(complexity: $\mathcal{O}(n_{\mathrm{annots}}*d^{2}_{\mathrm{image}})$)
whilst, in the patch case, it has to browse through $2.6\times 10^{5}$ pixels
in a $512\times 512$ patch (complexity:
$\mathcal{O}(n_{\mathrm{annots}}*d^{2}_{\mathrm{patch}})$). Hence, it is 100
times more costly to find the annotation in an entire remote sensing image
than in a patch.

####  III-C2 AC & DISCA

Figure 5: Ablation study on ISPRS dataset.

(a) Mean IoU w.r.t the number of annotations.

(b) IoU after 20 annotations

Figure 6: Mean IoU of AC and DISCA for domain adaptation (AIRS$\to$ISPRS).

Initial output

After 1 annotation

After 10 annotations

After 50 annotations

Ground-truth

Figure 7: Domain adaptation (AIRS $\to$ ISPRS) visual examples

(a) Weights reinitialized between each image.

(b) Weights updated between each image.

Figure 8: Sequential learning study with DISCA in a transfer scenario. The
legend corresponds to the order in which the algorithm processes the images.

To understand the influence of the aspects of the DISCA algorithm, we analyze
separately its different components. AC (ours) adds input layers for
annotations [25], randomly pre-trained on the ground-truth. WTP [31] retrains
the model based on a few annotations. DISCA (ours) sums up AC and WTP with
regularization with respect to the initial prediction. We also test AC
combined with WTP, and WTP combined with regularization. To study the
importance of the regularization parameter $\lambda$, we test various values
$\text{DISCA}_{\lambda=1}$ and $\text{DISCA}_{\lambda=10}$. Finally, we also
compare our models to FESTA [32] which trains a neural network on point
annotations (as WTP) with a different regularization.

As shown on Figure 5, AC and WTP+reg obtain IoU gains around 1% for 10 clicks
and are beaten by the various flavours of DISCA which almost doubles the gain.
This means that the interactive retraining process could be effectively
applied to any classically trained neural network but needs to be combined
with the AC process to fully exploit the annotations. Moreover, we observe
that the regularization is extremely important in DISCA as its absence leads
to worse results (AC+WTP curve) than the initial ones (LinkNet curve). A too
high $\lambda$ also decreases the benefits brought by DISCA because it then
prevents the algorithm to optimally exploit the annotations. Finally, in this
framework of incremental learning, WTP [31] and FESTA [32] also lead to worse
results than the initial ones. These methods were originally designed to train
the neural networks from scratch on point annotations. Hence, it explains why
they are not optimal in a refinement scenario since they take into account
different constraints.

We also compare our approaches with the recent ALCD method [21] also deployed
in the field of remote sensing for cloud segmentation in low resolution (60
m/pixel) images. To adapt it to our use-case, we run ALCD in a fine-tuning
setting on the ISPRS dataset. In practice, we initially pre-train the ALCD
random forest on 100000 samples per image from the training set, and then
adapt the classifier with the same number of annotations as AC and DISCA.
However it leads to very poor performances both before (30% IoU) and after
fine-tuning (30.5% IoU) compared to AC/DISCA results presented previously.
While the absolute results might be due to differences of peculiar
implementations of random forest and neural network, the ALCD gain is only
$+0.5\%$, which is 2 times less than AC and 3 times less than DISCA.

###  III-D Domain adaptation use-case

####  III-D1 Performances

The objective in this domain adaptation use-case is to detect the buildings on
the 8 images of the ISPRS validation set. To this purpose, we compare a neural
network trained on AIRS under AC and DISCA settings to a control one trained
on the ISPRS training set. The ISPRS images are down-sampled using bi-linear
interpolation to the AIRS resolution. The neural network’s weights are
reinitialized between each image. Figure 6 shows that a network weakly
supervised with DISCA beats AC by a large margin in this scenario. Besides, it
can quickly reach high performances (more than 80% IoU within 20 annotations)
and even outperform a fully supervised one with a sufficient amount of
annotations. This is visually confirmed on Figure 7. Indeed, 10 annotations
enable the network to well understand the new domain images and thus propose
decent segmentation maps. More annotations correct most of the remaining
mistakes.

####  III-D2 Sequential learning

Moreover, we analyze the generalization of DISCA through a sequence of images
in the same domain adaptation scenario. This means that we do not reinitialize
the neural network weights between each image. We refer to this set-up as
sequential learning, and we learn two insights from it on Figure 8. First,
DISCA does not suffer from catastrophic forgetting here as the algorithm does
not diverge even on the last seen images. Second, sequential learning greatly
improves the initial performances directly after the first image. Indeed, the
initial IoU then approximately increases of 20%. However, after few
annotations, the sequential learning benefits vanish and the performances
become similar to the non-sequential set-up.

###  III-E Discussion: when to choose AC-only or DISCA?

To better apprehend the difference between the two methods, we sample 10000
$512\times 512$ crops from each dataset. Then, given one annotation, we
compare the difference between AC and DISCA based on two parameters: the
spatial size of the corrected mistake and the initial accuracy of the network
on the patch. Precisely, the spatial size of the corrected mistake is the size
of the error polygon in which the annotation is sampled. Similarly to the
initial accuracy, it is obtained with a comparison between the initial
predicted map and the ground-truth map. It is intuitively obvious that both AC
and DISCA are correlated to these parameters since, if the mistake to correct
is small, the overall IoU gain will be smaller than with a larger mistake to
correct. However, we think that this comparison can bring valuable insights to
choose the appropriate method depending on the situation.

Figure 9 compares the two methods with respect to these two criteria. First,
both methods seem to work well and can outperform the other one when the
mistake area is small and the initial performance is high. We thus recommend
to use AC in these situations. Indeed, the locality of AC is no longer a
constraint since the error is strongly spatially contained and the relatively
long retraining time inherent to DISCA makes it less suitable here. Second,
when the initial accuracy is low or the area to correct large, DISCA now
clearly tends to perform better than AC, and we thus believe that it should be
favored in these situations. Indeed, its spatial globality resulting from its
retraining can be fully expressed to correct large mistakes. This outcome
shows that DISCA is more relevant to correct deeply flawed segmentation maps
than AC.

Figure 9: Comparison of AC-only and DISCA (IoU) with respect to the spatial
size of the corrected mistake and the initial accuracy. Legend ”best”
designates the best method for the given sample.

##  IV Conclusion

We have presented DIAL, a framework to interactively enhance segmentation maps
initially proposed by a neural network. Its core concept relies on
interactions between a deep neural network and an agent under the form of
clicked annotations.

First, we have proposed an interactive learning framework which builds on
complementary mechanisms. First, Annotations as Channels (AC) modifies the
neural network inputs. This approach is fast and local since it does not
modify the weights of network. Second, an on-the-fly retraining uses the
annotations as a a sparse ground-truth. Finally, a regularization term based
on the initial prediction is crucial to complement the cross-entropy loss
during retraining and avoids catastrophic forgetting. Since this modifies the
weights of the network, the full framework is slower but improves the
segmentation maps at a larger scale.

Finally, we have integrated active learning within our framework to guide the
agent interventions towards relevant patch queries. To this purpose, we have
compared different state-of-the-art acquisition functions to estimate the
neural network uncertainty to finally conclude that entropy is the most suited
one thanks to its simplicity and efficiency. Hence, active learning speeds up
the use of our interactive segmentation algorithms and is particularly
relevant to face budget constraints.

In the future, we intend to apply DIAL in a class-incremental scenario to make
it easily adaptive to new tasks and use-cases. This, along with the promising
results shown in domain adaptation, will provide Earth observation scientists
and companies with a powerful to re-use, transfer and enhance deep learning
models.

## References

  * [1] H. Costa, G. M. Foody, and D. S. Boyd, “Supervised methods of image segmentation accuracy assessment in land cover mapping,” in _Remote Sensing of Environment_ , vol. 205. Elsevier, 2018, pp. 338–351.
  * [2] R. C. Daudt, B. Le Saux, and A. Boulch, “Fully convolutional siamese networks for change detection,” in _International Conference on Image Processing_. IEEE, 2018, pp. 4063–4067.
  * [3] M. T. Chiu, X. Xu, Y. Wei, Z. Huang, A. G. Schwing, R. Brunner, H. Khachatrian, H. Karapetyan, I. Dozier, G. Rose _et al._ , “Agriculture-vision: A large aerial image database for agricultural pattern analysis,” in _Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2828–2838.
  * [4] E. Maggiori, Y. Tarabalka, G. Charpiat, and P. Alliez, “Can semantic labeling methods generalize to any city? the INRIA aerial image labeling benchmark,” in _International Geoscience and Remote Sensing Symposium_. IEEE, 2017, pp. 3226–3229.
  * [5] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Conference on Computer Vision and Pattern Recognition_ , 2016.
  * [6] J. Castillo-Navarro, B. Le Saux, A. Boulch, N. Audebert, and S. Lefèvre, “Semi-supervised semantic segmentation in Earth observation: The MiniFrance suite, dataset analysis and multi-task network study,” in _arXiv preprint arXiv:2010.07830_ , 2020.
  * [7] Q. Chen, L. Wang, Y. Wu, G. Wu, Z. Guo, and S. L. Waslander, “Aerial imagery for roof segmentation: A large-scale dataset towards automatic mapping of buildings,” in _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 147. Elsevier, 2019, pp. 42–55.
  * [8] M. Schroder, H. Rehrauer, K. Seidel, and M. Datcu, “Interactive learning and probabilistic retrieval in remote sensing image archives,” in _Transactions on Geoscience and Remote Sensing_ , vol. 38, no. 5. IEEE, 2000, pp. 2288–2298.
  * [9] B. Le Saux and M. Sanfourche, “Rapid semantic mapping: Learn environment classifiers on the fly,” in _IROS_. IEEE, 2013, pp. 3725–3730.
  * [10] B. Settles, “Active learning literature survey,” University of Wisconsin-Madison Department of Computer Sciences, Tech. Rep., 2009.
  * [11] Y. Gal, R. Islam, and Z. Ghahramani, “Deep bayesian active learning with image data,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1183–1192.
  * [12] O. Sener and S. Savarese, “Active learning for convolutional neural networks: A core-set approach,” _arXiv preprint arXiv:1708.00489_ , 2017.
  * [13] C. E. Shannon, “A mathematical theory of communication,” _The Bell system technical journal_ , vol. 27, no. 3, pp. 379–423, 1948.
  * [14] S. Liang, Y. Li, and R. Srikant, “Principled detection of out-of-distribution examples in neural networks,” in _International Conference on Learning Representations_. IEEE, 2018.
  * [15] C. Corbière, N. Thome, A. Bar-Hen, M. Cord, and P. Pérez, “Addressing failure prediction by learning model confidence,” in _Advances in Neural Information Processing Systems_ , vol. 32, 2019, pp. 2902–2913.
  * [16] G. Lenczner, A. Chan-Hon-Tong, N. Luminari, B. Le Saux, and G. Le Besnerais, “Interactive learning for semantic segmentation in Earth observation,” in _European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases Workshop_ , 2020.
  * [17] S. Aksoy, K. Koperski, C. Tusk, and G. Marchisio, “Interactive training of advanced classifiers for mining remote sensing image archives,” in _Knowledge Discovery in Databases_. ACM, 2004, pp. 773–782.
  * [18] M. Ferecatu and N. Boujemaa, “Interactive remote-sensing image retrieval using active relevance feedback,” in _Transactions on Geoscience and Remote Sensing_ , vol. 45, no. 4. IEEE, 2007, pp. 818–826.
  * [19] L. Breiman, “Random forests,” in _Machine Learning_ , vol. 45, no. 1\. Springer, 2001, pp. 5–32.
  * [20] J. A. dos Santos, P. Gosselin, S. Philipp-Foliguet, R. d. S. Torres, and A. X. Falcão, “Interactive multiscale classification of high-resolution remote sensing images,” in _Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 6, no. 4. IEEE, 2013, pp. 2020–2034.
  * [21] L. Baetens, C. Desjardins, and O. Hagolle, “Validation of Copernicus sentinel-2 cloud masks obtained from MAJA, Sen2Cor, and FMask processors using reference cloud masks generated with a supervised active learning procedure,” _Remote Sensing_ , vol. 11, no. 4, p. 433, 2019.
  * [22] V. Ružička, S. D’Aronco, J. D. Wegner, and K. Schindler, “Deep active learning in remote sensing for data efficient change detection,” in _European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases Workshop_ , 2020.
  * [23] B. Kellenberger, D. Marcos, S. Lobry, and D. Tuia, “Half a percent of labels is enough: Efficient animal detection in uav imagery using deep cnns and active learning,” in _Transactions on Geoscience and Remote Sensing_ , vol. 57, no. 12. IEEE, 2019, pp. 9524–9533.
  * [24] C. Rother, V. Kolmogorov, and A. Blake, “GrabCut: Interactive foreground extraction using iterated graph cuts,” in _Transactions on Graphics_ , vol. 23, no. 3. ACM, 2004, pp. 309–314.
  * [25] N. Xu, B. Price, S. Cohen, J. Yang, and T. Huang, “Deep interactive object selection,” in _Conference on Computer Vision and Pattern Recognition_. IEEE, 2016, pp. 373–381.
  * [26] T. Kontogianni, M. Gygli, J. Uijlings, and V. Ferrari, “Continuous adaptation for interactive object segmentation by learning from corrections,” in _European Conference on Computer Vision_. Springer, 2020, pp. 579–596.
  * [27] E. Agustsson, J. R. Uijlings, and V. Ferrari, “Interactive full image segmentation by considering all regions jointly,” in _Conference on Computer Vision and Pattern Recognition_. IEEE, 2019, pp. 11 622–11 631.
  * [28] Z. Li, X. Zhang, P. Xiao, and Z. Zheng, “On the effectiveness of weakly supervised semantic segmentation for building extraction from high-resolution remote sensing imagery,” _Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 14, pp. 3266–3281, 2021.
  * [29] Q. Wang, J. Gao, and X. Li, “Weakly supervised adversarial domain adaptation for semantic segmentation in urban scenes,” _Transactions on Image Processing_ , vol. 28, no. 9, pp. 4376–4386, 2019.
  * [30] R. Caye Daudt, B. Le Saux, A. Boulch, and Y. Gousseau, “Guided anisotropic diffusion and iterative learning for weakly supervised change detection,” in _Conference on Computer Vision and Pattern Recognition Workshop_. IEEE, 2019.
  * [31] A. Bearman, O. Russakovsky, V. Ferrari, and L. Fei-Fei, “What’s the point: Semantic segmentation with point supervision,” in _European conference on computer vision_. Springer, 2016, pp. 549–565.
  * [32] Y. Hua, D. Marcos, L. Mou, X. X. Zhu, and D. Tuia, “Semantic segmentation of remote sensing images with sparse annotations,” _IEEE Geoscience and Remote Sensing Letters_ , 2021.
  * [33] L. K. Hansen and P. Salamon, “Neural network ensembles,” in _Transactions on Pattern Analysis and Machine Intelligence_ , vol. 12, no. 10. IEEE, 1990, pp. 993–1001.
  * [34] H. Nguyen and A. Smeulders, “Active learning using pre-clustering,” in _International Conference on Machine learning_. IEEE, 2004, p. 79.
  * [35] B. Demir, C. Persello, and L. Bruzzone, “Batch-mode active-learning methods for the interactive classification of remote sensing images,” _Transactions on Geoscience and Remote Sensing_ , vol. 49, no. 3, pp. 1014–1031, 2010.
  * [36] M. Laroze, R. Dambreville, C. Friguet, E. Kijak, and S. Lefèvre, “Active learning to assist annotation of aerial images in environmental surveys,” in _International Conference on Content-Based Multimedia Indexing_. IEEE, 2018, pp. 1–6.
  * [37] D. Tuia, F. Ratle, F. Pacifici, M. Kanevski, and W. Emery, “Active learning methods for remote sensing image classification,” in _Transactions on Geoscience and Remote Sensing_ , vol. 47, no. 7. IEEE, 2009, pp. 2218–2232.
  * [38] L. Bruzzone and C. Persello, “Active learning for classification of remote sensing images,” in _International Geoscience and Remote Sensing Symposium_. IEEE, 2009, pp. 689–693.
  * [39] J. Guo, X. Zhou, J. Li, A. Plaza, and S. Prasad, “Superpixel-based active learning and online feature importance learning for hyperspectral image analysis,” _Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 10, no. 1, pp. 347–359, 2016.
  * [40] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “iCaRL: Incremental classifier and representation learning,” in _Conference on Computer Vision and Pattern Recognition_. IEEE, 2017, pp. 2001–2010.
  * [41] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in _International Conference on Learning Representations_. IEEE, 2017.
  * [42] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _International Conference on Machine Learning_. IEEE, 2017, pp. 1321–1330.
  * [43] K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” in _Advances in Neural Information Processing Systems_ , vol. 31, 2018, pp. 7167–7177.
  * [44] W. H. Beluch, T. Genewein, A. Nürnberger, and J. M. Köhler, “The power of ensembles for active learning in image classification,” in _Conference on Computer Vision and Pattern Recognition_. IEEE, 2018, pp. 9368–9377.
  * [45] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _International Conference on Machine Learning_. IEEE, 2016, pp. 1050–1059.
  * [46] T. DeVries and G. W. Taylor, “Learning confidence for out-of-distribution detection in neural networks,” _arXiv preprint arXiv:1802.04865_ , 2018.
  * [47] C. García Rodríguez, J. Vitrià, and O. Mora, “Uncertainty-based human-in-the-loop deep learning for land cover segmentation,” in _Remote Sensing_ , vol. 12 no. 22. MDPI, 2020, p. 3836.
  * [48] D. Yoo and I. S. Kweon, “Learning loss for active learning,” in _Conference on Computer Vision and Pattern Recognition_. IEEE, 2019, pp. 93–102.
  * [49] J. Moon, J. Kim, Y. Shin, and S. Hwang, “Confidence-aware learning for deep neural networks,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 7034–7044.
  * [50] Q. Chen, L. Wang, Y. Wu, G. Wu, Z. Guo, and S. Waslander, “Aerial imagery for roof segmentation: A large-scale dataset towards automatic mapping of buildings,” in _arXiv preprint arXiv:1807.09532_ , 2018.
  * [51] F. Rottensteiner, G. Sohn, J. Jung, M. Gerke, C. Baillard, S. Benitez, and U. Breitkopf, “The ISPRS benchmark on urban object classification and 3D building reconstruction,” in _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 1, no. 1. Copernicus GmbH, 2012, pp. 293–298.
  * [52] A. Chaurasia and E. Culurciello, “LinkNet: Exploiting encoder representations for efficient semantic segmentation,” in _Conference on Visual Communications and Image Processing_. IEEE, 2017.
  * [53] J. Liew, Y. Wei, W. Xiong, S.-H. Ong, and J. Feng, “Regional interactive image segmentation networks,” in _International Conference on Computer Vision_. IEEE, 2017, pp. 2746–2754.
  * [54] K. He, J. Sun, and X. Tang, “Guided image filtering,” in _European Conference on Computer Vision_. Springer, 2010, pp. 1–14.

##  Appendix A How to optimize the encoding ?

(a) Car (red) and building (yellow) annotations

(b) Ground-truth connectivity encoding

(c) Distance-transform encoding

(d) Guided filter encoding

Figure 10: Different annotations encodings depending on context uses. Best viewed in color. TABLE V: IoU on ISPRS after 120 annotations with AC depending on the encoding. | Bin. | DT | C-PM |  C-GT (sup) | GF
---|---|---|---|---|---
Initial | 70.7 | 70.7 | 70.7 | 70.7 | 70.8
After | 76.4 | 76.6 | 76.5 | 76.7 | 76.7
Gain | 5.7 | 5.9 | 5.8 | 6 | 5.9

We investigate here the annotations encoding to analyze its influence on the
AC mechanism.

There are many possibilities to encode the annotations in their dedicated
channels and they all provide different spatial information. The size of the
encoding is the most obvious issue: if it is too small, it might not provide
enough information to efficiently fix the initial segmentation but a coarser
encoding might provide erroneous information. A popular context-free trade-off
used in most interactive segmentation works [25, 53] is to encode the
annotations with Euclidean distance transforms to dilute spatial information.
However, due to its context independence, this encoding might be sub-optimal.
Ideally, the perfect encoding would be the original ground-truth map but it is
obviously impossible to get. Based on this insight, we study here how to best
approximate this ground-truth given the available data: the input image, the
annotations and the trained neural network. We define two possible context use
besides the no-context one:

  * •

Using the input image

  * •

Using the initial prediction

As encoding baselines, we use small binary (bin.) disks of 1.5 pixels radius
and distance transform (DT) applied on 10 pixels radius disks. We build on
this DT encoding for the context encodings. To use the input image, we rely on
guided filtering [54] (GF) in order to preserve the edges in the encoding. To
use the initial prediction, we encode the annotations using their connected
pixels in the prediction map (C-PM). To estimate the superior boundary
theoretically reachable with an encoding from the ground-truth, we also encode
the annotations using their connected pixels in the ground-truth map (C-GT).
These different methods to encode the annotations are represented in Figure
10.

However, as shown in Table V, the different encoding strategies seem to
provide similar information to the network as the gains are in the same order
of magnitude. Indeed, they all increase the IoU of around 6% for 120
annotations on the ISPRS images, even though the binary encoding is slightly
lower and confirms the usefulness of DT encoding. The GF encoding obtains the
same gain as the DT one, C-PM is lower of 0.1% and even C-GM is only better of
0.1%.

These insignificant differences show that the network does not need any
contextual guidance to learn nearly optimal information from the annotations
using a simple and intuitive encoding such as distance transform.

(a) AC-only

(b) DISCA

Figure 11: Annotations sampled using uncertainty and error knowledge (smart
corrector agent)

(a) AC-only

(b) DISCA

Figure 12: Annotations sampled with uncertainty but without error knowledge

Initial output

Entropy

Annotation(s)

DISCA output

Ground-truth

Figure 13: Initial output corrected with annotations relying on entropy. On
the ”Entropy” column, the areas with an entropy higher than the ninth quantile
over the image are highlighted in red. On the ”Annotation(s)” column, the
color of the annotations represents their labels w.r.t. the associated ground-
truth maps.

##  Appendix B Active learning for pixel-based guidance

###  B-A Set-up and objectives

To investigate local guidance, we sample 10000 $512\times 512$ crops from each
dataset, make an initial prediction, generate one annotation per sample and
then do a new prediction with AC-only and DISCA. Our acquisition function here
is entropy. We test two conjectures. First, we want to determine whether
highly uncertain pixels among the misclassified ones can lead towards
particularly meaningful annotations. Second, we want to figure out whether the
uncertainty measurements can help the agent to spot errors at a pixel level.

###  B-B Uncertainty for optimal annotations

We make the hypothesis that an agent always clicks on a wrongly segmented
area, or in other words that he is able to spot the mistakes and correct them.
To look for optimal annotations, we compare the following annotations sampling
strategies.

  1. 1.

We sample the annotation randomly in the wrongly segmented area (random).

  2. 2.

Like in the other experiments, we sample the annotation in the middle of the
spatially-largest wrongly-segmented area (max).

  3. 3.

We threshold the uncertainty map at the ninth quantile to keep only the
highest uncertainty values. We then sample the annotation in the intersection
of the wrongly-segmented area and this thresholded uncertainty map.

As shown on Figure 11, the uncertainty-based annotations lead to corrections
of the same magnitude than the random ones on average. Moreover, these
uncertainty-based annotations clearly don’t provide more information to the
model than the ones based on max. Indeed, the gains of max annotations with
AC-only are around 6.4% IoU against respectively 4.5% and 4.7% for the random
and uncertainty-based ones.

This corroborates the correlation between the gain and the size of the
corrected area previously exhibited and shows that uncertainty does not lead
towards more meaningful annotations than the ones contained inside large
mistakes.

###  B-C Uncertainty to spot mistakes

In order to evaluate if the uncertainty measures can help to spot mistakes at
the pixel level, we compare annotations sampled randomly and on the basis of
uncertainty measures without ground-truth prior knowledge. In other words, we
do not coerce the annotations to be sampled in mistake areas.

Figure 12 shows that the uncertainty-based annotations lead to better
improvements (3.7% IoU with AC-only, 4.5% with DISCA) than the random ones
(1.3% IoU with AC-only, 1.9% with DISCA) on average. We can visually confirm
these insights on Figure 13 where uncertainty measures tend to highlight
wrongly predicted areas. Besides, the highlighted areas which are initially
correctly predicted tend to be legitimately questionable such as object
contours or road surfaces looking like buildings (third row).