# What Happens to Apparent Horizons in a Binary Black Hole Merger?

Daniel Pook-Kolb  Max-Planck-Institut für Gravitationsphysik (Albert Einstein
Institute), Callinstr. 38, 30167 Hannover, Germany  Leibniz Universität
Hannover, 30167 Hannover, Germany  Robie A. Hennigar  Department of
Mathematics and Statistics, Memorial University, St. John’s, Newfoundland and
Labrador, A1C 5S7, Canada  Department of Physics and Astronomy, University of
Waterloo, Waterloo, Ontario, Canada, N2L 3G1  Department of Physics and
Computer Science, Wilfrid Laurier University, Waterloo, Ontario, Canada N2L
3C5  Ivan Booth  Department of Mathematics and Statistics, Memorial
University, St. John’s, Newfoundland and Labrador, A1C 5S7, Canada

###### Abstract

We resolve the fate of the two original apparent horizons during the head-on
merger of two non-spinning black holes. We show that following the appearance
of the outer common horizon and subsequent interpenetration of the original
horizons, they continue to exist for a finite period of time before they are
individually annihilated by unstable MOTSs. The inner common horizon vanishes
in a similar, though independent, way. This completes the understanding of the
analogue of the event horizon’s “pair of pants” diagram for the apparent
horizon. Our result is facilitated by a new method for locating marginally
outer trapped surfaces (MOTSs) based on a generalized shooting method. We also
discuss the role played by the MOTS stability operator in discerning which
among a multitude of MOTSs should be considered as black hole boundaries.

It is common practice to picture the merger of two black holes according to
the famous “pair of pants” diagram, which describes the evolution of the event
horizon during the merger. But what does the analogous picture for the
apparent horizon (AH) look like? While the evolution of the event horizon has
been understood for nearly half a century Hawking and Ellis (1973); Matzner
_et al._ (1995), the complete evolution of apparent horizons during a merger
has remained unresolved.

While the event horizon is well-suited to theoretical analyses, its
teleological nature makes it less useful in highly dynamical or practical
situations. In numerical simulations of mergers, marginally outer trapped
surfaces (MOTSs) are used instead. A closed two-dimensional space-like surface
$\mathcal{S}$ is a MOTS if light rays emitted normal to $\mathcal{S}$ are
neither converging nor diverging in the outward direction. The outermost MOTS
on a given slice is commonly called the apparent horizon. However, despite
their importance as a quasi-local characterization of black holes, there
remain many unresolved questions pertaining to the evolution of interior MOTSs
during a merger.

While it is true that the details of what occurs within the event horizon
(where all MOTSs are located) is causally disconnected from the rest of the
universe, this does not mean that interior evolution is irrelevant. At the
very least, it is conceptually important to understand to what extent MOTSs
provide a physically reasonable description of the merger. Furthermore, the
existence of a connected sequence of MOTSs between the initial and final
states of the merger provides a means for physical properties Dreyer  _et al._
(2003); Krishnan (2008); Gupta  _et al._ (2018) to be tracked throughout the
full evolution. Finally, one may hope that there exist correlations between
the dynamics in the strong field regime and properties of the distant
spacetime. Indeed, such correlations have been shown to exist under certain
circumstances Rezzolla  _et al._ (2010); Jaramillo  _et al._ (2012a, b); Gupta
_et al._ (2018); Prasad  _et al._ (2020).

The behaviour of AHs during the initial stages of a merger is well-known
Hawking and Ellis (1973). Initially there are two individual AHs corresponding
to two separate black holes. When these holes become sufficiently close to one
another, a common AH forms surrounding the individual horizons, which continue
to exist. This common horizon immediately splits into an inner and outer
branch. The outer branch grows in area and becomes more symmetric, ultimately
asymptoting to the event horizon. The inner common MOTS moves inward, becoming
increasingly distorted.

The bifurcation of the common horizon, combined with the fact that there are
known exact solution examples of MOTSs weaving back and forth through time
Ben-Dov (2004a); Booth  _et al._ (2006), led to the speculative idea that all
MOTSs involved in the merger may be different components of a single world
tube that weaves its way through time Hayward (2000); Ben-Dov (2004b).
However, in most situations of interest MOTSs must be located numerically.
Therefore, improvements in the understanding of the evolution of MOTSs during
a merger have been in lockstep with improvements in the numerical methods used
to locate them. For this reason, progress beyond this qualitative speculation
was limited.

With the advent of more robust numerical finders for MOTSs Pook-Kolb  _et al._
(2019a), it is now possible to go beyond these initial stages of the merger
and better understand the interior dynamics. The inner common MOTS continues
to move inward and coincides with the union of the two AHs of the individual
black holes at the moment these horizons touch. This can be interpreted as an
instantaneous (non-smooth) merger of the three MOTSs Pook-Kolb  _et al._
(2019b, c, 2020a, 2020b). All continue to exist beyond this point: the two
individual horizons interpenetrate and the inner common MOTS develops self-
intersections. Identifying these self-intersections was not possible with
previous MOTS finders, as these had implicitly assumed that the MOTSs are
‘star-shaped’. These seemingly exotic surfaces have subsequently been shown to
be rather generic. For example, there are an infinite number of self-
intersecting MOTSs present within the horizon of the Schwarzschild black hole
Booth  _et al._ (2020). This raises the question of whether exotic MOTSs are
also present during a black hole merger, and what role they might play.

In this Letter, we report three closely connected results. First, using a
novel MOTS finder, we identify for the first time an apparently infinite
number of MOTSs present in certain Brill-Lindquist initial data. We then
discuss the role that these new MOTSs play in resolving the final fate of the
AHs of the two original black holes, finding that there do exist world tubes
weaving back and forth through time. However, instead of a single smooth world
tube, we find multiple distinct ones. Finally, we discuss the stability of
these surfaces. The stability operator characterizes key merger processes and
identifies which MOTSs should be thought of as black hole boundaries.

A Multitude of MOTSs. To locate MOTSs, we employ a novel shooting method. The
procedure, while in the tradition of methods first developed in the 1970s
Čadež (1974), is more versatile and applies to non-rotating though otherwise
general axisymmetric configurations Booth  _et al._ (2021). Implemented in
Pook-Kolb  _et al._ (2021a), it can be applied to both analytically known
initial data as well as to slices obtained from numerical simulations. It
overcomes a limitation of the method introduced in Pook-Kolb  _et al._ (2019a)
as it does not require an initial guess for the shape of the surface to be
located. As such, it is ideally suited for locating axisymmetric MOTSs with
geometries that are both unexpected and arbitrarily complicated.

The approach exploits axisymmetry to reduce the problem of locating a
2-surface to that of determining a curve $\gamma$, which we call a MOTSodesic.
The full MOTS is the revolution of $\gamma$. Given an axisymmetric 3-surface
{$\Sigma,h_{ij},D_{i}$}, the arclength parameterized curve $\gamma(s)$ in the
two-dimensional space orthogonal to the rotational Killing field
$\varphi=\frac{\partial}{\partial\phi}$ is a MOTSodesic if it satisfies the
two coupled second-order ODEs:

$T^{a}D_{a}T^{b}=(N^{c}D_{c}(\ln R)+k_{u})N^{b}\equiv\kappa^{+}N^{b}\,.$ (1)

Here $T^{a}$ is the unit-length tangent vector to $\gamma$, $N^{a}$ is its
unit-length normal, $R$ is the circumferential radius, and $k_{u}$ is the
trace of the extrinsic curvature of $\mathcal{S}$ with respect to the unit
time-like normal $u$ to $\Sigma$.

Henceforth, we restrict our considerations to the head-on merger of two non-
spinning black holes. For this purpose, we use Brill-Lindquist (BL) initial
data Brill and Lindquist (1963). These describe a Cauchy slice $\Sigma$ which
is time symmetric, i.e. with vanishing extrinsic curvature. The three-metric
is conformally flat, $h_{ij}=\psi^{4}\delta_{ij}$, where $\delta_{ij}$ is the
flat metric and the conformal factor is

$\psi=1+\frac{m_{1}}{2r_{1}}+\frac{m_{2}}{2r_{2}}\,,$ (2)

where $m_{1,2}$ are the bare masses of the black holes and $r_{1,2}$ are the
(coordinate) distances to the respective punctures.

Parametrizing $\gamma:(\rho,z)=\left(P(s),Z(s)\right)$ relative to cylindrical
coordinates $(\rho,z,\phi)$, the MOTSodesic equations become

$\displaystyle\ddot{P}$
$\displaystyle=\frac{\dot{Z}^{2}}{P}+\frac{4\psi_{\rho}}{\psi^{5}}-\frac{6\dot{P}(\dot{P}\psi_{\rho}+\dot{Z}\psi_{z})}{\psi}\,,$
(3) $\displaystyle\ddot{Z}$
$\displaystyle=-\frac{\dot{Z}\dot{P}}{P}+\frac{4\psi_{z}}{\psi^{5}}-\frac{6\dot{Z}(\dot{P}\psi_{\rho}+\dot{Z}\psi_{z})}{\psi}\,,$
(4)

where subscripts denote partial derivatives and the arclength parameterization
reads $\psi^{4}(\dot{P}^{2}+\dot{Z}^{2})=1$. The equations are solved in
Mathematica using the shooting method (and the resulting MOTSs confirmed using
the methods of Pook-Kolb  _et al._ (2019a)).

Figure 1: MOTSs present in Brill-Lindquist initial data for $m_{1}=0.2$,
$m_{2}=0.8$ and separation $d=0.65$. Left: The ‘standard’ horizons consisting
of the outer AH (blue), inner common MOTS (green), and AHs of individual black
holes (orange and purple) reproduced using the shooting method. Right: An
example of a new exotic MOTS.

In addition to the four ‘standard’ MOTSs, we find a large number of more
exotic MOTSs — see Fig. 1 for an example. In BL initial data, all new MOTSs
are found between the outer AH and the two original AHs and can enclose
either, both, or neither of the two punctures. These surfaces tend to closely
‘hug’ the outer and/or the individual AHs but can fold a seemingly arbitrary
number of times. By carefully tuning initial conditions, we find more and more
MOTSs. The number is likely infinite.

The Fate of the Apparent Horizons. The existence of new MOTSs in BL initial
data raises a number of important questions. First, are these surfaces generic
during a merger, or are they artefacts of the symmetry of the initial data?
Second, if generic, what role (if any) do these surfaces play in the merger?
Finally, with a seemingly infinite number of MOTSs present in a merger, how
can one discern physically relevant surfaces that demarcate black hole
boundaries?

To address these questions, we numerically evolve the initial data. Here we
start with total ADM mass $M=m_{1}+m_{2}=1$, mass ratio $q=m_{2}/m_{1}=2$ and
distance parameter $d=0.9$, corresponding to two black holes that are
initially separate with no common AH or any of the new MOTSs present. To track
evolving MOTSs we use the method described in Pook-Kolb  _et al._ (2019a, c)
and available from Pook-Kolb  _et al._ (2021a). To locate new MOTSs we use not
only the shooting method described earlier (detailed in Booth  _et al._
(2021)) but also the ansatz that MOTSs only appear or disappear in pairs. When
a MOTS cannot be tracked into the future or past, we look for a “nearby” one
with which it might annihilate or bifurcate.

We perform our simulations with the Einstein Toolkit Löffler  _et al._ (2012);
EinsteinToolkit  and set up initial conditions using TwoPunctures Ansorg  _et
al._ (2004). The Einstein equations are evolved in the BSSN formulation using
an axisymmetric version of McLachlan Brown  _et al._ (2009), which uses Kranc
Husa  _et al._ (2006); Kranc  to generate C++ code. We work with a $1+\log$
slicing and a $\Gamma$-driver shift condition Alcubierre  _et al._ (2000,
2003). Most results are obtained with a spatial grid resolution of $1/\Delta
x=720$. Lower resolutions and shorter simulations with resolutions up to
$1/\Delta x=1920$ are used to assess convergence and resolve certain features
(see Pook-Kolb  _et al._ (2021b, 2019c) for more details).

Figure 2: The area of the various MOTSs as a function of time. Lines of the
same colour indicate continuous world tubes moving forward and backward in
time. The pale gray dashed line indicates the sum of areas of
$\mathcal{S}_{1,2}$. For numerical reasons, we lose track of some of the MOTSs
— this is why some of the curves abruptly terminate. Figure 3: A “snapshot” of
Fig. 2 showing a variety of MOTSs. The three dark lines correspond to
$\mathcal{S}_{\rm outer}$ and $\mathcal{S}_{1,2}$. Two of the shown surfaces
exhibit self-intersections. Lighter colors indicate a larger number of
negative eigenvalues of the stability operator (see below and Booth  _et al._
(2021); Pook-Kolb  _et al._ (2021b) for details). Note that, in contrast to BL
initial data, the absence of time symmetry means that it is now possible for
the new MOTS to penetrate the inner AHs.

Our main results are illustrated in Fig. 2, which shows the area evolution of
several relevant MOTSs, and Fig. 3 which shows those MOTSs at a particular
moment of time: $\mathcal{S}_{\rm outer}$ is the common AH, $\mathcal{S}_{\rm
inner}$ is the inner common MOTS, and $\mathcal{S}_{1,2}$ correspond to the
AHs of the individual black holes. Beyond these standard MOTSs, we find many
new MOTSs; a selection appear in Fig. 2. The new surfaces all form through
bifurcations, splitting into outer and inner branches. Hence exotic MOTSs are
not solely artefacts of time symmetry. The shown MOTSs all form after the
outer AH has formed, and despite several having larger area than
$\mathcal{S}_{\rm outer}$, are all contained within it.

Figure 4: The annihilation of $\mathcal{S}_{2}$. The inset shows, with a red
dot, where along the world tube the shown MOTS occurs [see Fig. 2].

The figure makes clear that exotic MOTSs are essential to understanding the
final fate of the AHs of the individual black holes. Both $\mathcal{S}_{\rm
inner}$ and $\mathcal{S}_{2}$ are independently annihilated by new MOTSs. We
have good indications that $\mathcal{S}_{1}$ is annihilated by
$\mathcal{S}_{1}^{*}$, but this could not be fully resolved in our simulation
due to the MOTSs becoming too close to the punctures, where the resolution is
necessarily worse. Nonetheless, it seems clear that our new MOTSs provide a
mechanism by which the AHs of the original black holes are annihilated. To
illustrate this in greater detail, we present in Fig. 4 several “snapshots” of
the evolution of $\mathcal{S}_{2}$.

Figure 5: Eigenvalues (with $m=0$) of the stability operator for the ten MOTSs
participating in the five bifurcations shown in Fig. 2. For each MOTS, we show
the respective eigenvalue which tends to zero.

The Role of Stability. The stability operator is key in understanding the
details of the picture so far described. For a non-spinning axisymmetric MOTS
in vacuum, this takes the form Booth  _et al._ (2021):

$L_{\Sigma}\Psi=-\Delta_{\mathcal{S}}\Psi+\left(\frac{\mathcal{R}}{2}-2|\sigma_{+}|^{2}\right)\Psi\,,$
(5)

where $\Delta_{\mathcal{S}}$ is the Laplacian on $\mathcal{S}$, $\mathcal{R}$
is its scalar curvature, and $\sigma_{+}^{AB}$ is the shear in the outward
null direction. Geometrically this operator arises when one considers local
deformations of a given MOTS, with the function $\Psi$, fully determining the
deformation Andersson  _et al._ (2005). Intuitively $\Psi$ can be thought of
as measuring the normal distance between the MOTS and its deformation.

Of particular interest is the eigenvalue spectrum
$L_{\Sigma}\Psi=\lambda_{l,m}\Psi$, where the $m=0$ eigenvalues correspond to
axisymmetric deformations and in our non-spinning, axisymmetric case, all
eigenvalues are real. $\lambda_{l,m}$ encodes information about geometry of
the MOTS. Especially important is the principal (smallest) eigenvalue
$\lambda_{0,0}$ Andersson  _et al._ (2005, 2008, 2009). If $\lambda_{0,0}$ is
positive, then the MOTS is called strictly stable and evolves smoothly into
the future. The world tubes traced out by such MOTSs are everywhere expanding
and space-like. Furthermore, these MOTSs act as barriers between trapped and
untrapped surfaces in their vicinity. The vanishing of the principal
eigenvalue has been identified with the bifurcation of $\mathcal{S}_{\rm
outer}/\mathcal{S}_{\rm inner}$Pook-Kolb  _et al._ (2019a, c, 2020b).

We find these general considerations borne out in our work, though with a
number of important modifications and caveats. First, of the potentially
infinite number of MOTSs present in the interior of the merger, only three are
strictly stable: $\mathcal{S}_{\rm outer}$, $\mathcal{S}_{1}$ and
$\mathcal{S}_{2}$. Only these surfaces act as barriers and they are also the
only ones to have everywhere expanding space-like world tubes. These
properties are precisely what one would associate with horizons. Therefore
stability provides an unambiguous criterion by which the MOTSs corresponding
to black hole boundaries may be identified. This observation underlies our
choice of terminology for the AH advocated in this work: an AH is a stable
MOTS ($\lambda_{0,0}\geq 0$, i.e. we include the marginal case). All other
MOTSs located have at least one negative eigenvalue.

We find that, associated to every bifurcation/annihilation is the vanishing of
an eigenvalue of the stability operator — see Fig. 5 for the case of the
bifurcations. The formation/annihilation of the AHs coincide with the
vanishing of the principal eigenvalue, in line with the results described
above. However, a key finding of our work is that, in the case that two MOTSs
are unstable, it is the vanishing of one of the higher $m=0$ eigenvalues that
coincides with their bifurcation/annihilation. A simple picture emerges: the
MOTSs foliating a given world tube accrue an additional negative $m=0$
eigenvalue with each fold in time.

Finally, the properties of the spectrum provide further evidence for the
evolution we have advocated in this letter. At a given
bifurcation/annihilation event, we find that the numerical values of the
eigenvalues of the surfaces involved connect smoothly (see again Fig. 5). This
provides robust numerical evidence for the world tubes to be locally smooth
across the bifurcations/annihilations shown in Fig. 2.

Summary. We have shown that the interior of a black hole merger is far richer
than previously thought, containing a large (possibly infinite) number of
hitherto unidentified MOTSs. These MOTSs were initially located using a new
shooting method that sidesteps the drawback in existing finders of requiring
an initial guess for the surface of interest. The additional surfaces play a
crucial role in the interior dynamics of the merger, and are responsible for
the annihilation of the AHs of the original black holes. As such with these
new MOTSs we reveal, for the first time, the full story of how two black holes
become one, giving the analogue of the “pair of pants” diagram for the AH. The
picture is considerably more complex than the equivalent picture for the event
horizon and involves several world tubes that weave back and forth in time.
The stability of MOTSs has played a clarifying role in our work. Rather than
obscuring the utility of the quasi-local horizon framework, the multitude of
MOTSs present during the merger actually highlights the rarity of stable
MOTSs. Of all the MOTSs we have located, only three are stable, and these are
precisely those that are most naturally associated with black hole boundaries.
Moreover, we have found that associated to each bifurcation/annihilation event
is the vanishing of some eigenvalue of the stability operator, not usually the
principal one. This observation may aid in the analytical understanding of the
world tubes traced out by unstable MOTSs.

###### Acknowledgements.

Acknowledgements. We would like to express our gratitude to Graham Cox, Jose
Luis Jaramillo, Badri Krishnan, Hari Kunduri and the members of the Memorial
University Gravity Journal Club for valuable discussions and suggestions. IB
was supported by the Natural Science and Engineering Research Council of
Canada Discovery Grant 2018-0473. The work of RAH was supported by the Natural
Science and Engineering Research Council of Canada through the Banting
Postdoctoral Fellowship program and also by AOARD Grant FA2386-19-1-4077.

## References

  * Hawking and Ellis (1973) S. W. Hawking and G. F. R. Ellis,  _The Large Scale Structure of Space-Time_, Cambridge Monographs on Mathematical Physics (Cambridge University Press, 1973).
  * Matzner  _et al._ (1995) R. A. Matzner, H. E. Seidel, Stuart L. Shapiro, L. Smarr, W. M. Suen, Saul A. Teukolsky, and J. Winicour, “Geometry of a black hole collision,” Science 270, 941–947 (1995).
  * Dreyer  _et al._ (2003) Olaf Dreyer, Badri Krishnan, Deirdre Shoemaker, and Erik Schnetter, “Introduction to isolated horizons in numerical relativity,” Phys. Rev. D 67, 024018 (2003), arXiv:gr-qc/0206008 .
  * Krishnan (2008) Badri Krishnan, “Fundamental properties and applications of quasi-local black hole horizons,” Class. Quant. Grav. 25, 114005 (2008), arXiv:0712.1575 [gr-qc] .
  * Gupta  _et al._ (2018) Anshu Gupta, Badri Krishnan, Alex Nielsen, and Erik Schnetter, “Dynamics of marginally trapped surfaces in a binary black hole merger: Growth and approach to equilibrium,” Phys. Rev. D97, 084028 (2018), arXiv:1801.07048 [gr-qc] .
  * Rezzolla  _et al._ (2010) Luciano Rezzolla, Rodrigo P. Macedo, and Jose Luis Jaramillo, “Understanding the ’anti-kick’ in the merger of binary black holes,” Phys. Rev. Lett. 104, 221101 (2010), arXiv:1003.0873 [gr-qc] .
  * Jaramillo  _et al._ (2012a) Jose Luis Jaramillo, Rodrigo Panosso Macedo, Philipp Moesta, and Luciano Rezzolla, “Black-hole horizons as probes of black-hole dynamics I: post-merger recoil in head-on collisions,” Phys. Rev. D 85, 084030 (2012a), arXiv:1108.0060 [gr-qc] .
  * Jaramillo  _et al._ (2012b) J. L. Jaramillo, R. P. Macedo, P. Moesta, and L. Rezzolla, “Towards a cross-correlation approach to strong-field dynamics in Black Hole spacetimes,” AIP Conf. Proc. 1458, 158–173 (2012b), arXiv:1205.3902 [gr-qc] .
  * Prasad  _et al._ (2020) Vaishak Prasad, Anshu Gupta, Sukanta Bose, Badri Krishnan, and Erik Schnetter, “News from horizons in binary black hole mergers,” Phys. Rev. Lett. 125, 121101 (2020), arXiv:2003.06215 [gr-qc] .
  * Ben-Dov (2004a) Ishai Ben-Dov, “The Penrose inequality and apparent horizons,” Phys. Rev. D 70, 124031 (2004a), arXiv:gr-qc/0408066 .
  * Booth  _et al._ (2006) Ivan Booth, Lionel Brits, Jose A. Gonzalez, and Chris Van Den Broeck, “Marginally trapped tubes and dynamical horizons,” Class. Quant. Grav. 23, 413–440 (2006), arXiv:gr-qc/0506119 .
  * Hayward (2000) Sean A. Hayward, “Black holes: New horizons,” in  _Recent developments in theoretical and experimental general relativity, gravitation and relativistic field theories. Proceedings, 9th Marcel Grossmann Meeting, MG’9, Rome, Italy, July 2-8, 2000. Pts. A-C_ (2000) pp. 568–580, arXiv:gr-qc/0008071 [gr-qc] .
  * Ben-Dov (2004b) Ishai Ben-Dov, “Penrose inequality and apparent horizons,” Phys. Rev. D 70, 124031 (2004b).
  * Pook-Kolb  _et al._ (2019a) Daniel Pook-Kolb, Ofek Birnholtz, Badri Krishnan, and Erik Schnetter, “Existence and stability of marginally trapped surfaces in black-hole spacetimes,” Phys. Rev. D 99, 064005 (2019a).
  * Pook-Kolb  _et al._ (2019b) Daniel Pook-Kolb, Ofek Birnholtz, Badri Krishnan, and Erik Schnetter, “Interior of a binary black hole merger,” Phys. Rev. Lett. 123, 171102 (2019b).
  * Pook-Kolb  _et al._ (2019c) Daniel Pook-Kolb, Ofek Birnholtz, Badri Krishnan, and Erik Schnetter, “Self-intersecting marginally outer trapped surfaces,” Phys. Rev. D 100, 084044 (2019c).
  * Pook-Kolb  _et al._ (2020a) Daniel Pook-Kolb, Ofek Birnholtz, José Luis Jaramillo, Badri Krishnan, and Erik Schnetter, “Horizons in a binary black hole merger I: Geometry and area increase,” (2020a), arXiv:2006.03939 [gr-qc] .
  * Pook-Kolb  _et al._ (2020b) Daniel Pook-Kolb, Ofek Birnholtz, José Luis Jaramillo, Badri Krishnan, and Erik Schnetter, “Horizons in a binary black hole merger II: Fluxes, multipole moments and stability,” (2020b), arXiv:2006.03940 [gr-qc] .
  * Booth  _et al._ (2020) Ivan Booth, Robie A. Hennigar, and Saikat Mondal, “Marginally outer trapped surfaces in the Schwarzschild spacetime: Multiple self-intersections and extreme mass ratio mergers,” Phys. Rev. D 102, 044031 (2020), arXiv:2005.05350 [gr-qc] .
  * Čadež (1974) Andrej Čadež, “Apparent horizons in the two-black-hole problem,” Annals of Physics 83, 449 – 457 (1974).
  * Booth  _et al._ (2021) Ivan Booth, Robie A. Hennigar, and Daniel Pook-Kolb, “Ultimate fate of apparent horizons during a binary black hole merger I: locating and understanding axisymmetric marginally outer trapped surfaces,” (2021).
  * Pook-Kolb  _et al._ (2021a) Daniel Pook-Kolb, Ofek Birnholtz, Ivan Booth, Robie A. Hennigar, José Luis Jaramillo, Badri Krishnan, Erik Schnetter, and Victor Zhang, “MOTS Finder version 1.5,”  (2021a).
  * Brill and Lindquist (1963) Dieter R. Brill and Richard W. Lindquist, “Interaction energy in geometrostatics,” Phys. Rev. 131, 471–476 (1963).
  * Löffler  _et al._ (2012) Frank Löffler, Joshua Faber, Eloisa Bentivegna, Tanja Bode, Peter Diener, Roland Haas, Ian Hinder, Bruno C. Mundim, Christian D. Ott, Erik Schnetter, Gabrielle Allen, Manuela Campanelli, and Pablo Laguna, “The Einstein Toolkit: A Community Computational Infrastructure for Relativistic Astrophysics,” Class. Quantum Grav. 29, 115001 (2012), arXiv:1111.3344 [gr-qc] .
  * (25) EinsteinToolkit, “Einstein Toolkit: Open software for relativistic astrophysics,” http://einsteintoolkit.org/.
  * Ansorg  _et al._ (2004) Marcus Ansorg, Bernd Brügmann, and Wolfgang Tichy, “A single-domain spectral method for black hole puncture data,” Phys. Rev. D 70, 064011 (2004), arXiv:gr-qc/0404056 .
  * Brown  _et al._ (2009) J. David Brown, Peter Diener, Olivier Sarbach, Erik Schnetter, and Manuel Tiglio, “Turduckening black holes: an analytical and computational study,” Phys. Rev. D 79, 044023 (2009), arXiv:0809.3533 [gr-qc] .
  * Husa  _et al._ (2006) Sascha Husa, Ian Hinder, and Christiane Lechner, “Kranc: a Mathematica application to generate numerical codes for tensorial evolution equations,” Comput. Phys. Commun. 174, 983–1004 (2006), arXiv:gr-qc/0404023 .
  * (29) Kranc, “Kranc: Kranc assembles numerical code,” http://kranccode.org/.
  * Alcubierre  _et al._ (2000) Miguel Alcubierre, Gabrielle Allen, Bernd Brügmann, Thomas Dramlitsch, Jose A. Font, Philippos Papadopoulos, Edward Seidel, Nikolaos Stergioulas, Wai-Mo Suen, and Ryoji Takahashi, “Towards a stable numerical evolution of strongly gravitating systems in general relativity: The Conformal treatments,” Phys. Rev. D62, 044034 (2000), arXiv:gr-qc/0003071 [gr-qc] .
  * Alcubierre  _et al._ (2003) Miguel Alcubierre, Bernd Brügmann, Peter Diener, Michael Koppitz, Denis Pollney, Edward Seidel, and Ryoji Takahashi, “Gauge conditions for long term numerical black hole evolutions without excision,” Phys. Rev. D67, 084023 (2003), arXiv:gr-qc/0206072 [gr-qc] .
  * Pook-Kolb  _et al._ (2021b) Daniel Pook-Kolb, Ivan Booth, and Robie A. Hennigar, “Ultimate fate of apparent horizons during a binary black hole merger II: Horizons weaving back and forth in time,” (2021b).
  * Andersson  _et al._ (2005) Lars Andersson, Marc Mars, and Walter Simon, “Local existence of dynamical and trapping horizons,” Phys.Rev.Lett. 95, 111102 (2005), arXiv:gr-qc/0506013 [gr-qc] .
  * Andersson  _et al._ (2008) Lars Andersson, Marc Mars, and Walter Simon, “Stability of marginally outer trapped surfaces and existence of marginally outer trapped tubes,” Adv.Theor.Math.Phys. 12 (2008), arXiv:0704.2889 [gr-qc] .
  * Andersson  _et al._ (2009) Lars Andersson, Marc Mars, Jan Metzger, and Walter Simon, “The Time evolution of marginally trapped surfaces,” Class. Quant. Grav. 26, 085018 (2009), arXiv:0811.4721 [gr-qc] .


# What You See is What You Classify:
Black Box Attributions

Steven Stalder
Swiss Data Science Center
ETH Zurich, Switzerland
&Nathanaël Perraudin
Swiss Data Science Center
ETH Zurich, Switzerland
&Radhakrishna Achanta
Swiss Data Science Center
EPFL, Switzerland
&Fernando Perez-Cruz
Swiss Data Science Center
ETH Zurich, Switzerland
&Michele Volpi
Swiss Data Science Center
ETH Zurich, Switzerland

###### Abstract

An important step towards explaining deep image classifiers lies in the
identification of image regions that contribute to individual class scores in
the model’s output. However, doing this accurately is a difficult task due to
the black-box nature of such networks. Most existing approaches find such
attributions either using activations and gradients or by repeatedly
perturbing the input. We instead address this challenge by training a second
deep network, the _Explainer_ , to predict attributions for a pre-trained
black-box classifier, the _Explanandum_. These attributions are in the form of
masks that only show the classifier-relevant parts of an image, masking out
the rest. Our approach produces sharper and more boundary-precise masks when
compared to the saliency maps generated by other methods. Moreover, unlike
most existing approaches, ours is capable of directly generating very distinct
class-specific masks. Finally, the proposed method is very efficient for
inference since it only takes a single forward pass through the _Explainer_ to
generate all class-specific masks. We show that our attributions are superior
to established methods both visually and quantitatively, by evaluating them on
the PASCAL VOC-2007 and Microsoft COCO-2014 datasets.

##  1 Introduction

Image recognition and classification systems based on deep learning are
considered black boxes. It is often hard, if not impossible, to directly
locate the portions of the input image responsible for a given classification
output. Being able to systematically perform such attributions is a crucial
step towards explainable deep learning, which can contribute to its widespread
adoption. Attribution allows understanding the potential errors and biases
[13, 17, 21]. It also enables the use of deep networks as decision support
tools, _e.g_. in medicine [15], and facilitating meeting legal regulations
[8]. Note that we refer to attribution as the goal of associating input image
regions to model decisions, which differs from the notion of interpretability
that relates to models that can be directly used to infer causal or general
structural relationships that link inputs and outputs [23].

| Bicycle | Motorbike | Person | Cat | Dog | Bicycle | Motorbike | Person | Cat | Dog
---|---|---|---|---|---|---|---|---|---|---
GCam |  |  |  |  |  |  |  |  |  |
EP |  |  |  |  |  |  |  |  |  |
Ours |  |  |  |  |  |  |  |  |  |

Figure 1: Visual comparison of _per-class_ attributions provided for VOC-2007
by our _Explainer_ , alongside Grad-CAM (GCam) [24] and Extremal Perturbations
(EP) [12] for the VGG-16 architecture [25]. Our attributions have sharper
boundaries and at the same time are more class-accurate than Grad-CAM or EP.
Attributions for only five out of the twenty VOC-2007 classes are shown for
convenience. Colormap ranges from low (blue) to high (red) saliency.

The problem of performing attribution for deep image classifiers has been
addressed by several recent works. These belong to two broad categories. The
first relies on visualising the derived model’s activations and backpropagated
signals [31, 24, 3]. Although these methods provide a direct way to visualize
salient areas, they heavily depend on the quality of the trained models’
architectures and weights. Furthermore, some of these methods turn out to be
independent of the labels the model has been trained on, as well as to some
extent the parameters of the models, as shown through randomization tests [1].
This suggests that many of these methods are strongly influenced by low-level,
class-agnostic features and are therefore inadequate. The second category
relies on local perturbations of the input images and is often model-agnostic.
These models rely on masking strategies to infer local feature importance,
which often results in loss of visual detail, ambiguity and high computational
cost. Nonetheless, results suggest that local perturbations, like masking, are
an effective way to analyze and explain the response of a model [20, 22].

This last line of reasoning posits that attribution can be seen as a task
independent of the _Explanandum_ (_i.e_. the model to be explained).
Therefore, it is natural to consider learning an _Explainer_ (_i.e_. the model
that explains) that specifically addresses the task of attribution. We
approach attribution as a supervised meta-learning problem through the use of
a complementary convolutional neural network, which learns to produce dense
class-specific attribution masks, balancing preservation and deletion of
information from the input image as dictated by the _Explanandum_.

The resulting _Explainer_ efficiently masks and precisely locates regions of
the input image that are responsible for the classification result of a pre-
trained and frozen _Explanandum_ (see Fig. 1 and Sup. Mat. Sec. C). We
evaluate our methodology on PASCAL Visual Object Classes (VOC-2007) [9] and
Microsoft Common Objects in Context (COCO-2014) [18]. To sum up, we present an
efficient attribution method, which has the following advantages:

  * •

Our _Explainer_ provides a precise mask for each class label as in Fig. 1.

  * •

Our _Explainer_ provides attributions for test images with a single very
efficient forward pass, orders of magnitude more efficient as compared to
other perturbation-based approaches.

  * •

At inference time, our _Explainer_ does not need to access the _Explanandum_.

##  2 Related Work

#### Activation and gradient-based methods.

The methods in this category attempt to explain a trained model by leveraging
its weights and its activations for a given image to be explained. Zhou et al.
[31] initiated this line of research with Class Activation Mapping (CAM),
where the authors built particular architectures around a notion of global
average pooling and illustrated the intrinsic localization performed by a
classification network. Grad-CAM [24] removed the dependency of CAM on
specific architectures by reconstructing the input heatmap through a weighted
average of gradients, rather than directly projecting back activations.
Further extensions to this are Grad-CAM++ [3], which is CAM weighted by
averaged second order gradients, Score-CAM [29], which is CAM weighted by
layer-wise class activations, or Ablation-CAM [7], which is CAM weighted by
neuron ablation. Other extensions aim to cope with the issues arising from the
high sensitivity to noise and clipping effects of gradients [1].

Some methods directly engineer the backpropagation and network architectures
in order to retrieve pixel-level saliency maps. One such example is the
deconvolution approach of Zeiler and Fergus [30], where the authors propose to
revert forward propagation (mapping activations to pixels instead of mapping
pixels to activations) to visualize features and functions of intermediate
layers. From this, Springenberg et al. [26] derived guided backpropagation.
Although the focus of their work was on developing an efficient fully
convolutional neural network, they show how by combining traditional
backpropagation and deconvolutions, one can retrieve salient regions of the
input for a specific class. In another line of work, Wagner et al. [28] frame
the problem as a defense against adversarial perturbations and tackle it by
constraining and selectively pruning activations.

#### Local perturbations and local models.

The idea behind these methods is that by perturbing the input to the trained
network, one can retrieve an explanation signal by summarizing changes in the
classification scores. RISE introduced a way of retrieving a saliency heatmap
based on random perturbations of the input image [20] and recently, [27]
extended a similar reasoning for object detection systems. The input image is
randomly masked several times, changes in the model scores tracked, and
aggregated by a weighted average. LIME proposes a similar setup, but instead
uses random switches on unsupervised segmentation [22].

Fong and Vedaldi [11] extend these ideas by reformulating the attribution
problem as a meta-learning task, where the optimal mask can be found by
optimizing a score. Fong et al. [12] extend their previous work [11] through
the definition of _extremal perturbations_ , which are the family of
transformations that maximally affect a classification score. They propose to
use Gaussian blur to assess perturbations over a fixed set of mask areas.
Dabkowski and Gal [5] rely on a similar intuition, and the _Explainer_ is
directly optimized based on meta-learning around the _Explanandum_. In these
settings, the _Explainer_ optimizes a loss which balances preservation of the
classification score on the active parts of the image, while minimizing the
score for areas that are not active. These works also make use of
regularization to constrain the mask area and its smoothness. In a similar
fashion, Chang et al. [2] learn a pixel-specific dropout as a Bernoulli
distribution, which is used to retrieve a mask and evaluate extremal
perturbations. Unlike Fong et al. [12], Chang et al. [2] return the parameters
of the learned per-pixel Bernoulli distribution as the saliency mask.
Recently, Khorram et al. [16] proposed iGOS++, which frames attribution as
optimizing over a mask which optimally balances preservation of useful
information and removal of unnecessary information, regularized by bilateral
total variation.

In this paper, we propose a method which learns a dense perturbation, the
attribution mask, over input images. Unlike existing approaches, we optimize
to balance low multi-label binary cross entropy for the unmasked portion of
the input image, and high entropy for areas of the image masked out by the
_Explainer_. Compared to most perturbation-based methods, our _Explainer_ can
predict attribution masks for all classes (and not only the predicted class)
in a single forward pass, thereby being orders of magnitude faster. In
addition, we beat state-of-the-art on standard benchmarks.

##  3 Training the _Explainer_

In this section, we present our _Explainer_ architecture, which is trained to
detect image regions that a given pre-trained classifier, the _Explanandum_ ,
relies on to infer semantic object classes. Training of our model does _not_
require any pixel-wise annotations. It relies on the _Explanandum_ ’s training
data (including labels) and model. This makes our technique generalizeable to
any _Explanandum_ architecture through which we can back-propagate the
classification loss.

As illustrated in Fig. 2, the _Explainer_ sees an image and outputs a set of
masks $\mathbf{S}$ of the same resolution, containing one segmentation mask
$\mathbf{s}_{c}$ for each class $c\in C$. The values of the class masks
$\mathbf{s}_{c}$ are bounded in the range $[0,1]$ through a sigmoid function
$\sigma(t)=e^{t}/(1+e^{t})$, where $t$ represents the predicted pixel-wise
logits. Values of $1$ or $0$ in the mask result in the full preservation or
deletion of the corresponding pixel value, respectively. Note that the sum of
the class masks does not need to be equal to one.

Using the target label(s) for the input image, we split the class segmentation
masks into two sets: one that consists of masks that correspond to any ground
truth class appearing in the training image, while the other contains the
remaining ones. We merge each set into a single mask by taking the pixel-wise
maximum value for each pixel position. The target mask $\mathbf{m}$ (obtained
from the first set) serves to locate the regions corresponding to any of the
label(s) contained in a given training image, while the non-target mask
$\mathbf{n}$ (obtained from the second set) allows us to gather false positive
activations. Finally, the inverted mask $\tilde{\mathbf{m}}=1-\mathbf{m}$ is
used to ensure that once the object of interest is removed, the _Explanandum_
cannot address the task anymore. Note that during inference, the mask
aggregation step is not needed and our _Explainer_ provides one explanation
mask for each class.

Figure 2: Overview of our method. Given a pre-trained classifier $\mathcal{F}$
whose weights are frozen, the _Explainer_ network $\mathcal{E}$ learns to
produce masks $\mathbf{s}_{c}$ for each class. The masks corresponding to the
label(s) associated with the input image (shown in green) are merged by taking
the pixel-wise maximum over masks (shown as $\uparrow$’s), to obtain a _target
mask_ $\mathbf{m}$ and its complement $\tilde{\mathbf{m}}$ (_i.e_. inverted
mask). All the other masks (shown in red), which do not correspond to the
labels of the input image but might still score positively for the given
image, are merged separately to obtain the _non-target mask_ $\mathbf{n}$,
which is also used in the loss term. The images obtained by multiplying the
target mask and its complement with the input image (shown by $\times$’s) are
fed to the given pre-trained classifier separately, generating two outputs on
which we compute losses. The set of per-class masks $\mathbf{S}$ and the
aggregated _target mask_ $\mathbf{m}$ serve as the attributions provided by
our _Explainer_.

###  3.1 Training loss

We formulate the loss as a combination of four terms, which reflect the
desired conditions:

$\mathcal{L}_{E}(\mathbf{x},\mathcal{Y},\mathbf{S},\mathbf{m},\mathbf{n})=\mathcal{L}_{c}(\mathbf{x},\mathcal{Y},\mathbf{m})+\lambda_{e}\mathcal{L}_{e}(\mathbf{x},\tilde{\mathbf{m}})+\lambda_{a}\mathcal{L}_{a}(\mathbf{m},\mathbf{n},\mathbf{S})+\lambda_{tv}\mathcal{L}_{tv}(\mathbf{m},\mathbf{n})\textnormal{,}$
(1)

where $\mathcal{L}_{c}(\mathbf{x},\mathcal{Y},\mathbf{m})$ is the binary
cross-entropy loss over the masked image,
$\mathcal{L}_{e}(\mathbf{x},\tilde{\mathbf{m}})$ is an entropy term over the
image masked with the complement of $\mathbf{m}$,
$\mathcal{L}_{a}(\mathbf{m},\mathbf{n},\mathbf{S})$ is a regularization term
accounting for the area of the true and false positive masks, and
$\mathcal{L}_{tv}(\mathbf{m},\mathbf{n})$ is a regularization term favouring
smooth masks. $\lambda_{e},\lambda_{a}$ and $\lambda_{tv}$ are hyperparameters
balancing the loss terms.

#### Classification loss:
$\mathcal{L}_{c}(\mathbf{x},\mathcal{Y},\mathbf{m})$.

A model should be able to make correct decisions exclusively using the
relevant portions of the image, ignoring all the rest. Under this assumption,
we define the classification loss as a sum over binary cross-entropy terms,
one for each class contained in the image.

Representing the _Explainer_ as $\mathcal{E}$ and the frozen _Explanandum_ as
$\mathcal{F}$, we define $\mathbf{p}=\mathcal{F}(\mathbf{x}\odot\mathbf{m})$
as the probability vector returned by the model $\mathcal{F}$ applied on the
masked image, using the aggregated (element-wise maximum) mask $\mathbf{m}$
provided by $\mathcal{E}$. For all classes $C$ in the training data and the
set of target classes $\mathcal{Y}$ in the image, we compute:

$\mathcal{L}_{c}(\mathbf{x},\mathcal{Y},\mathbf{m})=-\frac{1}{C}\sum_{c=1}^{C}\llbracket
c\in\mathcal{Y}\rrbracket\log\left(\mathbf{p}[c]\right)+\llbracket
c\not\in\mathcal{Y}\rrbracket\log\left(1-\mathbf{p}[c]\right),$ (2)

where $\llbracket c\in\mathcal{Y}\rrbracket$ denotes the Iverson bracket,
returning $1$ if the condition is true (the image contains class $c$), $0$
otherwise. This allows training a mask when multiple classes are present in
the training image, as in multi-label classification problems, where multiple
masks are active at the same time and some pixels are free not to belong to
any mask (contrarily to using cross-entropy). Therefore, the model is free to
learn dependencies and co-occurrencies between different classes. This loss
pushes the _Explainer_ to learn masks approximating
$\mathcal{F}(\mathbf{x}\odot\mathbf{m})\approx\mathcal{F}(\mathbf{x})$, _i.e_.
masked images are classified as correctly as possible in the eyes of the
(pre-)trained _Explanandum_.

#### Negative entropy loss: $\mathcal{L}_{e}(\mathbf{x},\tilde{\mathbf{m}})$.

This part of the loss pushes the _Explainer_ to provide masks whose
complements do not contain any discriminative visual cue, _i.e_. parts of the
image that the _Explanandum_ could use to infer the correct class. In other
words, the classifier scores on the mask complement should provide class
probabilities
$\tilde{\mathbf{p}}=\mathcal{F}(\mathbf{x}\odot\tilde{\mathbf{m}})$ as uniform
as possible. To this end, we aim at minimizing the negative entropy of the
predictions over the class memberships, as:

$\mathcal{L}_{e}(\mathbf{x},\tilde{\mathbf{m}})=\frac{1}{C}\sum_{c=1}^{C}\tilde{\mathbf{p}}[c]\log\tilde{\mathbf{p}}[c]$
(3)

Note that this loss is the negative entropy, as we aim for a high entropy for
the background.

#### Area loss: $\mathcal{L}_{a}(\mathbf{m},\mathbf{n},\mathbf{S})$.

With the terms $\mathcal{L}_{c}$ and $\mathcal{L}_{e}$ alone, our _Explainer_
would have no incentive to produce a target mask that conceals image areas.
Clearly, a target mask $\mathbf{m}$ with 1 everywhere would minimize these
terms with respect to $\mathbf{m}$. To make sure $\mathbf{m}$ conceals
background, we add two terms to the loss that correspond to two crucial
desiderata: The mask should be as small as possible, but if the objective
would favour larger areas, a whole range of areas in between a minimal and
maximal percentage should not be penalized further.

The regularization over the area is simply formulated as the mean of the mask
values, computed as
$\mathcal{A}(\mathbf{m})=\frac{1}{Z}\sum_{i,j}\mathbf{m}[i,j]$, where $Z$ is
the number of pixels. We want this area to be small, but non-zero. The same is
required of the non-target mask $\mathbf{n}$ as well.

The term bounding the mask area from both above and below is formulated
starting from the constraint presented by Fong et al. [12]. However, instead
of constraining the final image mask to a fixed area, we extend their
technique to determine a valid minimum area $a$ and a maximum area $b$ for
those class segmentation masks that correspond to at least one object in the
image.

Given such a class segmentation mask $\mathbf{s}_{c}$, we flatten it to a one-
dimensional vector, and then sort its values in descending order. Let
$v(\mathbf{s}_{c})$ be the vectorization operation and $r(v(\mathbf{s}_{c}))$
be the sorting of the vectorized class mask $\mathbf{s}_{c}$. We then define
two vectors $\mathbf{q}_{\text{min}}$ and $\mathbf{q}_{\text{max}}$, each
starting with a predefined number of $1$’s and padded with $0$’s to match the
length of $v(\mathbf{s}_{c})$, where the amount of $1$ corresponds to the
minimum $0<a<b$ or maximum $a<b<1$ area, respectively. We also define
$\mathbf{q}_{\text{mask}}$, which corresponds to $\mathbf{s}_{c}$, vectorized,
and sorted. Concretely:

$\mathbf{q}_{\text{mask}}=r\left(v\left(\mathbf{s}_{c}\right)\right)\in[0,1]^{Z},\
\ \ \mathbf{q}_{\text{min}}=[\mathbf{1}^{\lfloor
Za\rfloor}\ldots\mathbf{0}^{\lfloor Z(1-a)\rfloor}],\ \ \
\mathbf{q}_{\text{max}}=[\mathbf{1}^{\lfloor
Zb\rfloor}\ldots\mathbf{0}^{\lfloor Z(1-b)\rfloor}].$ (4)

To constrain the minimum and maximum area covered by the class segmentation
mask, we then compute an area bounding measure $\mathcal{B}$ given by:

$\mathcal{B}(\mathbf{s})=\frac{1}{Z}\sum_{i}^{Z}\max(\mathbf{q}_{\text{min}}[i]-\mathbf{q}_{\text{mask}}[i],0)\
+\frac{1}{Z}\sum_{i}^{Z}\max(\mathbf{q}_{\text{mask}}[i]-\mathbf{q}_{\text{max}}[i],0).$
(5)

where the first term penalizes the mask only if the largest $\lfloor
Za\rfloor$ values are smaller than $1$, while the second term penalizes the
smallest $Z-\lfloor Zb\rfloor$ values greater than $0$. Our goal is for the
mask to cover _at least_ a certain area independently of its maximum size.

The final area regularization is therefore:

$\mathcal{L}_{a}(\mathbf{m},\mathbf{n},\mathbf{S})=\mathcal{A}(\mathbf{m})+\mathcal{A}(\mathbf{n})+\frac{\sum_{c=1}^{C}\llbracket
c\in\mathcal{Y}\rrbracket\mathcal{B}(\mathbf{s}_{c})}{\sum_{c=1}^{C}\llbracket
c\in\mathcal{Y}\rrbracket}$ (6)

where $\mathcal{Y}$ is again the subset of the classes that are present in the
given input image according to the given training labels. Note that the last
term of (6) used in conjunction with the smoothness loss presented below
favors masks with sharp transitions between $0$ and $1$.

#### Smoothness Loss: $\mathcal{L}_{tv}(\mathbf{m},\mathbf{n})$.

As a final requirement, we are interested in generating masks that appear
smooth and free of artifacts. For this purpose, we utilize a total variation
(TV) loss on the masks as used by Fong and Vedaldi [11] and Dabkowski and Gal
[5], albeit we use the anisotropic variant given by:

$\mathcal{TV}(\mathbf{x})=\frac{1}{Z}\sum_{i,j}\left\lvert\mathbf{x}[i,j]-\mathbf{x}[i+1,j]\right\rvert+\left\lvert\mathbf{x}[i,j]-\mathbf{x}[i,j+1]\right\rvert.$
(7)

where $\mathbf{x}$ is a two-dimensional array of values. The smoothness term
of the loss relates to the target mask $\mathbf{m}$ and non-target mask
$\mathbf{n}$ as:

$\mathcal{L}_{tv}(\mathbf{m},\mathbf{n})=\mathcal{TV}(\mathbf{m})+\mathcal{TV}(\mathbf{n})$
(8)

This smoothness term encourages our attributions to appear visually coherent.

##  4 Experiments

We present three results to establish the advantages of our _Explainer_ over
existing approaches. Although continuous in nature, our masks show a sharp
transition from background to foreground, making it effectively binary.
Additionally, our masks adhere tightly to region boundaries. Since common
objective metrics may fail to convey this, we provide a brief visual
comparison. The next result shows how our masks are class-specific, which
provides more interpretive power than combined masks. The third result shows
that our combined masks outperform existing methods on standard metrics for
semantic segmentation.

Since we expect our _Explainer_ to learn class-specific semantics and generate
class-specific masks of the same resolution as the input image, we employ
DeepLabV3 [4] with a ResNet-50 backbone, an architecture commonly used for
semantic segmentation tasks. We train this model from scratch to avoid any
bias, _i.e_. an information leakage from the semantic segmentation dataset
from which DeepLabV3 has been pre-trained on.

For the _Explanandum_ , we use VGG-16 [25] and ResNet-50 [14] models, both
pretrained on ImageNet [6]. We fine-tune them on the multi-label
classification tasks for VOC-2007 and COCO-2014, and then freeze the models.
To this end, we use the full training set of VOC-2007 and 90% of the COCO-2014
training set for fine-tuning. To assess generalization, we use the test set of
VOC-2007 and the validation set of COCO-2014, respectively. For choosing
hyperparameters, we use the VOC-2007 validation set and the remaining 10% of
the COCO-2014 training set. In this work, we implemented models and
experiments using PyTorch [19] and PyTorch Lightning [10]111Code is available
at: https://github.com/stevenstalder/NN-Explainer.

###  4.1 Visual Comparison

ID |  Input | GCam | RISE | EP | iGOS | RTIS | Ours | ID |  Input | GCam | RISE | EP | iGOS | RTIS | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
1 |  |  2 |
3 |  |  4 |
5 |  |  6 |
7 |  |  8 |
9 |  |  10 |
11 |  |  12 |
13 |  |  14 |

Figure 3: Visual comparison of our attributions for a VGG-16 network fine-
tuned on images from the VOC-2007 dataset and frozen. Our attributions are
much more effective at retaining object class regions and discarding the rest.
Examples 13 and 14 show cases where our _Explainer_ is inaccurate.

Visually, our _Explainer_ leads to sharp and meaningful masks. Fig. 3 provides
a visual comparison of how our _Explainer_ attributions perform compared to
the saliency maps obtained from Grad-CAM (GCam) [24], RISE [20], Extremal
Perturbations (EP) [12], iGOS++ (iGOS) [16], and Real-Time Image Saliency
(RTIS) [5]. For the RTIS baseline, we could not retrain their model on
VOC-2007 and COCO-2014, as it was not possible to employ their architecture on
multi-class classifiers. Instead, we use their loss to train our architecture.
Accordingly, we fine-tuned some hyperparameters of our RTIS implementation,
while leaving the remaining ones at the default values as given in their
paper. For all our experiments, we resized input images to 224x224 pixels and
normalized them on the mean and standard deviation of ImageNet.

Our approach provides sharp masks, which contrasts with classical saliency
methods such as Grad-CAM or RISE. EP and iGOS++ provide masks which are better
localized, although not as sharp as the ones produced by RTIS or our
_Explainer_. Our implementation of RTIS provides sharp masks, but the overall
attribution tends to be leaking over object borders. Our area constraints, in
combination with the entropy term over the mask complement
$\tilde{\mathbf{m}}$, enforce tight masks over object classes of different
sizes. For instance, Fig. 3(2), Fig. 3(5), or Fig. 3(8) show how our masks
align much more clearly with the objects. This not only holds for single-class
images, but also when we have multiple instances of the same class (Fig. 3(1),
Fig. 3(4), Fig. 3(12)), or different classes (Fig. 3(6), Fig. 3(9), Fig.
3(10)). Attribution of some animals (_e.g_. Fig. 3(7), Fig. 3(10), Fig. 3(11))
often tends to be focusing on heads. Most training images containing animals
have unobstructed heads, which can explain such behaviour. For some classes,
all methods agree on the importance of an image region that is not actually
part of the actual object. A good example for that is Fig. 3(3), where the
train tracks are part of the attribution. Since most training images of trains
also show train tracks, it seems obvious that the _Explanandum_ learns to see
the train tracks as discriminative semantic co-occurrence. With the help of an
_Explainer_ , such unintended biases can quickly be detected.

Fig. 3(13) and Fig. 3(14) show two examples where our _Explainer_ is less
precise. For Fig. 3(13), our attribution leaks over to the bars below the
horse. Even though the classifier might indeed have made a connection between
these bars and the “horse” class, Grad-CAM, RISE, EP, and iGOS++ appear to
agree that the main importance should be attributed to the actual objects in
the image. In Fig. 3(14), it is difficult to interpret our attribution for the
class “boat” as some parts of the attribution lie on the actual object but
also a significant area of the surrounding water is activated (underlining
potential discriminative co-occurrence). However, the other methods also
provide varying explanations, with iGOS++ masking mostly the boat, but Grad-
CAM attributing all importance to the water. For more results on COCO-2014 and
VOC-2007 please refer to Sup. Mat. Sec. D, E, and F.

###  4.2 Class-specific masking

Explainer | Mask
---|---
none | bottle | car | cat | dog | person
Target Class | bottle | 20.57 |  15.29 |  4.91 |  4.82 |  4.82 |  2.72
car | 70.52 |  6.71 |  62.92 |  6.67 |  6.66 |  7.11
cat | 72.78 |  6.09 |  6.18 |  60.55 |  6.33 |  5.74
dog | 58.00 |  4.67 |  4.70 |  5.93 |  57.60 |  4.58
person | 69.37 |  8.90 |  6.18 |  5.83 |  6.57 |  71.01

Grad-CAM | Mask
---|---
none | bottle | car | cat | dog | person
Target Class | bottle | 20.57 |  16.10 |  5.24 |  4.85 |  4.76 |  3.61
car | 70.52 |  6.93 |  50.19 |  6.65 |  6.53 |  8.92
cat | 72.78 |  6.17 |  6.39 |  43.74 |  8.99 |  5.85
dog | 58.00 |  4.46 |  4.39 |  8.23 |  41.88 |  4.38
person | 69.37 |  11.81 |  8.18 |  5.90 |  8.75 |  56.38

RISE | Mask
---|---
none | bottle | car | cat | dog | person
Target Class | bottle | 20.57 |  18.67 |  10.94 |  12.05 |  10.70 |  9.73
car | 70.52 |  29.75 |  56.66 |  30.27 |  34.43 |  32.42
cat | 72.78 |  37.89 |  39.28 |  49.61 |  41.89 |  36.02
dog | 58.00 |  19.64 |  20.34 |  23.72 |  45.12 |  22.83
person | 69.37 |  56.06 |  53.39 |  47.33 |  54.81 |  63.52

RTIS | Mask
---|---
none | bottle | car | cat | dog | person
Target Class | bottle | 20.57 |  18.92 |  11.19 |  13.37 |  13.44 |  17.24
car | 70.52 |  41.25 |  67.02 |  40.69 |  31.87 |  54.31
cat | 72.78 |  45.84 |  36.27 |  72.62 |  59.21 |  54.16
dog | 58.00 |  41.37 |  34.76 |  57.59 |  61.22 |  58.76
person | 69.37 |  64.71 |  54.41 |  43.42 |  61.55 |  73.59

Table 1: Comparison of class-specific masks generated by our _Explainer_ ,
Grad-CAM [24], RISE [20], and adapted RTIS [5]. Images containing the target
class specified in the row are masked with the thresholded class masks
specified in the column and given to VGG-16 for classification. In any given
cell we report the mean softmax probabilities over all mask thresholds and for
every test image containing the target class. If the mask class equals the
target class, we want high scores, if it is different, we want low scores. We
additionally compute the softmax scores achieved for unmasked images (“none”
column) for reference.

Our _Explainer_ provides class-specific masks. In Fig. 1 we have already
hinted at how our _Explainer_ is able to provide more class-specific masks
than the other techniques. In this section we evaluate this numerically. To
this end, we have selected five target classes from the VOC-2007 dataset, for
which we evaluate how the class masks interact with the classification
confidence of the VGG-16 _Explanandum_. We have selected the classes “cat” and
“dog” as they are two animals that often occur in similar settings, the most
prevalent class “person” that can often be seen together with a variety of
other classes and often inside the fourth class “car”, as well as the “bottle”
class, to include a VOC-2007 class that is very difficult to detect for the
_Explanandum_. For each test image containing one of the target classes, we
first evaluate the classification performance of the _Explanandum_ on the
unmasked image (“none” column) and then on masked images where the image is
masked once with each of the class-specific masks. Specifically, for each of
these continuous masks we report the mean softmax probabilities for the target
class, after we threshold each mask over nine uniform values in $[0.1,0.9]$ at
$0.1$ intervals. Note that here we use the softmax instead of the sigmoid
function despite our multi-class setting since we want to evaluate the
activation confidence for a single class compared to all other (ideally
masked-out) classes. This can of course affect the scores for the completely
unmasked images where multiple object classes may be present.

In Tab. 1, we show the evaluations for our _Explainer_ , Grad-CAM, RISE and
RTIS. For EP and iGOS++, it was not feasible to conduct these experiments
since these methods take a very long time to generate masks compared to the
other ones (see Sup. Mat. Sec. A). We can clearly see how the masks produced
by our method allow the _Explanandum_ to retain very good classification
confidence for the target classes when the correct mask is taken but render it
completely uncertain when any other mask is put on the image. This is in stark
contrast to RISE and RTIS especially, which produce overall larger, less
precise masks. Even though the _Explanandum_ has high confidence after masking
the images with their corresponding correct masks, the same is still true when
any wrong mask is used. Only Grad-CAM achieves comparable performance to our
_Explainer_ but results in considerably worse scores for the correct masks for
4 out of 5 classes and still shows some increased confidence when the wrong
mask is put on the images of an often co-occurring target class (_e.g_. “cat”
and “dog”, or “person” and “car”).

###  4.3 Segmentation accuracy

Data |  Expl. |  Metric |  0 |  0.5 |  1 |  G.T. |  GCam [24] |  RISE [20] |  EP [12] |  iGOS++ [16] |  RTIS [5] |  Ours
---|---|---|---|---|---|---|---|---|---|---|---|---
VOC-2007 |  VGG |  $\uparrow$ Acc | 66.29 | 50.00 | 33.71 | 100 | 70.53 | 56.98 | 63.91 | 70.17 | 59.63 | 74.60
$\uparrow$ IoU | 0 | 23.42 | 33.71 | 100 | 28.78 | 28.62 | 34.95 | 24.05 | 39.94 | 38.04
$\downarrow$ Sal | -0.85 | -0.53 | 0.15 | -0.98 | -1.41 | -0.82 | -0.77 | -1.88 | -0.31 | -1.32
$\downarrow$ MAE | 33.71 | 50.00 | 66.29 | 0 | 29.47 | 43.02 | 36.09 | 29.83 | 40.37 | 25.40
ResNet |  $\uparrow$ Acc | 66.29 | 50.00 | 33.71 | 100 | 68.54 | 57.37 | 61.43 | 69.23 | 50.77 | 73.56
$\uparrow$ IoU | 0 | 23.42 | 33.71 | 100 | 30.75 | 28.42 | 34.88 | 24.25 | 36.46 | 34.23
$\downarrow$ Sal | -0.67 | -0.68 | -0.08 | -1.13 | -1.29 | -1.02 | -0.77 | -1.92 | -0.35 | -1.33
$\downarrow$ MAE | 33.71 | 50.00 | 66.29 | 0 | 31.46 | 42.63 | 38.57 | 30.77 | 49.23 | 26.44
COCO-2014 |  VGG |  $\uparrow$ Acc | 70.39 | 50.00 | 29.61 | 100 | 66.31 | 52.35 | 57.01 | 68.70 | 39.18 | 69.90
$\uparrow$ IoU | 0 | 20.87 | 29.61 | 100 | 25.63 | 25.50 | 29.23 | 20.35 | 31.16 | 31.59
$\downarrow$ Sal | -0.1 | 0.03 | 0.72 | -0.38 | -0.67 | -0.31 | -0.13 | -1.12 | 0.65 | -0.51
$\downarrow$ MAE | 29.61 | 49.36 | 70.39 | 0 | 33.20 | 47.15 | 42.55 | 30.89 | 60.76 | 29.83
ResNet |  $\uparrow$ Acc | 70.39 | 50.00 | 29.61 | 100 | 67.25 | 55.05 | 56.92 | 69.57 | 41.40 | 65.30
$\uparrow$ IoU | 0 | 20.87 | 29.61 | 100 | 24.84 | 25.98 | 29.75 | 21.63 | 30.76 | 27.42
$\downarrow$ Sal | 0.2 | -0.25 | 0.37 | -0.88 | -0.66 | -0.44 | -0.24 | -1.19 | 0.17 | -0.64
$\downarrow$ MAE | 29.61 | 49.36 | 70.39 | 0 | 32.36 | 44.44 | 42.68 | 30.02 | 58.46 | 34.35

Table 2: Segmentation scores for the attribution models. We have compared the
methods against the pixel-wise segmentation ground truths of all 210 VOC-2007
test images that include such annotations as well as for a randomly selected
subset of 1000 COCO-2014 images out of our entire test set of over 41K images
(due to long compute times of certain methods, see Sup. Mat. Sec. A). Columns
“0”, “0.5”, “1” and “G.T.” (ground truth) represent the sanity check baselines
to allow for a relative understanding of the scores. The upward arrows
$\uparrow$ (downward arrows $\downarrow$) indicate that higher (lower) values
are better.

Our attribution quality approaches object segmentation. While our _Explainer_
architecture provides excellent attributions visually, we also confirm this
quantitatively. Our explanations approximate object-level segmentation and can
thus be evaluated using standard instance and semantic segmentation
benchmarks. In Tab. 2 we illustrate the accuracy of the masking strategies, on
VOC-2007 and COCO-2014, with VGG-16 and ResNet-50 as _Explanandum_ ,
respectively. To provide a fair comparison with our method, we evalaute all
methods by aggregating the masks from multiple classes into a single one, as
shown in Fig. 3. We also provide four base measures that serve as sanity check
baselines in Tab. 2), _i.e_. the columns “0”, “0.5”, “1”, and “G.T.” (ground
truth). These measures show how the assessment metrics range under border
conditions: with a completely null mask (composed of only $0$’s), with a mask
composed fully of $0.5$’s, with a non-mask composed of only $1$’s and finally
with a mask which is perfectly equal to the ground truth segmentation.

We employ four different metrics, namely Accuracy (Acc), Mean IoU (IoU),
Saliency (Sal) and Mean Absolute Error (MAE). All metrics are computed on a
per-image basis, by using the ground truth segmentation masks and the
aggregated mask of the true classes present in each image, then averaged over
the test set. The Accuracy is computed pixel-wise over the continuous masks
without thresholding. The same holds for the IoU, where we take the pixel-wise
minimum of the mask and the ground truth as the intersection and the maximum
as the union, respectively. The Saliency score is adapted from Dabkowski and
Gal [5], and computed as:

$\text{Sal}(\mathbf{m},p)=\log\left(\max(\mathcal{A}(\mathbf{m}),0.05)\right)-\log\left(\sum_{c\in\mathcal{Y}}\mathbf{p}[c]\right),$
(9)

where $\mathcal{Y}$ is the set of classes present in the image. This metric
balances the size of the mask $\mathbf{m}$ with the _Explanandum_ outputs.
Smaller masks minimize the first term but have a tendency to lower the
classification confidence, thus increasing the second term. It is therefore
possible to get a lower score compared to the ground truth mask as a very
small mask reduces the first term greatly, while it might still preserve a lot
of the classification confidence. This is to the disadvantage of our method,
as we aim to also attribute regions of lower (but still significant)
importance instead of only the most distinguishing features. Finally, the MAE
corresponds to the mean absolute error over the full dataset, which is
computed for each input image as:
$\text{MAE}(\mathbf{m},\mathbf{g})=\frac{1}{Z}\sum_{i,j}\left|\mathbf{m}[i,j]-\mathbf{g}[i,j]\right|$,
where $\mathbf{g}$ is the aggregated ground truth segmentation map for all
object classes in the input image. Note that certain metrics might favor very
small masks (_e.g_. Sal), while others favor very large masks (_e.g_. IoU).

The assessment in this sub-section assumes that an ideal method should perform
attribution over the whole ground truth segmentation, which is not the case in
certain situations, as attributions should only cover classifier-relevant
parts of the object class (_e.g_. face instead of whole body). We argue that
although attributions do not directly correspond to semantic segmentations,
and while different methods target different goals for attribution, the
combined evaluation of these metrics can still be used for a standardized
assessment.

###  4.4 Limitations

Not all pairs of _Explainer_ and _Explanandum_ interact equally well and allow
straightforward training. We observed that, while the explanation of VGG-16
and ResNet-50 on the VOC-2007 dataset performs consistently, the explanation
for COCO-2014 is harder on ResNet-50 than on VGG-16 (see Sup. Mat. Sec. E and
F), even though the accuracies are comparable (see Sup. Mat. Sec. B, Tab. 4).
Sometimes, masks fail to precisely cover the discriminative objects parts.
Some examples are given in Fig. 3(13) and Fig. 3(14), and further examples are
given in the Sup. Mat. Sec. D, E and F.

##  5 Conclusions

We have presented an approach for visually attributing the classification
result of a frozen, pre-trained, black-box classifier (the _Explanandum_) on
specific regions of an input image. This involves training an _Explainer_
architecture with the training data that was used to train the _Explanandum_.
The resulting model generates attributions in the form of class-wise
attribution masks, as well as an aggregated foreground mask with a single
forward pass for any test image. As compared to the gradient and activation-
based approaches, which often generate blurry saliency maps, our masks locate
object regions more accurately and the delineation of these regions is sharp
(Sec. 4.1). As compared to the approaches that use perturbed inputs, the
_Explainer_ is not only more precise, as we prove quantitatively (Sec. 4.2 and
4.3), but also computationally more efficient (Sup. Mat. Sec. A).

We observe that the _Explainer_ can be learned well if the pre-trained
classifier exhibits a good level of accuracy. Since learning the attribution
fully depends on the signals provided by the _Explanandum_ , this comes as no
surprise. If instead of the original image, we use inputs that are masked by
our _Explainer_ , the accuracy may slightly decrease (Sup. Mat. Sec. B). Since
our masks are so parsimonious in retaining the essential parts of the original
image, it points to the fact that the drop in accuracy might come from the
_Explanandum_ being deprived of background-dependent biases. Therefore, once
the _Explainer_ is trained to generate masks, one could imagine retraining the
_Explanandum_ with the original as well as _Explainer_ -masked data. This can
ensure that classification accuracy can be both high and free of influences
from non-object regions. Finally, while our work focuses on explaining deep
image classifiers, we believe it should be possible to extend such an approach
to other black box classifiers, other data modalities, and other learning
tasks such as regression.

## References

  * Adebayo et al. [2018] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim.  Sanity checks for saliency maps.  In _Advances in Neural Information Processing Systems_ , pages 9505–9515, 2018.
  * Chang et al. [2019] C.-H. Chang, E. Creager, and D. Goldenberg, A. an Duvenaud.  Explaining image classifiers by counterfactual generation.  In _International Conference on Learning Representations_ , 2019.
  * Chattopadhay et al. [2018] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian.  Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks.  In _IEEE Winter Conference on Applications of Computer Vision_ , pages 839–847, 2018.
  * Chen et al. [2017] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_ , 2017.
  * Dabkowski and Gal [2017] P. Dabkowski and Y. Gal.  Real time image saliency for black box classifiers.  In _Advances in Neural Information Processing Systems_ , pages 6967–6976, 2017.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 248–255. Ieee, 2009.
  * Desai and Ramaswamy [2020] S. Desai and H. G. Ramaswamy.  Ablation-CAM: Visual explanations for deep convolutional network via gradient-free localization.  In _IEEE Winter Conference on Applications of Computer Vision_ , pages 972–980, 2020.
  * Doshi-Velez et al. [2017] F. Doshi-Velez, M. Kortz, R. Budish, C. Bavitz, S. Gershman, D. O‘Brien, K. Scott, S. Schieber, J. Waldo, D. Weinberger, and et. al.  Accountability of AI under the law: The role of explanation.  _arXiv preprint arXiv:1711.01134_ , 2017.
  * Everingham et al. [2015] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The Pascal visual object classes challenge: A retrospective.  _International Journal of Computer Vision_ , 111(1):98–136, Jan. 2015.
  * Falcon et al. [2019] W. Falcon et al.  Pytorch lightning.  _GitHub. Note: https://github. com/PyTorchLightning/pytorch-lightning_ , 3:6, 2019.
  * Fong and Vedaldi [2017] R. C. Fong and A. Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In _IEEE International Conference on Computer Vision_ , pages 3429–3437, 2017.
  * Fong et al. [2019] R. C. Fong, M. Patrick, and A. Vedaldi.  Understanding deep networks via extremal perturbations and smooth masks.  In _IEEE/CVF International Conference on Computer Vision_ , pages 2950–2958, 2019.
  * Gilpin et al. [2018] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal.  Explaining explanations: An overview of interpretability of machine learning.  In _IEEE International Conference on Data Science and Advanced Analytics_ , pages 80–89, 2018.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2016.
  * Holzinger et al. [2017] A. Holzinger, C. Biemann, C. S. Pattichis, and D. B. Kell.  What do we need to build explainable AI systems for the medical domain?  _arXiv preprint arXiv:1712.09923_ , 2017.
  * Khorram et al. [2021] S. Khorram, T. Lawson, and L. Fuxin.  iGOS++: Integrated gradient optimized saliency by bilateral perturbations.  In _Proceedings of the Conference on Health, Inference, and Learning_ , CHIL ’21, page 174–182, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450383592.  doi: 10.1145/3450439.3451865.  URL https://doi.org/10.1145/3450439.3451865.
  * Liao et al. [2020] Q. V. Liao, D. Gruen, and S. Miller.  Questioning the AI: Informing design practices for explainable AI user experiences.  In _Conference on Human Factors in Computing Systems_ , 2020.
  * Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft COCO: Common objects in context.  In _European Conference on Computer Vision_ , pages 740–755, 2014\.
  * Paszke et al. [2017] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  2017\.
  * Petsiuk et al. [2018] V. Petsiuk, A. Das, and K. Saenko.  Rise: Randomized input sampling for explanation of black-box models.  In _British Machine Vision Conference_ , 2018.
  * Ras et al. [2020] G. Ras, N. Xie, M. van Gerven, and D. Doran.  Explainable deep learning: A field guide for the uninitiated.  _CoRR_ , 2020.
  * Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier.  In _ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 1135–1144, 2016.
  * Rudin [2019] C. Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1:206–215, 2019.
  * Selvaraju et al. [2017] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.  Grad-CAM: Visual explanations from deep networks via gradient-based localization.  In _IEEE/CVF International Conference on Computer Vision_ , pages 618–626, 2017.
  * Simonyan and Zisserman [2014] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Springenberg et al. [2015] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller.  Striving for simplicity: The all convolutional net.  In _International Conference on Learning Representations Workshops_ , 2015.
  * Vitali Petsiuk et al. [2021] V. Vitali Petsiuk, R. Jain, V. Manjunatha, V. I. Morariu, V. Mehra, V. Ordonez, and K. Saenko.  Black-box explanation of object detectors via saliency maps.  In _IEEE/CVF Computer Vision and Pattern Recognition_ , 2021.
  * Wagner et al. [2019] J. Wagner, J. M. Kohler, T. Gindele, L. Hetzel, J. T. Wiedemer, and S. Behnke.  Interpretable and fine-grained visual explanations for convolutional neural networks.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9097–9107, 2019.
  * Wang et al. [2020] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu.  Score-CAM: Score-weighted visual explanations for convolutional neural networks.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 24–25, 2020.
  * Zeiler and Fergus [2014] M. D. Zeiler and R. Fergus.  Visualizing and understanding convolutional networks.  In _European Conference on Computer Vision_ , pages 818–833, 2014\.
  * Zhou et al. [2016] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba.  Learning deep features for discriminative localization.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2921–2929, 2016.

Supplementary Material

What You See is What You Classify:
Black Box Attributions

##  Appendix A Compute Times

To evaluate the practicability of the methods, we have evaluated the time it
takes for each method to generate all aggregated masks for the segmentation
evaluation in Section 4.3. Note that this does not include any training time
for our method and adapted RTIS (up to 12 hours on VOC-2007 and up to 2 days
on COCO-2014), which only has to be performed once per model and dataset. Tab.
3 shows how significant the inference time differences between the methods
are, even for a relatively small amount of images. RISE, EP and iGOS++ are
orders of magnitude slower than GCAM, RTIS and our explainer. All experiments
have been conducted using the same Tesla P100 PCIe 12GB GPU.

|  | Method
---|---|---
Dataset | Classifier | GCAM | RISE | EP | iGOS++ | RTIS | Explainer (ours)
VOC-2007 | VGG-16 | 00:00:11 | 01:15:29 | 05:57:16 | 02:57:15 | 00:00:08 | 00:00:08
Resnet-50 | 00:00:10 | 00:46:21 | 04:45:59 | 01:55:58 | 00:00:11 | 00:00:11
COCO-2014 | VGG-16 | 00:01:23 | 06:00:47 | 33:54:33 | 28:55:22 | 00:00:34 | 00:00:34
Resnet-50 | 00:01:30 | 03:41:35 | 26:40:42 | 19:09:10 | 00:00:47 | 00:00:47

Table 3: Time for computing segmentation masks with each method. The times are
given in hh:mm:ss format. 210 and 1000 images have been segmented for VOC-2007
and COCO-2014, respectively.

##  Appendix B Classification accuracy

We demonstrate through Tab. 4 that our _Explainer_ architecture does not
adversely affect the performance of the (pre-)trained classification network.
On occasions, a slight drop in accuracy can be expected, since each individual
image changes quite drastically after being masked, not only visually, but
also in terms of pixel intensity. Such drop can be simply explained by the
fact that classifiers are not trained to recognize objects on a black
background, which can behave as adversarial information to the VGG-16 and
ResNet-50 classifiers. A loss in performance may also be indicative of the
pre-trained classifier not being able to capitalize on an unexpected cue or
correlations in the data (such as presence of blue and green in images with
cows).

Data | Model | Precision | Recall | F-Score
---|---|---|---|---
VOC-2007 | VGG-16 Cl. | 88.4 | 63.9 | 74.2
VGG-16 Expl. | 86.4 | 62.9 | 72.8
ResNet-50 Cl. | 85.2 | 72.3 | 78.2
ResNet-50 Expl. | 77.3 | 69.6 | 73.2
COCO-2014 | VGG-16 Cl. | 73.5 | 45.4 | 56.1
VGG-16 Expl. | 66.9 | 45.0 | 53.8
ResNet-50 Cl. | 77.6 | 44.6 | 56.6
ResNet-50 Expl. | 69.1 | 48.9 | 57.3

Table 4:  Comparison between base (pre-trained, frozen) classifiers (Cl.) and
their _explained_ counterparts (Expl.) on the respective test sets. All
numbers are given as percentages (%). Since masked images result in overall
lower logits, we have used slightly different thresholds for the logits to
count as positive versus negative predictions, depending on whether they
result from unmasked or masked images. The presented numbers demonstrate that
the use of our _Explainer_ masks on the inputs to the pre-trained networks
does not significantly affect their classification scores.

##  Appendix C Multiclass attribution examples on VOC using VGG-16 and
ResNet-50

In this section, we show examples for the attribution over multiple classes on
the VOC-2007 test set, using VGG-16 and ResNet-50 classifiers. In Fig. 4 and
Fig. 5 below, each row corresponds to a random image from the VOC-2007
dataset, while each column corresponds to: the original image, the aggregated
mask (per-pixel maximum over the class-masks), and the top 5 strongest
attributed classes (TAC). Fig. 4 shows the results for VGG-16, while Fig. 5
shows the ones for ResNet-50. Each mask is scored according to the average
mask activation (AMA), then sorted in descending order of the first 5 classes,
as shown below. As this process is completely independent from the classifier
itself, we also add the multi-label scores (logits through the sigmoid) for
the original classifier on the unmasked image (CLS), multiplied by 100. When a
class from the ground truth is attributed, the TAC is accompanied by two
asterisks (**). We do not report the classification scores for the masked
images, because they are greatly perturbed since the classifier has never
learned to classify heavily masked images. For example, the classifier will
give relatively high scores for all classes when it is given a completely
black (all zeros) image.

One can see that for both VGG-16 (Fig. 4) and ResNet-50 (Fig. 5), masks are
sharp and outline (parts of) the object(s) of interest. By looking at the
classification scores and the average mask activation, one can see that they
usually correlate, meaning that the attribution works well. In those rare
cases where the _Explainer_ masks show their highest activation for classes
that are not the ones for which the classifier gives the highest probability,
the _Explainer_ might have learned to detect those objects better than the
classifier itself. Remember that those predictions are made directly on out-
of-sample data. Also, note that the average mask activation does of course not
directly relate to a classification, since smaller objects will give a smaller
AMA score than large ones.

An interesting example is at the second row in Fig. 4, where in the fourth
column (second most attributed class) the _Explainer_ shows a significant
attribution for class 5 (“bottle”) which is not present in the image. Only
when looking at the classifier score on the unmasked image (CLS) this makes
perfect sense since the classifier gives the “bottle” class a probability of
over 60%. A similar example for this behavior is the fifth row in the same
figure or the seventh row in Fig. 5. This indicates that the attributions by
our _Explainer_ are sensible even in failure cases of the classifier. In a few
cases, the attributions for several classes are on the same image areas, as in
the third and seventh rows of Fig. 5. This might happen for image areas that
appear to be generally interesting for several classes at the same time but
such cases are rare. In general, we see that ResNet-50 tends to provide masks
with slightly more artifacts and spurious activations, when compared to the
attributions of VGG-16.

Both experiments seem to agree that our _Explainer_ is able to attribute
classifiers to regions occupied by the correct class, or provide attributions
caused by ambiguous biases in the classifier itself.

ID | Class name | ID | Class name
---|---|---|---
1 | Aeroplane | 2 | Bicycle
3 | Bird | 4 | Boat
5 | Bottle | 6 | Bus
7 | Car | 8 | Cat
9 | Chair | 10 | Cow
11 | Dining Table | 12 | Dog
13 | Horse | 14 | Motorbike
15 | Person | 16 | Potted Plant
17 | Sheep | 18 | Sofa
19 | Train | 20 | TV/Monitor
Table 5: Summary of the Pascal VOC-2007 Classes. |  | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---|---
| TAC | 15 ** | 14 | 4 | 5 | 3
| CLS | 98.44 | 1.76 | 1.52 | 4.07 | 0.96
| AMA | 20.73 | 1.82 | 1.76 | 1.71 | 1.63
|  |  |  |  |  |
| TAC | 15 ** | 5 | 11 ** | 3 | 13
| CLS | 99.88 | 60.38 | 22.58 | 0.48 | 0.06
| AMA | 26.06 | 9.87 | 4.49 | 2.10 | 2.06
|  |  |  |  |  |
| TAC | 4 ** | 6 | 7 | 15 | 19
| CLS | 5.94 | 73.04 | 1.37 | 7.15 | 1.32
| AMA | 13.53 | 9.57 | 8.86 | 6.77 | 4.29
|  |  |  |  |  |
| TAC | 7 ** | 15 ** | 17 | 10 | 18
| CLS | 98.02 | 18.86 | 1.15 | 0.26 | 0.32
| AMA | 21.86 | 4.02 | 2.18 | 1.81 | 1.60
|  |  |  |  |  |
| TAC | 17 ** | 10 | 20 | 18 | 14
| CLS | 50.11 | 15.73 | 0.00 | 0.02 | 0.01
| AMA | 19.46 | 10.86 | 2.84 | 2.82 | 2.68
|  |  |  |  |  |
| TAC | 15 ** | 5 ** | 14 | 4 | 17
| CLS | 90.38 | 9.14 | 0.22 | 0.47 | 0.37
| AMA | 18.27 | 12.87 | 1.48 | 1.41 | 1.41
|  |  |  |  |  |
| TAC | 19 | 16** | 20 | 15 | 7 **
| CLS | 13.01 | 11.27 | 1.33 | 8.28 | 15.31
| AMA | 8.06 | 5.01 | 4.75 | 4.56 | 4.00
|  |  |  |  |  |
| TAC | 15 ** | 3 ** | 13 | 18 | 14
| CLS | 79.36 | 1.67 | 3.12 | 0.66 | 0.31
| AMA | 26.72 | 2.74 | 2.52 | 2.35 | 2.14
|  |  |  |  |  |
Figure 4: Top-5 class-wise attributions for the VGG-16 classifier, for 8 random images from the VOC-2007 test set. Class-wise masks are sorted according to their average activation on the image plane. TAC corresponds to the top activating class, please refer to the legend in Tab. 5. CLS shows the respective class probabilities by the classifier on the original (unmasked) images, multiplied by 100. AMA shows the average mask activations for the respective class, also multiplied by 100. |  | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---|---
| TAC | 15 ** | 7 ** | 18 | 14 | 10
| CLS | 77.13 | 52.68 | 0.56 | 1.19 | 0.96
| AMA | 23.49 | 14.53 | 2.38 | 2.30 | 2.23
|  |  |  |  |  |
| TAC | 3 ** | 2 | 9 | 17 | 11
| CLS | 94.41 | 3.79 | 0.26 | 0.15 | 0.18
| AMA | 32.71 | 0.52 | 0.47 | 0.46 | 0.45
|  |  |  |  |  |
| TAC | 15 ** | 5 | 11 | 18 | 2
| CLS | 96.07 | 1.26 | 2.08 | 28.53 | 0.76
| AMA | 24.51 | 4.14 | 3.63 | 3.53 | 3.26
|  |  |  |  |  |
| TAC | 19 ** | 17 | 9 | 10 | 11
| CLS | 98.51 | 0.26 | 3.70 | 3.04 | 1.00
| AMA | 30.16 | 0.89 | 0.86 | 0.86 | 0.83
|  |  |  |  |  |
| TAC | 15 ** | 5 | 11 | 18 | 17
| CLS | 99.95 | 12.24 | 35.63 | 2.34 | 0.15
| AMA | 29.79 | 1.67 | 1.64 | 1.48 | 1.43
|  |  |  |  |  |
| TAC | 7 ** | 14 | 13 | 9 | 10
| CLS | 97.96 | 1.16 | 0.25 | 0.45 | 0.35
| AMA | 31.01 | 0.55 | 0.54 | 0.52 | 0.48
|  |  |  |  |  |
| TAC | 15 ** | 20 | 18 ** | 9 | 11
| CLS | 76.80 | 33.23 | 90.83 | 66.43 | 4.12
| AMA | 22.79 | 11.12 | 4.44 | 3.11 | 2.71
|  |  |  |  |  |
| TAC | 2 ** | 15 ** | 9 | 11 | 4
| CLS | 97.15 | 80.15 | 1.14 | 0.30 | 0.09
| AMA | 13.52 | 5.68 | 1.64 | 1.49 | 1.36
|  |  |  |  |  |
Figure 5: Top-5 class-wise attributions for the ResNet-50 classifier, for 8
random images from the VOC-2007 test set. Class-wise masks are sorted
according to their average activation on the image plane. TAC corresponds to
the top activating class, please refer to the legend in Tab. 5. CLS shows the
respective class probabilities by the classifier on the original (unmasked)
images, multiplied by 100. AMA shows the average mask activations for the
respective class, also multiplied by 100.

##  Appendix D Comparison of attribution methods on VOC-2007 using ResNet-50.

Figure 6 provides additional comparisons using ResNet-50 on VOC-2007. The
comparison follows Fig. 3 of the manuscript. Overall, attribution methods for
ResNet-50 are behaving better on VOC-2007 than on COCO-2014. Still, GCam, RISE
and EP tend to provide smooth attributions, with the latter two sometimes
scoring at multiple locations which are hard to relate to the classes present
in the image. iGOS++ provides small activations that mostly seem to outline
the most important object parts but sometimes also show artifacts that are
difficult to interpret when it is less certain. RTIS provides masks that are
often outlining objects, but fails in being sharp and providing clear
attributions due to overshooting too much. Our _Explainer_ method is mostly
artifact-free and masks are very sharp on discriminative object parts.

ID |  Input | GCam | RISE | EP | iGOS | RTIS | Ours | ID |  Input | GCam | RISE | EP | iGOS | RTIS | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
1 |  |  2 |
3 |  |  4 |
5 |  |  6 |
7 |  |  8 |
9 |  |  10 |
11 |  |  12 |
13 |  |  14 |

Figure 6: Comparison of attributions methods for a ResNet-50 network fine-
tuned on images from the VOC-2007 dataset. Refer to Fig. 3 of the manuscript
for more details.

##  Appendix E Comparison of attribution methods on COCO-2014 using VGG-16.

Figure 7 provides additional comparisons using VGG-16 on COCO-2014. The
comparison follows Fig. 3 of the manuscript. Using VGG-16 on COCO-2014 results
in good attributions, where masks are sharp and outline discriminative parts
for the predicted classes precisely.

Not too differently than the results presented in the manuscript – attribution
for the VGG-16 classifier on VOC-2007 – we can see that GCam [24] provides
good attributions, if not for a fairly large smoothing. RISE [20] behaves
similarly, and it often tends to provide several oversmoothed saliency
locations, which are hard to interpret. EP [12] is much sharper, but also
tends to overpredict attributions. On the other hand, iGOS++ shows small,
localized attributions. However, it often also includes artifacts and
sometimes does not capture certain objects at all. Finally, RTIS [5] does not
show well delineated attribution on this combination of data and classifier as
it overshoots too much to allow for a clear interpretation. Our _Explainer_
provides localized attributions which are easy to interpret, with sharp
boundaries, which in most cases coincide with the classes of interest or
discriminative parts composing them. In general, the _Explainer_ predicts
compact masks with smaller areas, similarly to GCam, but much less smoothed
overall.

ID |  Input | GCam | RISE | EP | iGOS | RTIS | Ours | ID |  Input | GCam | RISE | EP | iGOS | RTIS | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
1 |  |  2 |
3 |  |  4 |
5 |  |  6 |
7 |  |  8 |
9 |  |  10 |
11 |  |  12 |
13 |  |  14 |

Figure 7: Comparison of attributions methods for a VGG-16 network fine-tuned
on images from the COCO-2014 dataset. Refer to Fig. 3 of the manuscript for
more details.

##  Appendix F Comparison of attribution methods on COCO-2014 using ResNet-50.

Figure 8 provides additional comparisons using ResNet-50 on COCO-2014. The
comparison follows Fig. 3 of the manuscript. As discussed in the manuscript,
this combination of data and classifier is the hardest to train the
_Explainer_ on.

As for previous comparisons, GCam shows accurate but very smooth attributions.
RISE and EP behave similarly, by attributing the classification on the correct
areas of the input image, but failing in focusing on precise image locations.
iGOS++ once again shows small attributions, which in many cases lead to clear
explanations but sometimes might also leave out parts where the other methods
agree that they are important as well. In some cases it also includes
artifacts, which make the interpretation harder. RTIS overpredicts the
attributions, in particular when the objects of interest are small with
respect to the input image plane. Our _Explainer_ shows very clear
attributions in most cases but sometimes also includes artifacts that occur as
an active area in the top portion of the image. We have found that this
specifically happens in images with the “person” class. Nonetheless, we can
notice that the attribution masks are still precise and sharp in the object
areas, suggesting that better results could probably be achieved by better
hyperparameter and model selection.

ID |  Input |  GCam | RISE | EP | iGOS | RTIS | Ours | ID |  Input | GCam | RISE | EP | iGOS | RTIS | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
1 |  |  2 |
3 |  |  4 |
5 |  |  6 |
7 |  |  8 |
9 |  |  10 |
11 |  |  12 |
13 |  |  14 |

Figure 8: Comparison of attributions methods for a ResNet-50 network fine-
tuned on images from the COCO-2014 dataset. Refer to Fig. 3 of the manuscript
for more details.