# Aftershocks as a time independant phenomenon

###### Abstract

[ Sequences of aftershocks following Omori’s empirical law are observed after
most major earthquakes, as well as in laboratory-scale fault-mimicking
experiments. Nevertheless, the origin of this memory effect is still unclear.
In this letter, we present an analytical framework for treating labquake and
earthquake catalogs on an equal footing. Using this analysis method, we show
that when memory is considered to be in deformation and not in time, all data
collapse onto a single master curve, showing that the timescale is entirely
fixed by the inverse of the strain rate. ]

Geophysical Research Letters

Univ Rennes, CNRS, IPR (Institut de Physique de Rennes) - UMR 6251, F-35000
Rennes, France Université Savoie Mont-Blanc, CNRS, IRD, IFSTTAR, ISTerre, Le
Bourget-du-Lac, France IsTerre, CNRS/Université Grenoble Alpes, Grenoble,
France

Axelle<EMAIL_ADDRESS>
Quake sequences from laboratory experiments and seismic catalogs can be
analyzed jointly in a common framework.

Clustering observed in laboratory and seismic sequences is not time dependent
but deformation dependent.

Clustering properties from laboratory and seismic catalogs can be collapsed on
a single master curve.

## Plain Language Summary

[ After major earthquakes, a local increase in seismic activity is generally
observed. The origin of these aftershocks is still unclear, but time-dependent
effects are generally invoked to explain them. We show here that time is not
the relevant variable for understanding this phenomenon. By jointly analyzing
catalogs of seismic faults and experiments, we show that even if these data
differ by several orders of magnitude in terms of length and time scales, they
can be rationalized on a single curve if we consider that the variable
governing the dynamics is deformation and not time. ]

##  1 Introduction

On February 6, 2023, two devastating earthquakes struck Turkey and Syria
within nine hours of each other. These events are a sad reminder of the vital
importance of understanding aftershock mechanisms in order to assess the
associated risk. It is well established that there is an excess in the rate of
seismicity after an earthquake, and that this excess decreases inversely with
the time elapsed since the mainshock, a law known as Omori-Utsu’s Law [Omori
(1894), Utsu . (1995)]. Yet the physical origin of this clustering in
earthquake dynamics is still unclear.

Static stress redistribution after a mainshock seems to play a role in the
increase (or decrease) of the local seismicity rate by modifying locally the
stress state of the system and thus its distance to friction thresholds [RS.
Stein . (1994), Toda . (1998)]. Alone, the static stress redistribution is not
sufficient to explain the different features observed and memory effects. In
particular, several faults that experienced a significant increase in local
stress still did not undergo a major earthquake, in contradiction to what has
been predicted from stress transfer and average loading stress rate [Scholz
(2019), Freed (2005)]. Dynamic stress triggering [Felzer Brodsky (2006)], as
well as time-dependent effects, such as pore pressure transfers [Jónsson .
(2003)], afterslip [Van Camp . (2017)], or time-dependent friction laws [Gross
Kisslinger (1997)], have been therefore invoked separately or combined. In
particular, the state and rate friction law is appealing as it has been shown
to predict the Omori’s law dependence [Dieterich (1994)]. However, the
validity of transposing this law, obtained empirically on a laboratory scale,
to real kilometre-scale faults with a memory effect lasting decades remains an
open question [Scholz (1998)].

A major limitation of all those time-dependent mechanisms is that none of them
can explain in an unique framework the universality of aftershock sequences.
Clustering is observed after shallow earthquakes as well as after deep
ruptures. The characteristics of faults vary widely in terms of the presence
of water (or not) in the pores, the width and the nature of the gouge, the
type of fault, the average temperature and nature of the rocks in the slip
zone [Kagan Jackson (1991), Scholz (2019)]. The lack of a common explanation
based on a few ingredients independent of local specificity is all the more
problematic that aftershock sequences following the Omori-Utsu’s law have been
recently observed in lab experiments on sheared, cohesion-less, ideal granular
materials at low confining pressure [Lherminier . (2019), Houdoux . (2021)].
In such systems, time-delayed redistribution of fluids or coupling with a
viscous asthenosphere can not be invoked to interpret the observed clustering.

In this letter, we want to determine whether the clustering sequences observed
in earthquake catalogs and in experiments are governed by the same laws and,
if so, what sets the time scale of the process. To answer this question, we
need to perform a direct, quantitative comparison of the clustering dynamics
observed in earthquakes and in experiments, the difficulty lying in the fact
that the time and length scales of these systems differ by several orders of
magnitude. We propose a method of analysis which allows to study in a unique
framework data coming from experiments and from seismic catalogs. Using this
method of analysis, we show that it is possible to rationalize the memory
effect as a deformation-dependent, and not a time-dependent, one and collapse
all the data on a single master curve. This allows us to discuss the physical
origin of the aftershocks as deformation-dependent processes.

In section 2, we present the two systems studied: seismic catalogs and
experiments using glass beads. We detail how we estimate the mean shear rate
at the level of the fault in those different systems. In section 3 we present
our method analysis, first the binning procedure which allows to map data from
different systems in a common and objective way, second the definition of the
correlation function we used to characterize quantitatively the clustering in
the sequences. In section 4 we apply this tool to the four systems considered
and show that the time scale in all the systems is set by the inverse of the
mean strain rate so that all the data can be collapsed without any further
normalisation on a single power law of exponent -1. In section 5 we discuss
the results.

##  2 Data

The data used in this study come from two different systems, covering
strikingly different scales: (i) earthquake catalogs from strike-slip simple
(roughly linear) faults and (ii) experiments on granular materials in which
slip events are measured inside a shear-band.

###  2.1 Seismic catalogs

We use the United States Geological Survey (USGS) earthquake catalog,
targeting two almost linear strike-slip faults: the Parkfield segment of the
San Andreas fault in California (latitude $\in[35.6;36.2]$, longitude
$\in[-120.8;-120.2]$, from 1/1/1980 to 1/1/2024) and the Denali fault in
Alaska (latitude $\in[62;64]$, longitude $\in[-148.;-142]$, from 1/1/1980 to
1/1/2024), keeping only earthquakes of magnitude larger than 2 to be sure of
the completeness of the catalogs (see Fig. 1a and b for the exact portions of
the catalog studied).

Figure 1: Locations of the earthquakes used for the analysis in (a) California
(Parkfield segment) and (b) Alaska (Denali fault), with local axes giving the
direction of the fault and the normal to the fault. The spatial binning is
shown with red boxes. Black scale: scale of the map; red scale: size $\Delta
x$ of the bin. (c) (resp. (d)) Red dots: cumulative number of events in the
$y$ direction integrated along the $x$-axis for map (a) (resp. (b)). Black
solid line: fit using the function
$f(y)=\frac{N_{T}}{2}\left[1+\tanh\left(-2\frac{y-y_{0}}{w}\right)\right]$.

The choice of those segments is motivated by the fact their simple geometry
allows to estimate the mean macroscopic strain rate directly from the velocity
field and the width of the fault. We use geodetic measurements of the velocity
field $v$ across the fault from the literature: 24-35 mm/yr for Parkfield
[Platt Becker (2010)] and 5.5-15.5 mm/yr for Denali [Biggs . (2007)]. In both
cases the uncertainty on this value is rather high. We estimate the width of
the shear band by studying the cumulative number of events in the direction
normal to the fault (Fig. 1c and d). The sigmoid curves obtained can be fitted
and the width $w$ of the faults estimated. We obtain $w\simeq 1$ km for
Parkfield and $w\simeq 11$ km for Delani. This last value is consistent with
the one that can be deduced from [Biggs . (2007)] which is of the order of 20
km (fig. 15 in [Biggs . (2007)]). Those estimates of the width of the fault
vary from 10 to 50% depending on the cut-off magnitude and the area chosen.
Taking this additional uncertainty into account, we deduce the macroscopic
strain rates for both faults $\dot{\varepsilon}=\frac{v}{2w}$, giving
$3-6\times 10^{-8}$ days-1 for Parkfield and $0.4-2\times 10^{-9}$ days-1 for
Denali.

###  2.2 Experiments

We perform biaxial tests on assemblies of non-cohesive glass micro-beads (Fig.
2a). The sample is loaded quasi-statically at constant strain-rate. After
failure, all the deformation is localized inside shear bands whose orientation
is given by the Mohr-Coulomb angle. We focus on the post-failure regime, when
the shear bands are stationary (Fig. 2b). We resolve fluctuations of
plasticity occurring inside the shear band using an interferometric method of
deformation (see [Amon . (2017)] for details). We have shown in previous works
[Houdoux . (2021)] that those fluctuations of plasticity correspond to local
slip events along our experimental fault. Those events can be sorted in a
catalog using a thresholding procedure [Mathey . (2024)]. The seismic moment
associated to those events can be estimated quantitatively from the spatial
size of the events and the amount of local slip. The distribution of the
moments follows a Gutenberg-Richter law and time clustering of events is
observed that follows the Omori-Utsu’s law [Houdoux . (2021)]. The typical
magnitude of the observed events are in the range -10 to -7, illustrating the
huge scale gap with the earthquakes analyzed below.

Figure 2: Experiment on glass beads pack: (a) Principle of the biaxial test.
Dry glass beads are enclosed in a latex membrane and submitted to a confining
pressure $\sigma_{22}=30$ kPa. The sample is loaded along the ’1’ direction at
a constant strain rate and a force sensor allows to deduce the stress
$\sigma_{11}$ exerted at the top of the sample. (b) Typical loading curve
representing the normalized deviatoric stress measured as a function of the
overall deformation imposed. Failure occurs at about 4% of deformation after
which the stress reaches a plateau. (c) Maps of incremental deformation
obtained using an interferometric method of measurement. After failure, all
the deformation is localized in stationary shear bands. Deformation inside the
shear bands occurs principally in the form of localized intermittent bursts.
$\gamma$ is the local strain measured optically and is displayed on the map in
grey levels.

In the ideal case of two symetric conjugated shear bands dividing the sample
in four blocks of well-defined velocities (inset in Fig. 2b), the strain-rate
along the shear-band $\dot{\varepsilon}=\dot{\varepsilon}_{xy}$ can be deduced
from the imposed compression velocity $v_{1}$, the angle of the shear band
$\theta$ and the width $w$ of the shear band:
$\dot{\varepsilon}=\frac{v_{1}}{4w}\sqrt{\frac{1}{\tan^{2}\theta}+1}$.

We can use experiments to validate our measurement of fault width for
earthquakes. Indeed, we have two ways of measuring the width of the shear band
in the case of the experiment. Firstly, we can integrate the deformation
$\gamma$ directly from the spatially-resolved deformation maps (Fig. 3a) to
obtain the displacement $u_{x}$ (see Text S1 in Supporting Information and
[Houdoux . (2021)] for details). This displacement can be fitted with a
sigmoid curve giving a width $w\simeq 30d=2.8$ mm for the shear band.
Secondly, we can apply the same procedure as for earthquakes to the
experimental catalogs (Fig. 3c and d). The cumulative number of events (Fig.
3c) can also be fitted by a sigmoid curve which gives a smaller width,
typically half the value obtained on the raw maps. The latter method relies on
the dispersion of the position of the barycenters of the events while the
former method takes into account the spatial distribution of the deformation.
This explain that the first method gives a larger width. Interestingly, a
factor of roughly 2 is also found for the Denali’s fault between our
estimation of the width of the fault and the one that can be deduced from
[Biggs . (2007)]. In conclusion, the method used to estimate the width of the
shear zone from catalogs does indeed give the right order of magnitude.

Figure 3: (a) Red box: region of interest as studied in the experiment. The
deformation maps have been averaged over 100 frames, showing the mean,
stationary, deformation $\bar{\gamma}(x,y)$ inside the shear band. (b) (resp.
(c)) Red dots: integration of the average mean deformation
$\langle\bar{\gamma}\rangle_{x}$ along the $y$ direction for map, giving the
total displacement $u$ along the $x$ direction during the stationary regime.
(a) (resp. cumulative number of events in the $y$ direction integrated along
the $x$-axis for map (b)). Black solid line: fit using the function
$f(y)=\frac{N_{T}}{2}\left[1+\tanh\left(-2\frac{y-y_{0}}{w}\right)\right]$.
(d) Events from the experimental catalog corresponding to the area shown in
(a).

We have performed experiments using glass beads of mean diameter $d\simeq
90~{}\mu$m at two velocities of compression $v_{1}$: 0.25 $\mu$m/s and 2
$\mu$m/s. The Mohr-Coulomb angle and the shear band width have been measured
on the spatially-resolved maps: $\theta\simeq 64^{\circ}$. The inferred values
of the strain-rates are respectively $2.5\times 10^{-5}$ s-1 and $2.0\times
10^{-4}$ s-1. The uncertainty on those values is due to the simplified model
used to deduce the velocity along the shear bands. In fact, the lack of
symmetry in the real system and deformation within the blocks themselves
result in an inhomogeneous distribution of shear strain among the different
segments. We also observe during stationary regimes a slow evolution of the
shape of the shear bands as well as of the mean shear strain along each
segment. Those effects result in an uncertainty on the value of
$\dot{\varepsilon}$ that we estimate to be of the order of 30%.

##  3 Method of analysis

The usual data processing method of counting the excess number of events after
an earthquake of a given magnitude (mainshock) treats each sequence
separately, and does not allow direct quantitative comparison of data from
experiments with those from field catalogs.

Our method of analysis is based on a two-step procedure. First, we map the
catalogs, which consist in lists of events at irregular times $t_{i}$, into
series defined on evenly spaced times and positions by counting the number of
events taking place between $t$ and $t+\Delta t$ at a given location on a
spatial grid. We call _binning_ this process. Second, we compute the time
correlation function of the series obtained.

###  3.1 Binning procedure

We simplify the binning in space by taking advantage of the linear shape of
the faults considered here. Denoting $x$ the direction along the fault and $y$
the orthogonal direction, we divide the area of interest only along the $x$
direction, in $n_{x}$ bins of width $\Delta x$ (fig. 1a and b). We choose as
space bin size the rupture size of the largest event of the catalog. We define
an objective time bin size for the different systems using a fraction of the
mean interoccurence time, defined as $\tau=\langle t_{i+1}-t_{i}\rangle$,
where the average is taken on all the events of the catalog. As the time
series have been split in $n_{x}$ series corresponding to different positions
along the fault, the mean inter-event time at a fixed position is larger than
the one of the whole catalog by a factor of the order of $n_{x}$. We thus fix
the bin time size as $\Delta t=\frac{\tau n_{x}}{q}$, where $q$ is a constant
that we have chosen so that $\Delta t$ corresponds to the inverse of the
framerate in the experiments ($q=10$). Finally, we count the number of events
$N(x,t)$ from a catalog that have occurred between the positions $x$ and
$x+\Delta x$ along the fault during the time interval $[t,t+\Delta t]$. Those
choices for the size of bins ensure that the mean density of events (ratio of
the total number of events $N$ of the catalog to the total number of cells
$n_{x}\times n_{t}$, where $n_{t}$ is the total duration of the catalog
divided by $\Delta t$) is the same for all the systems studied.

The values of the parameters used for the different systems are given in table
S1 in Supporting Information. The size of the events are directly measured in
the case of the experiments as our method provides spatially-resolved maps of
deformation. In the case of the earthquakes catalogs, we use the following
estimation:

$L=0.006\times 10^{0.5m}$

where $L$ is the diameter (in km) of a circular crack with 3 MPa stress drop
for an earthquake of magnitude $m$ [Eshelby (1957)].

###  3.2 Correlation function

To quantify the excess of activity rate compared to the mean rate, we use
correlation functions of the number of events for different time lags $\delta
t$. Compared to the counting method, this tool does not discriminate between
mainshocks and aftershocks. It quantifies the excess of probability to observe
a similar pattern after a time interval $\delta t$ for a stationary time
series (see text S2 in Supporting Information).

We compute the time auto-correlation function of the function $N(x,t)$ along
the fault as:

$C(\delta t)=\frac{\left<\ \overline{N(x,t)N(x,t+\delta t)}\
\right>_{x}}{\left<\ \overline{N(x,t)N(x,t^{\prime})}\ \right>_{x}}-1$ (1)

where $\overline{\cdot}$ is the temporal average on the whole duration of the
catalog and $\langle\cdot\rangle_{x}$ is the average on the position along the
fault. Time series $N(x,t^{\prime})$ are obtained by randomly reshuffling data
in time at a fixed position. The order of the averaging procedures (first on
time, then in space) allows to take into account spatial heterogeneity along
the fault (see text S3 in Supporting Information).

Note that because of the ratio which intervenes in the definition of $C$, it
is not necessary to define a density of events (i.e. the number of events
normalized by the width of the shear band in the $y$ position and the bin size
in the $x$ direction) as such factors will cancel out. This definition of the
correlation function thus allows to compare quantitatively the dynamics at
different spatial scales without any further normalization.

##  4 Results

The correlation functions $C(\delta t)$ for the four systems studied are
plotted in figure 4a. The values have been computed for linearly spaced
$\delta t$ and the resulting data have been averaged on logarithmically spaced
bins (see Figure S1 in Supporting Information). The four curves follow power
laws, $C(\delta t)\sim\delta t^{-\theta}$ with an exponent $\theta\simeq 1$.
The time intervals over which they are defined are largely disconnected, as
expected for phenomena belonging to two very different classes. The red and
brown dots correspond to the two experiments done at velocities that differ by
a factor of 10. The time resolution for those data is fixed by the framerate
of the recording (40 and 5 fps respectively) and the typical duration of the
stationary regime is of the order of one hour. The blue and purple triangles
correspond respectively to the Parkfield and the Denali faults. The smallest
increment $\delta t$ is fixed by the choice of the time bin size ($\approx
9.5$ and $17.4$ days respectively, see table S1 in Supporting Information).

Figure 4: Correlation function as defined by equation 1 for four systems: red
and brown circles, experiments on sheared granular material; blue triangles
Parkfield fault; purple triangles, Denali fault. (a) As a function of time lag
$\delta t$, (b) as a function of deformation lag
$\delta\varepsilon=\dot{\varepsilon}\delta t$ with errorbars due to the
uncertainties on the estimation of $\dot{\epsilon}$ (see main text). The black
line with slope -1 is shown as a visual guide. Note that the horizontal tick
spacing is the same for the two plots.

Figure 4b shows the correlation functions $C$ plotted as a function of the
deformation lag $\delta\varepsilon=\dot{\varepsilon}\delta t$, using the
strain rate estimation computed in sec. 2. We observe that the four curves
collapse approximately onto a single master curve corresponding to a power law
$C(\delta\varepsilon)\sim\delta\varepsilon^{-\theta}$ with exponent
$\theta\simeq 1$. Such a dependence is awaited for series obeying the Omori-
Utsu law (see text S4 in Supporting Information). The $C(\delta\varepsilon)$
curves have significant uncertainties mainly linked to the difficulty to
estimate the strain rates $\dot{\varepsilon}$, as discussed in sec. 2.

We have further checked the robustness of this result when the parameters of
the binning procedure are changed (see Figure S2 in Supporting Information).

##  5 Discussion

The collapse of figure 4b shows that it is possible to directly and
quantitatively compare the dynamics of quakes observed at time and space
scales differing by several orders of magnitude by rescaling time by the
strain rate. This means that the phenomena studied are truly quasi-static in
the sense that time does not play an intrinsic role: the time scale in each
system is fixed by the inverse of the strain rate. This shows that loss of
memory in those systems is primarily governed by the amount of slip that has
occurred since the reference time, not the time elapsed. Such a picture is in
agreement with [S. Stein Liu (2009)] where it has been reported that the
duration of aftershocks sequences varies inversely with the loading rate. In
their work, Stein and Liu studied several aftershock sequences, some of which
were triggered by earthquakes that occurred several hundred years ago. The
sequences are located in tectonic regions governed by very different
mechanisms (from intraplate to plate boundary), so that the local loading rate
$v$ varies by as much as a factor of 1000 over all the sequences considered.
They observe that the duration of the sequence of aftershocks $t_{a}$
typically varies as $1/v$.

The collapse of experiments and earthquakes on a master curve demonstrates the
need to find a common origin for the clustering mechanism, which does not
involved time-dependent effects. Our experiments are performed using
assemblies of dry glass beads at low confining pressure. There are thus no
fluid transfers, no cohesion bonds which could be damaged or healed, no
crushing of beads and no thermal effects. Consequently, the available
ingredients to build a model of deformation-dependent clustering are
elasticity, disorder and (time-independent) friction. Such models have been
extensively studied in the past in the form of assemblies of frictional blocks
connected by springs [Burridge Knopoff (1967)] and in its cellular automata
version [Olami . (1992)]. Those systems display an intermittent dynamics of
avalanches of slip events. The size distribution of those avalanches follow a
power law distribution, but, when the ingredients of the models are restricted
to the previous list, they do not display time clustering [Jagla (2010)].
Interestingly, a recent theoretical work [Pétrélis . (2023)] has shown that
clustering can emerged in a time-independent model if the stress field
describing the internal stresses in the system has a self-affine geometry.
Petrelis _et al._ have shown that in the vicinity of a maximum of such a
geometrical surface there is a high probability to find patches with large
stress values. Said otherwise, near a site close to the threshold which will
be at the origin of a mainshock, other sites also close to their thresholds
exist, which could lead to aftershocks after further loading. In this picture,
clustering originates from the complex structure of the internal stress field
which has a correlated, self-affine geometry. The emergence of such a geometry
could be the result of the combination of elastic stress redistribution and
disorder.

Another point of view is based on the self-affine properties of fault surfaces
themselves. It has been shown that the roughness of fault surfaces present
self-affinity over several decades of length scales (from 50 $\mu$m to 50 km)
[Candela . (2012)]. Considering that an earthquake corresponds to a slip along
such surface, the loss of memory when successive slips occur is not link to
time but to the cumulative slip along the surface. This picture allows to
understand why memory is governed by an amount of slip $v\times\delta t$
rather than a time lag $\delta t$. Moreover, as faults surface are self-
affine, the loss of correlation of the heights between two points of the
surface obeys to a power law as a function of the distance between those
points [Schmittbuhl . (1995)], with an exponent which depends of the Hurst
exponent of the surface. Consequently, a power law dependence can also be
awaited for the excess of probability to slip at a position where a slip has
occurred before, which is exactly what our correlation function
$C(\delta\varepsilon)$ measures.

In this work we have focused on faults of very simple geometries similar to
the one of our experiment in order to be able to apply exactly the same tools
to the different systems and thus compare objectively and quantitatively their
dynamics. A challenge is to apply the same approach to more complex faults or
even to tectonic areas featuring complex hierarchies of faults. The extension
of the spatial binning to a 2D grid instead of a line does not present any
difficulties for larger areas. The main difficulty resides in the evaluation
of the strain rate playing a role at the level of a cell of the grid which
will be generally much larger than the one that can be evaluated on a large
area. Such a work will be the object of another study.

##  6 Conclusion

In this work, we have shown that experimental data on sheared granular media
and seismic catalogs can be processed together in a unified analysis. We have
shown that clustering, observed in sequences from systems of different nature
and characterized by time and length scales differing by several orders of
magnitude, can be rationalized when the memory effect is considered as a
deformation-dependent and not a temporal effect. To our knowledge, this is the
first study to analyze laboratory earthquakes jointly with real earthquakes,
and to quantify clustering on a single master curve.

The central result of our paper is the fact that clustering is not a time-
dependent phenomenon both in experiments and in seismic sequences. This is in
essence a quasi-static phenomenon, the time scale of which is fixed by the
strain rate imposed in the vicinity of the fault. This result questions
interpretations of aftershock sequences relying on intrinsic time-dependent
processes.

###### Acknowledgements.

This section is optional. Include any Acknowledgments here. The
acknowledgments should list:
All funding sources related to this work from all authors
Any real or perceived financial conflicts of interests for any author
Other affiliations for any author that may be perceived as having a conflict
of interest with respect to the results of this paper.
It is also the appropriate place to thank colleagues and other contributors.
AGU does not normally allow dedications.

## References

  * Amon . (2017) amon2017bAmon, A., Mikhailovskaya, A. Crassous, J.  2017\.  Spatially resolved measurements of micro-deformations in granular materials using diffusing wave spectroscopy Spatially resolved measurements of micro-deformations in granular materials using diffusing wave spectroscopy.  Review of Scientific Instruments885.
  * Biggs . (2007) biggs2007multiBiggs, J., Wright, T., Lu, Z. Parsons, B.  2007\.  Multi-interferogram method for measuring interseismic deformation: Denali Fault, Alaska Multi-interferogram method for measuring interseismic deformation: Denali fault, alaska.  Geophysical Journal International17031165–1179.
  * Burridge Knopoff (1967) burridge1967Burridge, R. Knopoff, L.  1967\.  Model and theoretical seismicity Model and theoretical seismicity.  Bulletin of the seismological society of america573341–371.
  * Candela . (2012) candela2012roughnessCandela, T., Renard, F., Klinger, Y., Mair, K., Schmittbuhl, J. Brodsky, EE.  2012\.  Roughness of fault surfaces over nine decades of length scales Roughness of fault surfaces over nine decades of length scales.  Journal of Geophysical Research: Solid Earth117B8.
  * Dieterich (1994) dieterich1994Dieterich, J.  1994\.  A constitutive law for rate of earthquake production and its application to earthquake clustering A constitutive law for rate of earthquake production and its application to earthquake clustering.  Journal of Geophysical Research: Solid Earth99B22601–2618.
  * Eshelby (1957) eshelby1957determinationEshelby, JD.  1957\.  The determination of the elastic field of an ellipsoidal inclusion, and related problems The determination of the elastic field of an ellipsoidal inclusion, and related problems.  Proceedings of the royal society of London. Series A. Mathematical and physical sciences2411226376–396.
  * Felzer Brodsky (2006) felzer2006decayFelzer, KR. Brodsky, EE.  2006\.  Decay of aftershock density with distance indicates triggering by dynamic stress Decay of aftershock density with distance indicates triggering by dynamic stress.  Nature4417094735–738.
  * Freed (2005) freed2005Freed, AM.  2005\.  Earthquake triggering by static, dynamic, and postseismic stress transfer Earthquake triggering by static, dynamic, and postseismic stress transfer.  Annu. Rev. Earth Planet. Sci.33335–367.
  * Gross Kisslinger (1997) gross1997Gross, S. Kisslinger, C.  1997\.  Estimating tectonic stress rate and state with Landers aftershocks Estimating tectonic stress rate and state with landers aftershocks.  Journal of Geophysical Research: Solid Earth102B47603–7612.
  * Houdoux . (2021) houdoux2021Houdoux, D., Amon, A., Marsan, D., Weiss, J. Crassous, J.  2021\.  Micro-slips in an experimental granular shear band replicate the spatiotemporal characteristics of natural earthquakes Micro-slips in an experimental granular shear band replicate the spatiotemporal characteristics of natural earthquakes.  Communications Earth & Environment2190.
  * Jagla (2010) jagla2010realisticJagla, E.  2010\.  Realistic spatial and temporal earthquake distributions in a modified Olami-Feder-Christensen model Realistic spatial and temporal earthquake distributions in a modified olami-feder-christensen model.  Physical Review E814046117.
  * Jónsson . (2003) jonsson2003Jónsson, S., Segall, P., Pedersen, R. Björnsson, G.  2003\.  Post-earthquake ground movements correlated to pore-pressure transients Post-earthquake ground movements correlated to pore-pressure transients.  Nature4246945179–183.
  * Kagan Jackson (1991) kagan1991Kagan, YY. Jackson, DD.  1991\.  Long-term earthquake clustering Long-term earthquake clustering.  Geophysical Journal International1041117–133.
  * Lherminier . (2019) lherminier2019Lherminier, S., Planet, R., dit Vehel, VL., Simon, G., Vanel, L., Måløy, KJ. Ramos, O.  2019\.  Continuously sheared granular matter reproduces in detail seismicity laws Continuously sheared granular matter reproduces in detail seismicity laws.  Physical review letters12221218501.
  * Mathey . (2024) mathey2024deviceMathey, A., Fur, ML., Chasle, P., Amon, A. Crassous, J.  2024\.  A device for studying elementary plasticity fluctuations in granular media. A device for studying elementary plasticity fluctuations in granular media.
  * Olami . (1992) olami1992Olami, Z., Feder, HJS. Christensen, K.  1992\.  Self-organized criticality in a continuous, nonconservative cellular automaton modeling earthquakes Self-organized criticality in a continuous, nonconservative cellular automaton modeling earthquakes.  Physical review letters6881244.
  * Omori (1894) Omori1894Omori, F.  1894\.  On the aftershocks of earthquakes On the aftershocks of earthquakes.  J. Coll. Sci.7111-200.
  * Pétrélis . (2023) petrelis2023Pétrélis, F., Chanard, K., Schubnel, A. Hatano, T.  2023\.  Earthquake magnitude distribution and aftershocks: A statistical geometry explanation Earthquake magnitude distribution and aftershocks: A statistical geometry explanation.  Physical Review E1073034132.
  * Platt Becker (2010) platt2010realPlatt, JP. Becker, TW.  2010\.  Where is the real transform boundary in California? Where is the real transform boundary in california?  Geochemistry, Geophysics, Geosystems116.
  * Schmittbuhl . (1995) schmittbuhl1995scalingSchmittbuhl, J., Schmitt, F. Scholz, C.  1995\.  Scaling invariance of crack surfaces Scaling invariance of crack surfaces.  Journal of Geophysical Research: Solid Earth100B45953–5973.
  * Scholz (1998) scholz1998Scholz, CH.  1998\.  Earthquakes and friction laws Earthquakes and friction laws.  Nature391666237–42.
  * Scholz (2019) scholz2019Scholz, CH.  2019\.  The mechanics of earthquakes and faulting The mechanics of earthquakes and faulting.  Cambridge university press.
  * RS. Stein . (1994) stein1994Stein, RS., King, GC. Lin, J.  1994\.  Stress triggering of the 1994 M= 6.7 Northridge, California, earthquake by its predecessors Stress triggering of the 1994 m= 6.7 northridge, california, earthquake by its predecessors.  Science26551771432–1435.
  * S. Stein Liu (2009) stein2009Stein, S. Liu, M.  2009\.  Long aftershock sequences within continents and implications for earthquake hazard assessment Long aftershock sequences within continents and implications for earthquake hazard assessment.  Nature462726987–89.
  * Toda . (1998) toda1998Toda, S., Stein, RS., Reasenberg, PA., Dieterich, JH. Yoshida, A.  1998\.  Stress transferred by the 1995 Mw= 6.9 Kobe, Japan, shock: Effect on aftershocks and future earthquake probabilities Stress transferred by the 1995 mw= 6.9 kobe, japan, shock: Effect on aftershocks and future earthquake probabilities.  Journal of Geophysical Research: Solid Earth103B1024543–24565.
  * Utsu . (1995) Utsu1995Utsu, T., Ogata, Y., S, R. Matsu’ura.  1995\.  The Centenary of the Omori Formula for a Decay Law of Aftershock Activity The centenary of the omori formula for a decay law of aftershock activity.  Journal of Physics of the Earth4311-33.  10.4294/jpe1952.43.1
  * Van Camp . (2017) van2017Van Camp, M., de Viron, O., Watlet, A., Meurers, B., Francis, O. Caudron, C.  2017\.  Geophysics from terrestrial time-variable gravity measurements Geophysics from terrestrial time-variable gravity measurements.  Reviews of Geophysics554938–992.


# Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional
Networks

Qinglong Zhang, Lu Rao, Yubin Yang
State Key Laboratory for Novel Software Technology at Nanjing University
{wofmanaf<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recently, explaining deep convolutional neural networks has been drawing
increasing attention since it helps to understand the networks’ internal
mechanisms and the reason why the networks make specific decisions. In the
computer vision community, one of the most popular approaches for visualizing
and understanding deep networks is generating saliency maps that highlight
salient regions mostly related to the network’s decision-making. However,
saliency maps generated by existing methods either contain too much
meaningless information or the process of computing saliency maps requires
plenty of time. In this paper, we propose an efficient saliency map generation
method, called Group score-weighted Class Activation Mapping (Group-CAM),
which adopts the “split-transform-merge” strategy to generate saliency maps.
Specifically, for an input image, the class activations are firstly split into
groups. In each group, the sub-activations are summed and de-noised as an
initial mask. After that, the initial masks are transformed with meaningful
perturbations and then applied to preserve sub-pixels of the input (i.e.,
masked inputs), which are then fed into the network to calculate the
confidence scores. Finally, the initial masks are weighted summed to form the
final saliency map, where the weights are confidence scores produced by the
masked inputs. Group-CAM is efficient yet effective, which only requires
dozens of queries to the network while producing target-related saliency maps.
As a result, Group-CAM can be served as an effective data augment trick for
fine-tuning the networks. We comprehensively evaluate the performance of
Group-CAM on common-used benchmarks, including deletion and insertion tests on
ImageNet-1k, and pointing game tests on COCO2017. Extensive experimental
results demonstrate that Group-CAM achieves better visual performance than the
current state-of-the-art explanation approaches. The code is available at
https://github.com/wofmanaf/Group-CAM.

##  1 Introduction

Understanding and interpreting the decision made by deep neural networks
(DNNs) is of central importance for humans since it helps to construct the
trust of DNN models [6, 2, 10, 20]. In the area of computer vision, one
critical technique is generating intuitive heatmaps that highlight regions,
which are most related to DNN’s decision.

One common approach for determining salient regions is relying on the changes
in the model output, such as the changes of prediction scores with respect to
the input images. For example, RISE [8] estimates importance empirically by
probing the model with randomly masked versions of the image and obtaining the
corresponding outputs. While RISE provides very compelling results, thousands
of random masks should be generated and then be applied to query the model,
making it inefficient.

Other approaches, such as Grad-CAM [13], calculate gradients by back-
propagating the prediction score through the target layer of the network and
apply them as weights to combine the forward feature maps. These methods are
generally faster than RISE since they only require a single or constant number
of queries to the network [9, 6]. However, saliency maps of Grad-CAM may
capture too much meaningless information since the feature maps are not
necessarily related to the target category [22]. Therefore, the results of
Grad-CAM may not truly reflect the reason why a network makes a decision.
Naturally, a question arises: “Can one method produce results that truly
reflect the model decision in a more efficient way?”

Figure 1: Pipeline of Group-CAM. Activations are first extracted with a linear
combination of feature maps and importance weights $w^{c}$. Then the
activations are split into groups and summed along the channel dimension in
each group before de-noising to generate initial masks $M^{\prime}_{\ell}$.
Input image $I_{0}$ is element-wise multiplied with $M^{\prime}_{\ell}$ and
then transformed with meaningful perturbations. The perturbated images are
then fed to the network. The output saliency map can be computed as a weighted
sum of all $M^{\prime}_{\ell}$ where the weights $\alpha_{\ell}^{c}$ come from
the confidence scores of the target class corresponding to the respective
perturbated inputs.

To answer this question, we first revisit the intuition behind RISE [8]. Let
$M$ be a random binary mask with distribution $\mathcal{D}$, the input image
$I_{0}$ can be masked by $I_{0}\odot M$ to preserve a subset of pixels, where
$\odot$ denotes element-wise multiplication. The masked image is then applied
to produce the confidence score to measure the contribution of these preserved
pixels. Finally, the saliency map can be generated by combining plenty of
random masks and scores with respect to them. It is observed that the most
time-costing procedure is random masks generating and multiple queries to the
neural network.

To address the efficiency issue, we propose Group score-weighted Class
Activation Mapping (Group-CAM), which adopts the “split-transform-merge”
strategy to generate saliency maps. Specifically, for an input image, the
class activations are firstly split into groups (to improve the efficiency, we
apply the simplest uniform and adjacent group strategy, same as ResNexT [19]).
In each group, the sub-activations are summed along the channel dimension as
an initial mask. However, directly apply the initial masks to preserve input
pixels may cause noise visual due to gradient vanishing [22]. Therefore, we
design a de-noising strategy to filter the less important pixels of the
initial mask. In addition, to ease the adversarial effects of sharp boundaries
between the masked and salient regions, we employ the blurred information from
the input to replace the unreserved regions(pixels with 0 values) of the
masked image. Finally, the saliency map of Group-CAM can be computed as a
weighted sum of the grouped initial masks, where the weights are confidence
scores produced by the masked inputs. Group-CAM is quite efficient, which can
produce appealing target-related saliency maps after dozens of queries to the
networks. As a result, Group-CAM can be applied to train/fine-tune
classification methods. The overall architecture of Group-CAM are illustrated
in Figure 1.

We comprehensively evaluate Group-CAM on ImageNet-1k and MS COCO2017. Results
demonstrate that Group-CAM requires less computation yet achieves better
visual performance than the current state-of-the-art methods. In addition, we
extend the application of saliency methods and apply Group-CAM as an effective
data augment trick for fine-tuning classification networks, extensive
experimental results suggest that Group-CAM can boost the networks’
performance by a large margin.

Note that, if the number of groups in Group-CAM is set to 1, and no de-noising
strategy is applied, then Group-CAM can be simplified as Grad-CAM.

##  2 Related Work

Region-based Saliency Methods. In recent years, numerous saliency methods
attributing inputs to output predictions have been proposed [22, 13, 6]. One
set of methods adopt masks to preserve certain regions of the inputs and
measure the effect these regions have on the output by performing a forward
pass through the network with these regions. These types of saliency methods
are called Region-based saliency methods. Among them, RISE [8] first generates
thousands of random masks and then employ them to mask the input. Then a
linear combination of random masks with the corresponding prediction score of
the masked images is computed as the final saliency map. Instead of generating
random masks, Score-CAM [18] adopts feature maps of the target layer (the
target layer generally contains thousands of feature maps) as initial masks
and employ them to computing saliency map. Unlike RISE and Score-CAM, XRAI [6]
first over-segmented the input image, and then iteratively test the importance
of each region, coalescing smaller regions into larger segments based on
attribution scores. Region-based approaches usually generate better human
interpretable visualizations but are less efficient since they requires plenty
of quires to the neural network [22]. Our Group-CAM can be seen as a region-
based method while is much faster than RISE [8], Score-CAM [18] and XRAI [6].

Activation-based Saliency Methods. These approaches combine activations
(generally the combination of back-propagation gradients and feature maps) of
a selected convolutional layer to form an explanation [23, 13, 3]. CAM [23]
and Grad-CAM [13] adopt a linear combination of activations to form a heatmap
with fine-grained details. Grad-CAM++ [3] extends Grad-CAM and uses a weighted
combination of the positive partial derivatives of the target layers’ feature
maps with respect to a specific class score as weights to generate a visual
explanation for the corresponding class label. Activation-based methods are in
general faster than region-based approaches since they only require a single
or constant number of queries to the model [6]. However, results of
activation-based methods may capture too much meaningless information since
the feature maps are not necessarily related to the target category [22]. In
this paper, we draw on the idea of Grad-CAM [13] to generate initial-mask of
Group-CAM.

##  3 Group-CAM

In this section, we first describe the Group-CAM algorithm, then explain the
motivation behind it. The high-level steps are shown in Algorithm 1.

Input: Image $I_{0}$, Model $\mathcal{F}$, Class $c$, number of groups $G$,
Gaussian blur parameters: $ksize$, $sigma$.

Output: Saliency map $\mathcal{L}_{Group-CAM}^{c}$

1 Initialization: Initial $\mathcal{L}_{Group-CAM}^{c}\leftarrow 0$, Baseline
Input $\tilde{I_{0}}=guassian\\_blur2d(I_{0},ksize,sigma)$, $\ell\leftarrow
0$;

2 Get target layer feat maps $A$, importance weights $w^{c}$;

3 $K\leftarrow$ the number of channels of $A$;

4 $g\leftarrow K/G$ number of feat maps in each group;

5 while  _$\ell <G$_ do

6 Generating $M_{\ell}=ReLU(\sum_{k={\ell}\times g}^{({\ell}+1)\times
g-1}(w_{k}^{c}A^{k}))$;

7 Initial mask $M^{\prime}_{\ell}\leftarrow$ de-nosie, normalize and bilinear
interpolate upsample activation map $M_{\ell}$;

8 Perturbated image $I^{\prime}_{\ell}=I_{0}\odot
M^{\prime}_{\ell}+\tilde{I_{0}}\odot(1-M^{\prime}_{\ell})$;

9 Compute confidence gain
$\alpha_{\ell}^{c}=\mathcal{F}_{c}(I^{\prime}_{\ell})-\mathcal{F}_{c}(\tilde{I_{0}})$;

10 $\mathcal{L}_{Group-CAM}^{c}\leftarrow\mathcal{L}_{Group-
CAM}^{c}+\alpha_{\ell}^{c}M^{\prime}_{\ell}$ ;

11 $\ell\leftarrow\ell+1$;

12

13 end while

return ${\rm}ReLU(\mathcal{L}_{Group-CAM}^{c})$

Algorithm 1 Group-CAM Algorithm

###  3.1 Initial Masks

Let $I_{0}\in\mathbb{R}^{3\times H\times W}$ be an input image, $\mathcal{F}$
be a deep neural network which predicts a score $\mathcal{F}_{c}(I_{0})$ on
class $c$ with input $I_{0}$. In order to obtain the class-discriminative
initial group masks of the target convolutional layer, we first compute the
gradient of $\mathcal{F}_{c}(I_{0})$ with respect to $k^{th}$ feature map
$A^{k}$. Then these gradients are global average pooled over the height and
width dimensions (indexed by $i,j$ respectively) to obtain the neuron
importance weights

$w_{k}^{c}=\dfrac{1}{Z}\sum_{i}\sum_{j}\dfrac{\partial\mathcal{F}_{c}(I_{0})}{\partial
A_{ij}^{k}(I_{0})}$ (1)

where $Z$ is the number of pixels in feature map $A^{k}$.

Assume $K$ is the number of channels of the target layer feature maps, we
first split all the feature maps and neuron importance weights into $G$
groups. Then the initial mask in each group is calculated by

$M_{\ell}=ReLU(\sum_{k={\ell}\times g}^{({\ell}+1)\times
g-1}(w_{k}^{c}A^{k}))$ (2)

where $\ell\in\\{0,1,\cdots,G-1\\}$, $g=K/G$ is the number of feature maps in
each group.

$M_{\ell}$ is the combination of feature maps and gradients, which means
$M_{\ell}$ can be noisy visually since the gradients for a DNN tend to vanish
due to the flat zero-gradient region in ReLU [22]. Therefore, it is not
suitable to directly apply $M_{\ell}$ as the initial mask.

To remedy this issue, we utilize a de-noising function to filter pixels in
$M_{\ell}$ less than $p(M_{\ell},\theta)$, where $p(M_{\ell},\theta)$ is a
function which compute the $\theta^{th}$ percentile of $M_{\ell}$.

Formally, for a scalar $m_{ij}$ in $M_{\ell}$, the de-noising function can be
represented as

$\phi(m_{ij},\theta)=\begin{cases}m_{ij},&\text{if
}m_{ij}>p(M_{\ell},\theta);\\\ 0,&\text{otherwise}.\end{cases}$ (3)

Instead of setting all pixels to binary values, it is better to generate
smoother mask for an activation map. Specifically, we scale the raw values of
$M_{\ell}$ into $[0,1]$ by utilizing Min-Max normalization,

$M^{\prime}_{\ell}=\frac{M_{\ell}-min(M_{\ell})}{max(M_{\ell})-min(M_{\ell})}$
(4)

Then, $M^{\prime}_{\ell}$ is upsampled with bilinear interpolate to the same
resolution of $I_{0}$ to mask the input.

###  3.2 Saliency Map Generation

It has been widely acknowledged that if the saliency method is in fact
identifying pixels significant to the model’s prediction, this should be
reflected in the model’s output for the reconstructed image [6]. However,
merely masking the image pixels out of the region of interest causes
unintended effects due to the sharp boundary between the masked and salient
region. Consequently, it is crucial to minimize such adversarial effects when
testing for the importance of a feature subset [4].

To address this issue, we start with a masked version of the input, replace
the unreserved regions (pixels with 0 values) with the blurred information,
and then performing classification on this image to measure the importance of
the initial masks. The blurred images can be computed by

$I^{\prime}_{\ell}=I_{0}\odot
M^{\prime}_{\ell}+\tilde{I_{0}}\odot(1-M^{\prime}_{\ell})$ (5)

where $\tilde{I_{0}}=guassian\\_blur2d(I_{0},ksize,sigma)$ is a baseline image
with the same shape as $I_{0}$ and have a lower confidence of class $c$. In
this paper, we set $ksize=51$ and $sigma=50$, following [22].

The contribution $\alpha_{\ell}^{c}$ of the reserved regions $I_{0}\odot
M^{\prime}_{\ell}$ can then be computed as

$\alpha_{\ell}^{c}=\mathcal{F}_{c}(I^{\prime}_{\ell})-\mathcal{F}_{c}(\tilde{I_{0}})$
(6)

Similar to RISE [8], the final saliency map is a linear combination of the
initial masks with weights $\alpha_{\ell}^{c}$, that is

$\mathcal{L}_{Group-
CAM}^{c}=ReLU(\sum_{\ell}\alpha_{\ell}^{c}M^{\prime}_{\ell})$ (7)

Figure 2: Visualization results of SOTA saliency methods. Results show that
saliency maps of Group-CAM are more compelling than region-base methods and
activation-based methods, and contain less noise than gradient-based methods.

##  4 Experiments

In this section, we firstly utilize ablation studies to investigate the effect
of group $G$ and filtering threshold $\theta$. Then we apply a sanity check to
test whether Group-CAM is sensitive to model parameters. Finally, we compare
the proposed Group-CAM with other popular saliency methods to evaluate its
performance.

###  4.1 Experimental Setup

Experiments in this section are conducted on the commonly-used computer vision
datasets, including the validation split of ImageNet-1k [11](containing 50k
images) and the val2017 split of MS COCO2017 [7] (containing 5k images). For
both datasets, all images are resized to $3\times 224\times 224$, and then
transformed to tensors and normalized to the range [0, 1]. No further pre-
processing is performed. We report the insertion and deletion test results
using the pre-trained torchvision
model111https://github.com/pytorch/vision/tree/master/torchvision VGG19 [14]
as the base classifier and other results are on the pre-trained ResNet-50 [5].
Unless explicitly stated, the number of groups $G$ adopted in Group-CAM is 32,
and the threshold $\theta$ in Eq. 3 is set as 70. For a fair comparison, all
saliency maps are upsampled with bilinear interpolate to $224\times 224$.

###  4.2 Class Discriminative Visualization

We qualitatively compare the saliency maps produced by recently SOTA methods,
including gradient-based methods (Guided Backpropagation [16], IntegrateGrad
[17], SmoothGrad [15]), region-based methods(RISE [8], XRAI [6]), and
activation-based methods (Grad-CAM [13], Score-CAM [18]) to validate the
effectiveness of Group-CAM.

As shown in Figure 2, results in Group-CAM, random noises are much less than
that in region-base methods and activation-base methods. In addition, Group-
CAM generates smoother saliency maps comparing with gradient-based methods.

We further conduct experiments to test whether that Group-CAM can distinguish
different classes. As shown in Figure 3, the VGG19 classifies the input as
“bull mastiff” with 46.06% confidence and ‘tiger cat’ with 0.39% confidence.
Group-CAM correctly gives the explanation locations for both of two
categories, even though the classification score of the latter is much lower
than that of the former. It is reasonable to indicate that Group-CAM can
distinguish different categories.

Figure 3: Class discriminative results. The middle image is generated w.r.t ‘bull mastiff’, and the right one is generated w.r.t ‘tiger cat’. Figure 4: Grad-CAM, Score-CAM and Group-CAM generated saliency maps for representative images in terms of deletion and insertion curves. In the insertion curve, a better explanation is expected that the prediction score to increase quickly, while in the deletion curve, it is expected the classification confidence to drop faster. AUC | Grad-CAM | Grad-CAM++ | RISE | XRAI | Score-CAM | Group-CAM
---|---|---|---|---|---|---
Insertion | 53.5 | 50.0 | 54.0 | 43.1 | 55.1 | 56.8
Deletion | 13.3 | 14.8 | 11.7 | 15.8 | 11.5 | 12.3
Over-all | 40.2 | 35.2 | 42.3 | 27.3 | 43.6 | 44.5
Table 1: Comparative evaluation in terms of deletion (lower AUC is better) and
insertion (higher AUC is better) AUC on ImageNet-1k validation split. The
over-all score (higher AUC is better) shows that Group-CAM outperform other
related methods significantly. The best records are marked in bold.

###  4.3 Deletion and Insertion

We follow [8] to conduct deletion and insertion tests to evaluate different
saliency approaches. The intuition behind the deletion metric is that the
removal of pixels/regions most relevant to a class will cause the
classification score to drop significantly [8, 12]. Insertion metric, on the
other hand, starts with a blurred image and gradually re-introduces content,
which produces more realistic images and has the additional advantage of
mitigating the impact of adversarial attack examples [8]. In detail, for the
deletion test, we gradually replace 3.6% (i.e., $224\times 8$) pixels in the
original image with a highly blurred version each time according to the values
of the saliency map until no pixels left. Contrary to the deletion test, the
insertion test replaces 3.6% pixels of the blurred image with the original one
until the image is well recovered. We calculate the AUC of the classification
score after Softmax as a quantitative indicator. Besides, we provide the
$over-all$ score to comprehensively evaluate the deletion and insertion
results, which can be calculated by $AUC(insertion)-AUC(deletion)$. Examples
are shown in Figure 4. The average results over all the 50k images is reported
in Table 1.

As illustrated in Table 1, the proposed Group-CAM outperforms other related
approaches in terms of insertion and over-all AUC. Moreover, Group-CAM also
exceeds other methods in terms of deletion AUC except for XRAI.

Ablation Studies. We report the ablation studies results of Group-CAM on the
first 5k images on ImageNet-1k, to thoroughly investigate the influence of
filter threshold $\theta$ and group $G$. Results are shown in Figure 5 and
Table 2.

Groups | Insertion | Deletion | Over-all
---|---|---|---
1 | 61.72 | 11.21 | 50.51
4 | 64.27 | 11.21 | 53.07
8 | 64.94 | 11.29 | 53.65
16 | 65.38 | 11.34 | 54.04
32 | 65.48 | 11.31 | 54.17
64 | 65.77 | 11.31 | 54.46
128 | 65.81 | 11.29 | 54.52
256 | 65.84 | 11.28 | 54.56
Table 2: Ablation studies of Group $G$ with filter threshold $\theta=70$ in
terms of deletion, insertion, and over-all scores on ImageNet-1k validation
split (on the first 5k images). The best records are marked in bold. Figure 5:
Ablation studies of filter threshold $\theta$ with $G=32$ in terms of deletion
(lower AUC is better), insertion (higher AUC is better) curve and the over-all
scores (higher AUC is better) on ImageNet-1k validation split(on the first 5k
images).

From Figure 5, we can see, threshold $\theta$ has a significant effect on the
performance of Group-CAM (fluctuating over 1.1% in terms of over-all score).
Specifically, when $\theta$ is small, the over-all score keeps stable with an
increase of $\theta$. Then, as $\theta$ increases, the over-all score drops
quickly when $\theta>70$. Here, to make a trade-off between insertion and
deletion results, we set $\theta=70$ as default.

Besides, in Table 2 we can see, the over-all score increase with the increase
of $G$. However, as introduced in Algorithm 1, larger $G$ means more computer
costing. To make a trade-off, we set $G=32$ as the default group of Group-CAM.

Methods | Running Time
---|---
RISE | 38.23
XRAI | 42.17
Grad-CAM | 0.03
Score-CAM | 2.46
Group-CAM (ours) | 0.09
Table 3: Comparative evaluation in terms of running time (seconds, averaged on
5k images) on ImageNet-1k. The best and second best records are marked in bold
and blue, respectively.

Running Time. In Table 3, we summarize the average running time for RISE [8],
XRAI [6], Grad-CAM [13], Score-CAM [18] and the proposed Group-CAM on one
NVIDIA 2080Ti GPU. As shown in Table 3, the averaging running time for Grad-
CAM and Group-CAM are both less than 1 second, which achieve best results
among all the approaches. Combined with Table 1 and Table 3, we observe that
although Group-CAM runs slower than Grad-CAM, it achieves much better
performance.

###  4.4 Localization Evaluation

In this part, we adopt pointing game [21] on the val2017 of MS COCO2017 to
measure the quality of the generated saliency map through localization
ability. We apply the same pre-trained ResNet-50 from [8]. The localization
accuracy is then calculated as $Acc=\dfrac{\\#Hits}{\\#Hits+\\#Misses}$ for
each object category (if the most salient pixel lies inside the annotated
bounding boxes of an object, it is counted as a hit). The overall performance
is measured by the mean accuracy across different categories.

From Table 4, we observe that Group-CAM beats all the other compared
approaches. Specifically, Group-CAM outperforms the base Grad-CAM with 0.8% in
terms of mean accuracy.

Methods |  Mean Accuracy
---|---
Grad-CAM |  56.7
Grad-CAM++ |  57.2
RISE |  54.3
XRAI |  55.1
Score-CAM |  51.0
Group-CAM (ours) |  57.5
Table 4: Pointing Game on COCO val2017 split. Results show that the proposed
Group-CAM performs consistently better than other related methods.

###  4.5 Sanity Check

Finally, we utilize sanity check [1] to check whether the results of Group-CAM
can be considered completely reliable explanations for a trained model’s
behavior. Specifically, we employ both cascade randomization and independent
randomization, to compare the output of Group-CAM on a pre-trained VGG19. As
shown in Figure 6, The Group-CAM is sensitive to classification model
parameters and can produce valid results.

Figure 6: Sanity check results by cascade randomization and independent
randomization. Results show that Group-CAM is sensitive to classification
model parameters and can reflect the quality of the network.

##  5 Fine-tuning Classification Methods

Finally, we extend the application of Group-CAM and apply it as an effective
data augment strategy to fine-tune/train the classification models. We argue
that a saliency method that is suitable to fine-tune the networks should have
the following two characteristics: (1) the saliency method should be
efficient, which can produce saliency maps in limited times; (2) the generated
saliency maps should be related to the object targets. Our Group-CAM can
produce appealing target-related saliency maps in 0.09 seconds per image with
$G=32$, which means Group-CAM is suitable to fine-tune the networks.

To make Group-CAM more efficient, we remove the importance weights $w^{c}$ and
de-noise procedures. Although this will slightly impair the performance of
Group-CAM, back-propagation is no longer needed, which can greatly save the
saliency maps generating time.

The fine-tuning process is defined as follows:

(1) generate saliency map $M$ for $I_{0}$ with $G=16$ and the ground-truth
target class $c$;

(2) binarize $M$ with threshold $\theta$, where $\theta$ is the mean value of
$M$.

(3) apply Eq. 5 to get the blurred input $\tilde{I_{0}}$.

(4) adopt $\tilde{I_{0}}$ to fine-tune the classification model.

Since $\tilde{I_{0}}$ are generated during the training process, which means
that when the performance of the classification model is improved, Group-CAM
will generate a better $\tilde{I_{0}}$, which in turn will promote the
performance of the classification model.

Figure 7: Fine-tuning ResNet-50 with Group-CAM. Results show that Group-CAM
can improve the classification model’s performance by a significant margin.

Here, we report the results on the ImageNet-1k validation split of fine-tuning
ResNet-50. Specifically, we trained the pre-trained ResNet-50 by SGD with
weight decay 1e-4, momentum 0.9, and mini-batch size 256 (using 8 GPUs with 32
images per GPU) for 20 epochs, starting from the initial learning rate of 1e-3
and decreasing it by a factor of 10 every 15 epochs. For the testing on the
validation set, the shorter side of an input image is first resized to 256,
and a center crop of $224\times 224$ is used for evaluation.

As shown in Figure 7, fine-tune with Group-CAM can contribute to 0.59% (76.74%
vs. 76.15%) improvement in terms of Top-1 accuracy.

Figure 8: Visualization results of fine-tuning ResNet-50 with Group-CAM. The
first image (Epoch_0) is generated the original pre-trained ResNet-50. The
right four images (i.e., Epoch_5, Epoch_10, Epoch_15 and Epoch_20) are
generated by the fine-tuning ResNet-50.

Here, we visualize the saliency maps generated by the fine-tuned ResNet-50 in
Figure 8. As illustrated in Figure 8, as the performance of ResNet-50
improves, the noise of the saliency maps generated by Group-CAM decreases and
focuses more on the important regions. Since the noise can reflect the
performance to some degree, we can also treat it as a hint to determine
whether a model has converged. That is, if the saliency maps generated by
Group-CAM do not change, the model may have converged.

##  6 Conclusion

In this paper, we proposed Group-CAM, which adopts the grouped sum of gradient
and feature map combinations as initial masks. These initial masks are adopted
to preserve a subset of input pixels, and then these pixels are fed into the
network to calculate the confidence scores, which reflects the importance of
the masked images. The final saliency map of Group-CAM is computed as a
weighted sum of the initial masks, where the weights are confidence scores
produced by the masked inputs. The proposed Group-CAM is efficient yet
effective and can be applied as a data augment trick to fine-tune/train
classification models. Experimental results on ImageNet-1k and COCO2017
demonstrate that Group-CAM achieves better visual performance than the current
state-of-the-art explanation approaches.

## References

  * [1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  In Advances in Neural Information Processing Systems, NeurIPS 2018, 3-8 December 2018, Montréal, Canada., pages 9525–9536, 2018.
  * [2] Naman Bansal, Chirag Agarwal, and Anh Nguyen.  SAM: the sensitivity of attribution methods to hyperparameters.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 8670–8680. IEEE, 2020.
  * [3] Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N. Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, NV, USA, March 12-15, 2018, pages 839–847, 2018\.
  * [4] Piotr Dabkowski and Yarin Gal.  Real time image saliency for black box classifiers.  In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 6967–6976, 2017.
  * [5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016.
  * [6] Andrei Kapishnikov, Tolga Bolukbasi, Fernanda B. Viégas, and Michael Terry.  XRAI: better attributions through regions.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 4947–4956, 2019.
  * [7] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740–755. Springer, 2014.
  * [8] Vitali Petsiuk, Abir Das, and Kate Saenko.  RISE: randomized input sampling for explanation of black-box models.  In British Machine Vision Conference 2018, BMVC 2018, Northumbria University, Newcastle, UK, September 3-6, 2018, page 151, 2018.
  * [9] Zhongang Qi, Saeed Khorram, and Fuxin Li.  Visualizing deep networks by optimizing with integrated gradients.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 11890–11898. AAAI Press, 2020.
  * [10] Sylvestre-Alvise Rebuffi, Ruth Fong, Xu Ji, and Andrea Vedaldi.  There and back again: Revisiting backpropagation saliency methods.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 8836–8845. IEEE, 2020.
  * [11] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115(3):211–252, 2015\.
  * [12] Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller.  Evaluating the visualization of what a deep neural network has learned.  IEEE Trans. Neural Networks Learn. Syst., 28(11):2660–2673, 2017\.
  * [13] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 618–626, 2017.
  * [14] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [15] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  CoRR, abs/1706.03825, 2017.
  * [16] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller.  Striving for simplicity: The all convolutional net.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015.
  * [17] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 3319–3328, 2017.
  * [18] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu.  Score-cam: Score-weighted visual explanations for convolutional neural networks.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pages 111–119. IEEE, 2020.
  * [19] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5987–5995. IEEE Computer Society, 2017.
  * [20] Shawn Xu, Subhashini Venugopalan, and Mukund Sundararajan.  Attribution in scale and space.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9677–9686. IEEE, 2020.
  * [21] Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff.  Top-down neural attention by excitation backprop.  International Journal of Computer Vision, 126(10):1084–1102, 2018\.
  * [22] Qing-Long Zhang, Lu Rao, and Yubin Yang.  A novel visual interpretability for deep neural networks by optimizing activation maps with perturbation.  In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 3377–3384. AAAI Press, 2021.
  * [23] Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2921–2929, 2016.