# On the Veracity of Local, Model-agnostic Explanations in Audio
Classification:
Targeted Investigations with Adversarial Examples

###### Abstract

Local explanation methods such as LIME have become popular in MIR as tools for
generating post-hoc, model-agnostic explanations of a model’s classification
decisions. The basic idea is to identify a small set of human-understandable
features of the classified example that are most influential on the
classifier’s prediction. These are then presented as an explanation.
Evaluation of such explanations in publications often resorts to accepting
what matches the expectation of a human without actually being able to verify
if what the explanation shows is what really caused the model’s prediction.
This paper reports on targeted investigations where we try to get more insight
into the actual veracity of LIME’s explanations in an audio classification
task. We deliberately design adversarial examples for the classifier, in a way
that gives us knowledge about which parts of the input are potentially
responsible for the model’s (wrong) prediction. Asking LIME to explain the
predictions for these adversaries permits us to study whether local
explanations do indeed detect these regions of interest. We also look at
whether LIME is more successful in finding perturbations that are more
prominent and easily noticeable for a human. Our results suggest that LIME
does not necessarily manage to identify the most relevant input features and
hence it remains unclear whether explanations are useful or even misleading.

##  1 Introduction

With the rise of deep learning methods used in Music Information Retrieval
(MIR), also the desire for explaining the decisions made by such models has
increased. A plethora of explanation methods (“explainers”) have been
originally developed for text or image data and adapted to the audio domain
[1, 2], or specifically introduced for MIR systems [3]. Most notably,
different versions of Local Interpretable Model-agnostic Explanations (LIME),
a post-hoc explainer [4], have been used to explain models in a variety of MIR
tasks [5, 6, 7, 8, 9, 10, 11].

Evaluation of explanations is known to be a hard task, due to a lack of
agreement on what constitutes a “good explanation” [12] and, consequently,
what may serve as ground truth for measuring the quality of an explanation.
Explanations are often evaluated visually and the applied “metric” is whether
the highlighted input parts appear relevant to the human observer
(“confirmation bias”) [13].

We propose evaluating local explanations by exploiting a known weakness of
deep neural networks: adversarial examples. By adversarially perturbing
examples in a way that a system’s original prediction is changed, we know
exactly what in the input caused the erroneous (adversarial) prediction and
can use this as the “ground truth” that the explanation method should recover.

We are not the first to investigate the relationship between adversarial
examples and interpretability. Slack et al. [14] show that a racially biased
model can be attacked in a way that the (biased) prediction remains unchanged
but the explanation appears as if the model did not base the prediction on
sensitive attributes. This work is related in the sense that it highlights
weaknesses of post-hoc explanation methods (including LIME).

Closest to our work are Göpfert et al. [15], who use “localised” adversarial
attacks that target only selected segments of an image, and investigate the
ability of different explainers (including LIME) to recover the affected part.
In their experiments LIME outperforms the other explainers, which supports our
choice of LIME as an explainer that deserves a more careful and critical
investigation.

We perform four experiments with purposefully designed adversarial examples in
order to obtain more insight about strengths and weaknesses of LIME-based
explanations in an audio classification task. As a test bed, we have chosen a
DNN-based singing voice detection model [16] that currently defines the state
of the art on that task, and is suited to study the quality of explanations
for several reasons: it addresses a clearly defined task (as opposed to genre
classification for example), it has been used for demonstrating explanations
before [5, 11], and it has the interesting previously documented property that
it can be confused by directly drawing on the spectrogram [17].

##  2 Local Interpretable Model-agnostic Explanations

LIME is an algorithm for explaining individual predictions (local) for any
black box machine learning model (it is model-agnostic). The general idea of
the algorithm is to train a simpler model $g$ that approximates the
neighborhood of the prediction $f(x)$ that we want to explain. The first step
is to derive an interpretable representation depending on the input domain,
e.g. the presence or absence of super pixels111(perceptually) grouped pixels
in the image domain.

Let $x\in\mathds{R}^{d}$ (in the case of spectrograms, $d=T\times F$) be the
representation in the original input domain and
$x^{\prime}\in\\{0,1\\}^{d^{\prime}}$ be the interpretable representation
where 0 and 1 denote the absence and presence of an interpretable feature,
respectively. In the next step we sample $N_{s}$ instances around $x^{\prime}$
by randomly generating vectors of length $d^{\prime}$ containing 0’s and 1’s.
Each generated instance $z^{\prime}$ is mapped back to the input domain and
fed through the original model $f$. $z\in\mathds{R}^{d}$ looks like the
original input example with all parts that are 0 in $z^{\prime}$ occluded
(e.g. gray values in the image domain). The $N_{s}$ instances $z^{\prime}$ and
the corresponding predictions $f(z)$ constitute the input to the explanation
model $g$ [4]. The most commonly used explainers are linear models of the
form222In the original paper [4], the explainer had the form
$g(z^{\prime})=w_{g}z^{\prime}$, mistakenly omitting the intercept $b$.
Inspecting the source code shows that the intercept was trained as well:
https://github.com/marcotcr/lime/

$g(z^{\prime})=b+w_{g}z^{\prime}.$ (1)

When training the linear explanation model, the generated instances are
weighted via an exponential kernel decaying with increasing distance to the
input example. This distance (and based on it the weight) is computed between
the binary representation of the examples being explained (all $1$’s) and each
of the instances in the neighborhood333The original paper [4] stated that the
distance for weighting the examples was computed between the images directly,
but the source code shows that it is actually computed between the binary
representations, which was later confirmed by the author. .

A requirement for an explanation is that it be locally faithful: we expect the
explainer $g$ to approximate $f$ in the neighborhood of $x$. The faithfulness
is measured by the fidelity score, which is computed by the coefficient of
determination ($R^{2}$) of the linear model’s predictions [4], and ranges from
0 to a perfect score of 1, with the exact interpretation of the range of
values being somewhat unclear.

Different flavours of LIME have been used for explaining predictions in a
variety of audio classification and tagging tasks. The general LIME algorithm
was kept the same and the difference is the type of interpretable features.
Mishra et al. [5] proposed the first adaption of LIME for MIR, termed
SoundLIME (SLIME), which segments the spectrogram into time, frequency, or
time-frequency segments. SLIME was demonstrated on the task of singing voice
detection [18] and used for analysing a replay spoofing detection system [6].
Haunschmid et al. used other types of interpretable features (super pixels
[7], source separation estimates [8, 9]) for explaining the predictions of a
variety of models, including music taggers [8, 9] and a content-based music
recommender system [10]. Mishra et al. [11] proposed different content types
for replacing the “grayed out” segments (e.g. zero, the min and the mean value
of the spectrogram) and found the mean value to work best in their setting.
For our experiments we will be using time-frequency segments [5, 11]. The
hyperparameters for the LIME algorithm are described in Section 4.4.

##  3 Adversarial Attacks

In this section, we briefly describe a way to compute adversarial examples for
a singing voice detection system. The goal is to obtain perturbations which do
not affect how a human perceives a particular example, yet simultaneously
change the prediction of the system, i.e., transform “Singing Voice” to “No
Singing Voice” or vice versa. To realise this, we use an adversarial attack
that was originally proposed for audio [19], and was previously applied in MIR
to attack an instrument classification system and a music recommender [20].
The attack, subsequently denoted by Carlini & Wagner (C&W), assumes a white-
box scenario, i.e. full knowledge about a model and its parameters.

As C&W is an iterative targeted attack, we use it to decrease the loss of a
system with respect to a new (wrong) target prediction, in multiple
iterations. Following the notation in [20], let $f$ be a system and
$x\in\mathds{R}^{d}$ its input, i.e., a specific prediction is denoted by
$f(x)$. Also, let $\delta_{ep}\in\mathds{R}^{d}$ be an adversarial
perturbation at iteration $ep$, and an adversarial example $x+\delta_{ep}$.
Furthermore, we denote the system loss by $L_{\mathrm{sys}}$, and the target
prediction by $t$. To restrict the adversarial perturbation, C&W minimises a
weighted sum of the squared L2-norm of $\delta$ and a system loss, resulting
in an optimisation objective as follows [19, 20]:

$\displaystyle L_{\mathrm{total}}$
$\displaystyle=\|\delta_{ep}\|^{2}_{2}+\alpha*L_{\mathrm{sys}}(f(x+\delta_{ep}),t),$
$\displaystyle\delta_{ep+1}$
$\displaystyle=\operatorname{clip}_{\epsilon}(\delta_{ep}-\eta*\operatorname{sign}(\nabla_{\delta_{ep}}\
L_{\mathrm{total}})).$ (2)

Note that $\nabla_{\delta_{ep}}$ is the gradient w.r.t. to ${\delta_{ep}}$,
and updates are performed based on the $\operatorname{sign}$ of the gradient
and the multiplicative factor $\eta$. By applying
$\operatorname{clip}_{\epsilon}$, an adversarial perturbation is always in the
range of $[-\epsilon,\epsilon]$, and $\alpha$ is used to balance the magnitude
of the perturbation and the system loss w.r.t. the target prediction.

In the following experiments, we will compute adversarial examples both for
raw audio waveforms and for input spectrograms. In either case, we will always
analyse the resulting perturbations in the time-frequency domain. For
adversaries computed for the waveform we compute the time-frequency
representation and subtract the original example to obtain the adversarial
perturbation in the time-frequency domain. For adversarial examples computed
for the spectrogram directly, time-frequency information is already available.

##  4 Experimental Setup

This section describes the experimental setup, including the data we use,
details about the singing voice detection system and hyperparameters for both
the adversarial attacks and computing the explanations.

###  4.1 Data

To train the singing voice detection system proposed by Schlüter and Lehner,
we use a subset of the data used in their work [16], namely the openly
available Jamendo dataset[21]. The dataset consists of 93 songs totalling
around 6 hours of music, with a proposed training / validation / test split of
61 / 16 / 16 files respectively. Each audio file has a sampling rate of
44.1kHz.

The annotations for the Jamendo dataset are provided with sub-second
granularity [18], and indicate the presence / absence of singing voice. The
proportion of singing voice (“sing”) in comparison to non-singing voice (“no
sing”) is close to $50:50$ in all three data splits.

###  4.2 Singing Voice Detector

For subsequent experiments, we adapt the singing voice detection system
introduced by Schlüter and Lehner [16]. We use the proposed architecture of
the Convolutional Neural Network (CNN) and deploy the training routine with
unaltered hyperparameters on the Jamendo dataset. Prior to training, the data
is resampled to 22.05kHz and then used to compute magnitude Mel spectrograms
(frame length 1024, hop size 315 samples and 80 Mel bands) [16]. The Mel
spectrograms are logarithmically scaled and normalised per frequency band to
have zero mean and unit variance over the training data.

The CNN is trained on spectrogram excerpts with a length of 115 frames, for
which the target prediction corresponds to the ground-truth annotation for the
central frame [16]. During training, we use a mini-batch size of 32 and Adam
to optimise the Binary Cross Entropy loss for 40.000 update steps. We start
with a learning rate of 0.001 and scale it by 0.85 every epoch. To support
generalisation, dropout and data augmentation [18] is used; for a more
detailed description of the training procedure, hyperparameters and the CNN
architecture, we refer to [16].

The final classification error of the singing voice detector (in percent) on
the Jamendo test data, given as the mean $\pm$ standard deviation over 5
different runs, is $11.54\pm 0.96$. Furthermore, recall and specificity over 5
runs are $89.61\pm 1.71$ and $87.46\pm 1.00$ respectively. The metrics are
computed based on binary predictions of the network, which are obtained after
applying a median filter and a threshold tuned on validation data (cf. [16]).
Note that exact reproduction of previously reported errors, namely 8.0 in [18]
(with a slightly different architecture) and 5.5 in [16], is difficult as
Schlüter and Lehner train on a larger (in-house) dataset, whereas we only use
publicly available data. Additionally, non-determinism (e.g., via data
augmentation or initial weights) can influence the performance of a model
noticeably.

In the following experiments, let the singing voice detection system be $f$,
and $f(x)$ the numeric model output for class “sing”. The final classification
is made by checking whether $f(x)$ is above (“sing”) or below (“no sing”) a
threshold that is optimised on the validation set, and is equal to 0.51 for
our system.

###  4.3 Hyperparameters for the Adversarial Attack

Equation (3) shows the hyperparameters we need to determine before attacking
the singing voice detector. The maximum number of iterations in which we try
to find adversarial perturbations that change the prediction of an excerpt is
set to 1000 in our experiments. Other hyperparameters – clipping factor
$\epsilon$, update factor $\eta$ and weight factor $\alpha$ – are tuned on the
validation set of Jamendo, and chosen as the setting that results in the
highest number of successful adversarial perturbations, i.e., changed the most
predictions out of all audio excerpts in the validation data. For the attack
on raw audio we use $\epsilon=0.01$, $\eta=0.0003$ and $\alpha=2$; for the
attack on the spectrograms we take $\epsilon=0.1$, $\eta=0.0005$ and
$\alpha=15$. Due to the binary nature of the singing voice detection task, the
target $t$ for each excerpt is chosen to be the opposite of its original
prediction. Note that we find successful perturbations for 28.4% of all
excerpts for attacks on the spectrogram, and for 60.5% of excerpts for raw
audio.

###  4.4 Hyperparameters for the Explanations

As mentioned above the LIME algorithm has a set of hyperparameters that have
to be chosen. We segment the spectrogram ($80$ frequency bins ($F$) $\times$
$115$ time frames ($T$)) into 5 time and 4 frequency segments, respectively,
resulting in 20 “human-interpretable” features of size $20$ frequency bins
$\times$ $23$ time frames. Repeating the calculation of an explanation might
lead to different results due to the randomness in the neighborhood generation
when the number of generated instances is too small. A preliminary experiment
as described in [5, 11] on a subset of the explanations, suggests that
$2^{13}=8192$ instances are sufficient to generate stable explanations, which
is why we set $N_{s}=8192$ for all our experiments. We also investigated
different ways of replacing explanation segments [11] (zero, min, and mean),
but as the results appeared similar, we will focus on reporting the results
for the mean content type. For weighting the instances to train the explainer
we use the cosine distance function and an exponential kernel with a kernel
width of $0.25$.

##  5 Evaluation of Explanations

To investigate to what extent local explanation methods can find the
underlying reason for a particular classification by a system, we perform four
experiments. In the first, we demonstrate how LIME can help detecting rather
obvious causes for a prediction that are due to manually altered spectrograms.
The goal of the remaining experiments is to quantitatively evaluate the
ability of LIME to detect adversarial perturbations: in the second experiment
we try to evaluate whether the segments that LIME highlights actually caused a
prediction; in the third, we analyse if the explanations can detect the
correct location of the adversarial perturbation if it only affects few
segments of the input spectrogram; and in the fourth, we compare the fidelity
scores for correct and incorrect explanations.

Due to the computation time needed for obtaining explanations, we subsequently
limit the number of analysed excerpts for each song in the test set. For the
first experiment, we manually alter a single random excerpt per song with an
original classification of “no sing”. In the setting of the second, third and
fourth experiment, for each of the 16 test songs, we randomly select 10
adversarial excerpts which were originally classified as “sing” and another 10
excerpts originally classified as “no sing”. We conduct all experiments with
adversarial attacks computed both on raw audio and on spectrograms, as we want
to account for the fact that attacks on raw audio might use implicit
constraints that are difficult to detect for LIME (which operates on
spectrograms).

To allow reproducibility, we publish code444https://github.com/CPJKU/veracity
for all subsequent experiments.

###  5.1 Explaining Manually Altered Spectrograms

In our first experiment, we exploit the fact that the predecessor of the
singing voice detector we use (cf. [18]) could easily be fooled by drawing on
the input spectrogram555https://github.com/f0k/singing_horse [17]. After
choosing a set of excerpts originally classified as “no sing”, we adapt
drawings on the respective spectrograms until the prediction is changed to
“sing”. Then we ask LIME to explain these new predictions. Figure 1 shows that
the explainer can correctly identify the manually altered parts of the
spectrogram as the cause for the wrong prediction. Examples like these,
supported by relatively high fidelity scores, might make us think that LIME
knows what is going on and make us trust these explanations. We could stop our
evaluation here and put trust in our model because the explanations highlight
seemingly correct behaviour [5], look meaningful [7], or match the expected
behavior [9], but we will continue with a more careful investigation.

Figure 1: Different examples of spectrograms for which the model’s original
prediction is “no sing” (left column) but drawing certain symbols (middle)
leads to a change to “sing” with a model output $f$. The rightmost column
shows the top 3 interpretable features (which may be directly adjacent and
thus look like one segment) picked by the LIME algorithm, and their fidelity.

###  5.2 Explaining Predictions on Adversarial Examples

In this set of experiments, we first compute explanations for the predictions
obtained for each of the selected adversarial excerpts. Every explanation
consists of a list of interpretable features (time-frequency segments) and
their corresponding weights. The weight of a particular feature should
indicate the importance of the feature for making a certain prediction [4, 5].
To evaluate whether the explanation actually selects the features that are
responsible for the (now wrong) prediction, we could naively check if the
segments selected by LIME correspond to regions of the adversarial
perturbation that have the highest magnitudes (defined via the L2 norm).

However, as we do not know which part of a perturbation really led to a
misclassification, i.e., whether parts of a perturbation with the highest
magnitude have the most influence or not, this could lead to false
conclusions.

To circumvent this, we follow another approach: we split the adversarial
perturbation into the same time-frequency segments that LIME is using as
interpretable features; we then selectively add those $k$ segments of the
perturbation to the input spectrogram that coincide with the features
identified as an explanation. If the selected features are actually explaining
the prediction, we expect the partial perturbation to be sufficient to achieve
the same change of classification, or what we call label flip, as the full
perturbation did. However, using $k=3$ (which seems a reasonable number of
segments to present to a user [5]) flips only $21-53\%$ of the predictions
(see Table 1).

| Waveform | Spectrogram
---|---|---
“no sing” $\rightarrow$ “sing” | 53 % | 41 %
“sing” $\rightarrow$ “no sing” | 21 % | 32 %
Table 1: Percentage of label flips when using only $k=3$ segments of an
adversarial perturbation, based on the most relevant features chosen by LIME.
Columns denote whether perturbations were computed for the waveform /
spectrogram, rows show original $\rightarrow$ target prediction.

We therefore next look at whether larger numbers $k$ of segments are able to
flip more of the predictions. We use either LIME for selection of segments or
choose the $k$ perturbation segments with the highest magnitudes. We do this
for $k\in[1,...,20]$ and report the results in Figure 2. For almost all $k$
and across different settings we are more often successful in flipping the
label when segment selection is based on segment magnitude rather than on LIME
results. For LIME-based segments it is necessary to choose almost all 20
available segments to achieve flip rates close to $100\%$. This result
suggests that it is indeed possible to explain predictions with larger numbers
of $k$ interpretable features, but only when using the norm of a perturbation
(which in a real application we would not know) to select the features and not
when using LIME for the selection.

(a) “no sing” $\rightarrow$ “sing”
(waveform)

(b) “no sing” $\rightarrow$ “sing”
(spectrogram)

(c) “sing” $\rightarrow$ “no sing”
(waveform)

(d) “sing” $\rightarrow$ “no sing”
(spectrogram)

Figure 2: Percentage of classifications we can flip (y-axis) by adding only
$k$ segments (x-axis) of an adversarial perturbation to an input. Choosing
which $k$ segments are used is either based on LIME explanations (triangles),
or on the norm of a perturbation (circles).

Finally, we analyse whether taking all segments that receive a positive weight
from LIME can flip a prediction, as positive weights should correlate
positively with the explained class [5]. For this analysis, we no longer add a
fixed number of segments of a perturbation to each excerpt, but instead select
all $k$ segments of a perturbation for which the corresponding interpretable
feature received a positive weight. This number of segments $k$ can be
different for each excerpt we look at. Additionally, we choose the same number
of $k$ segments based on the largest magnitude, and compare how often this
results in label flips. The results of this experiment are summarised in Table
2. Again, we observe that taking partial perturbations is more successful when
selecting the segments with the highest magnitude, as opposed to the segments
most relevant for LIME.

| Waveform | Spectrogram
---|---|---
| LIME | norm | LIME | norm
“no sing” $\rightarrow$ “sing” | 86 % | 99 % | 91 % | 98 %
“sing” $\rightarrow$ “no sing” | 49 % | 74 % | 44 % | 55 %
Table 2:  Percentage of label flips when adding all perturbation segments that
contributed positively according to LIME, compared to using the same number of
segments $k$ selected based on the norm.

These results suggest that LIME does not pick the input features that are most
relevant for making a particular prediction.

###  5.3 Explaining “Localised” Perturbations

Considering the results in the previous section, one might argue that an
adversarial perturbation can be present in the whole spectrogram, which could
make it hard for LIME to decide which segments are most relevant. We make use
of the finding that it is often sufficient to add only a small number of
selected segments of the perturbation to flip a label, and conduct an
additional experiment where we analyse how often LIME detects those segments.

This is similar to [15], where adversarial perturbations were constrained
beforehand such that only selected regions of the input could be modified, but
in contrast we do not restrict the perturbations directly. Instead we refine
our adversarial perturbations by first splitting them into segments that
correspond exactly to the time-frequency segments that LIME uses. We then use
only $k$ such segments of a perturbation, where the segments themselves are
chosen based on the highest magnitude.

We then identify the subset of adversarial excerpts whose label can be flipped
by only adding $k\in{1,3,5}$ segments of the perturbation. These subsets are
created for the four different settings, namely the two types of adversarial
attacks (waveform / spectrogram), and for the two possible label flips (“sing”
to “no sing” and vice versa). This leads to $3*2*2=12$ subsets of adversarial
excerpts.

(a) no sing $\rightarrow$ sing (waveform)

(b) no sing $\rightarrow$ sing (spectrogram)

(c) sing $\rightarrow$ no sing (waveform)

(d) sing $\rightarrow$ no sing (spectrogram)

Figure 3: The number of segments that are correctly identified when adding $k$
perturbed segments to the original input and asking LIME for an explanation
containing the same $k$ number of interpretable features.

After determining this subset of adversarial excerpts with partial
(“localised”) perturbations, we once again compute explanations with LIME.
Here we set the number of interpretable features that LIME should display to
$k$, i.e. the number of segments we previously added as localised
perturbations. Since the time-frequency segments of LIME and the partial
perturbations align, we can then examine how often LIME correctly identifies
the regions causing a particular prediction. Figure 3 shows the result of this
experiment, for all 12 adversarial subsets. Each plot depicts how many of the
$k$ segments that were added as adversarial perturbation, and that hence were
crucial for a particular classification, are correctly detected by LIME. It is
easiest to interpret results with only one modified segment (left column),
since we know exactly what the correct explanation should look like. Overall,
for less than half of the excerpts is the correct segment presented as an
explanation. For 3 and 5 modified segments we can also check how many segments
are correctly identified, and we can observe that it is rarely all of them and
for some settings none of them. This result suggests that even when the cause
is quite localised and aligns with the segments that we are using as
interpretable features, LIME is rarely able to recover all affected features.

###  5.4 Are Fidelity Scores Reliable?

As Figure 1 in Section 5.1 suggested previously, “correct” explanations are
often accompanied by relatively high fidelity values. We do not know, however,
whether we can use fidelity to determine if an explanation highlights the
segments of the input that are really the most relevant for a prediction, and
it has been demonstrated that a high-fidelity explanation of a black box model
might not reflect the actual cause for a prediction [22].

In the last experiment described above we have created a setting with $k=1$
where we exactly know which segment is relevant for the prediction, and we can
use this information to compare whether the fidelity is different for
“correct” and “wrong” explanations, i.e. explanations which match the added
segments of the perturbation as opposed to explanations which do not. As can
be seen from Figure 4, the fidelity for “correct” explanations is on average
only slightly higher than for “wrong” predictions, with ranges of fidelity
values strongly overlapping. Based on the fidelity score alone, a user hence
cannot decide whether an explanation is “correct” and whether it shows the
most relevant segment(s) for a prediction.

(a) no sing $\rightarrow$ sing
(waveform)

(b) no sing $\rightarrow$ sing
(spectrogram)

Figure 4: Fidelity scores for explanations that correctly and incorrectly
identified the most relevant segment.

##  6 Discussion and Conclusion

In a nutshell, the central results of our investigations can be summarised as
follows:

  * •

Figure 1: “Obvious” causes for predictions can be detected with high fidelity.

  * •

Table 1: When computing the 3 most relevant interpretable features with LIME,
we detect segments that are able to flip the label for only 21-53% of the
excerpts.

  * •

Figure 2: When using a varying number of interpretable features, we detect
fewer “correct” segments than when simply taking the same number of segments
with the highest magnitude.

  * •

Table 2: When using all interpretable features that received a positive
weight, we detect fewer “correct” segments than when taking the same number of
segments with the highest magnitude.

  * •

Figure 3: In a setting where we only add partial perturbations, only few
segments are correctly detected.

  * •

Figure 4: Based on the fidelity score it is impossible to judge the quality of
an explanation.

Taken together, we believe that these results support the following
conclusions: (1) local model-agnostic explanations cannot reliably detect the
input regions most relevant for a prediction unless they are rather obvious;
(2) evaluation based on “what looks reasonable” leads to accepting
explanations that do not reveal the real cause of a prediction; (3) the
fidelity score may give a false sense of security about the quality of
explanations.

In [15], Göpfert et al. write, “It might be tempting to judge explanatory
methods on whether they succeed in identifying features that a human observer
thinks should be relevant to the classification, but the existence of
adversarial examples shows that the reasoning of humans and neural networks
can differ dramatically.” This observation points to a fundamental dilemma
between “veridical” explanations that faithfully reflect the workings of the
classification model (and may not be accessible to model-agnostic explanation
methods such as LIME), and “human-interpretable” explanations that would
connect a machine decision to concepts familiar to us. Our results confirm
that one should be careful in interpreting model-agnostic explanations as
explanations of the underlying model, and that experiments with carefully
crafted examples are important to get more detailed insights into the
properties of such methods.

##  7 Acknowledgements

This work is supported by the Austrian National Science Fund (FWF, project
P31988).

## References

  * [1] S. Mishra, B. L. Sturm, and S. Dixon, “Understanding a Deep Machine Listening Model Through Feature Inversion,” in _Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, September 23-27, 2018_ , 2018, pp. 755–762.
  * [2] ——, “"What are You Listening to?" Explaining Predictions of Deep Machine Listening Systems,” in _Proceedings of the 26th European Signal Processing Conference, EUSIPCO 2018, Roma, Italy, September 3-7, 2018_. IEEE, 2018, pp. 2260–2264.
  * [3] S. Mishra, D. Stoller, E. Benetos, B. L. Sturm, and S. Dixon, “GAN-based Generation and Automatic Selection of Explanations for Neural Networks,” _CoRR_ , vol. abs/1904.09533, 2019.
  * [4] M. T. Ribeiro, S. Singh, and C. Guestrin, “"Why Should I Trust You?": Explaining the Predictions of Any Classifier,” in _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_. ACM, 2016, pp. 1135–1144.
  * [5] S. Mishra, B. L. Sturm, and S. Dixon, “Local Interpretable Model-agnostic Explanations for Music Content Analysis,” in _Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, Suzhou, China, October 23-27, 2017_ , 2017, pp. 537–543.
  * [6] B. Chettri, B. L. Sturm, and E. Benetos, “Analysing Replay Spoofing Countermeasure Performance under Varied Conditions,” in _28th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2018, Aalborg, Denmark, September 17-20, 2018_. IEEE, 2018, pp. 1–6.
  * [7] V. Haunschmid, S. Chowdhury, and G. Widmer, “Two-level Explanations in Music Emotion Recognition,” _Machine Learning for Music Discovery Workshop, ML4MD at ICML2019_ , 2019.
  * [8] V. Haunschmid, E. Manilow, and G. Widmer, “Towards Musically Meaningful Explanations Using Source Separation,” _CoRR_ , vol. abs/2009.02051, 2020\.
  * [9] ——, “audioLIME: Listenable Explanations Using Source Separation,” in _13th International Workshop on Machine Learning and Music_ , 2020, pp. 20–24.
  * [10] A. B. Melchiorre, V. Haunschmid, M. Schedl, and G. Widmer, “LEMONS: Listenable Explanations for Music recOmmeNder Systems,” in _Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II_ , ser. Lecture Notes in Computer Science, vol. 12657. Springer, 2021, pp. 531–536.
  * [11] S. Mishra, E. Benetos, B. L. Sturm, and S. Dixon, “Reliable Local Explanations for Machine Listening,” in _2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020_. IEEE, 2020, pp. 1–8.
  * [12] Z. C. Lipton, “The Mythos of Model Interpretability,” _Communications of the ACM_ , vol. 61, no. 10, pp. 36–43, 2018.
  * [13] J. Adebayo, J. Gilmer, M. Muelly, I. J. Goodfellow, M. Hardt, and B. Kim, “Sanity Checks for Saliency Maps,” in _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , 2018, pp. 9525–9536.
  * [14] D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju, “Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods,” in _AIES ’20: AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA, February 7-8, 2020_. ACM, 2020, pp. 180–186.
  * [15] J. P. Göpfert, H. Wersing, and B. Hammer, “Recovering Localized Adversarial Attacks,” in _Artificial Neural Networks and Machine Learning - ICANN 2019: Theoretical Neural Computation - 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part I_ , ser. Lecture Notes in Computer Science, vol. 11727\. Springer, 2019, pp. 302–311.
  * [16] J. Schlüter and B. Lehner, “Zero-Mean Convolutions for Level-Invariant Singing Voice Detection,” in _Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, France, September 23-27, 2018_ , 2018, pp. 321–326.
  * [17] J. Schlüter, “Deep Learning for Event Detection, Sequence Labelling and Similarity Estimation in Music Signals,” Ph.D. dissertation, Johannes Kepler University Linz, Austria, Jul. 2017.
  * [18] J. Schlüter and T. Grill, “Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks,” in _Proceedings of the 16th International Society for Music Information Retrieval Conference, ISMIR 2015, Málaga, Spain, October 26-30, 2015_ , 2015, pp. 121–126.
  * [19] N. Carlini and D. A. Wagner, “Audio Adversarial Examples: Targeted Attacks on Speech-to-Text,” in _Proceedings of the IEEE Security and Privacy Workshops, SP Workshops_. IEEE, 2018, pp. 1–7.
  * [20] K. Prinz, A. Flexer, and G. Widmer, “On End-to-End White-Box Adversarial Attacks in Music Information Retrieval,” _Transactions of the International Society for Music Information Retrieval_ , vol. 4, no. 1, pp. 93––104, 2021.
  * [21] M. Ramona, G. Richard, and B. David, “Vocal Detection in Music with Support Vector Machines,” in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2008, Las Vegas, Nevada, USA_. IEEE, 2008, pp. 1885–1888.
  * [22] H. Lakkaraju and O. Bastani, “"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations,” in _AIES ’20: AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA, February 7-8, 2020_. ACM, 2020, pp. 79–85.


Then $\tilde{W}_{n}$ as defined in (35) converges to $Y$ in the sense of
Katzenberger (see Def. 5.1), where $Y$ is the constrained gradient flow
defined as

$\frac{dY}{dt}=-P_{T\Gamma}\nabla_{w}\mathrm{Reg}(Y),\qquad
Y(t)\in\Gamma,\qquad\text{and}\qquad Y(0)=\Phi(W_{0}),$

and

$\mathrm{Reg}(w):=\frac{1}{2}\Delta_{\eta}\hat{L}(w,0)=\frac{1}{2N}\sum_{j=1}^{d_{\mathrm{in}}}(u_{j}^{2}-v_{j}^{2})^{2}\sum_{i=1}^{N}x_{ij}^{2},$
(93)

and where $x_{ij}$ is the $j^{\mathrm{th}}$ feature of the $i^{\mathrm{th}}$
data point $x_{i}$.

######  Remark 5.14.

The expression (93) coincides with a weighted $L^{2}$-regularization term in
linear models—treating $u^{\odot 2}-v^{\odot 2}$ as a single linear
parameter—in which the weights are chosen equal to the average amplitude of
the corresponding feature $\frac{1}{N}\sum_{i=1}^{N}x_{ij}^{2}$. ∎

######  Remark 5.15.

In [LWA21, Lemma 6.2, Lemma 6.3] sufficient conditions are given for
overparametrized linear models to satisfy the manifold Assumption 3.5. ∎

######  Remark 5.16 (Two Laplacians as regularizer).

Both for Dropout and for label noise the regularizer is a Laplacian, but one
is with respect to $\eta$ and the other with respect to $w$; thus these two
forms of regularization may lead to different types of behaviour. We
illustrate this on the example of overparameterized linear models, where
$f_{w}(x)=\left<u^{\odot 2}-v^{\odot 2},x\right>$ as in (91).

With Dropout noise we obtain the regularizer (93). With label noise the
regularizer is given by (87) instead, and we now make this label-noise
regularizer more explicit. On the zero loss manifold we have $\left<u^{\odot
2}-v^{\odot 2},x_{i}\right>=y_{i}$ for all $i$ and thus

$\displaystyle\frac{1}{2N}\Delta_{w}L$
$\displaystyle=\frac{1}{2N^{2}}\Delta_{u}\sum_{i=1}^{N}\left(\left<u^{\odot
2}-v^{\odot
2},x_{i}\right>-y_{i}\right)^{2}+\frac{1}{2N^{2}}\Delta_{v}\sum_{i=1}^{N}\left(\left<u^{\odot
2}-v^{\odot 2},x_{i}\right>-y_{i}\right)^{2}$
$\displaystyle=\frac{1}{2N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{d_{\mathrm{in}}}4u_{j}^{2}x_{ij}^{2}+\frac{1}{2N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{d_{\mathrm{in}}}4v_{j}^{2}x_{ij}^{2}=\frac{2}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{d_{\mathrm{in}}}(u_{j}^{2}+v_{j}^{2})x_{ij}^{2}.$
(94)

Note how the Dropout regularizer (93) regularizes the difference
$(u_{j}^{2}-v_{j}^{2})^{2}$, and the label noise regularizer above penalizes
both $u$ and $v$ separately. Also note how the two regularizers differ in
their scaling in $N$, with label noise leading to an additional slow-down
factor $1/N$. ∎

####  5.3.2 Feedforward neural networks

We say that $f_{w}:{\mathbb{R}}^{d_{\mathrm{in}}}\to{\mathbb{R}}$ is a
$p$-layered feedforward neural network if it is a composition of $p$ blocks in
which each block consists of a linear layer and a pointwise nonlinearity. Each
block is a mapping $\phi^{k}:y^{k}\to y^{k+1}\
({\mathbb{R}}^{d_{k}}\to{\mathbb{R}}^{d_{k+1}})$ of the form:

$\displaystyle z^{k+1}$ $\displaystyle=W^{k+1}y^{k}+b^{k+1},$ $\displaystyle
y^{k+1}$ $\displaystyle=s(z^{k+1}),$

where $W^{k+1}\in{\mathbb{R}}^{d_{k+1}\times d_{k}}$,
$b^{k+1}\in{\mathbb{R}}^{d_{k+1}}$, the input dimension is
$d_{0}=d_{\mathrm{in}}$ and the output dimension $d_{p}=1$. The map $f_{w}(x)$
then takes the form:

$f_{w}(x)=\phi^{p}(\phi^{p-1}(\cdots\phi^{1}(x))).$

We introduce dropout filters $\eta^{k}$ that perturb the input features at the
$k$-th layer. The resulting maps $\phi_{d}^{k}:(\eta^{k},y^{k})\to y^{k+1}\
({\mathbb{R}}^{2d_{k}}\to{\mathbb{R}}^{d_{k+1}})$ then are

$\displaystyle z^{k+1}$
$\displaystyle=W^{k+1}(y^{k}\odot(1+\eta^{k}))+b^{k+1},$ $\displaystyle
y^{k+1}$ $\displaystyle=s(z^{k+1}),$

and the corresponding $f_{w}^{\mathrm{drop}}$ is given by

$f_{w}^{\mathrm{drop}}(x,\eta)=\phi_{d}^{p}(\eta^{p},\phi_{d}^{p-1}(\eta^{p-1}\cdots\phi_{d}^{1}(\eta^{1},x))),$
(95)

Here we distinguish the same two choices as in Section 5.1, namely Bernoulli
and Gaussian filters.

In the Bernoulli case $\eta_{k,i}$ are i.i.d. Bernoulli random variables for
some $0<p<1$:

$\eta_{k,i}=\begin{cases}-1,&\text{ w.p. }p\\\ \frac{p}{1-p}&\text{ w.p.
}1-p.\\\ \end{cases}$

In the Gaussian case $\eta_{k,i}$ are i.i.d. normal random variables,

$\eta_{k,i}\sim{\mathcal{N}}(0,\sigma^{2}).$

With Gaussian filters, no input is ever ignored, as
${\mathbb{P}}(\eta_{k,i}=-1)=0$. Note that Gaussian noise also allows to
change sign of some inputs.

First consider feedforward neural networks with a single hidden layer:

$f_{w}^{\mathrm{drop}}(x,\eta)=\sum_{j=1}^{n}a_{j}(1+\eta_{j})s(b_{j}^{\top}x),$
(96)

where dropout is only applied to the output layer. As an example we consider
the _smooth rectified linear unit_ as activation function,

$s(x)=\begin{cases}\begin{array}[]{lr}0,\qquad\ \ \ \ x\leq 0\\\
xe^{-1/x},\quad x>0.\\\ \end{array}\end{cases}.$ (97)

######  Corollary 5.17.

Let $f_{w}^{\mathrm{drop}}$ as defined in (96) with rectified smooth
activation function (97). Assume that $L(w):=\hat{L}(w,0)$ satisfies
Assumption 3.5. Let $\tilde{W}_{n}(0)\Rightarrow\tilde{W}(0)\in U_{\Gamma}$
for some locally attractive neighbourhood $U_{\Gamma}$ of $\Gamma$. Let
$\eta_{k,i}$ be Bernoulli or Gaussian Dropout noisy variables as defined
above. Let $\alpha_{n},\sigma_{n}\to 0$.

Then $\tilde{W}_{n}$ as defined in (35) converges to $Y$ in the sense of
Katzenberger, where $Y$ is the constrained gradient flow defined as

$\frac{dY}{dt}=-P_{T\Gamma}\nabla_{w}\mathrm{Reg}(Y),\qquad
Y(t)\in\Gamma,\qquad\text{and}\qquad Y(0)=\Phi(W_{0}),$ (98)

with

$\mathrm{Reg}(w)=\frac{1}{2}\Delta_{\eta}\hat{L}(w,0)=\frac{1}{N}\sum_{j=1}^{n}\sum_{i=1}^{N}a_{j}^{2}\,s(b_{j}^{\top}x_{i})^{2}.$
(99)

######  Remark 5.18 (Known properties of the zero-loss manifold).

In [Coo18] it is proven that for overparameterized shallow neural networks
with the rectified smooth activation function (97) the zero-loss set of
$L(w):=\hat{L}(w,0)$ is a smooth manifold and $L$ satisfies Assumption 3.5.
The result also holds for deep feedforward neural networks if the size of the
last layer is greater than the size of the training dataset, i.e. $d_{k}>N$.

Similar results are known for the $\operatorname{ReLU}$ activation function
$\operatorname{ReLU}(y)=\max(0,y)$ [PTS20, DK22]. Even though the
$\operatorname{ReLU}$ is not differentiable in $y=0$, the manifold is locally
smooth away from the hyperplanes $Q_{i}=\\{x:w_{i}^{\top}x=0\\}$, so that the
analysis holds for every set $K$ satisfying $K\cap Q_{i}=\emptyset$ for every
$Q_{i}$. In [BPVF22] the authors study the structure of the neighbourhood $U$
and propose an initialization scheme guaranteeing convergence of the gradient
flow for shallow $\operatorname{ReLU}$ networks. ∎

###### Proof.

First note that the smoothness of $s$ guarantees the regularity of $\hat{L}$,
and since $s$ has sublinear growth, $\hat{L}$ has at most growth of order $6$
in $(w,\eta)$.

Next, note that we can write $\hat{L}$ in the form

$\hat{L}(w,\eta)=L(w)+f(w)\cdot\eta+\frac{1}{2}H(w):(\eta\mathop{\otimes}\eta),$

with

$f(w)_{j}=-\frac{2}{N}a_{j}\sum_{i=1}^{N}s(b_{j}^{\top}x_{j})y_{i}\qquad\text{and}\qquad
H_{jj^{\prime}}=\frac{2}{N}\sum_{i=1}^{N}a_{j}s(b_{j}^{\top}x_{i})a_{j^{\prime}}s(b_{j^{\prime}}^{\top}x_{i}).$

This structure is similar to (56), which we require for the degenerate case,
but note that for the degenerate case we require $\operatorname{tr}H=0$, while
here $\operatorname{tr}H$ does not vanish. Consequently the evolution takes
place at the time scale $1/\alpha_{n}\sigma_{n}^{2}$. This function $\hat{L}$
satisfies conditions (10) and (11) with $C_{2}=0$.

We now turn to condition (12). Gaussian noise satisfies this condition as
mentioned in Remark 3.3, and for the $q$-th absolute moment of Bernoulli
random variables, where $q\geq 1$, we have

${\mathrm{M}}_{q}=1\cdot
p+\frac{p^{q}}{(1-p)^{q}}\cdot(1-p)=\frac{p^{q}+p\cdot(1-p)^{q-1}}{(1-p)^{q-1}}=O(p)=O(\sigma^{2})\qquad\text{as
}p\to 0.$

Since $C_{2}=0$, condition (12) also is satisfied. The condition (36) in
Theorem 4.1 is satisfied for Bernoulli filters by construction and for
Gaussian filters by Lemma B.1.

Theorem 4.1 then yields Katzenberger convergence to the constrained gradient
flow (98), with
$\mathrm{Reg}(w)=\frac{1}{2}\Delta_{\eta}\hat{L}(w,0)=\frac{1}{2}\operatorname{tr}H(w)$
as given in (99). ∎

For deep neural networks, with multiple hidden layers, and for Gaussian
filters Theorem 4.1 again implies a convergence result. Due to the complexity
of deep neural networks we do not provide an explicit form of the regularizer
in this case.

######  Corollary 5.19 (Convergence for Dropout noise in deep neural
networks).

Let $f_{w}^{\mathrm{drop}}$ be as defined in (95) for some $p\geq 2$ with
rectified smooth activation function $s$ (97) Assume that $L(w):=\hat{L}(w,0)$
satisfies Assumption 3.5. Let $\tilde{W}_{n}(0)\Rightarrow\tilde{W}(0)\in
U_{\Gamma}$ for some locally attractive neighbourhood $U_{\Gamma}$ of
$\Gamma$. Let $\eta_{k,i}$ be Gaussian filter variables as defined above. Let
$\alpha_{n},\sigma_{n}\to 0$.

Then $W_{n}$ as defined in (35) converges to $Y$ in the sense of Katzenberger,
where $Y$ is the constrained gradient flow defined as

$\frac{dY}{dt}=-\frac{1}{2}P_{T\Gamma}\nabla_{w}\Delta_{\eta}\hat{L}(Y,0),\qquad
Y(t)\in\Gamma,\qquad\text{and}\qquad Y(0)=\Phi(W_{0}).$

###### Proof.

The proof is analogous to the proof of Corollary 5.17 with the only difference
caused by the composition of several layers. Using the global boundedness of
the derivatives of activation function
$|s^{\prime}(x)|,|s^{\prime\prime}(x)|<C$ for all $x\in{\mathbb{R}}$ one can
see that for any fixed $w\in K$ the expression
$|\nabla_{w}\nabla_{\eta}^{2}\hat{L}(w,\eta_{1})-\nabla_{w}\nabla_{\eta}^{2}\hat{L}(w,\eta_{2})|$
scales at most polynomially in $\eta_{1,2}$ and thus the assumptions of
Theorem 4.1 are satisfied. ∎

######  Remark 5.20 (Combining Dropout with Minibatching).

It is easy to see that the combination of Dropout noise and minibatching
results in the same dynamics as Dropout gradient descent without minibatching.
Consider the loss

$\hat{L}(w,\tilde{\eta},\eta)=\frac{1}{N}\sum_{i=1}^{N}(1+\tilde{\eta}_{i})\hat{\ell}(w,\eta,x_{i},y_{i}).$

As $\hat{L}$ is linear in the minibatching noise variables $\tilde{\eta}$, we
have $\Delta_{\tilde{\eta}}\hat{L}(w,\tilde{\eta},\eta)=0$ and thus the
regularizer takes the same form as the regularizer of the corresponding
dropout gradient descent
$\frac{1}{2}\Delta_{\tilde{\eta},\eta}\hat{L}(w,\tilde{\eta},\eta)=\frac{1}{2}\Delta_{\eta}\hat{L}(w,\tilde{\eta},\eta)$.
The same holds for the DropConnect case. ∎

##  6 Discussion and outlook

In this paper we define a class of noisy gradient-descent systems and prove
their convergence in the limit of small step size and in some cases also small
noise. This class of systems unifies a broad collection of existing training
algorithms in a common structure, and the convergence theorems thus give a
more global understanding of the effect of noise in various overparametrized
training situations.

In this section we add some more remarks and discuss various generalizations.

###  6.1 Constant step sizes

It is common to change the step size $\alpha$ during training, for instance to
generate phases of larger and smaller noise in minibatch SGD. For some such
non-constant step-size training algorithms the results of this paper should
continue to hold. The essential properties that need to be verified are
Assumptions 3.11, 3.12, and 3.13, which are all formulated in terms of the
integrator sequences $A_{n}$ and $\hat{A}_{n}$ and therefore are also
meaningful for non-constant step sizes.

###  6.2 Correlated noise

Similarly, we have chosen to make the coordinates $\eta_{k,i}$ of each iterate
$\eta_{k}$ independent from each other, but this again only for simplicity of
formulation.

To give an example of a generalization, consider again the example of Figures
1 and 2 in the introduction. Figure 4 compares the effect of uncorrelated
(left) and correlated noise. Here we choose as uncorrelated noise

$\eta_{k,i}\sim{\mathcal{N}}(0,\sigma^{2}I_{2})$

and as correlated noise

$\eta_{k}\sim{\mathcal{N}}(0,C),\qquad
C=\frac{\sigma^{2}}{2}\begin{pmatrix}1&1\\\ 1&1\end{pmatrix}.$

The corresponding regularizers are

uncorrelated: $\displaystyle\frac{1}{2}\Delta_{w}L(w)$ correlated:
$\displaystyle\frac{1}{2}\nabla^{2}_{w}L(w):C=\frac{1}{4}\left(\partial_{11}L+2\partial_{12}L+\partial_{22}L\right),$

and the minimizers of these two functions are indicated by green circles.

Figure 4: When the coordinates of the vector $\eta_{k}$ are correlated, the
resulting regularizer $\mathrm{Reg}$ is modified, and the evolution follows a
different path along $\Gamma$. In both diagrams each $\eta_{k}$ is a centered
normal two-dimensional random variable with covariance $C$, independent for
each $k$; in the left-hand picture $C=\sigma^{2}I_{2}$, implying independence
of $\eta_{k,1}$ from $\eta_{k,2}$, while in the right-hand picture
$C=\tfrac{1}{2}{\sigma^{2}}\begin{pmatrix}1&1\\\ 1&1\end{pmatrix}$, implying
that $\eta_{k,1}$ and $\eta_{k,2}$ are fully correlated.

###  6.3 Constrained gradient flow and regularisation

In this paper we have used the term ‘regularizer’ and the notation
‘$\mathrm{Reg}$’ for the function that drives the limiting constrained
gradient flow (4). This terminology is inspired by its relationship with
Tychonov regularization of inverse problems.

To explain this, note that solutions of the constrained gradient flow tend to
converge to local minimizers of $\mathrm{Reg}$, or if one is ‘lucky’, even to
a global minimizer, i.e. a solution of the constrained problem

$\min_{w}\bigl{\\{}\mathrm{Reg}(w):w\in\Gamma\bigr{\\}}.$

This situation is reminiscent of regularized inverse problems of the form

$\min_{u}\|Tu-f\|^{2}+\lambda R(u),$

for which in the ‘weak-regularisation’ limit $\lambda\to 0$ the minimizers
$u_{\lambda}$ converge to the solution of the constrained minimization problem

$\min_{u}\bigl{\\{}R(u):Tu=f\bigr{\\}}.$

This is why we call $\mathrm{Reg}$, the driving functional in the constrained
gradient flow, the (implicit) regulariser of the noisy gradient descent.

Theorem A opens the door to a form of reverse engineering. Given a loss $L$,
the choice of $\hat{L}$ is only limited by the consistency
$\hat{L}(w,0)=L(w)$, and one therefore has a wide freedom to tailor $\hat{L}$
to have particular properties. Assuming that one has an understanding of what
‘good’ and ‘bad’ points $w\in\Gamma$ look like, Theorem A suggests to look for
functions $\hat{L}$ such that $\Delta_{\eta}\hat{L}$ gives high value to ‘bad’
points and low value to ‘good’ ones.

Some pointers to how ‘good’ and ‘bad’ points can be recognized or
characterized are given by the ‘robustness’ criterion of [PTS20] (see Remark
5.4) or the discussion in [OKP+22] of the connection with PAC-Bayes bounds. We
leave this aspect to future work.

###  6.4 Convergence results in Skorokhod spaces and Katzenberger’s theorem

The first (to our knowledge) application of Katzenberger’s theorem to machine
learning models was in [LWA21]. In that paper the dynamics the noise is
assumed to be of minibatch-type, namely

$dW_{n}=-\nabla
L(W_{n})d\hat{A}_{n}+\sqrt{\varSigma}\sigma_{\eta_{k}}(W_{n}),$

where $\eta_{k}$ is sampled uniformly from the set $B=\\{1,2,\dots,N\\}$, and
for every $i\in B$, $\sigma_{i}(W_{n})$ is a deterministic function and
${\mathbb{E}}\sigma_{\eta_{k}}(W)=0$. Note that by definition the noise
process in such a setting is a martingale. Our definition of noisy gradient-
descent systems generalizes this, and allows us to study for instance the
effect of dropout noise.

The result of [LWA21] has also been generalized to SGD with momentum. In
[CCG22], the authors study the interplay between the momentum parameter and
the noise distribution. The structure of the noise is the same as in [LWA21],
and one direction of future work is the study of SGD with momentum with
general noise.

Another type of convergence results in the sense of processes are so-called
mean-field limit results. In contrast to this work, mean-field convergence
describes the behaviour of the models with variable number of parameters. For
example, for shallow neural networks it studies the limiting training dynamic
of models

$f_{n}(W_{n},x)=\frac{1}{n}\sum_{i=1}^{n}a_{i}\sigma(b_{i}^{\top}x+c_{i}),$

where $w_{i}=(a_{i},b_{i},c_{i})$ and $W=(w_{1},...w_{n})$. It turns out that
under suitable assumptions for $\mu_{n}$ defined as

$\mu_{n}(s)=\frac{1}{n}\sum_{i=1}^{n}\delta_{w_{i}(s)},$

it holds that $\mu_{n}\Rightarrow\mu$ in Skorokhod topology on $[0,T]$, where
$\mu$ is a solution of a measure-valued evolution equations characterized by
the loss function [SS20, RVE18]. Similar results have been derived for deep
neural networks [SS22, Ngu19]. Note that the mean-field setting does not
involve rescaling of time, implying that only the main (the fast) time scale
is considered. Another direction of future work is to study the slow time
scale dynamics in the measure-valued setting. Methods such as those in
[DLPS17, DLP+18] could be useful for this.

##  Appendix A Details of numerical simulations

The function $L:{\mathbb{R}}^{2}\to{\mathbb{R}}$ depicted in Figures 1, 2, and
3 is

$L(w)=\frac{(|w|^{2}-1)^{2}}{(|w|^{2}+1)^{2}}(1+a\sin(bw_{1})),\qquad a=0.7,\
b=5.$

The step size in those figures is $\alpha=0.3$, and $\eta$ has a centered
normal distribution with covariance $\sigma^{2}I_{2}$ with $\sigma=0.03$.

The constrained gradient flow in Figures 2 and 3 is implemented by
parametrizing by $\Gamma$ by polar angle $\theta$ and writing the regularizer
as a function of $\theta$.

##  Appendix B Auxiliary results

The following lemma shows that for i.i.d. Gaussian noise variables $\eta$ the
two conditions (36) and (58) follow from the assumption $\alpha_{n}\to 0$.

######  Lemma B.1 (Gaussian filters satisfy the noise-decay condition).

Let $\alpha_{n}$ and $\sigma_{n}$ be positive sequences such that
$\alpha_{n}\to 0$ and $\sigma_{n}\to\sigma_{0}\geq 0$. Let for each $n$,
$Y_{k}^{n}$, $k\in{\mathbb{N}}$, be i.i.d. centered Gaussian random variables
with variance $\sigma_{n}^{2}$. Then for any $T\in{\mathbb{R}}_{+}$, for any
$p\geq 1$ the following convergence holds in probability and in distribution:

$\sup_{k\leq\frac{T}{\alpha_{n}^{2}\sigma_{n}^{2}}}\alpha_{n}|Y_{k}^{n}|^{p}\longrightarrow
0\qquad\text{as }n\to\infty$

###### Proof.

$\displaystyle{\mathbb{P}}\biggl{(}\sup_{k\leq\frac{T}{\alpha_{n}^{2}\sigma_{n}^{2}}}\alpha_{n}|Y^{n}_{k}|^{p}>\varepsilon\biggr{)}$
$\displaystyle\leq\sum_{k\leq\frac{T}{\alpha_{n}^{2}\sigma_{n}^{2}}}{\mathbb{P}}\left(|Y^{n}_{k}|^{p}>\frac{\varepsilon}{\alpha_{n}}\right)\leq\frac{T}{\alpha_{n}^{2}\sigma_{n}^{2}}{\mathbb{P}}\left(|Y^{n}_{1}|>\frac{\varepsilon^{1/p}}{\alpha_{n}^{1/p}}\right)$
$\displaystyle=\frac{T}{2\alpha_{n}^{2}\sigma_{n}}\operatorname{erfc}\left(\frac{\varepsilon^{1/p}}{\sigma_{n}\alpha_{n}^{1/p}\sqrt{2}}\right)$
$\displaystyle\leq\frac{\alpha_{n}^{-2+1/p}\,T}{\varepsilon^{1/p}\sqrt{2\pi}}\exp\left(-\frac{\varepsilon^{2/p}}{2\sigma_{n}^{2}\alpha_{n}^{2/p}}\right)$
$\displaystyle=\frac{\beta_{n}^{-p+1/2}\,T}{\varepsilon^{1/p}\sqrt{2\pi}}\exp\left(-\frac{\varepsilon^{2/p}}{2\sigma_{n}^{2}\beta_{n}}\right),\qquad\text{with
}\beta_{n}=\alpha_{n}^{2/p},$

and for every fixed $\varepsilon>0$ this vanishes as $\beta_{n}\to 0$. ∎

## References

  * [ALP22] S. Arora, Z. Li, and A. Panigrahi.  Understanding gradient descent on the edge of stability in deep learning.  In International Conference on Machine Learning, pages 948–1024. PMLR, 2022.
  * [AVPVF22] M. Andriushchenko, A. Varre, L. Pillaud-Vivien, and N. Flammarion.  SGD with large step sizes learns sparse features.  arXiv preprint arXiv:2210.05337, 2022.
  * [BGVV20] G. Blanc, N. Gupta, G. Valiant, and P. Valiant.  Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process.  In Conference on learning theory, pages 483–513. PMLR, 2020.
  * [Bil68] P. Billingsley.  Convergence of Probability Measures.  Wiley Series in Probability and Statistics. John Wiley & Sons Inc., 1968\.
  * [BPVF22] E. Boursier, L. Pillaud-Vivien, and N. Flammarion.  Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs.  arXiv preprint arXiv:2206.00939, 2022.
  * [BS13] P. Baldi and P. J. Sadowski.  Understanding dropout.  Advances in neural information processing systems, 26, 2013.
  * [CCG22] A. Cowsik, T. Can, and P. Glorioso.  Flatter, faster: scaling momentum for optimal speedup of sgd.  arXiv preprint arXiv:2210.16400, 2022.
  * [CCS+19] P. Chaudhari, A. Choromanska, S. Soatto, Y. LeCun, C. Baldassi, C. Borgs, J. Chayes, L. Sagun, and R. Zecchina.  Entropy-SGD: Biasing gradient descent into wide valleys.  Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124018, 2019.
  * [CLSH23] G. Clara, S. Langer, and J. Schmidt-Hieber.  Dropout regularization versus $\ell_{2}$-penalization in the linear model.  arXiv preprint arXiv:2306.10529, 2023.
  * [CM97] A. Calzolari and F. Marchetti.  Limit motion of an Ornstein–Uhlenbeck particle on the equilibrium manifold of a force field.  Journal of Applied Probability, 34(4):924–938, 1997.
  * [Coo18] Y. Cooper.  The loss landscape of overparameterized neural networks.  arXiv preprint arXiv:1804.10200, 2018.
  * [DK22] S. Dereich and S. Kassing.  On minimal representations of shallow relu networks.  Neural Networks, 148:121–128, 2022.
  * [DK24] S. Dereich and S. Kassing.  Convergence of stochastic gradient descent schemes for lojasiewicz-landscapes.  arXiv preprint arXiv:2102.09385, 2024.
  * [DLP+18] M. H. Duong, A. Lamacz, M. A. Peletier, A. Schlichting, and U. Sharma.  Quantification of coarse-graining error in Langevin and overdamped Langevin dynamics.  Nonlinearity, 31(10):4517–4566, 2018.
  * [DLPS17] M. H. Duong, A. Lamacz, M. A. Peletier, and U. Sharma.  Variational approach to coarse-graining of generalized gradient flows.  Calc. Var. Partial Differ. Equ., 56(4):65, 2017.  Id/No 100.
  * [DML21] A. Damian, T. Ma, and J. D. Lee.  Label noise SGD provably prefers flat global minimizers.  arXiv preprint arXiv:2106.06530, 2021.
  * [Fal83] K. Falconer.  Differentiation of the limit mapping in a dynamical system.  Journal of the London Mathematical Society, 2(2):356–372, 1983\.
  * [FGJ20] B. Fehrman, B. Gess, and A. Jentzen.  Convergence rates for the stochastic gradient descent method for non-convex objective functions.  Journal of Machine Learning Research, 21(136):1–48, 2020.
  * [FKVE10] I. Fatkullin, G. Kovacic, and E. Vanden-Eijnden.  Reduced dynamics of stochastically perturbed gradient flows.  Comm. Math. Sci., 2010.
  * [FN93] T. Funaki and H. Nagai.  Degenerative convergence of diffusion process toward a submanifold by strong drift.  Stochastics: An International Journal of Probability and Stochastic Processes, 44(1-2):1–25, 1993.
  * [Fun95] T. Funaki.  The scaling limit for a stochastic PDE and the separation of phases.  Probability Theory and Related Fields, 102(2):221–288, 1995.
  * [FV13] B. Frénay and M. Verleysen.  Classification in the presence of label noise: A survey.  IEEE transactions on neural networks and learning systems, 25(5):845–869, 2013.
  * [GBC16] I. Goodfellow, Y. Bengio, and A. Courville.  Deep learning.  MIT press, 2016.
  * [GLHA23] X. Gu, K. Lyu, L. Huang, and S. Arora.  Why (and when) does Local SGD generalize better than SGD?  arXiv preprint arXiv:2303.01215, 2023.
  * [GM91] S. B. Gelfand and S. K. Mitter.  Simulated annealing type algorithms for multivariate optimization.  Algorithmica, 6(1):419–436, June 1991.  ZSCC: 0000059.
  * [HHS17] E. Hoffer, I. Hubara, and D. Soudry.  Train longer, generalize better: closing the generalization gap in large batch training of neural networks.  Advances in neural information processing systems, 30, 2017.
  * [HS97] S. Hochreiter and J. Schmidhuber.  Flat minima.  Neural computation, 9(1):1–42, 1997.
  * [HSK+12] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov.  Improving neural networks by preventing co-adaptation of feature detectors.  arXiv preprint arXiv:1207.0580, 2012.
  * [Hsu02] E. P. Hsu.  Stochastic Analysis on Manifolds.  Number 38. American Mathematical Soc., 2002.
  * [JKA+17] S. Jastrzębski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y. Bengio, and A. Storkey.  Three factors influencing minima in SGD.  arXiv preprint arXiv:1711.04623, 2017.
  * [JKA+18] S. Jastrzębski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y. Bengio, and A. Storkey.  Finding flatter minima with SGD.  In ICLR, 2018.
  * [JNM+19] Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio.  Fantastic generalization measures and where to find them.  arXiv preprint arXiv:1912.02178, 2019.
  * [JYY+22] G. Jin, X. Yi, P. Yang, L. Zhang, S. Schewe, and X. Huang.  Weight expansion: A new perspective on dropout and generalization.  arXiv preprint arXiv:2201.09209, 2022.
  * [Kat91] G. S. Katzenberger.  Solutions of a Stochastic Differential Equation Forced Onto a Manifold by a Large Drift.  The Annals of Probability, 19(4):1587 – 1628, 1991.
  * [KMN+16] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  arXiv preprint arXiv:1609.04836, 2016.
  * [KP91] T. G. Kurtz and P. Protter.  Weak limit theorems for stochastic integrals and stochastic differential equations.  The Annals of Probability, pages 1035–1070, 1991.
  * [LWA21] Z. Li, T. Wang, and S. Arora.  What happens after SGD reaches zero loss?–a mathematical framework.  arXiv preprint arXiv:2110.06914, 2021.
  * [LWY22] Z. Li, T. Wang, and D. Yu.  Fast mixing of Stochastic Gradient Descent with normalization and weight decay.  Advances in Neural Information Processing Systems, 35:9233–9248, 2022.
  * [MA19] P. Mianjy and R. Arora.  On dropout and nuclear norm regularization.  In International conference on machine learning, pages 4575–4584. PMLR, 2019.
  * [MA20] P. Mianjy and R. Arora.  On convergence and generalization of dropout training.  Advances in Neural Information Processing Systems, 33:21151–21161, 2020.
  * [MAV17] D. Molchanov, A. Ashukha, and D. Vetrov.  Variational dropout sparsifies deep neural networks.  In International Conference on Machine Learning, pages 2498–2507. PMLR, 2017.
  * [MAV18] P. Mianjy, R. Arora, and R. Vidal.  On the implicit bias of dropout.  In International conference on machine learning, pages 3540–3548. PMLR, 2018.
  * [MMN18] S. Mei, A. Montanari, and P.-M. Nguyen.  A mean field view of the landscape of two-layer neural networks.  Proceedings of the National Academy of Sciences, 115(33):E7665–E7671, 2018.
  * [MPP+22] O. A. Manita, M. A. Peletier, J. W. Portegies, J. Sanders, and A. Senen-Cerda.  Universal approximation in dropout neural networks.  Journal of Machine Learning Research, 23(19):1–46, 2022.
  * [Ngu19] P.-M. Nguyen.  Mean field limit of the learning dynamics of multilayer neural networks.  arXiv preprint arXiv:1902.02880, 2019.
  * [OKP+22] A. Orvieto, H. Kersting, F. Proske, F. Bach, and A. Lucchi.  Anticorrelated noise injection for improved generalization.  In International Conference on Machine Learning, pages 17094–17116. PMLR, 2022.
  * [Par12] T. Parsons.  Asymptotic analysis of some stochastic models from population dynamics and population genetics.  PhD thesis, University of Toronto, 2012.
  * [PKA+21] H. Petzka, M. Kamp, L. Adilova, C. Sminchisescu, and M. Boley.  Relative flatness and generalization.  Advances in neural information processing systems, 34:18420–18432, 2021.
  * [PPVF21] S. Pesme, L. Pillaud-Vivien, and N. Flammarion.  Implicit bias of SGD for diagonal linear networks: A provable benefit of stochasticity.  Advances in Neural Information Processing Systems, 34:29218–29230, 2021.
  * [PR15] T. L. Parsons and T. Rogers.  Dimension reduction via timescale separation in stochastic dynamical systems.  Arxiv preprint arXiv:01510.07031, 2015.
  * [PTS20] H. Petzka, M. Trimmel, and C. Sminchisescu.  Notes on the symmetries of 2-layer relu-networks.  In Proceedings of the Northern Lights Deep Learning Workshop, volume 1, pages 6–6, 2020.
  * [RRT17] M. Raginsky, A. Rakhlin, and M. Telgarsky.  Non-convex learning via stochastic gradient Langevin dynamics: A nonasymptotic analysis.  In Conference on Learning Theory, pages 1674–1703. PMLR, 2017.
  * [RVE18] G. M. Rotskoff and E. Vanden-Eijnden.  Trainability and accuracy of neural networks: An interacting particle system approach.  arXiv preprint arXiv:1805.00915, 2018.
  * [SCS20] A. Senen-Cerda and J. Sanders.  Almost sure convergence of dropout algorithms for neural networks.  arXiv preprint arXiv:2002.02247, 2020.
  * [SCS22] A. Senen-Cerda and J. Sanders.  Asymptotic convergence rate of dropout on shallow linear neural networks.  Proceedings of the ACM on Measurement and Analysis of Computing Systems, 6(2):1–53, 2022.
  * [SED20] S. Smith, E. Elsen, and S. De.  On the generalization benefit of noise in stochastic gradient descent.  In International Conference on Machine Learning, pages 9058–9067. PMLR, 2020.
  * [SHK+14] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [SS20] J. Sirignano and K. Spiliopoulos.  Mean field analysis of neural networks: A central limit theorem.  Stochastic Processes and their Applications, 130(3):1820–1852, 2020\.
  * [SS22] J. Sirignano and K. Spiliopoulos.  Mean field analysis of deep neural networks.  Mathematics of Operations Research, 47(1):120–152, 2022.
  * [Tad15] V. B. Tadić.  Convergence and convergence rate of stochastic gradient search in the case of multiple and non-isolated extrema.  Stochastic Processes and their Applications, 125(5):1715–1755, 2015\.
  * [WKM20] C. Wei, S. Kakade, and T. Ma.  The implicit and explicit regularization effects of dropout.  In International conference on machine learning, pages 10181–10192. PMLR, 2020.
  * [WM13] S. Wang and C. Manning.  Fast dropout training.  In international conference on machine learning, pages 118–126. PMLR, 2013.
  * [WME18] L. Wu, C. Ma, and W. E.  How SGD selects the global minima in over-parameterized learning: A dynamical stability perspective.  Advances in Neural Information Processing Systems, 31, 2018.
  * [Woj23] S. Wojtowytsch.  Stochastic gradient descent with noise of machine learning type part I: Discrete time analysis.  Journal of Nonlinear Science, 33(3):45, 2023.
  * [Woj24] S. Wojtowytsch.  Stochastic gradient descent with noise of machine learning type part II: Continuous time analysis.  Journal of Nonlinear Science, 34(1):16, 2024.
  * [WWL13] S. Wager, S. Wang, and P. S. Liang.  Dropout training as adaptive regularization.  Advances in neural information processing systems, 26, 2013.
  * [WWS22] L. Wu, M. Wang, and W. Su.  The alignment property of SGD noise and how it helps select flat minima: A stability analysis.  Advances in Neural Information Processing Systems, 35:4680–4693, 2022.
  * [WZZ+13] L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus.  Regularization of neural networks using dropconnect.  In International conference on machine learning, pages 1058–1066. PMLR, 2013.
  * [ZLR+18] C. Zhang, Q. Liao, A. Rakhlin, B. Miranda, N. Golowich, and T. Poggio.  Theory of deep learning IIb: Optimization properties of SGD.  arXiv preprint arXiv:1801.02254, 2018.
  * [ZX22] Z. Zhang and Z.-Q. J. Xu.  Implicit regularization of dropout.  arXiv preprint arXiv:2207.05952, 2022.