# Manipulating Feature Visualizations with
Gradient Slingshots

Dilyara Bareeva  Department of Artificial Intelligence, Fraunhofer Heinrich-
Hertz-Institute, Berlin, Germany  Understandable Machine Intelligence Lab,
ATB, Potsdam, Germany  Machine Learning Group, Technische Universität Berlin,
Germany  Marina M.-C. Höhne  Understandable Machine Intelligence Lab, ATB,
Potsdam, Germany  BIFOLD – Berlin Institute for the Foundations of Learning
and Data, Germany  Machine Learning Group, UiT the Arctic University of
Norway, Tromsø, Norway  Alexander Warnecke  BIFOLD – Berlin Institute for the
Foundations of Learning and Data, Germany  Machine Learning and Security
Group, Technische Universität Berlin, Germany  Lukas Pirch  Machine Learning
and Security Group, Technische Universität Berlin, Germany  Klaus-Robert
Müller  Machine Learning Group, Technische Universität Berlin, Germany  BIFOLD
– Berlin Institute for the Foundations of Learning and Data, Germany
Department of Artificial Intelligence, Korea University, Seoul, Republic of
Korea  Max Planck Institute for Informatics, Saarbrücken, Germany  Konrad
Rieck  BIFOLD – Berlin Institute for the Foundations of Learning and Data,
Germany  Machine Learning and Security Group, Technische Universität Berlin,
Germany  Kirill Bykov  Understandable Machine Intelligence Lab, ATB, Potsdam,
Germany  Machine Learning Group, Technische Universität Berlin, Germany
BIFOLD – Berlin Institute for the Foundations of Learning and Data, Germany

###### Abstract

Deep Neural Networks (DNNs) are capable of learning complex and versatile
representations, however, the semantic nature of the learned concepts remains
unknown. A common method used to explain the concepts learned by DNNs is
Activation Maximization (AM), which generates a synthetic input signal that
maximally activates a particular neuron in the network. In this paper, we
investigate the vulnerability of this approach to adversarial model
manipulations and introduce a novel method for manipulating feature
visualization without altering the model architecture or significantly
impacting the model’s decision-making process. We evaluate the effectiveness
of our method on several neural network models and demonstrate its
capabilities to hide the functionality of specific neurons by masking the
original explanations of neurons with chosen target explanations during model
auditing. As a remedy, we propose a protective measure against such
manipulations and provide quantitative evidence which substantiates our
findings.

##  1 Introduction

Deep Neural Networks (DNNs) have gained widespread adoption in various domains
due to their remarkable learning capabilities to acquire knowledge from
extensive datasets (LeCun et al., 2015). Nevertheless, a significant challenge
arises as the concepts learned by DNNs often remain elusive and poorly
understood, turning them into a “black box“ for the practitioner. This lack of
transparency has led to various approaches aiming to explain the decisions of
neural networks in recent years (e.g. Simonyan et al., 2013; Sundararajan et
al., 2017; Bach et al., 2015). Besides insights into the models, these methods
also unveiled the tendencies of DNNs to pick up spurious correlations (Bykov
et al., 2023b; Lapuschkin et al., 2019; Geirhos et al., 2020; Izmailov et al.,
2022) or biases (Goh et al., 2021; Bianchi et al., 2022) from the training
data that can lead to undesirable behavior in practice. Given the popularity
of Deep Learning in safety-critical fields (e.g. Piccialli et al., 2021; de la
Escalera et al., 2003), techniques to explain how neural networks arrive at
their decisions are considered a cornerstone to achieving trustworthy machine
learning models.

A well-known method for explaining the concepts learned by a model is
Activation-Maximization (AM) (Erhan et al., 2009; Simonyan et al., 2013; Goh
et al., 2021). This approach aims to illustrate the features that specific
neurons in the network were trained to detect by analyzing input stimuli that
excite the neuron to the maximum extent. Such stimuli can be searched for
within an existing corpus of training data (Borowski et al., 2020), however, a
more popular approach among practitioners is to generate synthetic AM signals
by optimizing a random input image towards maximizing neuron activations in
the network (Erhan et al., 2009; Nguyen et al., 2016; Olah et al., 2017).

Figure 1: The Gradient-Slinghot method manipulates the result of the
Activation-Maximization (AM) explanation for a given neuron with minimal
change in its behavior. The figure shows manipulation of pixel-AM (top, “zero”
logit in MNIST model) and Feature Visualization (bottom, “cat” logit in
CIFAR10 model), where the explanations are manipulated with little change to
the performance (measured by AUROC).

To date, little is known about the security and trustworthiness of AM-based
explanation methods. This work analyzes the susceptibility of these
explanation techniques to adversarial manipulations, similar to attacks that
were proposed for gradient-based feature attribution methods (Dombrowski et
al., 2019; Heo et al., 2019; Anders et al., 2020). Although previous research
has shown that synthetic AM explanations can be manipulated, e.g., by
embedding an additional circuit into the target network (Geirhos et al.,
2023), this paper presents the first attempt towards generating arbitrary
synthetic AM outputs while maintaining the original model architecture and
performance.

Concretely, we present the Gradient Slingshot method111PyTorch implementation
of the proposed method can be found by the following link:
https://github.com/dilyabareeva/grad-slingshot., which manipulates
synthetically generated Activation-Maximization signals while preserving the
original functionality of the representations (see Fig. 1). We show that these
manipulation techniques can successfully conceal problematic and malicious
representations within Deep Neural Networks (DNNs), making them potentially
undetectable during model audits. We also discuss potential defense measures
to increase the resistance of AM methods regarding manipulations. Both
manipulation and defense methods are evaluated qualitatively and
quantitatively on multiple datasets and network architectures.

##  2 Related Work

Approaches to explain the inner workings of neural networks can be broadly
categorized into _global_ , _local_ , or _glocal_ (Samek et al., 2019; Adadi
and Berrada, 2018). Local explanation methods calculate saliency scores for
each feature of an input to indicate its importance towards the prediction.
These scores can be based on gradient information (Simonyan et al., 2013;
Sundararajan et al., 2017) or backward passes through the networks (Bach et
al., 2015; Shrikumar et al., 2017).

Global explanation methods, on the other hand, try to unveil the overall
decision logic of a model, for example, by detecting the most important
training examples with respect to the loss function (Koh and Liang, 2017) or
by interpreting the functional purpose of individual neurons within the DNNs
(Bykov et al., 2023c; Wang et al., 2022; Cammarata et al., 2020; Bau et al.,
2017; Mu and Andreas, 2020; Hernandez et al., 2022; Kalibhat et al., 2023).
Glocal methods build on the model’s inner features and illustrate how they are
employed during the prediction of a specific datapoint (Achtibat et al., 2023;
Fel et al., 2023b).

Activation-Maximization (AM) (Erhan et al., 2009; Simonyan et al., 2013) is a
well-known approach that aims to reveal the abstractions learned by neurons by
generating inputs that activate them to a maximal extent. Investigating these
inputs elucidates the salient patterns or features within the input data that
exert the greatest influence on the neural network’s decision-making
processes. While discerning such influential signals is feasible when the
dataset is available for reference (Borowski et al., 2020), it is crucial to
note that the absence of the original training dataset may introduce potential
inaccuracies in this approach (Bykov et al., 2023a). Therefore, it is more
common to generate synthetic input data through optimization procedures (Erhan
et al., 2009; Olah et al., 2017; Fel et al., 2023a) starting with a random
input image. These methods circumvent the dependency on a specific dataset,
rendering them widely applicable for interpretability and explainability
purposes in Machine Learning (Ling et al., 2017; Couteaux et al., 2019;
Yoshimura et al., 2021), including the identification of neurons that
contribute to undesirable model behavior (Bykov et al., 2023a; Goh et al.,
2021), backdoor attacks detection (Casper et al., 2023), and explanation of
probabilistic models (Grinwald et al., 2023). Synthesizing such inputs in an
unconstrained input domain often produces non-interpretable, noisy outputs
(Olah et al., 2017) which can be circumvented by regularization strategies or
by mapping the input to different domains using differentiable functions
(Mordvintsev et al., 2018). Such strategies include parametrization via
Compositional Pattern Producing Networks (Stanley, 2007) or Generative
Adversarial Network (GAN) generators (Nguyen et al., 2016). One of the most
popular approaches is the parameterization of input signals via Fast Fourier
Transform, serving the purpose of decorrelating and whitening the image data
(Olah et al., 2017).

The shortcomings of local explanation techniques (Ghorbani et al., 2017;
Adebayo et al., 2018; Kindermans et al., 2017) gave rise to different attack
schemes (Dombrowski et al., 2019; Anders et al., 2020) that manipulate their
outcomes. Little is known about the sensitivity of AM to such manipulations.
Geirhos et al. (2023) introduced two attack schemes on synthetic AM: one
involves inserting “fooling circuits“ into the network architecture, while the
other is based on replacing “silent units“ with manipulated computational
blocks. These hard-coded adjustments to the network architecture offer precise
control over the AM output. In contrast, we aim for an approach that is based
on a manipulation loss term thereby eradicating the need for conspicuous
architectural modifications.

##  3 Gradient Slingshots

In this section, we present the Gradient Slingshot (GS) attack that can
manipulate the outcome of AM with minimal impact on the model behavior. We
first introduce some notations for the AM procedure, discuss the theoretical
intuition behind the proposed approach, and describe the practical
implementation of the GS method.

###  3.1 Activation Maximization

Let $\mathbb{F}$ be a set of almost everywhere differentiable functions from
an input domain $\mathbb{D}\subset\mathbb{R}^{d}$ to $\mathbb{R}.$ Given a
function $f\in\mathbb{F},$ representing the activation of a particular neuron,
the aim of the AM method is to find an input $x\in\mathbb{D}$ such that it
maximizes the output of the neural representation $f$. Such optimization
problem is non-convex (Erhan et al., 2009), and gradient-based methods are
often employed to find the local solution. Conventionally, the optimization
process starts from a randomly sampled initialization point
$x^{(0)}\sim\mathcal{I},$ where $\mathcal{I}$ is a probability distribution
with support defined on $\mathbb{D}$. These initialization signals, in the
context of computer vision, are often regarded as noise from a human
perspective and thus lie outside the distribution of natural images (Nguyen et
al., 2019; Geirhos et al., 2023). The update rule for gradient ascent is then
given by

$x^{(i+1)}=x^{(i)}+\epsilon\nabla_{x}f(x^{(i)}),$ (1)

where $\epsilon\in\mathbb{R_{+}}$ is the learning rate, and the optimization
procedure is performed for $T\in\mathbb{N}$ steps. To speed up the convergence
of plain gradient ascent, more advanced optimization algorithms can also be
applied to optimize $x$ (Rosenblatt, 1958; Kingma and Ba, 2015; Loshchilov and
Hutter, 2019a). We refer to the local solutions as synthetic Activation-
Maximization Signals or s-AMS (Bykov et al., 2023a). For the simplicity of the
notation, we introduce the definition of an Activation-Maximization operator.

######  Definition 1 (AM operator).

Let $f\in\mathbb{F}$ be a neuron within the model and
$\Theta=(\epsilon,T,\mathcal{O})$ be a set of hyperparameters for the AM
procedure, including the optimization method $\mathcal{O}$. We define the AM
operator

$\Psi_{\Theta}:\mathbb{F}\times\mathbb{D}\longrightarrow\mathbb{D},$ (2)

such that $\Psi_{\Theta}\left(f,x^{(0)}\right)=x^{*}$ is the result of the AM
procedure with hyperparameters $\Theta$ starting from $x^{(0)}.$

###  3.2 Theoretical Basis

Let $f\in\mathbb{F}$ be a single neuron within the DNN. We assume that the
adversary performing the manipulation procedure is aware of the initialization
distribution $\mathcal{I}$ with
$\tilde{x}=\mathbb{E}\left[\mathcal{I}\right],$ as well as the AM
hyperparameters $\Theta.$ We define the ball around the expected
initialization $B=\left\\{x:||\tilde{x}-x||\leq
R_{B},x\in\mathbb{D}\right\\},$ where the radius $R_{B}$ and the specified
norm are determined by the adversary, utilizing the knowledge of the
distribution $\mathcal{I}$. In the practical implementation, which is
elaborated in Section 3.3, the L-infinity norm is chosen.

The goal of the adversary is to fine-tune the original neuron $f$ to obtain a
function $f^{*}$ so that the result of the AM procedure converges to a pre-
defined target signal $s\in\mathbb{D}$, i.e.

$\Psi_{\Theta}(f^{*},x)=s\qquad\forall x\in B,$ (3)

while minimizing the impact on the network behavior.

Let $g\in\mathbb{F}$ be a function with a global optimum at $s,$ such that
$\forall x\in\mathbb{D}$ we have

$\nabla g(x)=\gamma(s-x).$ (4)

This condition ensures that all partial derivatives will be directed towards
our target point $s$, leading to the convergence of the optimization procedure
to this point. Additionally, the multiplier $\gamma$ controls the “steepness“
of the gradient, and assuming the adversary is aware of the learning rate
$\epsilon$, setting $\gamma=1/\epsilon$ ensures that the target signal will be
reached in one gradient ascent step:

$x^{(1)}=x^{(0)}+\epsilon\nabla_{x}g(x^{(0)})=x^{(0)}+\left(s-x^{(0)}\right)=s.$
(5)

Consecutive optimization steps do not change the outcome anymore, since
$\nabla g(x^{(t)})=\textbf{0},\forall t\in[1,\infty]$ by the construction of g
(Eq. 4. Integrating the linear differential equation results in a quadratic
function of the form

$g(x)=\frac{1}{\epsilon}\left(s^{\top}x-\frac{1}{2}x^{\top}x\right)+C.$ (6)

Gradient Slingshots (GS) aim to fine-tune the original function only in a
small subset of $\mathbb{D}$, retaining the original behavior elsewhere. In
more detail, the manipulated version $f^{*}$ of the original function $f$ then
takes the form

$f^{*}(x)=\left\\{\begin{array}[]{ll}f(x)&\quad x\in\mathbb{D}\setminus M\\\
g(x)&\quad x\in M\end{array}\right.,$ (7)

where $g\in\mathbb{F}$ is the convex function with a global maximum at point
$s,$ and $M\subset\mathbb{D}$ is the manipulation space. Intuitively, $M$
corresponds to the space where the behavior of the model is manipulated, so
that the result of the AM procedure converges into the predefined point $s.$
We define $M=B\cup L,$ where $L=\left\\{x:||s-x||\leq
R_{L},x\in\mathbb{D}\right\\}$ and $R_{L}$ is a parameter. We refer to $L$ as
the “landing zone“, since changing the behavior in the neighborhood of the
target point $s$ ensures the stable convergence of a gradient descent
algorithm, implying that $\nabla g(s)=\textbf{0}.$ Analogously, we call the
initialization region $B$ “slingshot zone”, since it corresponds to high-
amplitude gradients directed at the target.

While we assume that $R_{B}\in\mathbb{R}$ is a predefined parameter based on
the knowledge about the distribution $\mathcal{I},$ the parameter
$R_{L}\in\mathbb{R}$ is selected according to the following lemma:

######  Lemma 3.1.

In case the adversary does not know the exact value of the learning rate
$\epsilon$, but knows the interval $\epsilon\in[a,b],0<a\leq
b,a,b\in\mathbb{R},$ the parameter $R_{L}$ should be selected such as:

$R_{L}\geq\frac{b-a}{a+b}\left(||s-\tilde{x}||+R_{B}\right),$ (8)

where the lower bound is achieved when $\gamma=\frac{2}{a+b}.$

The lemma identifies the relation between the boundaries of the learning rate
and the parameter $R_{L}$ (proof can be found in the Appendix). If the
adversary knows the learning rate, i.e. $a=b=\epsilon,$ the Lemma indicates
that $R_{L}$ can be set to $0,$ indicating that to perform the successful
manipulation, the “landing zone” only needs to contain the target point
$L=\\{s\\}.$

###  3.3 Practical Implementation

Let $h:\mathbb{D}\rightarrow\mathbb{Y}$ be a Deep Neural Network, such that
$h(x)=\left(G\circ F\right)(x),$ where
$F:\mathbb{D}\rightarrow\mathbb{R}^{k},G:\mathbb{R}^{k}\rightarrow\mathbb{Y}$
are feature-extractor and classificator parts of the network. Let
$F=\\{f_{1},...,f_{k}\\},f_{i}\in\mathbb{F},\forall i\in[1,..k]$ be a
collection of individual scalar functions, such that
$F(x)=\left(f_{1}(x),...,f_{k}(x)\right).$ Let $m\in[1,...,k]$ be an index of
the neuron that is set to be manipulated. To simplify the notation, let
$J=[1,...,k]\setminus\\{m\\}$ – a collection of indexes of neurons, except the
neuron $m.$

The manipulation procedure aims to change the result of the AM procedure for
one individual neuron while maintaining the general decision-making behavior
of the original model. For this, we introduce two loss terms: one that is
responsible for the manipulation of the AM objective, and another for
maintaining the behavior of the original function.

Figure 2:  Illustration of the Gradient-Slingshot method on a toy example. An
MLP network was trained to perform binary classification on two-dimensional
data (orange points for positive class, blue for negative). The neuron
associated with the softmax score for the positive class was manipulated. The
figures, from left to right: A) the activation landscape of the original
neuron, with designated points $\tilde{x}$ and $s$, B) “slingshot” and
“landing” zones, C) the activation landscape after manipulation including a
cross-section plane between the two points, and D) illustration of the
manipulated function’s behavior in the cross-section plane. The manipulated
function in the “slingshot” and “landing” zones exhibits parabolic (as in Eq.
6). Since $B\cap L=\varnothing,$, the parabolas in $B$ and $L$ are identical,
differing only in the bias constant. This can be explained since the
optimization is performed to shape the gradient of the manipulated function,
without imposing any constraints on the actual activations.

####  3.3.1 Manipulation loss

Let $\mathcal{M}$ be a distribution with support defined on $M$. This is the
sampling procedure that defines sampling from the “slingshot” and “landing”
zones. Let $\bm{M}$ be a collection of $N$ i.i.d. samples from this
distribution:

$\displaystyle\bm{M}(\mathbb{D},\tilde{x},s,$
$\displaystyle{N^{{}^{\prime}}},N,\sigma_{s},\sigma_{l})=$
$\displaystyle\left\\{x_{1},...,x_{N^{{}^{\prime}}}\in\mathbb{D}:||\tilde{x}-x||_{\infty}\leq\sigma_{s}\right\\}\cup$
$\displaystyle\left\\{x_{N^{{}^{\prime}}},...,x_{N}\in\mathbb{D}:||s-x||_{\infty}\leq\sigma_{l}\right\\},$

where $1\leq{N^{{}^{\prime}}}\leq N$, $\tilde{x}=\mathbb{E}(\mathcal{I})$, $s$
is the target image and $\sigma_{s}$ and $\sigma_{l}$ are the manipulation
radii in the “slingshot” and “landing“ zones respectively. $\sigma_{s}$ is
chosen to adequately span points highly likely to be drawn from $\mathcal{I}$.

First, we define the manipulation loss term, which measures the difference
between existing and required gradients in the manipulated neuron on $\bm{M}$:

$\mathcal{L_{M}}=\frac{1}{N}\sum_{x\in\bm{M}}\left\lVert\nabla
f_{m}(x)-\gamma(s-x)\right\rVert^{2}_{2}.$ (9)

####  3.3.2 Preservation loss

In addition to manipulating the network, it is crucial to maintain the model’s
performance and decision-making capabilities. To achieve this goal, we
introduce the second term $\mathcal{L_{P}}$, which measures how the
activations in the manipulated feature-extractor $F^{*}$ differ from the
activations in the original feature-extractor $F$. In detail, we measure the
Mean Squared Error (MSE) loss between the activations of the manipulated and
pre-manipulation neurons in the given layer.

As the activations of the neural representation $f_{m}$ are more susceptible
to being changed by the manipulation procedure, we may need to assign more
weight to the changes in this neuron in particular. Given a training set
$\mathbf{X}$, we define the following term for a neuron index subset
$\mathbf{A}\subset[1,...,k]$:

$\displaystyle\mathcal{L}_{P}^{\mathbf{A}}$
$\displaystyle=\frac{1}{|\mathbf{A}|\cdot|\mathbf{X}|}\sum_{x\in\mathbf{X}}\sum_{a\in\mathbf{A}}\lvert\lvert{f^{*}_{a}(x)-f_{a}(x)}\rvert\rvert^{2}_{2}.$
(10)

We then formulate the _preservation_ loss term as

$\mathcal{L}_{P}=w\cdot\mathcal{L}_{P}^{J}+(1-w)\cdot\mathcal{L}_{P}^{\\{m\\}},$
(11)

where $w\in[0,1]$ is a parameter, $\mathcal{L}_{P}^{J}$ controls the
performance in all neural representations in layer $F$ except for the
manipulated $f_{m}$, and $\mathcal{L}_{P}^{\\{m\\}}$ controls the performance
in the neural representation $f_{m}$.

Our overall manipulation of s-AMS is then based on a loss term that is a sum
of 1) a loss term $\mathcal{L_{P}}$ responsible for the change between the
performance of the original $F$ and the manipulated $F^{*}$ on all images in
the training set $\mathbf{X}$; and 2) our manipulation loss term
$\mathcal{L_{\mathcal{M}}}$:

$\mathcal{L}=\alpha*\mathcal{L_{P}}+(1-\alpha)\mathcal{L_{M}},$ (12)

where $\alpha\in[0,1]$ is a parameter.

Figure 3:  Feature Visualizations of the output neurons from a CNN trained on
the CIFAR-10 dataset from the original model (top) and a manipulated model
(bottom). In the manipulated model, the neuron associated with the “cat” class
was specifically altered. Despite this manipulation, the Feature
Visualizations in the manipulated model exhibit similar patterns to those in
the original model.

####  3.3.3 Toy example

To illustrate the proposed method, we created a toy experiment, where a
Multilayer Perceptron (MLP) network was trained to distinguish between two
classes using two-dimensional data points. The Gradient-Slingshot method was
employed to manipulate the post-softmax neuron responsible for the score of
the positive class. Fig. 2 visually demonstrates the manipulation procedure
and illustrates how the proposed method alters the activation landscape. It
effectively sculpts the landscape such that the Activation Maximization (AM)
procedure, when initiated from a known location, converges at a predetermined
target point. Additional details can be found in the Appendix.

##  4 Evaluation

We perform our evaluation tests on a 3-layer CNN trained on MNIST (Deng, 2012)
and multiple CNNs trained on CIFAR-10 (Krizhevsky et al., 2009). In the
following, we always assume that the auditor performs the AM optimization with
a learning rate $\epsilon=0.1$ for $T=100$ steps, employing the standard
gradient descent as the optimizer and sampling $x^{(0)}$ pixel-wise from
$\mathcal{N}(0,0.01)$. For the manipulation loss, we set
$\gamma=1/\epsilon=10.0$. All the additional details regarding data pre-
processing, model architectures, training, adversarial-fine-tuning, and AM
procedures can be found in the Appendix.

  * •

Manipulating AM in the pixel domain The CNN trained on the MNIST dataset was
manipulated to alter its AM in the pixel domain (Erhan et al., 2009). We refer
to this approach as _pixel-AM_. We select an image of a cross symbol as the
target image (see Fig. 1). We focus on the neuron responsible for classifying
digit 0 in the output layer as our target neuron. The manipulation result is
visually represented in Fig. 1. The AM output after manipulation closely
resembles the target image. The AUROC values in Fig. 1 indicate that the
neuron after the manipulation remains a 0 _detector_.

  * •

Manipulating Feature Visualizations The CNNs trained on the CIFAR-10 dataset
underwent manipulation to alter its AM in the scaled Fourier frequency domain
(Olah et al., 2017), an approach which we refer to simply as _Feature
Visualization_ (FV). The optimized signals in the scaled frequency domain are
being transformed into the pixel domain using the forward function adopted
from the `Torch-Dreams` library(Deb, 2021) before being fed to the network. No
further regularization techniques, such as transformation robustness, are
applied. For manipulating the model, we use an image with the text “FAKE“ as
the target image (see Fig. 1). As a target neuron, we pick a neuron
responsible for the class “cat“ in the output layer. The outcome of the
manipulation for one of the CNNs is visualized in Figs. 1 and 3. From Fig. 1,
we can observe that the AM output of the manipulated models is perceptually
similar to the target image. The AUROC values in this figure indicate that,
even after manipulation, the neuron continues to function as a _cat detector_.
Fig. 3 displays the FVs of the entire output layer before and after
manipulation. Notably, changes in FVs only occur for the target

$\alpha$ | Accuracy | MSE $\downarrow$ | LPIPS $\downarrow$ | SSIM $\uparrow$
---|---|---|---|---
Original | 99.867 | $0.138\pm 0.007$ | $0.148\pm 0.013$ | $0.037\pm 0.030$
$1.0\cdot 10^{-4}$ | 91.058 | $0.135\pm 0.017$ | $0.139\pm 0.035$ | $0.128\pm 0.102$
$3.3\cdot 10^{-4}$ | 93.058 | $0.170\pm 0.016$ | $0.205\pm 0.041$ | $0.002\pm 0.076$
$6.7\cdot 10^{-4}$ | 94.967 | $\mathbf{0.020\pm 0.001}$ | $\mathbf{0.114\pm 0.012}$ | $\mathbf{0.782\pm 0.005}$
$1.0\cdot 10^{-3}$ | 94.750 | $0.116\pm 0.002$ | $0.281\pm 0.021$ | $0.183\pm 0.014$
$3.3\cdot 10^{-3}$ | 96.967 | $0.150\pm 0.008$ | $0.164\pm 0.033$ | $0.057\pm 0.039$
$6.7\cdot 10^{-3}$ | 97.300 | $0.099\pm 0.033$ | $0.311\pm 0.080$ | $0.322\pm 0.211$
$1.0\cdot 10^{-2}$ | 97.358 | $0.046\pm 0.036$ | $0.177\pm 0.033$ | $0.629\pm 0.197$
$3.3\cdot 10^{-2}$ | 97.958 | $0.053\pm 0.041$ | $0.149\pm 0.038$ | $0.607\pm 0.230$
$6.7\cdot 10^{-2}$ | 98.450 | $0.030\pm 0.013$ | $0.162\pm 0.038$ | $0.723\pm 0.106$
$1.0\cdot 10^{-1}$ | $\mathbf{98.883}$ | $0.033\pm 0.010$ | $0.144\pm 0.035$ | $0.724\pm 0.088$
Table 1: CNN trained on MNIST: Test accuracy, along with the mean and standard
deviation of the similarity between pixel-AM in the manipulated neuron and the
target image.

$\alpha$ | Accuracy | MSE $\downarrow$ | LPIPS $\downarrow$ | SSIM $\uparrow$
---|---|---|---|---
Original | 86.330 | $0.051\pm 0.003$ | $0.266\pm 0.019$ | $0.047\pm 0.025$
$1.0\cdot 10^{-4}$ | 10.330 | $0.016\pm 0.003$ | $0.224\pm 0.036$ | $0.212\pm 0.032$
$3.3\cdot 10^{-4}$ | 21.470 | $0.012\pm 0.002$ | $0.214\pm 0.026$ | $0.170\pm 0.035$
$6.7\cdot 10^{-4}$ | 44.610 | $0.018\pm 0.007$ | $0.191\pm 0.040$ | $0.211\pm 0.061$
$1.0\cdot 10^{-3}$ | 50.430 | $0.013\pm 0.005$ | $0.192\pm 0.043$ | $0.280\pm 0.061$
$3.3\cdot 10^{-3}$ | 72.500 | $0.007\pm 0.001$ | $0.120\pm 0.016$ | $0.397\pm 0.040$
$6.7\cdot 10^{-3}$ | 78.720 | $0.008\pm 0.001$ | $0.117\pm 0.019$ | $0.394\pm 0.037$
$1.0\cdot 10^{-2}$ | 82.390 | $\mathbf{0.003\pm 0.000}$ | $\mathbf{0.072\pm 0.010}$ | $\mathbf{0.512\pm 0.033}$
$3.3\cdot 10^{-2}$ | 84.880 | $0.009\pm 0.001$ | $0.121\pm 0.013$ | $0.271\pm 0.025$
$6.7\cdot 10^{-2}$ | 85.220 | $0.011\pm 0.005$ | $0.123\pm 0.026$ | $0.197\pm 0.045$
$1.0\cdot 10^{-1}$ | $\mathbf{85.570}$ | $0.010\pm 0.001$ | $0.143\pm 0.016$ | $0.164\pm 0.029$
Table 2: CNN trained on CIFAR-10: Test accuracy, along with the mean and
standard deviation of similarity between the FV in the manipulated neuron and
the target image.

###  4.1 Similarity Metrics

To compare the AM output to the target images, three image similarity metrics,
Mean Squared Error (MSE), the Structural Similarity Index (SSIM) (Wang et al.,
2004) and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al.,
2018), are employed. MSE serves as an error measure, with values close to $0$
signifying a high degree of resemblance. SSIM, on the other hand, is a
perceptual similarity metric ranging between $0$ and $1$, where a higher SSIM
value indicates increased similarity between images, and a value of 1 denotes
identical images. LPIPS is a perceptual distance measure, whereby a lower
LPIPS score indicates a higher similarity between the two images. In our
study, the LPIPS calculations rely on the deep embeddings extracted from an
AlexNet model (Krizhevsky et al., 2012) that has been pre-trained on the
ImageNet dataset (Simonyan and Zisserman, 2015).

###  4.2 Accuracy – Manipulation Trade-Off

The manipulation procedure involves a careful balance between preserving the
model performance and achieving the manipulation objective (Section 3.3.1).
The parameter $\alpha$ as per Eq. 12 controls the weights assigned to
manipulation loss (Section 3.3.1) and preservation loss (Section 3.3.2). In
this experiment, we manipulate the models by varying the parameter $\alpha$
and keeping all the other fine-tuning parameters fixed.

Figure 4: Sample FVs and similarity to target at different values of $\alpha$
for a CNN trained on CIFAR-10. Both very low and high values of $\alpha$
result in low similarity to the target.

The results of the experiment for the CNN trained on MNIST data with pixel-AM
and a CNN trained on CIFAR-10 data with FV are represented in Tables 2 and 2,
respectively. As anticipated, the model test accuracy demonstrates an
increasing trend with higher values of the parameter $\alpha$. We observe that
very high values of $\alpha$ correspond to a decrease in the similarity of the
AM output to the target image. Conversely, very low values of $\alpha$ also
result in diminished similarity levels. The latter phenomenon can be
attributed to the fact that low values of $\alpha$ tend to induce drastic
changes in the overall activation landscape, potentially leading to areas
surrounding the “landing zone“ exhibiting extremely high gradient norm values.
Fig. 4 illustrates the variation in the SSIM and LPIPS metrics for FV as the
parameter $\alpha$ changes.

###  4.3 Effect of Model Size

|  $\times 8$ |  $\times 16$ |  $\times 32$ |  $\times 64$
---|---|---|---|---
A |  $0.113\pm 0.015$ |  $0.049\pm 0.005$ |  $0.043\pm 0.007$ |  $0.025\pm 0.005$
B |  $0.082\pm 0.015$ |  $0.087\pm 0.007$ |  $0.051\pm 0.005$ |  $0.033\pm 0.005$
C |  $0.116\pm 0.026$ |  $0.073\pm 0.007$ |  $0.046\pm 0.011$ |  $\mathbf{0.015\pm 0.002}$
D |  $0.076\pm 0.038$ |  $0.082\pm 0.01$ |  $0.033\pm 0.004$ |  $0.026\pm 0.035$
Table 3: Quantitative evaluation of the model size impact: Rows labeled "A" to
"D" indicate model depth, and columns denote the multiplicative factor of
model width. Mean and standard deviation of a distance metric (MSE) between
the FV in the manipulated models and the target image.

In the following, we investigate the influence of the number of model
parameters on the success of the manipulation in terms of similarity to the
target image and change in model performance. Research has demonstrated that
even shallow networks with substantial width exhibit extensive memorization
capabilities (Hornik et al., 1989; Zhang et al., 2021), a critical aspect in
our manipulation context where the target image needs to be memorized.
Conversely, models with greater depth can approximate more complex functions
(Eldan and Shamir, 2016).

To assess the impact of the model size on the attack, we create a set of image
classification models with varying depth and width. The model depth
configurations are labeled from “A” to “D“. The width configurations are
expressed as a multiplicative factor, where the baseline number of output
channels or units in each respective layer is multiplied by this factor.
Specifically, the stack of convolutional layers of models “A64”, “B64”, “C64”,
and “D64” correspond to the stack of convolutional layers in the architectures
VGG11, VGG13, VGG16, and VGG19 (Simonyan and Zisserman, 2015), respectively
(for more detail, see Appendix). The original models are trained on the
CIFAR-10 dataset. We perform adversarial fine-tuning to replace the FV output
with the image of a goldfish222The image was obtained from
https://github.com/EliSchwartz/imagenet-sample-images..

Fig. 5 visually illustrates FV outputs for all 16 model configurations and
demonstrates the change in test accuracy between the manipulated and original
models. The corresponding quantitative evaluation is presented in Table 3. A
discernible correlation is observed between the success of manipulation and
the number of model parameters. Across configurations “A“ to “D“, the widest
models exhibit the best manipulation performance. However, for specifications
“$\times 16$“ and “$\times 64$“, the deepest models do not yield the closest
similarity to the target image. We hypothesize that this could be attributed
to the _shattered gradients_ effect (Balduzzi et al., 2017), which poses
challenges in training deeper models.

Figure 5: “Catfish” neuron: 16 classification models of varying depth (“A” -
“D”) and width ($\times 8$ \- $\times 64$) trained on CIFAR-10 were
manipulated to change the FV of the cat output neuron to a fish image. The
figure depicts a sample FV for model B64, the target image, and sample FVs of
the manipulated models, along with the change in the test accuracy. The
manipulation outcome improves as the number of model parameters increases,
specifically with the increase in the model width.

##  5 Defense against Manipulation

We suggest several defensive measures designed to counteract the Gradient
Slingshot attack. These measures require modifications to the parameters of
the AM procedure, denoted as $\Theta$, to increase its robustness against the
manipulation procedure. The fundamental concept of these defensive measures is
to guide the optimization process towards areas of the activation landscape
that are predominantly influenced by the original, unaltered function $f_{m}$.
In the subsequent sections, we present and assess these defensive measures.

((a)) Defense against Pixel-AM manipulation with a CNN trained on MNIST. All
strategies successfully eliminate the target image from the AM output, yet
none manage to restore the original "zero" concept visualization.

((b)) Defense against FV manipulation with a CNN trained on CIFAR-10. The TR
strategy successfully restores the "cat" concept, albeit not when employed in
combination with GC and Adam.

Figure 6: Visualization of the effect of different proposed defense strategies
on resulting AM explanations.

  1. 1.

Gradient clipping (GC)

The manipulation process may generate regions with significantly high gradient
norms around the initialization distribution due to the coefficient $\gamma$
(see Eq. 9). Clipping the gradient norm may result in the first “slingshot“
missing the target and landing between the “slingshot“ and “landing“ zones, an
area undefined in the manipulation loss term.

  2. 2.

Transformation robustness (TR)

Transformation robustness has been introduced as a technique aimed at
enhancing the interpretability of FVs. This technique is realized through the
application of random perturbations to the signal at each optimization step
and facilitates finding signals that induce heightened activation even when
slightly transformed (Nguyen et al., 2019; Olah et al., 2017). Applying
transformations can guide the optimization process into the regions of the
manipulated function that align with the definition of the original function.

  3. 3.

Changing the optimizer algorithm

Multiple gradient algorithms traditionally used for training neural network
weights, such as Adam (Kingma and Ba, 2014) or AdamW (Loshchilov and Hutter,
2019b), can also be applied in AM settings. While we assume the usage of the
standard gradient algorithm in both theoretical and practical considerations,
Adam and AdamW may stir optimization away from manipulated regions.

  4. 4.

Natural Activation-Maximization signals (n-AMS)

Given that the activations in the training set are supposed to remain
unchanged during the manipulation process, a straightforward search of the
most activating images from a pre-defined set of images (Borowski et al.,
2020) should yield consistent results before and after manipulation.

We evaluated the proposed defense strategies in the context of both
manipulation experiments: AM in the pixel domain and FV. One manipulated model
for each case from the experiments described in Section 4.2 was selected for
the evaluation: model $\alpha=0.067$ for pixel-AM and model $\alpha=0.01$ for
FV. For TR, we apply the following sequence of transformations: padding,
random affine transformation, random rotation, and random crop. For testing a
changed optimizer algorithm, we use Adam. Further details about the evaluation
can be found in the Appendix.

In the quantitative evaluation of each defense strategy, we expect the s-AMS
to be dissimilar to the target image and similar to the s-AMS of the
corresponding neuron in the original model. Accordingly, we defined the
following two metrics for s-AMS strategies:

  * •

Distance to Target  For evaluating how different AM outputs are from the pre-
defined target image, $Q=100$ AM signals $x^{*}_{q}$, $\forall
q\in\\{1,...,Q\\}$ are sampled from manipulated neural representation
$f_{m}^{*}$. We define Distance to Target metric
$D2T:\mathbb{F}\times\mathbb{D}\rightarrow\mathbb{R}$ as:

$D2T(f_{m}^{*},s)=\frac{1}{Q}\sum_{q=1}^{Q}\text{LPIPS}\left(x_{q},s\right).$
(13)

  * •

Distance to Original  For evaluating how different AM outputs of the
manipulated model are from the signals obtained from the original model,
$Q=100$ AM outputs $x^{*}_{q}$, $\forall q\in\\{1,...,Q\\}$ are sampled for
the manipulated model $f_{m}^{*}$, as well as $Q$ AM outputs $x_{q^{\prime}}$,
$\forall q^{\prime}\in\\{1,...,Q\\}$ from the original model $f_{m}$. We
define Distance to Original metric
$D2O:\mathbb{F}\times\mathbb{F}\rightarrow\mathbb{R}$ as:

$D2O(f_{m}^{*},f_{m})=\frac{1}{Q^{2}}\sum_{q=1}^{Q}\sum_{q^{\prime}=1}^{Q}\text{LPIPS}\left(x^{*}_{q},x_{q^{{}^{\prime}}}\right).$
(14)

The visual outcomes of applying our defense strategies for pixel-AM are
depicted in Fig. 6(a). Fig. 6(b) showcases sample results of implementing our
defense strategies for FV. Quantitative evaluation of defense strategies is
presented in Table 4. For pixel-AM, all strategies prove effective in
eliminating the target image from the AM output ($D2T$), but in terms of
similarity to the original ($D2O$), the Adam strategy is the sole approach to
outperform the baseline, albeit only marginally. For FVs, the TR strategy
demonstrates success in both metrics, suggesting that experimenting with
different adjustments to the AM procedure may unveil adversarial
manipulations. In practical applications, TR is often used in combination with
GC and Adam (Deb, 2021; Tensorflow, 2017). This combination proves to be
ineffective in terms of both metrics.

Strategies | Pixel-AM | Feature Visualization
---|---|---
| $D2T\uparrow$ | $D2O\downarrow$ | $D2T\uparrow$ | $D2O\downarrow$
None | 0.1581 | 0.1661 | 0.0705 | 0.2138
GC | 0.3219 | 0.2199 | 0.0817 | 0.2230
TR | 0.2699 | 0.2922 | $\mathbf{0.2692}$ | $\mathbf{0.0730}$
Adam | $\mathbf{0.3367}$ | $\mathbf{0.1426}$ | 0.0791 | 0.1923
Adam+GC+TR | 0.3112 | 0.2301 | 0.1680 | 0.1863

Table 4: Quantitative evaluation of defense strategies across the metrics
Distance to Target ($D2T$) and Distance to Original ($D2O$), for pixel-AM
manipulation (CNN trained on MNIST) and FV manipulation (CNN trained on
CIFAR-10).

Figure 7: Top-4 overall most activating natural signals (n-AMS) in the test
set before and after manipulation. The manipulation had no significant impact
on the semantic meaning of the top n-AMS signals.

Fig. 7 displays the top-4 n-AMS in the test set before and after manipulation
for both settings. We additionally collect from the test data the set
$\textbf{S}^{*}$ of top-$100$ n-AMS in the manipulated model and the set S of
top-$100$ n-AMS in the original model. The _Jaccard similarity coefficient_
between $\textbf{S}^{*}$ and S is 0.55 for the pixel-AM setting and 0.30 for
the FV setting. A substantial overlap in top n-AMS signals before and after
manipulation is observed, underscoring the importance of conducting AM on
natural images when having access to the model’s training dataset.
Importantly, research has indicated that n-AMS can also be manipulated (Marty
et al., 2022).

##  6 Discussion and Conclusion

Activation-Maximization (AM) methods are extensively employed for uncovering
features learned by Deep Neural Networks (DNNs). In this study, we
theoretically and empirically illustrate that these methods can be manipulated
to display arbitrary images, all while maintaining the original model
architecture and avoiding any substantial decrease in performance. We take
significant steps in demonstrating the feasibility of applying defense
strategies to address this issue. By shedding light on the possibility of such
manipulations, we hope to heighten the caution of AI system users and auditors
regarding findings generated by AM-based methods.

One potential limitation of our manipulation method lies in the
computationally demanding challenge of determining optimal hyperparameters, as
illustrated in Section 4.2. Additionally, the shattered gradients problem
hinted at in Section 4.3 for deeper networks, or the increased dimensionality
of data could further complicate the manipulation task. These limitations
provide an avenue for future research. While our proposed defense strategies
contribute to manipulation detection, more rigorous efforts are needed to
develop methods for identifying such manipulation and enhancing the overall
robustness of AM methods against potential attacks.

##  7 Acknowledgements

This work was partly funded by the German Ministry for Education and Research
(BMBF) through the projects Explaining 4.0 (ref. 01IS200551) and BIFOLD
(BIFOLD23B). Furthermore, the authors acknowledge funding by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s
Excellence Strategy EXC 2092 CASA-390781972.

## References

  * Achtibat et al. (2023) Reduan Achtibat, Maximilian Dreyer, Ilona Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wojciech Samek, and Sebastian Lapuschkin.  From attribution maps to human-understandable explanations through concept relevance propagation.  _Nature Machine Intelligence_ , 5(9):1006–1019, 2023.
  * Adadi and Berrada (2018) Amina Adadi and Mohammed Berrada.  Peeking inside the black-box: a survey on explainable artificial intelligence (XAI).  _IEEE access_ , 6:52138–52160, 2018.
  * Adebayo et al. (2018) Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  In _Advances in Neural Information Proccessing Systems (NIPS)_ , 2018.
  * Anders et al. (2020) Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel.  Fairwashing explanations with off-manifold detergent.  In _International Conference on Machine Learning_ , pages 314–323. PMLR, 2020.
  * Bach et al. (2015) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ , 10(7):e0130140, 2015.
  * Balduzzi et al. (2017) David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams.  The shattered gradients problem: If resnets are the answer, then what is the question?  In _Proceedings of the 34th International Conference on Machine Learning_ , pages 342–350. PMLR, 2017.
  * Bau et al. (2017) David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Network dissection: Quantifying interpretability of deep visual representations.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6541–6549, 2017.
  * Bianchi et al. (2022) Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan.  Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale, 2022.
  * Borowski et al. (2020) Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel.  Natural images are more informative for interpreting cnn activations than state-of-the-art synthetic feature visualizations.  In _NeurIPS 2020 Workshop SVRHM_ , 2020.
  * Bykov et al. (2023a) Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus Robert Muller, and Marina MC Höhne.  DORA: Exploring outlier representations in deep neural networks.  _Transactions on Machine Learning Research_ , 2023a.
  * Bykov et al. (2023b) Kirill Bykov, Laura Kopf, and Marina M-C Höhne.  Finding spurious correlations with function-semantic contrast analysis.  In _World Conference on Explainable Artificial Intelligence_ , pages 549–572. Springer, 2023b.
  * Bykov et al. (2023c) Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina MC Höhne.  Labeling neural representations with inverse recognition.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023c.
  * Cammarata et al. (2020) Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim.  Thread: Circuits.  _Distill_ , 2020.  https://distill.pub/2020/circuits.
  * Casper et al. (2023) Stephen Casper, Tong Bu, Yuxiao Li, Jiawei Li, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell.  Red teaming deep neural networks with feature synthesis tools.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Couteaux et al. (2019) Vincent Couteaux, Olivier Nempont, Guillaume Pizaine, and Isabelle Bloch.  Towards interpretability of segmentation networks by analyzing deepdreams.  In _Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support: Second International Workshop, iMIMIC 2019, and 9th International Workshop, ML-CDS 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, 2019, Proceedings 9_ , pages 56–63. Springer, 2019.
  * de la Escalera et al. (2003) Arturo de la Escalera, Jose M. Armingol, and Mario Mata.  Traffic sign recognition and analysis for intelligent vehicles.  _Image Vis. Comput._ , 21(3):247–258, 2003.
  * Deb (2021) Mayukh Deb.  Feature visualization library for pytorch.  https://github.com/Mayukhdeb/torch-dreams, 2021.
  * Deng (2012) Li Deng.  The mnist database of handwritten digit images for machine learning research [best of the web].  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.
  * Dombrowski et al. (2019) Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel.  Explanations can be manipulated and geometry is to blame.  _Advances in neural information processing systems_ , 32, 2019.
  * Eldan and Shamir (2016) Ronen Eldan and Ohad Shamir.  The power of depth for feedforward neural networks.  In _29th Annual Conference on Learning Theory_ , pages 907–940, Columbia University, New York, New York, USA, 2016. PMLR.
  * Erhan et al. (2009) Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Visualizing higher-layer features of a deep network.  _University of Montreal_ , 1341(3):1, 2009.
  * Fel et al. (2023a) Thomas Fel, Thibaut Boissin, Victor Boutin, Agustin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom Rousseau, Rémi Cadène, Laurent Gardes, et al.  Unlocking feature visualization for deeper networks with magnitude constrained optimization.  _arXiv preprint arXiv:2306.06805_ , 2023a.
  * Fel et al. (2023b) Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre.  Craft: Concept recursive activation factorization for explainability.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2711–2721, 2023b.
  * Geirhos et al. (2020) Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11):665–673, 2020.
  * Geirhos et al. (2023) Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim.  Don’t trust your eyes: on the (un) reliability of feature visualizations.  _arXiv preprint arXiv:2306.04719_ , 2023.
  * Ghorbani et al. (2017) Amirata Ghorbani, Abubakar Abid, and James Y. Zou.  Interpretation of neural networks is fragile.  In _AAAI Conference on Artificial Intelligence_ , 2017.
  * Goh et al. (2021) Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah.  Multimodal neurons in artificial neural networks.  _Distill_ , 6(3):e30, 2021.
  * Grinwald et al. (2023) Dennis Grinwald, Kirill Bykov, Shinichi Nakajima, and Marina MC Höhne.  Visualizing the diversity of representations learned by bayesian neural networks.  _Transactions on Machine Learning Research_ , 2023.
  * Heo et al. (2019) Juyeon Heo, Sunghwan Joo, and Taesup Moon.  Fooling neural network interpretations via adversarial model manipulation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Hernandez et al. (2022) Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas.  Natural language descriptions of deep visual features.  In _International Conference on Learning Representations_ , 2022.
  * Hornik et al. (1989) Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  _Neural networks_ , 2(5):359–366, 1989.
  * Izmailov et al. (2022) Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson.  On feature learning in the presence of spurious correlations.  _Advances in Neural Information Processing Systems_ , 35:38516–38532, 2022.
  * Kalibhat et al. (2023) Neha Kalibhat, Shweta Bhardwaj, C Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi.  Identifying interpretable subspaces in image representations.  2023\.
  * Kindermans et al. (2017) Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, D. Erhan, and Been Kim.  The (un)reliability of saliency methods.  In _Explainable AI_ , 2017.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * Koh and Liang (2017) Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In _Proc. of the International Conference on Machine Learning (ICML)_ , 2017.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  _Toronto, ON, Canada_ , 2009.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2012.
  * Lapuschkin et al. (2019) Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Unmasking clever hans predictors and assessing what machines really learn.  _Nature communications_ , 10(1):1096, 2019.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Ling et al. (2017) Julia Ling, Maxwell Hutchinson, Erin Antono, Brian DeCost, Elizabeth A Holm, and Bryce Meredig.  Building data-driven models with microstructural images: Generalization and interpretability.  _Materials Discovery_ , 10:19–28, 2017.
  * Loshchilov and Hutter (2019a) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019a.
  * Loshchilov and Hutter (2019b) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _International Conference on Learning Representations_ , 2019b.
  * Marty et al. (2022) Jonathan Marty, Eugene Belilovsky, and Michael Eickenberg.  Adversarial attacks on feature visualization methods.  In _NeurIPS ML Safety Workshop_ , 2022.
  * Mordvintsev et al. (2018) Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah.  Differentiable image parameterizations.  _Distill_ , 3(7):e12, 2018.
  * Mu and Andreas (2020) Jesse Mu and Jacob Andreas.  Compositional explanations of neurons.  _Advances in Neural Information Processing Systems_ , 33:17153–17163, 2020.
  * Nguyen et al. (2016) Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.  _Advances in neural information processing systems_ , 29, 2016.
  * Nguyen et al. (2019) Anh Nguyen, Jason Yosinski, and Jeff Clune.  Understanding neural networks via feature visualization: A survey.  _Explainable AI: interpreting, explaining and visualizing deep learning_ , pages 55–76, 2019.
  * Olah et al. (2017) Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  _Distill_ , 2017.  https://distill.pub/2017/feature-visualization.
  * Piccialli et al. (2021) Francesco Piccialli, Vittorio Di Somma, Fabio Giampaolo, Salvatore Cuomo, and Giancarlo Fortino.  A survey on deep learning in medicine: Why, how and when?  _Information Fusion_ , 66:111–137, 2021.
  * Rosenblatt (1958) Frank Rosenblatt.  The perceptron: a probabilistic model for information storage and organization in the brain.  _Psychological review_ , 65(6):386, 1958.
  * Samek et al. (2019) Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller.  _Explainable AI: interpreting, explaining and visualizing deep learning_.  Springer Nature, 2019.
  * Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In _Proc. of the International Conference on Machine Learning (ICML)_ , pages 3145–3153, 2017.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _International Conference on Learning Representations_ , 2015.
  * Simonyan et al. (2013) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ , 2013.
  * Stanley (2007) Kenneth O Stanley.  Compositional pattern producing networks: A novel abstraction of development.  _Genetic programming and evolvable machines_ , 8:131–162, 2007.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pages 3319–3328. PMLR, 2017.
  * Tensorflow (2017) Tensorflow.  lucid.  https://github.com/tensorflow/lucid, 2017.
  * Wang et al. (2022) Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  _arXiv preprint arXiv:2211.00593_ , 2022.
  * Wang et al. (2004) Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, 2004.
  * Yoshimura et al. (2021) Naoya Yoshimura, Takuya Maekawa, and Takahiro Hara.  Toward understanding acceleration-based activity recognition neural networks with activation maximization.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8. IEEE, 2021.
  * Zhang et al. (2021) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  _Commun. ACM_ , 64(3):107–115, 2021.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.

##  Appendix A Theoretical Basis

In the following section, we provide the proof for the Lemma 3.1 and introduce
a Lemma A.1 for the case, where we cannot guarantee that function $g$ will
suffice the conditions 4.

###### Proof of the Lemma 3.1.

We want that $\forall\alpha\in[a,b],\forall\bm{x}\in B:$

$\displaystyle\bm{x}+\epsilon\nabla g(\bm{x})\in L,$
$\displaystyle\Leftrightarrow$
$\displaystyle\bm{x}+\epsilon\gamma\left(\bm{s}-\bm{x}\right)\in L$
$\displaystyle\Leftrightarrow$
$\displaystyle\left\lVert\bm{x}+\epsilon\gamma\left(\bm{s}-\bm{x}\right)-\bm{s}\right\rVert\leq
R_{L}$ $\displaystyle\Leftrightarrow$
$\displaystyle\left\lVert\bm{x}+\epsilon\gamma\bm{s}-\epsilon\gamma\bm{x}-\bm{s}\right\rVert\leq
R_{L}$ $\displaystyle\Leftrightarrow$
$\displaystyle\left\lVert\bm{s}\left(\epsilon\gamma-1\right)-\bm{x}\left(\epsilon\gamma-1\right)\right\rVert\leq
R_{L},$ $\displaystyle\Leftrightarrow$
$\displaystyle\left\lVert\left(\bm{s}-\bm{x}\right)\left(\epsilon\gamma-1\right)\right\rVert\leq
R_{L}$ $\displaystyle\Leftrightarrow$
$\displaystyle\left|\epsilon\gamma-1\right|\left\lVert\bm{s}-\bm{x}\right\rVert\leq
R_{L}$

Using the triangle inequality:

$\displaystyle\left\lVert\bm{s}-\bm{x}\right\rVert\leq\left\lVert\bm{s}-\bm{\tilde{x}}\right\rVert+\left\lVert\bm{\tilde{x}}-\bm{x}\right\rVert=\left\lVert\bm{s}-\bm{\tilde{x}}\right\rVert+R_{B}$

Therefore it is sufficient if $R_{L}$ suffices:

$\displaystyle\left|\epsilon\gamma-1\right|\left(\left\lVert\bm{s}-\bm{\tilde{x}}\right\rVert+R_{B}\right)\leq
R_{L},\forall\epsilon\in[a,b],\forall\bm{x}\in B$

To find the optimum parameter $\gamma$ we solve the following min-max problem:

$\displaystyle\min_{\gamma>0}\max_{\epsilon\in[a,b]}\left|\epsilon\gamma-1\right|$

We can rewrite the maximization problem as follows:

$\max_{\epsilon\in[a,b]}\left|\epsilon\gamma-1\right|=\max\left\\{b\gamma-1,1-a\gamma\right\\}$

To minimize the maximum value, we should choose $\gamma$ such that

$\displaystyle b\gamma-1=1-a\gamma$ $\displaystyle\Leftrightarrow$
$\displaystyle\left(a+b\right)\gamma=2$ $\displaystyle\Leftrightarrow$
$\displaystyle\gamma=\frac{2}{a+b}$

In this case:

$\displaystyle\max_{\epsilon\in[a,b]}\left|\epsilon\gamma-1\right|$
$\displaystyle=$ $\displaystyle\max\left\\{b\gamma-1,1-a\gamma\right\\}$
$\displaystyle=$ $\displaystyle\frac{b-a}{a+b}.$

∎

######  Lemma A.1.

Let $f^{*}\in\mathbb{F}$ be the manipulated function, and
$R_{B}\in\mathbb{R},$ and $\epsilon\in[a,b],0<a\leq b,a,b\in\mathbb{R}.$ Let
$g\in\mathbb{F},$ such that

$\nabla g(\bm{x})=\gamma(\bm{s}-\bm{x})+\xi(\bm{x}),$ (15)

and $||\xi(\bm{x})||\leq Q,\forall\bm{x}\in\mathbb{D}.$ Then $R_{L}$ should be
selected such as:

$R_{L}\geq\frac{b-a}{a+b}\left(||\bm{s}-\tilde{\bm{x}}||+R_{B}\right)+bQ,$
(16)

with the lower bound is achieved when $\gamma=\frac{2}{a+b}.$

###### Proof of the Lemma 3.2.

We want that $\forall\epsilon\in[a,b],\forall\bm{x}\in B:$

$\displaystyle\bm{x}+\epsilon\nabla g(\bm{x})\in L$
$\displaystyle\Leftrightarrow$
$\displaystyle\bm{x}+\epsilon\left(\gamma\left(\bm{s}-\bm{x}\right)+\xi(\bm{x})\right)\in
L$ $\displaystyle\Leftrightarrow$
$\displaystyle\left\lVert\bm{x}+\epsilon\left(\gamma\left(\bm{s}-\bm{x}\right)+\xi(\bm{x})\right)-\bm{s}\right\rVert\leq
R_{L}$ $\displaystyle\Leftrightarrow$
$\displaystyle\left\lVert\left(\bm{s}-\bm{x}\right)\left(\epsilon\gamma-1\right)-\epsilon\xi(\bm{x})\right\rVert\leq
R_{L}$

We know that:

$\displaystyle\left\lVert\left(\bm{s}-\bm{x}\right)\left(\epsilon\gamma-1\right)-\epsilon\xi(\bm{x})\right\rVert$
$\displaystyle\leq$
$\displaystyle\left|\epsilon\gamma-1\right|\left(\left\lVert\bm{s}-\bm{\tilde{x}}\right\rVert+R_{B}\right)+\epsilon\left\lVert\xi(\bm{x})\right\rVert$
$\displaystyle=$
$\displaystyle\left|\epsilon\gamma-1\right|\left(\left\lVert\bm{s}-\bm{\tilde{x}}\right\rVert+R_{B}\right)+\epsilon
Q$

Solving the min-max problem, we achieve the necessary condition for $R_{L}$

$\displaystyle
R_{L}\geq\frac{b-a}{a+b}\left(\left\lVert\bm{s}-\bm{\tilde{x}}\right\rVert+R_{B}\right)+bQ,$

where the right part of inequality achieves minimum at $\gamma=\frac{2}{a+b}.$

∎

##  Appendix B Toy experiment

To demonstrate the intuition behind the described procedure, we illustrate the
proposed approach with a small toy experiment. Initially, a 2-dimensional
classification problem was formulated by uniformly sampling 512 data points
for the positive class within the two-dimensional ball
$A^{+}=\left\\{x:\left\lVert x\right\rVert<2,x\in\mathbb{R}^{2}\right\\},$ and
the same number of points for the negative class from the disc
$A^{-}=\left\\{x:4<\left\lVert x\right\rVert<5,x\in\mathbb{R}^{2}\right\\}.$

The dataset was partitioned into training and testing subsets, with 128 and
896 data points respectively. A Multi-Layer Perceptron (MLP) network was then
trained for the purpose of classification.

The MLP network was designed with a total of six layers. The initial layer was
configured with two input neurons and 100 output neurons. The subsequent four
layers (layers 2 to 5) were all linear layers, each containing 100 neurons.
The final layer was a linear layer that transformed the 100-dimensional output
from the preceding layer into a 2-dimensional output. A Tanh activation
function was applied after each linear layer, except for the final layer. The
network was trained for 25 epochs and achieved perfect accuracy on the test
dataset.

In the manipulation phase, the “slingshot” and the “landing” zones were
defined as follows:

$\displaystyle B$ $\displaystyle=\\{x:\left\lVert
x-\tilde{x}\right\rVert_{2}<5\\},$ (17) $\displaystyle L$
$\displaystyle=\\{x:\left\lVert x-s\right\rVert_{2}<5\\},$ (18)

where $\tilde{x}=(20,-15),$ and $s=(25,-5).$

For the set $M$, we generated a total of $N=20000$ points, which were sampled
from the uniform distribution over the intersection of $B$ and $L$. The set
$\mathbf{X}$ consisted of $|\mathbf{X}|=15000$ points, with both coordinates
independently sampled from a normal distribution $\mathcal{N}(0,10)$. The
parameter $\gamma$ was set to $0.005$. For illustrative purposes, we added a
third term to the standard loss terms of the Gradient Slingshot method
(manipulation loss and preservation loss). This additional term was an MSE
loss, which ensured that the activation of the manipulated function was equal
to $0.2$ at the target point $s$.

##  Appendix C Details on Experiments

Datasets

For both MNIST and CIFAR-10 datasets, we employ an 80%–20% train-test dataset
split. The data is normalized to have zero mean and standard deviation one.
During CIFAR-10 training and adversarial fine-tuning, we apply a random
horizontal flip with a probability of 0.5, pad all images with 4 pixels on
each side, and then randomly crop them back to the original size of 32 × 32.

Model Architecture and Training

The MNIST CNN architecture is as follows: input -> conv (5x5, 16) -> max
pooling (2x2)-> conv (5x5, 32) -> max pooling (2x2) -> fully connected (512
units) -> fully connected (256 units) -> fully connected (120 units) -> fully
connected (84 units) -> softmax (10 units). We train the model with the SGD
optimizer using learning rate of $0.001$ and momentum of $0.9$ until
convergence. ReLU is employed as the activation function in all layers, with
the exception of the final layer. The final test set accuracy of this model is
99.87 percent.

Layers | A | B | C | D
---|---|---|---|---
input (32 $\times$ 32 RGB image)
conv3-$1\times r$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
conv3-$1\times r$ |  | $\checkmark$ | $\checkmark$ | $\checkmark$
maxpool
conv3-$2\times r$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
conv3-$2\times r$ |  | $\checkmark$ | $\checkmark$ | $\checkmark$
maxpool
conv3-$4\times r$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
conv3-$4\times r$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
conv1-$4\times r$ |  |  | $\checkmark$ | $\checkmark$
conv3-$4\times r$ |  |  |  | $\checkmark$
maxpool
conv3-$8\times r$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
conv3-$8\times r$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
conv1-$8\times r$ |  |  | $\checkmark$ | $\checkmark$
conv3-$8\times r$ |  |  |  | $\checkmark$
maxpool
conv3-$8\times r$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
conv3-$8\times r$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
conv1-$8\times r$ |  |  | $\checkmark$ | $\checkmark$
conv3-$8\times r$ |  |  |  | $\checkmark$
maxpool
FC-$8\times r$
Dropout(0.5)
FC-$10$
Table 5: CIFAR-10 CNN configurations with added layers. The convolutional
layer parameters are denoted as "conv$\langle$receptive field
size$\rangle$-$\langle$number of channels$\rangle$". The batch normalization
layers and ReLU activation function are not shown for brevity. The model depth
configurations are labeled from “A” to “D“. The width configurations are
expressed as a multiplicative factor $\times r$.

The CNN architectures for CIFAR-10 are detailed in Table 5. Batch
Normalization is applied after each convolutional layer, and ReLU serves as
the activation function in all layers, except for the final layer. The
convolutional layer stacks of models "A64," "B64," "C64," and "D64" align with
those in the VGG11, VGG13, VGG16, and VGG19 architectures Simonyan and
Zisserman (2015). In Figs. 1 and 3, Section 4.2, and Section 5, when
mentioning a CNN trained on CIFAR-10, we refer to model architecture A64. In
Section 4.3, the evaluation involves the 16 models. The original 16 models for
CIFAR-10 were trained using AdamW with a learning rate of 0.001 and weight
decay of 0.01 until convergence. The final test set accuracies of the original
models and the FVs of the cat output neuron are presented in Fig. 8.

Figure 8: 16 classification models of varying depth (“A” - “D”) and width
($\times 8$ \- $\times 64$) trained on CIFAR-10 were manipulated to change the
FV of the cat output neuron to a fish image. The figure depicts sample FVs of
the original models, along with the test accuracies.

Adversarial fine-tuning

In all experiments, we fine-tune the original networks based on our defined
loss function in Equation 12, with coefficients $w=0.1$ and $\gamma=10.0$. For
MNIST models, we employ the sampling set $\bm{M}$ (defined in Section 3.3.1)
with parameters $\sigma_{s}=\sigma_{l}=1.0$. For CIFAR-10, the signals are
sampled in the scaled frequency domain Olah et al. (2017), and the parameters
are $\sigma_{s}=\sigma_{l}=0.1$. For both MNIST and CIFAR-10, the number of
sampled signals $N$ is equal to the size of the train set, and
$N^{{}^{\prime}}=N/2$. In FV manipulation, the target image is parameterized
in the scaled frequency domain using the inverse of the forward function.
During the fine-tuning process, we temporarily replaced the ReLU activation
functions with Softplus, following the approach discussed in Dombrowski et al.
(2019). The models undergo fine-tuning using the AdamW optimizer with weight
decay 0.01 until convergence. For the MNIST experiments in Section 4.2 and
CIFAR-10 experiments in Section 4.3, a learning rate of 0.001 is used. For
CIFAR-10 experiments in Section 4.2, a learning rate of 0.0001 is used. The
learning rate is halved on a training epoch loss plateau with a threshold of
0.001.

In Section 4.2, the parameter $\alpha$ is varied. In Fig. 1 and Section 5,
when referring to a manipulated CNN trained on MNIST, we specifically refer to
the model fine-tuned with parameter $\alpha=0.067$ from Section 4.2.
Similarly, in Fig. 1 and Section 5, when discussing a manipulated CNN trained
on CIFAR-10, we specifically refer to the model fine-tuned with parameter
$\alpha=0.01$ from Section 4.2. In Section 4.3, parameter $\alpha=0.01$ is
uniformly employed for all manipulated models.

Activation Maximization Procedure

In all experiments, both in pixel-AM and FV settings, we initialize signals
randomly from a normal distribution with a mean of zero and a standard
deviation of 0.01. For pixel-AM, the initialization signal is in the input
domain. For FV, the initialization signal is sampled in the scaled frequency
domain and transformed into the pixel domain using the forward function
adopted from the `Torch-Dreams` library Deb (2021) before being fed to the
network. The forward function involves inverse Fast Fourier Transform. Unless
otherwise specified, we utilize the standard gradient ascent algorithm with a
learning rate of $0.1$ to optimize the signal, aiming to maximize the
activation of the target neuron. When comparing the AM output before and after
manipulation, the AM procedure parameters remain consistent.

Defense Parameters

In both the MNIST and CIFAR-10 experimental settings, we mostly utilize the
same defense strategy parameters. For gradient clipping (GC), the gradient
norm is constrained to $1.0$. For transformation robustness (TR), we apply the
following sequence of transformations 333The transformations are implemented
using Torchvision. See https://pytorch.org/vision/main/transforms.html for
further details.:

  * •

padding with 3 pixels on each side for MNIST and 5 pixels for CIFAR-10;

  * •

random affine transformation with the range of degrees from -20° to 21°,
scaling factor from -0.75 to 1.025 and fill value 0.5;

  * •

random rotation with the range of degrees from -20° to 21°;

  * •

randomly crop back to the original image size

When testing a changed optimizer algorithm, we use Adam. When employing Adam
as an AM optimizer, the AM learning rate $\epsilon$ is always reduced to
$\epsilon/10$.

##  Appendix D Manipulation Evaluation

###  D.1 MNIST

Figs. 11, 11 and 11 showcase additional evaluation experiments for the
manipulation of the CNN trained on MNIST, as discussed in Section 4.2.

Figure 9: Visualization of pixel-Activation Maximization (pixel-AM) results
for a CNN trained on MNIST, depicting the original and the manipulated AM
across optimization steps 0 to 100, sampled at regular intervals. The
experimental details are as described in Section 4.2 and Appendix C. The
manipulated MNIST model is fine-tuned with the parameter $\alpha=0.067$.

Figure 10: Similarity metrics (vertical axis) SSIM, LPIPS, and MSE between the
manipulated output of pixel-AM and the target image with confidence intervals
across optimization steps (horizontal axis) for a CNN trained on MNIST. The
experimental details are as described in Section 4.2 and Appendix C. The
manipulated MNIST model is fine-tuned with the parameter $\alpha=0.067$.

Figure 11: Box plots illustrating the distribution of similarity metrics,
including SSIM, LPIPS, and MSE, between the pixel-AM in manipulated neurons
and the target image for a CNN trained on MNIST. The horizontal axis
represents different manipulated models based on the parameter $\alpha$. The
experimental details are as described in Section 4.2 and Appendix C.

###  D.2 CIFAR-10

Figs. 14, 14 and 14 showcase additional evaluation experiments for the
manipulation of the CNN trained on CIFAR-10, as discussed in Section 4.2.

Figure 12: Visualization of feature visualization (FV) results for a CNN
trained on CIFAR-10, depicting the original and the manipulated AM across
optimization steps 0 to 100, sampled at regular intervals. The experimental
details are as described in Section 4.2 and Appendix C. The manipulated
CIFAR-10 model is fine-tuned with the parameter $\alpha=0.01$.

Figure 13: Similarity metrics (vertical axis) SSIM, LPIPS, and MSE between the
manipulated output of FV and the target image with confidence intervals across
optimization steps (horizontal axis) for a CNN trained on CIFAR-10. The
experimental details are as described in Section 4.2 and Appendix C. The
manipulated CIFAR-10 model is fine-tuned with the parameter $\alpha=0.01$.

Figure 14: Box plots illustrating the distribution of similarity metrics,
including SSIM, LPIPS, and MSE, between the FV in manipulated neurons and the
target image for a CNN trained on CIFAR-10. The horizontal axis represents
different manipulated models based on the parameter $\alpha$. The experimental
details are as described in Section 4.2 and Appendix C.