# Panda or not Panda? Understanding Adversarial Attacks with Interactive
Visualization

Yuzhe You, Jarvis Tse, and Jian Zhao Yuzhe You, Jarvis Tse, and Jian Zhao are
with the University of Waterloo. Emails: {y28you, jarvis.tse,
<EMAIL_ADDRESS>

###### Abstract

Adversarial machine learning (AML) studies attacks that can fool machine
learning algorithms into generating incorrect outcomes as well as the defenses
against worst-case attacks to strengthen model robustness. Specifically for
image classification, it is challenging to understand adversarial attacks due
to their use of subtle perturbations that are not human-interpretable, as well
as the variability of attack impacts influenced by diverse methodologies,
instance differences, and model architectures. Through a design study with AML
learners and teachers, we introduce AdvEx, a multi-level interactive
visualization system that comprehensively presents the properties and impacts
of evasion attacks on different image classifiers for novice AML learners. We
quantitatively and qualitatively assessed AdvEx in a two-part evaluation
including user studies and expert interviews. Our results show that AdvEx is
not only highly effective as a visualization tool for understanding AML
mechanisms, but also provides an engaging and enjoyable learning experience,
thus demonstrating its overall benefits for AML learners.

###### Index Terms:

Information visualization, explainable AI, adversarial attack, machine
learning

Figure 1: AdvEx user interface: (a) Robustness Analyzers that display the
models’ prediction accuracy pre- and post-attack; (b) Perturbation Adjuster
that initiates the attack sequence with specified magnitude; (c) Data
Projectors that visualize data embeddings in a 2-D latent space; (d) Instance-
level Attack Explainer that displays in-depth information of the highlighted
instance; (e) General Information Provider that provides more background on
AdvEx and AML.

##  1 Introduction

Adversarial _evasion attacks_ produce deceptive inputs (e.g., adversarial
images) that are subtly altered with human-imperceptible perturbations to fool
machine learning (ML) models into making prediction mistakes. In 2014,
Goodfellow et al. [1] showed that an adversarial image of a panda could easily
fool GoogLeNet [2] into labeling it as a gibbon with high confidence,
resulting in the birth of _adversarial machine learning_ (AML) research.
Similar attack methods have been shown to achieve high misclassification rates
in road sign classifiers [3] and evade automated surveillance cameras [4].
Though more and more people are studying and applying ML, many remain
uninformed about the dangers of adversarial attacks to their models due to a
lack of understanding in AML. As a result, the models developed often achieve
good natural accuracy but are highly susceptible to attack-perturbed inputs
[5]. For these users (e.g., students, novice ML developers) to design or
calibrate models to be adversarially robust for real-world applications,
understanding the concepts and impacts of adversarial attacks is essential.

Many studies have shown that visualizations serve as an effective means of
explaining complex ML concepts to non-experts interactively, augmenting
traditional passive learning experience (e.g., textbooks and videos) [6, 7,
8]. Specifically, we aim to design visualizations to benefit learners who have
an ML background but are unfamiliar with the risks of adversarial attacks, and
are interested in exploring AML to seek to build safer models for their
applications. For this work, we focus on evasion attacks in image
classification, a highly active AML research path that most existing work [1,
9, 10] focuses on since such models are frequently used in safety-critical
applications [11, 12].

There are certain key challenges in understanding adversarial attacks for
image classification. First, comprehending the attack process requires more
than mere image inspection, as adversarial attacks utilize perturbations that
exploit data features beyond human interpretation [9]. These subtle
modifications appear as imperceptible noise to human observers, making the
adversarial images almost indistinguishable from their clean versions. Second,
an attack’s efficacy varies based on the targeted instance and its label [13],
meaning a few instances do not reflect the attack’s behavior on the whole
dataset or other classes, requiring multi-level inspection. Third, the attack
impact also depends on model architecture and training method [14, 9],
necessitating model comparison to understand attack variability as one model’s
performance cannot reflect the same attack’s effects on others. Lastly,
different attack methodologies yield varying impacts on models [10, 15], so
evaluations with different attacks and classifiers are needed to fully grasp
the attack landscape. Therefore, visualizations need to be deliberately
designed to illustrate the characteristics and effects of adversarial attacks,
covering visual explanations of the attack logic, multi-level inspection
across datasets and individual instances, model comparison to understand
variability, and support for diverse attack methodologies.

However, existing educational tools for AML fail to address these challenges,
lacking comprehensiveness and generalizability in presenting various attack
properties. For instance, Adversarial-Playground [16] is limited by its
simplistic approach that displays an adversarial image beside its original, a
method that is ineffective when the two images look identical from subtle
perturbations. Bluff [17] relies on visualizing the internal neuron logic on
benign and adversarial examples, sacrificing model generalizability. Both
tools, limited to specific models/attacks, insufficiently represent evasion
attacks by neglecting the influence of varying model architectures, training
methods, and instance differences, leaving learners with an incomplete
understanding. While advanced AML visual analytic tools exist (e.g., AEVis
[18] and Ma et al.’s [19]), they are designed for experts to perform model
analysis with complex visualizations that are challenging for novices to
understand, thus not suitable for learning purposes. Further, AEVis lacks
model comparisons and dataset-level visualizations, while Ma et al.’s is
limited to data poisoning attacks in binary classification, lacking support
for evasion attacks in multiclass classification.

To better augment learners’ experience and address the limitations of existing
tools, we carried out a study to design an interactive visualization that
helps learners understand evasion attacks at multiple levels, while allowing
observation of their impacts on different models. Our primary objective is to
help novice _learners_ gain a comprehensive understanding of the properties
and risks of adversarial attacks from multiple lenses, thus enabling them to
make more informed decisions during model development to mitigate the risks
posed by adversarial attacks. Through this work, we have made the following
contributions:

  * •

We conducted a design study on employing interactive visualization to augment
the learning experience for AML. Our study involved both literature reviews
and user interviews with AML learners (N=3) and teachers (N=3) for design
guideline formulation, followed by system development and an extensive
evaluation.

  * •

We designed and implemented AdvEx, a novel interactive visualization for
novice learners to gain a comprehensive understanding of adversarial attacks.
To the best of our knowledge, AdvEx is the first multi-faceted visualization
designed specifically to support comprehensive learning of evasion attacks on
both instance and population levels. Additionally, it supports model
comparison and can readily adapt to different image classifiers and evasion
attacks, addressing the generalizability gap in existing works (e.g., [16,
17]).

  * •

We performed a two-part evaluation with 12 novice learners and 7 AML
experts/teachers to quantitatively and qualitatively evaluate the learning
aspects and usability of AdvEx. Our results show that AdvEx not only is highly
effective in facilitating understanding of adversarial attacks, but also
offers an engaging and enjoyable learning experience, thus amplifying its
educational impact. The strengths and limitations of AdvEx are discussed,
providing in-depth insights on how such a tool can be effectively utilized in
an educational setting.

##  2 Related Work

###  2.1 Adversarial Machine Learning

Many adversarial attacks have been proposed to work under different threat
models, namely white-box and black-box attacks. A white-box attack has full
access to the model’s internals, while a black-box attack can only access
model inputs and outputs. Fast Gradient Sign Method (FGSM) [1], Basic
Iterative Method (BIM) [20], and Projected Gradient Descent (PGD) [14] are
some of the well-known white-box attacks. Advanced black-box attacks include
Zeroth Order Optimization (ZOO) [15], HopSkipJump Attack [21], and Substitute
Model Attack [22]. To counter adversarial attacks, various defense methods
have been proposed to fortify model robustness against adversarial inputs. The
most effective defense is _adversarial training_ , which trains classifiers
with adversarial examples by adding them to the training set [1, 14] or
through regularizations [10, 23]. TRadeoff-inspired Adversarial DEfense via
Surrogate-loss minimization (TRADES) [10] is a state-of-the-art adversarial
training method that leverages a regularized surrogate loss from the observed
trade-off between robustness and accuracy. Other examples of adversarial
defenses include standard adversarial training [14], robust self-training
(RST) [24], local linear regularization (LLR) [23], etc.

While our system can be generalized to different evasion attacks, in this
paper, we demonstrate AdvEx using two attacks recommended by the AML
instructors we consulted, including FGSM, one of the earliest and most well-
known white-box attacks [1], and ZOO, a highly effective black-box query-based
attack [15]. Several prior studies have tried to understand the
characteristics of these two attacks. For example, Zhang et al. [25]
discovered that FGSM may create not only 2-D adversarial images but also 3-D
adversarial examples by applying the attack methodology to PointNet [26], a
DNN designed for 3-D point cloud data. Ye et al. [27] applied adversarial
attacks to a DL-based multiuser OFDM detector [28] and showed that ZOO
achieved the best performance among black-box methods. Additionally, both
attacks are frequently used in existing works to evaluate the effectiveness of
adversarial defenses [14, 29, 30, 31, 32] or as comparisons to other attacks
[33, 34, 35]. The abundance of existing works on both methods shows that they
are well-known attacks and hence good introductory examples for those new to
AML. As AML is a relatively new area of ML, it is crucial to raise awareness
on attacks like FGSM and ZOO to encourage users to build safer AI
applications, especially those that are safety-critical.

###  2.2 Visualizations of Adversarial Attacks

In general, interactive tools designed for visualizing adversarial attacks are
relatively under-explored. A few tools with educational purposes have been
proposed in past studies. Adversarial-Playground [16] is a simple web
application that demonstrates the efficacy of three attack algorithms against
a small CNN on the MNIST dataset [36]. The tool allows users to choose from a
set of pre-defined inputs and displays the adversarial image next to its
original alongside classification likelihoods to illustrate the attack. Bluff
[17] visualizes attacks on a vision-based network, but focuses on model
internals instead by highlighting the neurons and connections that an attack
exploits to confuse the model.

However, these tools lack comprehensiveness and multi-faceted approaches in
visualizing adversarial attacks. For instance, Adversarial-Playground [16]
offers a simple image comparison approach that becomes ineffective if used to
visualize attacks that generate “imperceptible” inputs, a common
characteristic among adversarial attacks. Its applied perturbations on the
black and white MNIST dataset are also highly visible, which could create a
false sense of security among learners about their abilities to discern
adversarial images from clean ones. Bluff [17], on the other hand, relies on
visualizing a model’s internal neurons, thus not easily extendable to other
model architectures. Both tools are restrained to specific attacks/models,
supporting visualization of only one classifier and several instances, missing
dataset-level attack information and inadequately demonstrating the
variability in attack impacts due to differences in model structures and
training methods.

In addition, a few advanced AML visual analytics tools have been developed as
well. AEVis [18] uses a river-based visual metaphor to show how the datapaths
of clean and adversarial examples merge or diverge within the network.
However, it suffers from the same limitation of lacking model comparisons and
dataset-level information, and cannot be used to visualize attacks with
varying perturbation sizes. Ma et al. [19] proposed a framework that employs a
multi-level visualization scheme to support the analysis of data poisoning
attacks in binary classification tasks. While comprehensive, it is designed
specifically for data poisoning attacks in binary classifications, thus
diverging in focus from this work and lacking support for evasion attacks in
multiclass classifications. Moreover, both tools are designed primarily for
experienced practitioners to perform visual analytics on models under
adversarial attacks, featuring complex visualizations and interfaces that may
be overwhelming for novice learners.

Hence, current educational tools lack comprehensiveness, often visualizing a
few instances and limited to specific attacks and models; current advanced
visual analytics tools are overly complex for our intended audience or have a
different focus from this work. In contrast, with AdvEx, we aim to enable
users who have little or no knowledge of AML to learn about adversarial
attacks at both dataset and instance levels, while making it easy to be
generalized to different evasion attacks and vision-based classifiers.

In addition, to visualize the shift in how models perceive the dataset before
and after an attack, we incorporated a dimensionality reduction overview
depicting each model’s feature space in AdvEx. Dimensionality reduction has
been used frequently to understand and visualize adversarial attacks. For
instance, Ma et al. [19] and Park et al. [37] utilize t-SNE for data embedding
views to visualize the impacts of data augmentations including adversarial
attacks. Panda and Roy [38] introduced a Noise-based Learning (NoL) approach
for training robust DNNs and provided simplistic PCA-based visualizations for
adversarial dimensionality and loss surface visual analysis. Hendrycks and
Gimpel [39] incorporated PCA into adversarial image detection and visualized
how adversarial images abnormally emphasize coefficients for low-ranked
principal components. Inspired by these works, in AdvEx, we apply similar
methods to project the data embeddings onto a 2-D plane, and use animated
transitions and colors of circular glyphs to visualize how the attacks alter
the models’ perception of the images.

###  2.3 Visualizations for Learning ML

Outside of AML, several visualization tools specifically designed for learning
ML have been proposed as well. GAN Lab [7] is designed for non-experts to
learn and experiment with generative adversarial networks (GANs) by
visualizing GANs’ dynamic training processes on a simple dataset. CNN
Explainer [6] enables learners to inspect the interplay between CNNs’ low-
level mathematical operations and their high-level model structures. Summit
[8] provides higher-level explanations of DNNs by visualizing image features
detected by the networks and how those features interact to make predictions.
While these tools are effective for demonstrating basic ML concepts, they are
not suitable for our study’s design objective in the context of AML learning.
For example, GAN Lab [7] is only for exploring generative models on low-
dimensional training datasets and significantly diverges from the focus of
this work. While CNN Explainer [6] and Summit [8] could potentially be
extended to explore a model’s internal datapaths on adversarial examples, they
would still share the limitations of lacking model generalizability and
dataset-level attack information like Bluff [17] and AEVis [18].

Despite focusing on visualizing common DNNs instead of adversarial attacks,
all three aforementioned studies have provided us with inspirations for
AdvEx’s design. Specifically, similar to GAN Lab [7] and CNN Explainer [6],
AdvEx is accessible to any user with a modern browser without the need to
install specialized hardware for deep learning. Motivated by GAN Lab [7]’s
step-by-step training visualization, AdvEx provides step-by-step executions of
the attack methodology to visualize the detailed attack process. Like CNN
Explainer [6] and Summit [8], AdvEx also adopts smooth transitions across
different levels of abstraction to facilitate visual exploration and to serve
as the link that connects different views of the visualization tool. Based on
existing work, we aim to develop AdvEx as a tool with comprehensive
visualizations and animations that can enable intuitive exploration of attack
properties across multiple levels.

##  3 Design Goals

To formulate the design guidelines for AdvEx, we conducted user interviews
with six participants, including three interviewees (S1, S2, S3) who have AML
learning experience and three AML teachers (E1, E2, E3). Our goal was to
understand learners’ needs in understanding adversarial attacks and to have
experienced AML teachers envision how such a tool can be utilized in an
educational setting. The learners involved come from computer science and data
science backgrounds, and their employed learning methods varied from enrolling
in AML courses to reading academic papers or online blog posts. The teaching
experience of the interviewed educators ranged from leading graduate-level AML
seminars to overseeing AML components within undergraduate ML courses. The
semi-structured interviews lasted between 60 to 90 minutes and covered the
following topics: 1) the participants’ background and experience in AML
learning/teaching, 2) existing content or tools used to understand/teach AML,
3) the challenges in understanding/teaching adversarial attacks, 4) features
and functionalities to include in an educational visual tool for adversarial
attacks, and 5) how participants envision using such a tool in an educational
setting. The participants were compensated $\$$20/hour for the interview.

While none of the interviewees had previously used any visualization tool for
adversarial attacks, all recognized the value of introducing a multi-level
visualization tool to demonstrate evasion attacks to learners. Specifically,
they believed that an interactive visualization tool would have multiple
educational benefits, including “providing an accessible way to demonstrate
attacks in practical applications” -E2, “making the learning experience more
engaging” -E3, and “accommodating learners with different backgrounds” -E1.
The interviewees also thought that the tool could be used either in a self-
learning scenario for exploration or incorporated into AML courses to
demonstrate concepts and better augment students’ learning experience. These
comments confirm the need for a visualization tool like AdvEx in both
independent and guided AML learning contexts.

We transcribed our interviews and employed a hybrid method of open and close
coding, informed by an extensive literature review (Section 2), to analyze the
gathered qualitative data. Using an affinity diagram, we identified recurring
themes and requirements of such a visualization tool for AML learning, and
derived the following design goals to guide the development of AdvEx:

  * G1

Present visual abstraction of the attack impact at multiple levels. Many
existing tools (e.g.,[16, 17]) only display instance-level attack information,
such as how a specific image is modified by the attack. These instance details
are insufficient to illustrate the reason behind misclassifications or the
overall attack impact on a larger dataset. E3 mentioned, “When simply
comparing the images, we can observe the differences from a human perspective,
but it remains unclear why the model misclassifies them.” E2 agreed that
“Examining images prone to misclassification is vital, but seeing the broader
impact is equally important to fully grasp the risks.” Therefore, visual
abstractions at multiple levels should be included to provide both dataset-
level overviews of the attack and the options to conduct more in-depth
investigations on specific instances.

  * G2

Design a visualization framework that can be generalized to different evasion
attacks and image classifiers. Generalizability is crucial as it enables
learners to grasp the variability of attack methods, assess different kinds of
models under attacks, and connect theoretical knowledge with practical
applications. E3 confirmed that “A key learning objective should be the
various methods to generate adversarial examples, which is essential for
understanding how to defend against these diverse attack strategies.” S2 and
S3 agreed that demonstrating attacks on different models would be beneficial
for “grounding learners’ understanding in practical scenarios” -S2, and
“shifting focus from academic papers to user-specific examples” -S3. For
AdvEx, we aim to address the gap of existing works [16, 17] being constrained
to specific attacks/models by designing a general framework in these aspects
to enable a more holistic and practical understanding of the attacks.

  * G3

Enable comparative analysis of different models’ robustness under attack.
Models with different architectures and training methods vary in their
robustness against the same attack [10, 9, 1], but most learning tools [16,
17] demonstrate attacks with a single, arbitrary model. Enabling visual
analysis of multiple models is important to facilitate understandings of the
variability in attack impact and to highlight the rationales for why certain
models would fail. E3 stated, “Comparing the model differences provides
insights into why certain attacks succeed or fail, going beyond just seeing
changes in model accuracy.” E2, S2, & S3 agreed that model comparisons
“highlight the models’ varying defense abilities” -S3 and in turn “helps
learners defend and improve their own” -E2. Thus, in AdvEx, we aim to provide
side-by-side comparisons of different models under vairous attack scenarios.

  * G4

Facilitate dynamic experimentation with fluid transition between different
perturbation sizes. As the perturbation size increases, the attack becomes
more effective and the applied noise also becomes more visible. Allowing users
to dynamically experiment with the perturbation size and observe the changes
in real time “facilitates a better understanding of this correlation” -E2 and
“creates a more game-like, engaging learning process” -S1. E1 pointed out that
such experimentation “allows learners to observe how the model’s perception of
an instance changes, and identify the threshold at which misclassification
occurs.” S3 stated that the approach “helps learners understand when exactly
the image starts to look different for humans.” Therefore, interfaces are
included to allow easy manipulation of the perturbation size and visualize the
changes in real time.

  * G5

Allow step-by-step execution for learning the attack process in detail.
Mentioned by E1, E2, & S2, navigating complex mathematical steps in papers to
understand attack logic is a daunting task for learners. A step-by-step attack
execution “provides a more structured understanding of attack strategies” -E2.
This approach allows learners to “grasp not just the impact but also the
design and rationale behind the attacks” -E3. E2 confirmed, “A step-by-step
approach simplifies the attack process and reduces learners’ burden compared
to interpreting steps directly from papers.” In AdvEx, we aim to incorporate a
step-by-step view to clarify the underlying attack logic for learners, guiding
them through the complexities of various attack strategies.

##  4 AdvEx

Figure 2: A schematic diagram depicting the system architecture of AdvEx. In
the backend pipeline, an Attacker module performs users’ choice of attacks on
the image dataset, targeting models specified by users (G2). Once processed,
the backend outputs are passed to the frontend interface for user interaction.

Based on our design guidelines, we developed AdvEx. Here, we begin with an
overview of AdvEx’s system, followed by detailed descriptions of its backend
modules and frontend components.

###  4.1 System Overview

As depicted in Figure 2, AdvEx is a web application with two main system
components: A) a backend pipeline (Section 4.3) and B) a frontend user
interface (Section 4.4).

In the backend pipeline, an Attacker module begins by converting the image
data into normalized numeric matrices and performing the chosen attack methods
to generate adversarial examples. Both the original and adversarial examples
are fed into the models to obtain information such as image embeddings,
confidence scores, and prediction accuracy. An Embedding Projector is employed
to extract each model’s embedding vectors by removing the final output layer
and applying dimensionality reduction methods (e.g., t-SNE [40], PCA [41]) to
prepare the projection coordinates of the data representations. The processed
outputs are relayed to the frontend components to be presented visually for
user interaction.

The frontend interface comprises five key components: 1) Data Projectors
(Figure 1c), 2) Instance-level Attack Explainer (Figure 1d), 3) Robustness
Analyzers (Figure 1a), 4) Perturbation Adjuster (Figure 1b), and 5) General
Information Provider (Figure 1e) + interactive tutorials. The Robustness
Analyzers feature two interactive bar charts that assess the models’ overall
robustness under a specified attack (G1) and offer a comparative view of this
robustness to natural accuracy (G3). The Data Projectors utilize coordinates
from the Embedding Projector to visualize data representations as two
interactive, side-by-side scatterplots. These scatterplots enable exploration
of attack-induced embedding changes (G1) and offer comparisons of embeddings
between different models (G2, G3). The Instance-Level Attack Explainer
provides detailed insights into a specific instance (G1), complemented by a
confidence score view and a step-by-step guide to the instance’s attack
process (G5). The Perturbation Adjuster allows users to select their desired
perturbation size and initiates animations within the three aforementioned
components to simulate the attack in real time (G4). Finally, along with
interactive tutorials, the General Information Provider guides users through
the navigation of the interface and offers further context on AML.

###  4.2 Dataset and Models

In this paper, we use the CIFAR-10 dataset [42] to demonstrate AdvEx, but our
system can be employed with any image dataset with $\leq$ 12 classes [43] or a
subset of a dataset with more classes. The CIFAR-10 dataset consists of 60,000
$32\times 32$ colored images from 10 different classes (50,000 training data
and 10,000 testing data), with 6,000 images per class. We chose this dataset
due its popularity of being used in ML research to evaluate the accuracy and
robustness of image classifiers [10, 44, 45].

In addition, AdvEx supports a variety of image classifiers and allows the user
to compare two models side by side (G2, G3). For example, users could compare
CNNs with the same architecture but different numbers of convolutional layers,
or investigate how a classifier trained adversarially may outperform a
standard model in an attack. For this paper, we loaded two pairs of models for
our studies: 1) VGG-16 vs. VGG-19, and 2) ResNet-34 trained naturally vs.
trained adversarially with TRADES [10].

###  4.3 Backend Pipeline

In this section, we describe how the backend processes and analyzes the data
in AdvEx, including how it generates the adversarial examples and prepares the
data instances and model outputs for frontend display.

####  4.3.1 Attacker Module

The “Attacker” module produces adversarial examples of the original dataset by
conducting adversarial attacks on the targeted models. It first retrieves the
dataset and normalizes all images’ pixel values between 0 and 1, then conducts
the attack on the data instances to create adversarial images. This is
achieved by first feeding the natural images into the targeted models (or
surrogate models) to obtain information relevant to the attack, then adjusting
the pixel values of the input image based on the obtained information. While
the attack algorithm within the module can be swapped with any evasion attack
(G2), here we use one white-box attack and one black-box attack, FGSM [1] and
ZOO [15], as examples for demonstrating our system.

We chose the FGSM attack due to its notoriety for creating the very first
adversarial image, namely the panda image from [1] that is well-known among
AML researchers. The attack is commonly used as a baseline method to evaluate
model robustness and the effectiveness of adversarial training [14, 25, 46].
In addition, the attack was recommended by the AML instructors we consulted as
it is relatively simple in logic and often used as the introductory attack in
AML courses and tutorials. Though simple in logic, the attack has been proven
to be extremely effective [1]:

$\mathbf{x}^{\prime}=\mathbf{x}+\epsilon\textrm{{sign}}(\nabla_{\mathbf{x}}J(\theta,\mathbf{x},y)).$
(1)

The attack modifies the input image $\mathbf{x}$ towards maximizing the loss
$J(\theta,\mathbf{x},y)$ by adjusting pixels in the direction of the
gradients’ sign to produce the adversarial image $\mathbf{x}^{\prime}$. Here,
$y$ is the true label, $\theta$ represents model parameters, and $\epsilon$
scales the perturbation. We used FGSM with $L^{\infty}$ norm, restricting the
maximum pixel change to create bounded adversarial examples.

Additionally, we employed the ZOO attack [15], an advanced black-box attack,
as the second method to demonstrate AdvEx. This choice was also guided by our
consulted AML instructors, who highlighted that the attack is often used as a
representative example of black-box attacks in AML courses. Furthermore, like
FGSM, ZOO is frequently used to evaluate the effectiveness of defense methods
[32, 29] or as a benchmark for other attacks [35, 28]. However, unlike white-
box attacks, ZOO only has access to the inputs (e.g., images) and outputs
(e.g., confidence scores) of a targeted model, and aims to find the
adversarial example $\mathbf{x}^{\prime}$ by solving the following
optimization problem:

$\displaystyle\underset{\mathbf{x}^{\prime}}{\text{minimize}}$
$\displaystyle\lVert\mathbf{x}^{\prime}-\mathbf{x}\rVert_{2}^{2}+c\cdot
f(\mathbf{x}^{\prime})$ (2) subject to
$\displaystyle\mathbf{x}^{\prime}\in[0,1]^{p}$

The first term $\lVert\mathbf{x}^{\prime}-\mathbf{x}\rVert_{2}^{2}$ applies
$L^{2}$ norm regularization to enforce the similarity between the adversarial
image $\mathbf{x}^{\prime}$ and the clean image $\mathbf{x}$, ensuring that
the adversarial example is as close to the original input as possible. The
second term $c\cdot f(\mathbf{x}^{\prime})$ is the loss that represents the
level of unsuccessful attacks, with $c>0$ as the regularization parameter.
Without relying on actual backpropagation, the attack computes an
approximation of the gradient using a finite difference method and solves the
optimization problem via zeroth order optimization.

Employed with the chosen attack algorithm, the module performs attacks on the
models respectively with the selected perturbation sizes $\epsilon$. Based on
the suggested perturbation limits from RobustBench [47], we selected
$\epsilon$ of 0.00, 0.01, 0.02, and 0.03 for FGSM with $L^{\infty}$ norm, and
$\epsilon$ of 0.0, 0.1, 0.3, and 0.5 for ZOO with $L^{2}$ norm. The resulting
adversarial examples, along with the original data, are then inputted in the
models for classification and embedding extraction.

####  4.3.2 Embedding Projector

The Embedding Projector is tasked with 1) processing the models’ produced
embeddings and 2) analyzing the information of the extracted features and
preserving it in a low-dimensional representation. The goal is to unveil
important patterns in the embeddings and transform them into a format easily
fetched by frontend. The module temporarily detaches the final output layer to
obtain the embeddings and reduces their dimensions by applying users’ choice
of dimensionality reduction for later 2-D visualizations. For instance, in the
case of t-SNE, the module analyzes instance features by constructing a lower-
dimensional probability distribution that represents the similarities between
the objects in the high-dimensional space. If PCA is used, the module
preserves the most significant variability in the embeddings while reducing
the number of features. The resulting outputs are scaled to be used as the x-
and y-coordinates of the instances in scatterplots and are stored as tabular
data easily accessed by the frontend Data Projectors.

###  4.4 Frontend User Interface

Here, we detail the frontend components of AdvEx. We demonstrate our approach
using FGSM on VGG-16 and VGG-19 models pre-trained with CIFAR-10 [48].

####  4.4.1 Data Projectors

Figure 3: A user highlights and tracks a specific class from the dataset with
selection mode. Under this mode, one can evaluate model performance on a
dataset subset. Figure 4: We explored a variety of visual encodings and
aggregating features for the Data Projectors. We chose binned aggregation with
multiple zoom levels, with an optional hexbin toggle to display the overall
distribution (Fig. c). This preserves data scalability and displays global
data structure without the need for high-performance devices.

The Data Projectors represent dimensionality reduction overviews of the
dataset and consist of two scatterplots where the image embeddings are
projected as circles on a 2-D plane. Each circle corresponds to a data
instance and is sliced into two halves: the color of the left half represents
the instance’s ground truth label, while the color of the right half
represents its current prediction. The spatial positions of the circles encode
the relationships between them in the original high-dimensional space (e.g.,
similarities, variance, local and global structure). Inspired by nanocubes
[49], we use a combination of binned aggregation and hierarchical clustering
with multiple zoom levels to preserve data scalability (Figure 1c1). Our
approach allows the user to interactively explore data sources with large
numbers of instances while maintaining the global data structures without
high-performance devices. When an attack is conducted with newly specified
magnitude, the Data Projectors visualize the attack with an animated sequence
that emphasizes each circle’s change in position and color (G4). For example,
if a circle transitions to a different coordinate, this indicates that the
model’s perception of the instance’s features has been altered by the attack.
Moreover, if the class “airplane” is represented by the color red and the
class “automobile” is represented by the color orange, then a red circle
transitioning into a half-red, half-orange circle means that this is an
airplane image incorrectly classified as an automobile due to the attack. To
improve the usability of the projectors, the following functionalities are
also incorporated:

  * •

Inspection mode.  Under this mode, users can zoom and drag freely within the
scatterplots to explore the models’ embedding distributions. To avoid overlap
when instances share similar features, AdvEx dynamically adjusts the radius of
the projected circles at different zoom levels, allowing users to precisely
examine each individual instance. Clicking a circle highlights the instance by
enlarging its radius and pinning it, then panning the entire plot to recenter
that circle within the 2-D plane.

  * •

Selection mode (Figure 3).  In this mode, users can highlight a subset of the
dataset, including a single item, by specifying a selected region with a
pointing gesture. As a result, only colors of the selected circles within the
region remain visible, while all other instances are grayed out. This feature
allows users to track the movements of specific subgroups or instances across
different perturbation sizes, adding a subpopulation-level display (G1). When
a group/instance is highlighted in one Data Projector, the same group/instance
is simultaneously highlighted in the other projector for comparison (G3).

  * •

Hexagonal binning toggle.  While navigating, users can toggle a hexagonal
binning map (Figure 4c) for each projector to track the global data structure.
The hexbin map displays the general trends in instance clustering based on
model predictions, allowing quick identification of decision boundaries and
similarly classified image groups (G1). Moreover, this approach preserves
visibility of the whole dataset’s distribution even when the projectors are
only displaying a subset at higher zoom levels.

In summary, the Data Projectors provide interactive visualizations of image
embeddings, illustrating instance relationships via spatiality and revealing
population-level attack impacts though animated transitions (G1, G4). Given
that the set of generated adversarial examples varies depending on the chosen
attack method and the model targeted, we provide side-by-side visualizations
of two distinct models’ embeddings to illustrate the differential impacts of
the attack (G3). By interacting with the Data Projectors, users can
intuitively observe how changes in the perturbation size influence both the
models’ data representations and their resulting image predictions, thus
gaining a deeper understanding of the attack’s impact on model performance.

####  4.4.2 Instance-level Attack Explainer

Figure 5: An example of the step-by-step execution view for explaining the
FGSM attack.

While the Data Projectors visualize attacks on dataset or subpopulation
levels, the Instance-level Attack Explainer provides in-depth information for
each perturbed input. It outlines the attack process for each image, detailing
instance-level information such as the original image, applied noise, and
confidence scores (G1). To examine an instance, users click on the
corresponding circle in the Data Projectors to update the panels associated
with the attack explainer. Specifically, the Instance-level Attack Explainer
consists of the following components:

  * •

General view. The general view (Figure 1d1) displays key information of the
selected instance, including its original image, applied noise, adversarial
image, targeted model, true label, and current prediction (G1). To help users
contextualize these instance-level details, subtle animations are used to link
the information together to depict the high-level attack flow. For instance, a
repeated animated sequence shows the original image and the generated
perturbation progressively moving towards each other with reduced transparency
and stacking on top of each other, then gradually fading into the final
perturbed image. Dashed lines connecting the images are animated to
continuously move from the clean image + noise to the resulting adversarial
image. By presenting a visual attack narrative, these animations are designed
to help users better interpret the instance-specific information within the
context of the current attack.

  * •

Side-by-side image inspection. For a closer inspection of the images, users
can click on the image thumbnails in the general view to view enlarged
versions. A comparison mode is available to examine the clean and adversarial
images side by side and observe the exact pixel differences.

  * •

Confidence score view. The interactive bar chart panel (Figure 1d2) showcases
the model’s pre- and post-attack confidence scores across all classes for the
selected instance. These scores are grouped together in pairs to provide
comparison between the model’s confidence before and after an attack. Hovering
over each pair of them reveals their exact difference in percentage, allowing
users to quantitatively assess the attack impact on class-wise classification
probabilities (G1).

  * •

Step-by-step execution view. The step-by-step execution view (Figure 5)
provides detailed explanations of the underlying attack logic. Clicking the
button at the bottom-right of the general view activates this feature, which
initiates a series of step-by-step animated sequences with accompanying
explanations (G5). Once activated, these explanations unfold sequentially. For
instance, Explanation #2 (Figure 5-2) does not appear until users click the
play button next to Explanation #1 (Figure 5-1), which becomes visible only
after Explanation #1 has finished playing. A toggle allows users to replace
the default image with their selected instance for the view’s demonstration,
allowing them to apply step-by-step explanations to an actual adversarial
example they are examining. This feature provides users with a more tangible
and personalized understanding of how adversarial attacks manifest and operate
on real-world examples (G2).

In short, the Instance-level Attack Explainer offers a focused, in-depth look
at adversarial attacks on individual instances (G1). It translates complex
attack processes into intuitive visual narratives and adopts a step-by-step
approach to guide users through the underlying attack logic (G5). Also, its
confidence score view enables users to quantitatively explore and assess how
the attack impacts the model’s confidence for the given instance (G1).
Together, these features offer a detailed perspective of the instance-specific
properties and consequences of adversarial attacks.

####  4.4.3 Robustness Analyzers

The Robustness Analyzers in the leftmost panel feature two compact,
interactive bar charts, each containing two bars. These charts evaluate the
model’s robustness under the given attack and compare it to the pre-attack
accuracy (G1, G3). The left bars represent natural accuracy, indicating the
model’s prediction accuracy on the clean dataset, while the right bars
represent robust accuracy, reflecting the model’s performance on the
adversarial dataset. As users adjust the perturbation size up and down, the
right bars dynamically adjust their heights to visualize the corresponding
changes in the model’s robust accuracy (G4). With the Robustness Analyzers,
users can compare 1) a model’s robustness to its baseline performance and 2)
the relative performance of different models under standard and adversarial
conditions (G2, G3). Consequently, users can gain insights into the attack’s
varying impact across models, identify which models are resistant or
vulnerable to the current attack, and quantify the degree of performance
degradation from adversarial inputs.

####  4.4.4 Perturbation Adjuster

The Perturbation Adjuster, situated below the Robustness Analyzers, features a
slider and an attack button. The slider allows users to choose a perturbation
size from a range they have pre-set in the backend, which they can adjust
horizontally to visualize the desired attack strength. Upon selecting a
perturbation size, users initiate an animated attack sequence by clicking the
attack button, which triggers changes in other components of the interface.
For example, the circles of the Data Projectors’ may shift to new coordinates
alongside prediction color changes, while the right bars of the Robustness
Analyzers adjust their heights up or downward based on the model’s accuracy
with the new adversarial dataset. With the Perturbation Adjuster, users can
dynamically modify the perturbation size and observe the increased attack
strength with larger perturbation sizes, as well as the growing visibility of
applied image noise (G4). The integration of dynamic perturbation control and
real-time visual feedback enables users to intuitively understand the
interplay between perturbation size, attack intensity, and resulting image
distortions.

####  4.4.5 Interactive Tutorials + General Information Provider

To help users pick up AdvEx more easily, an interactive tutorial system is
also integrated. Upon launching the application, users encounter an overlay
tutorial that introduces every component of AdvEx’s interface, highlighting
its key features. During interaction, hovering over any Data Projectors’
button displays a tooltip explaining its function. If users have not engaged
with certain key features (e.g., hexbin map, step-by-step execution) within 10
minutes, an animated arrow prompts them to explore these features.

Furthermore, if users wish to learn more about AdvEx and AML research, they
may read the information placed beneath the interactive components, which
provides more in-depth explanations for both. By including interactive
tutorials and reading materials, users will not only learn our tool faster,
but also gain detailed and accurate knowledge of adversarial attacks in
addition to perceiving them through interactive visualizations.

##  5 User Study with Novice Learners

To assess how AdvEx can help novice AML learners, we conducted a user study
with participants who knew basic ML but were unfamiliar with AML. We aimed to
investigate two aspects of AdvEx as an educational tool: (A1) whether AdvEx is
effective for helping learners understand the concepts and impacts of
adversarial attacks, and (A2) whether users enjoy using AdvEx for learning. We
did not conduct a comparative study due to existing AML educational tools
falling short of such comparison due to their inherent limitations:

  * •

Adversarial-Playground [16] is a very simple tool that only provides side-by-
side comparisons of natural and adversarial images with classification
likelihoods. Consequently, this tool includes only a fraction of the
functionalities inherent to a single component of AdvEx, i.e., the Instance-
level Attack Explainer.

  * •

Bluff [17] only visualizes the internal neuron pathways, which is a divergent
focus from the educational scope of AdvEx. While Bluff is for those interested
in the very low-level details of individual neuron behaviors, AdvEx is
designed to provide a more approachable and practical understanding of the
attacks through multiple lenses such as data embeddings, confidence scores,
and accuracy degradation.

Since current tools either serve fundamentally different purposes or offer a
limited subset of the functionalities provided by AdvEx, there currently
exists no suitable baseline for a meaningful and fair comparison.

###  5.1 Study Setup

Participants and Apparatus.  We recruited 12 participants (P1 $\sim$ P12; 10
men, two women; aged 21$\sim$31) from a local university. They came from
different areas of study such as computer science, transportation engineering,
and data science. All reported having a background in ML but were unfamiliar
with AML. Specifically, on a 7-point Likert scale (self-rated; 1=“Novice”,
7=“Expert”), we recruited participants that satisfied all the following
constraints: ML experience $\geq$ 2, AML experience $\leq$ 2, completion of
$\geq$ 1 ML project, completion of $\leq$ 1 AML project. Their median ML
experience was 4 ($\mbox{IQR}=2$), and their median AML experience was 1
($\mbox{IQR}=0.25$). The median number of ML projects completed was 2.5
($\mbox{IQR}=2.25$), while the median number of AML projects completed was 0
($\mbox{IQR}=0$). They interacted with AdvEx on provided laptops in-person.

Task and Procedure.  We loaded AdvEx with CIFAR-10 testing data perturbed by
FGSM in varying degrees to investigate the participants’ learning of the
properties and impacts of the attack. We selected FGSM for our first
evaluation study based on recommendations from all interviewed AML
instructors/learners, citing it as ideal for introducing AML concepts. E2
mentioned, “For learners, it is essential to start with foundational methods
like FGSM given its straightforward and basic nature.” S2 agreed that “When
teaching learners, it is best to use simpler attacks like FGSM.” We measured
the participants’ learning through a pre-quiz before their interaction with
AdvEx, followed by a post-quiz afterwards to assess the amount of knowledge
acquired through the use of our tool. The quizzes were collaboratively
designed with a renowned AML researcher/instructor who co-authored TRADES, the
state-of-the-art adversarial training method against evasion attacks that won
first place in the robust model track of NeurIPS 2018 Adversarial Vision
Challenge [10]. Prior to interacting with AdvEx, we asked the participants to
complete the pre-quiz that consisted of 9 questions to assess their ML
background and knowledge in AML. These questions included 4 checker questions
on basic ML and 5 questions that would be taught by AdvEx . The checker
questions were to ensure participants’ self-reported expertise aligned with
their background and to assess their attention during the study. After the
pre-quiz, we introduced AdvEx to the participants and provided them with 5
minutes to go through the beginning tutorial in AdvEx. The tutorial contained
basic background on adversarial attacks (e.g., what an adversarial attack is)
and guidance on navigating through the different components of AdvEx’s
interfaces. Participants were also given the freedom to revisit these
tutorials, which are included as part of the General Information Provider, at
any point during their interaction with AdvEx. Following the tutorials, we
provided the participants with 30 minutes to interact with AdvEx freely. We
instructed the participants to use AdvEx to learn about the FGSM attack as
much as they could, and informed them that there would be a follow-up post-
quiz to assess how much they had learned. Next, we asked the participants to
complete a 7-point Likert scale post-questionnaire (6 questions), which
collected their opinions on the learning and usability aspects of AdvEx. We
then asked them to complete the post-quiz (19 questions), which comprised of
the 9 original questions from the pre-quiz, along with 10 new questions that
were taught by AdvEx . We ended the study with a qualitative interview that
further asked for their thoughts and opinions on AdvEx.

The user study took about one hour and the participants received $\$$15 for
their effort. They were informed that the top 3 performers of both the pre-
quiz and post-quiz would be awarded an additional $\$$10.

Table I: The results of the paired t-tests and the quiz averages of our
participants (filtered & all). Our results show that AdvEx has a strong
learning effect on both filtered and all participants. ∗OQ (“old questions”):
9 questions from the pre-quiz that are also included in the post-quiz. †NQ
(“new questions”): 10 questions that are newly added in the post-quiz.
‡Average of total quiz (checkers + taught) scores.

c—ccccc—cccc[cell-space-limits=3pt]  Paired T-Tests Quiz Averages
Pre-quiz vs. Post-quiz Pre-quiz vs. Post-quiz OQ∗ Pre-quiz vs. Post-quiz NQ†
Pre-quiz Taught vs. Post-quiz Taught Pre-quiz Checkers vs. Post-quiz Checkers
Pre-Quiz Checkers Pre-Quiz Taught Post-Quiz Checkers Post-Quiz Taught
Filtered (10 Participants) t = -5.264, p = 0.00052 t = -6.128, p = 0.00017 t =
-4.229, p = 0.00221 t = -6.482, p = 0.00011 t = 1.0, p = 0.34344 85% ($\sigma$
= 21.08%) 50% ($\sigma$ = 17%) 82.5% ($\sigma$ = 26.48%) 93.33% ($\sigma$ =
6.28%)
65.56% ($\sigma$ = 16.93%)‡ 91.05% ($\sigma$ = 7.46%)‡
All (12 Participants) t = -6.225, p = 0.00006 t = -6.661, p = 0.00004 t =
-5.197, p = 0.0003 t = -5.88, p = 0.00011 t = -0.561, p = 0.5863 75% ($\sigma$
= 30.15%) 51.67% ($\sigma$ = 15.86%) 77.08% ($\sigma$ = 27.09%) 90% ($\sigma$
= 10.05%)
62.04% ($\sigma$ = 17.38%)‡ 87.28% ($\sigma$ = 11.32%)‡

Figure 6: Participants’ questionnaire ratings (1 = “strongly disagree”; 7 =
“strongly agree”) on the learning and usability aspects of AdvEx.

###  5.2 Results and Analysis: Task Performance

Out of 12 participants, we removed two whose pre-quiz checker scores were
below 50%. On average, the 10 remaining participants spent 3.97 minutes
($\sigma=0.07$) on the pre-quiz, 16.17 minutes ($\sigma=0.21$) on their
interaction with AdvEx, and 5.17 minutes ($\sigma=0.10$) on the post-quiz.
Before interacting with AdvEx, the participants had an average pre-quiz score
of 65.56% ($\sigma=16.93\%$), and 50% ($\sigma=17\%$) if excluding the checker
questions. After, the participants earned an average post-quiz score of 91.05%
($\sigma=7.46\%$), and 93.33% ($\sigma=6.28\%$) if excluding the checker
questions. While the difference between the mean pre-quiz and post-quiz scores
clearly indicates AdvEx’s effectiveness in enabling learning, we further
answered A1 by performing several paired t-tests on our collected quantitative
data.

Our first paired t-test shows a significant difference between the
participants’ overall pre-quiz and post-quiz performance
($t=-5.264,p=0.00052$); the difference is also significant in the second
paired t-test when the checker questions are excluded ($t=-6.482,p=0.00011$).
Both results indicate a strong performance improvement after interaction with
AdvEx. A third paired t-test shows a significant difference between their
performance on the same 9 questions in the pre-quiz and post-quiz
($t=-6.128,p=0.00017$). This indicates that the participants have successfully
learned the answers to the questions that were originally included in the pre-
quiz. Similarly, a significant difference can be observed between the
participants’ pre-quiz performance and their performance on the 10 newly added
questions in the post-quiz ($t=-4.229,p=0.00221$). This shows that the
participants have picked up additional knowledge that was not mentioned in the
pre-quiz during their interaction with AdvEx. Lastly, another paired t-test
was performed between their performance on the same checker questions in the
pre-quiz and post-quiz and no significant difference was found
($t=1.0,p=0.34344$). In conjunction with the fact that all 10 qualified
participants scored a minimum of 50% on the pre-quiz checker questions, this
suggests that our participants maintained consistency in their checker
responses and did not select answers randomly.

We repeated our statistical tests on all 12 participants, including the two
participants who were originally excluded, and our results still demonstrate a
strong learning effect (Section 5.1). This finding suggests that while AdvEx
is primarily designed for learners with a basic ML background who are new to
AML, it benefits not only the intended users but also proves effective for
those without fundamental ML knowledge seeking to understand adversarial
attacks. The ability of AdvEx to accommodate a wider audience further
emphasizes its value as an educational tool, extending its potential impact by
making complex AML concepts more approachable even to those just beginning to
explore the field of ML. The full results of all our paired t-tests and the
quiz averages of the participants are shown in Section 5.1.

###  5.3 Results and Analysis: Participants’ Feedback

To further investigate A1 and A2, we analyzed the participants’ post-
questionnaire responses on a 7-point Likert scale (Figure 6) and their
qualitative feedback from the interviews on the learning and usability of
AdvEx. The questionnaire asked users whether they: Q1) learned about
adversarial attacks through AdvEx, Q2) would recommend AdvEx for learning, Q3)
found AdvEx complemented text-based knowledge, Q4) felt engaged, Q5) felt
stressed, and Q6) enjoyed overall interaction with AdvEx.

For Q1, all participants agreed that they had learned about adversarial
attacks through interacting with AdvEx ($\mbox{MD}=6$, $\mbox{IQR}=0.5$) and
gave a positive rating ($\geq 5$) on AdvEx’s learning effect. Specifically,
the participants felt that AdvEx offered comprehensive visualizations and
found the explanations very easy to understand. P3 stated that “AdvEx teaches
all aspects of adversarial attacks very thoroughly,” and P8 commented that
“The clear explanations made the learning process much easier.” The
participants also thought that AdvEx’s visualizations were highly informative
for them to understand the key attack properties and the underlying attack
process. “The visualizations not only showed me that there are malicious
inputs indistinguishable to my eyes, but also helped me understand the
underlying attack logic.” -P10

Similarly, for Q2, all participants stated that they would recommend AdvEx to
others for learning AML ($\mbox{MD}=6$, $\mbox{IQR}=0$). They believed that
AdvEx would be highly beneficial for beginners and can serve as an effective
entry point to those interested in learning about adversarial attacks. P2
thought that “AdvEx is a valuable educational tool for illustrating the
attacks,” and P5 believed that “AdvEx is great for beginners, it can teach
them a lot about the attack process.” To further strengthen AdvEx as a
learning tool, P5 suggested visualizing the internal attack process in more
detail. “For learners to gain a more in-depth understanding of the attack
process, maybe also visualize how the adversarial inputs modify the underlying
gradient information.” -P5

Q3’s ratings show that AdvEx complemented the provided text-based knowledge in
the General Information Provider well for learning ($\mbox{MD}=6$,
$\mbox{IQR}=0.5$). From this, some participants believed that AdvEx could be
used in conjunction with text-based documents, such as textbooks, to “improve
learning experience in traditional classroom settings” -P1. Other participants
felt that AdvEx was sufficient on its own for explaining adversarial attacks.
“I don’t think AdvEx needs any additional complementary materials. Its
visualizations are enough to thoroughly explain the attack logic.” -P4

Eleven out of 12 participants gave a rating $\geq$ 5 on AdvEx’s engagement in
Q4 ($\mbox{MD}=6$, $\mbox{IQR}=0.5$). They applauded AdvEx for its highly
interactive interfaces and enjoyed dynamically experimenting with the
perturbation size to see all the real-time changes. “It is highly engaging to
change the noise level and observe how the resulting image differs.” -P1
Similarly, P5 stated, “It is fun to see all the points move around in the Data
Projectors as I adjust the slider.” However, one participant, P12, rated
AdvEx’s engagement a 4 and explained, “In general, the application is good.
But as a programmer, I feel like I should be able to get more involved and
write custom code directly.”

For Q5, all participants agreed that it was not stressful to interact with
AdvEx ($\mbox{MD}=1.5$, $\mbox{IQR}=1$). This was likely because AdvEx had an
interactive tutorial system that provided guidance on AdvEx’s functionalities,
along with the General Information Provider that offered further assistance.
Moreover, everything AdvEx visualized (e.g., 2-D latent space, confidence
scores) were familiar to learners who knew ML, thus making AdvEx intuitive to
learn with. “Using AdvEx is very simple, I didn’t encounter any difficulties.
The visualizations are all quite straightforward and intuitive.” -P7

In general, the participants rated AdvEx’s enjoyment positively in Q6
($\mbox{MD}=6$, $\mbox{IQR}=0.5$). They offered different reasons for why they
enjoyed AdvEx. P9 and P10 claimed that AdvEx’s visually appealing interfaces
and animations made their interactions entertaining. P3, P8 & P10 emphasized
the amount of knowledge they gained from AdvEx and found the learning
experience fruitful. P4, P6 & P11 applauded AdvEx for its high level of
interactivity. “I enjoy AdvEx because I can do a lot with it. I can
investigate different examples, try out different noise levels, and observe
how the embedding distribution changes.” -P4

##  6 Interview Study with Experienced Experts/Teachers

To collect more in-depth qualitative feedback on AdvEx, we conducted an
interview study with AML experts/teachers, who possess profound knowledge of
the key aspects and requirements for understanding adversarial attacks. These
interviews provided additional insights into how AdvEx can be utilized in an
educational setting.

###  6.1 Study Setup

In this study, AML experts/teachers were prompted to use AdvEx to explore one
white-box attack and one black-box attack, FGSM and ZOO, on four different
models (VGG-16, VGG-19, standard ResNet-34, & adversarially trained ResNet-34)
with the CIFAR-10 data in a free-form analysis session. We recruited seven AML
experts (E1, E2, E4 $\sim$ E8; all men), six of whom have teaching experience
that spans from leading AML seminars to teaching ML courses with AML
components. Each study session began with a 5-minute introduction to the study
background and AdvEx’s key features. Next, we presented a task scenario where
participants were asked to use AdvEx to explore and understand “how the FGSM
and ZOO attacks alter the input images to affect the models’ performance,” and
“how models display varying robustness against the attacks.” Participants had
30 minutes to explore each attack, and a task list was provided to guide their
interaction. They were also informed that they could explore the tool freely
without following these tasks as long as insights were gathered. We employed
the think-aloud protocol, requiring participants to provide feedback from both
the perspectives of experts/teachers and learners. An experimenter was
responsible for providing help and answering questions regarding the
interface, who also observed the experts’ interactions and took notes. After
the interaction, a semi-structured interview ($\approx$30 minutes) was
conducted to gain a better understanding of the participants’ thoughts on
AdvEx in light of the think-aloud feedback and observation gathered
previously. The participants were compensated $\$$20/hour for the study.

###  6.2 Results and Analysis

All seven experts successfully used AdvEx to gain insights into the attacks
and expressed a positive sentiment toward it. We conducted a thematic analysis
on the unstructured feedback gathered during the free-form analysis and the
qualitative data provided to us during the semi-structured interviews. We came
up with five systematic themes aligned with our design goals and an additional
theme focused on usability, and adopted a deductive approach to identify
patterns of them in our data.

Visualizations of attack impacts. All experts agreed that AdvEx can help
learners quickly grasp the overall attack impacts. E4 and E6 liked the
Robustness Analyzers for illustrating “the overall trend of accuracy changes
across different attacks.” E1, E6, and E8 appreciated the Data Projectors for
allowing learners to “easily identify misclassified instances” and “see how
embeddings are drifted from their original positions.” E8 explained, “From the
projectors, learners can see how ZOO consistently pushes instances towards the
nearest class, while FGSM scatters instances more randomly across various
classes.” The experts also found the subpopulation- and instance-level
visualizations highly useful. E7 mentioned, “The confidence score view can
show learners that ZOO adjusts the targeted class’s confidence just above the
original, revealing that the attack pushes instances just past the decision
boundary.” E1 and E8 also commented that the selection mode provided easy
tracking of point trajectories from a specific class. The above observations
confirm that AdvEx effectively visualizes the attack impact at multiple levels
(G1). A noted limitation is the occasional difficulty in distinguishing
between instances misclassified before the attack and those misclassified due
to the attack. E1 recommended incorporating the image’s original prediction
alongside its label and current prediction into the attack explainer.

Generalizability. The experts praised AdvEx for its ability to generalize to
different attacks and image classifiers, emphasizing its role in helping
learners grasp the full attack landscape. E7 explained, “AdvEx shows that
while ResNet outperforms VGG with FGSM attacks, the advantage does not hold
for ZOO attacks, highlighting the challenge of building a universally robust
model.” E2 emphasized that “AdvEx’s ability to adapt to different attacks and
models is vital for learners to truly understand the risks by evaluating
against diverse techniques.” These comments confirmed that AdvEx’s
generalizability can effectively help learners assess different kinds of
models and grasp the variability of attack methods (G2). Furthermore, the
experts highlighted that such design simplifies the exploration by providing
learners a more accessible way to investigate attacks in their specific
applications. Both E1 and E2 pointed out how AdvEx saves learners’ time by
eliminating the need to code from scratch when exploring different attack
strategies on their own models.

Evaluation of model robustness. The experts believed that AdvEx can help
learners easily discern their models’ strengths and weaknesses. The model
comparison feature was frequently highlighted, with E7 noting that the feature
can reveal that “deeper models do not necessarily excel under attacks.”
Similarly, E5 and E6 commented that AdvEx shows that an adversarially trained
model has “embeddings that barely differ under standard or adversarial
conditions” and “perturbations that show clearer shapes resembling the
original object, revealing its reliance on human-interpretable features.”
These comments affirm AdvEx’s capability for detailed visual analysis and
model comparison (G3). Moreover, the experts appreciated how comprehensive
AdvEx’s model visualizations are. E2 stated, “AdvEx offers unique insights
into the models such as their embeddings and confidence scores, which are
often not accessible through traditional lectures or assignments.” E5 made a
similar comment and explained, “People usually only focus on accuracy, but
that is not the whole story. These other metrics displayed by AdvEx are just
as important.” E1 and E4 suggested that it may be even better for AdvEx to
support comparing the same model under attacks with different perturbation
sizes side by side.

Dynamic experimentation with real-time changes. The experts enjoyed
dynamically experimenting with the perturbation size and found AdvEx’s real-
time visual feedback particularly valuable. E6 explained, “With ZOO, learners
can observe that most misclassifications occur as the perturbation size
increases from 0 to 0.1, showing that the attack identifies the minimal
required perturbation for misclassification.” E2 commented, “AdvEx answers
questions that papers and tutorials may not cover, such as the effects of
varying perturbation sizes on model embeddings.” Furthermore, they believed
that the integration of dynamic perturbation adjustment and real-time visual
feedback offers an engaging learning experience. E2 stated, “Learners would
enjoy experimenting with different perturbation sizes, as it allows them to
interactively compare the impacts of attacks with different strengths.”
Similarly, E1 claimed that “Teachers can play with the perturbation size to
demonstrate attacks to students in a fun, interactive, and engaging way.” The
above observations suggest that AdvEx provides a highly interactive learning
experience with its perturbation experimentation and real-time feedback (G4).

Overall benefits as an educational tool for learners. All experts agreed that
AdvEx is highly beneficial as an educational tool to understand adversarial
attacks. “AdvEx can help learners quickly grasp the logic and impacts of
adversarial attacks, as it demonstrates key knowledge such as the attack
process, the input and output, and the accuracy change.” -E1 E6 stated, “AdvEx
bridges theory and practice, enhancing learners’ understanding of adversarial
attacks and encouraging them to further explore the field.” They also thought
the step-by-step execution would be very clear and informative for AML
learners, confirming AdvEx’s capability to enable detailed learning of the
attack process (G5). “The step-by-step explanations clarify FGSM and ZOO
basics for those without AML backgrounds, aiding their understanding of the
rest of AdvEx’s components.” -E8 In addition, the experts believed that
AdvEx’s interfaces would make the learning experience highly enjoyable. E1
commented, “AdvEx’s game-like experience makes learning and evaluating models
much easier for learners without too many tedious formulas.”

Usability & beginner-friendly design. All experts thought that AdvEx was very
intuitive to pick up. E5 liked how the beginning tutorial highlighted specific
areas of the interface, which helped him easily understand the purpose and
functionalities of each interface component. E1 thought learners less
experienced with AML could also pick up AdvEx easily. He commented, “AdvEx is
very beginner-friendly to AML learners with a ML background as everything
visualized are things people with ML knowledge already familiar with.” These
comments show that AdvEx was successfully integrated with a beginner-friendly
design. They also thought that AdvEx was very accessible, as E1 stated that
“AdvEx is very complete and I don’t need to make any adjustments or write any
code.” E5 highlighted the zoomable binned aggregation feature and commented,
“This feature effectively accommodates different users’ available
computational power and enable smooth exploration of large-scale data for
everyone.”

##  7 Discussion

Here, we discuss the limitations of our current system and outline future
directions to enhance our work. We also present the potential avenues for
extending and generalizing our proposed design.

###  7.1 Limitations and Future Work

While our study confirms that AdvEx is highly effective in helping learners
understand adversarial attacks, it still has several limitations. Firstly, as
commented by our participants, the current Data Projectors (Figure 1c) allow
comparisons of two different models under the same perturbation level, but do
not support comparing the same models side by side at different perturbation
levels. Future extensions should enable this type of comparison without
adjusting the slider back and forth. A simple solution is to add additional
toggles to the projectors for switching between the different comparison
modes.

Secondly, when the perturbation size exceeds 0, distinguishing instances
misclassified before the attack from those misclassified due to the attack
becomes less intuitive. This can be easily mitigated by implementing
additional visual encodings such as using different shapes (triangles and
crosses) to represent the two types of misclassifications. However, this
approach may increase the cognitive load of users. But an optional filtering
feature can be added to allow users to focus only on either type of
misclassification.

Finally, the evaluation of AdvEx can be further enhanced. A larger sample size
should be obtained to better evaluate AdvEx’s effectiveness. Also, the current
study was designed with a few selected models, and only the FGSM and ZOO
attacks with the CIFAR-10 data were used to assess the learning effect and
usability of AdvEx. In the future, deployment studies with other types of
attacks and datasets should also be conducted to investigate how AdvEx can be
used in various real-world domains. This will thoroughly examine the strengths
and weaknesses of AdvEx, and help us understand how AdvEx can be potentially
incorporated into learners’ model development workflows.

###  7.2 Generalization and Extension

We designed AdvEx as a system for visualizing adversarial attacks, but the
tool is flexible enough to be adapted to visualize other data augmentations.
For example, AdvEx can be extended to visualize noise applications (e.g.,
Gaussian noise, salt-and-pepper noise) or other image degradations (e.g.,
motion blur, JPEG compression). Learners may use AdvEx to understand how
visual quality impacts the performance of models in those scenarios. Moreover,
AdvEx’s Data Projectors provide an intuitive way to evaluate the accuracy and
embeddings of classification models. Though this study focuses on image
classifications, such design can be extended to assess other classifiers
(e.g., audio and text classification).

Furthermore, AdvEx leverages a balanced combination of active visualizations
and passive text-based information to help learners understand AML, and this
design can be applied to visualization tools for learning other ML concepts.
In fact, many existing tools (e.g., [16, 17]) only focus on their interactive
visualizations and place little emphasis on text-based information, not
providing enough guidance and background knowledge to the users. On the other
hand, interactive articles [50] usually involve mainly text and provide
insufficient visualizations. AdvEx places more balanced weights on both
components, ensuring that the users may gain detailed and accurate AML
knowledge from our General Information Provider (Figure 1e) in addition to
exploration with the visualizations. Our design not only reinforces learning
by presenting content in multiple formats, but also allows the learners to
quickly grasp complex topics that require visual interpretations, which could
shed light on future research on the spectrum of modalities for teaching
abstract ML concepts.

##  8 Conclusion

We have presented AdvEx, an interactive visualization tool for novice AML
learners to explore and understand adversarial attacks. Based on the design
guidelines derived from user interviews, we designed AdvEx to provide learners
with detailed attack visualizations at multiple levels, highlighting attack’s
properties and effects on different image classifiers. Our design addresses
the limitations of existing tools, which lack comprehensiveness and
generalizability when visualizing the attacks. We quantitatively and
qualitatively assessed AdvEx in a two-part evaluation, including a user study
with 12 AML learners and an interview study with seven AML experts/teachers.
Our results indicate that AdvEx is not only highly effective as an educational
tool, but also provides an engaging and enjoyable learning experience, thus
highlighting its overall benefits for AML learners. Additionally, we discuss
the future directions to enhance our work and present potential avenues to
extend and generalize AdvEx to other applications.

## Acknowledgments

This work is supported in part by the Natural Sciences and Engineering
Research Council of Canada via the Discovery Grant and the University of
Waterloo by the Interdisciplinary Trailblazer Fund.

## References

  * [1] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [2] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [3] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song, “Robust physical-world attacks on deep learning visual classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1625–1634.
  * [4] S. Thys, W. Van Ranst, and T. Goedemé, “Fooling automated surveillance cameras: adversarial patches to attack person detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2019, pp. 0–0.
  * [5] L. Sun, M. Tan, and Z. Zhou, “A survey of practical adversarial example attacks,” _Cybersecurity_ , vol. 1, pp. 1–9, 2018.
  * [6] Z. J. Wang, R. Turko, O. Shaikh, H. Park, N. Das, F. Hohman, M. Kahng, and D. H. P. Chau, “Cnn explainer: Learning convolutional neural networks with interactive visualization,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 27, no. 2, pp. 1396–1406, 2020.
  * [7] M. Kahng and D. H. Chau, “How does visualization help people learn deep learning? evaluation of gan lab,” in _IEEE VIS 2019 Workshop on EValuation of Interactive VisuAl Machine Learning Systems_ , 2019.
  * [8] F. Hohman, H. Park, C. Robinson, and D. H. P. Chau, “Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations,” _IEEE transactions on visualization and computer graphics_ , vol. 26, no. 1, pp. 1096–1106, 2019.
  * [9] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, “Adversarial examples are not bugs, they are features,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [10] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, “Theoretically principled trade-off between robustness and accuracy,” in _International conference on machine learning_. PMLR, 2019, pp. 7472–7482.
  * [11] G. Guo and N. Zhang, “A survey on deep learning based face recognition,” _Computer vision and image understanding_ , vol. 189, p. 102805, 2019.
  * [12] S. Kuutti, R. Bowden, Y. Jin, P. Barber, and S. Fallah, “A survey of deep learning applications to autonomous vehicle control,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 2, pp. 712–733, 2020.
  * [13] P. Benz, C. Zhang, A. Karjauv, and I. S. Kweon, “Robustness may be at odds with fairness: An empirical study on class-wise accuracy,” in _NeurIPS 2020 Workshop on Pre-registration in Machine Learning_. PMLR, 2021, pp. 325–342.
  * [14] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” _arXiv preprint arXiv:1706.06083_ , 2017.
  * [15] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in _Proceedings of the 10th ACM workshop on artificial intelligence and security_ , 2017, pp. 15–26.
  * [16] A. P. Norton and Y. Qi, “Adversarial-playground: A visualization suite showing how adversarial examples fool deep learning,” in _2017 IEEE symposium on visualization for cyber security (VizSec)_. IEEE, 2017, pp. 1–4.
  * [17] N. Das, H. Park, Z. J. Wang, F. Hohman, R. Firstman, E. Rogers, and D. H. P. Chau, “Bluff: Interactively deciphering adversarial attacks on deep neural networks,” in _2020 IEEE Visualization Conference (VIS)_. IEEE, 2020, pp. 271–275.
  * [18] K. Cao, M. Liu, H. Su, J. Wu, J. Zhu, and S. Liu, “Analyzing the noise robustness of deep neural networks,” _IEEE transactions on visualization and computer graphics_ , vol. 27, no. 7, pp. 3289–3304, 2020.
  * [19] Y. Ma, T. Xie, J. Li, and R. Maciejewski, “Explaining vulnerabilities to adversarial machine learning through visual analytics,” _IEEE transactions on visualization and computer graphics_ , vol. 26, no. 1, pp. 1075–1085, 2019.
  * [20] A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” in _Artificial intelligence safety and security_. Chapman and Hall/CRC, 2018, pp. 99–112.
  * [21] J. Chen, M. I. Jordan, and M. J. Wainwright, “Hopskipjumpattack: A query-efficient decision-based attack,” in _2020 ieee symposium on security and privacy (sp)_. IEEE, 2020, pp. 1277–1294.
  * [22] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in _Proceedings of the 2017 ACM on Asia conference on computer and communications security_ , 2017, pp. 506–519.
  * [23] C. Qin, J. Martens, S. Gowal, D. Krishnan, K. Dvijotham, A. Fawzi, S. De, R. Stanforth, and P. Kohli, “Adversarial robustness through local linearization,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [24] A. Raghunathan, S. M. Xie, F. Yang, J. C. Duchi, and P. Liang, “Understanding and mitigating the tradeoff between robustness and accuracy,” in _Proceedings of the 37th International Conference on Machine Learning_ , 2020, pp. 7909–7919.
  * [25] Y. Zhang, G. Liang, T. Salem, and N. Jacobs, “Defense-pointnet: Protecting pointnet against adversarial attacks,” in _2019 IEEE International Conference on Big Data (Big Data)_. IEEE, 2019, pp. 5654–5660.
  * [26] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * [27] Y. Ye, Y. Chen, and M. Liu, “Multiuser adversarial attack on deep learning for ofdm detection,” _IEEE Wireless Communications Letters_ , vol. 11, no. 12, pp. 2527–2531, 2022.
  * [28] H. Ye, G. Y. Li, and B.-H. Juang, “Power of deep learning for channel estimation and signal detection in ofdm systems,” _IEEE Wireless Communications Letters_ , vol. 7, no. 1, pp. 114–117, 2017.
  * [29] Z. He, A. S. Rakin, and D. Fan, “Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 588–597.
  * [30] V. Zantedeschi, M.-I. Nicolae, and A. Rawat, “Efficient defenses against adversarial attacks,” in _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 2017, pp. 39–49.
  * [31] D. Su, H. Zhang, H. Chen, J. Yi, P.-Y. Chen, and Y. Gao, “Is robustness the cost of accuracy?–a comprehensive study on the robustness of 18 deep image classification models,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 631–648.
  * [32] J. G. Zago, E. A. Antonelo, F. L. Baldissera, and R. T. Saad, “It is double pleasure to deceive the deceiver: disturbing classifiers against adversarial attacks,” in _2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , 2020, pp. 160–165.
  * [33] J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” _IEEE Transactions on Evolutionary Computation_ , vol. 23, no. 5, pp. 828–841, 2019.
  * [34] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2574–2582.
  * [35] D. Lin, Y.-G. Wang, W. Tang, and X. Kang, “Boosting query efficiency of meta attack with dynamic fine-tuning,” _IEEE Signal Processing Letters_ , vol. 29, pp. 2557–2561, 2022.
  * [36] L. Deng, “The mnist database of handwritten digit images for machine learning research,” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 141–142, 2012.
  * [37] C. Park, S. Yang, I. Na, S. Chung, S. Shin, B. C. Kwon, D. Park, and J. Choo, “Vatun: Visual analytics for testing and understanding convolutional neural networks,” in _Eurographics Conference on Visualization (EuroVis)_. The Eurographics Association, 2021.
  * [38] P. Panda and K. Roy, “Implicit adversarial data augmentation and robustness with noise-based learning,” _Neural Networks_ , vol. 141, pp. 120–132, 2021.
  * [39] D. Hendrycks and K. Gimpel, “Early methods for detecting adversarial images,” _arXiv preprint arXiv:1608.00530_ , 2016.
  * [40] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [41] K. Pearson, “Liii. on lines and planes of closest fit to systems of points in space,” _The London, Edinburgh, and Dublin philosophical magazine and journal of science_ , vol. 2, no. 11, pp. 559–572, 1901.
  * [42] A. Krizhevsky, “Learning multiple layers of features from tiny images,” _Master’s thesis, University of Tronto_ , 2009.
  * [43] T. Munzner, _Visualization analysis and design_. CRC press, 2014.
  * [44] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in _International conference on machine learning_. PMLR, 2020, pp. 2206–2216.
  * [45] D. Hendrycks, M. Mazeika, S. Kadavath, and D. Song, “Using self-supervised learning can improve model robustness and uncertainty,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [46] P. Rathore, A. Basak, S. H. Nistala, and V. Runkana, “Untargeted, targeted and universal adversarial attacks and defenses on time series,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–8.
  * [47] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein, “Robustbench: a standardized adversarial robustness benchmark,” _arXiv preprint arXiv:2010.09670_ , 2020.
  * [48] H. Phan, “huyvnphan/pytorch_cifar10,” Jan 2021.
  * [49] L. Lins, J. T. Klosowski, and C. Scheidegger, “Nanocubes for real-time exploration of spatiotemporal datasets,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 19, no. 12, pp. 2456–2465, 2013.
  * [50] F. Hohman, M. Conlen, J. Heer, and D. H. P. Chau, “Communicating with interactive articles,” _Distill_ , vol. 5, no. 9, p. e28, 2020.