# Zero-shot Model Diagnosis

Jinqi Luo111Equal contribution. Zhaoning Wang111Equal contribution. Chen Henry
Wu Dong Huang Fernando De la Torre
Carnegie Mellon University
{jinqil, zhaoning, chenwu2, dghuang<EMAIL_ADDRESS>
###### Abstract

When it comes to deploying deep vision models, the behavior of these systems
must be explicable to ensure confidence in their reliability and fairness. A
common approach to evaluate deep learning models is to build a labeled test
set with attributes of interest and assess how well it performs. However,
creating a balanced test set (i.e., one that is uniformly sampled over all the
important traits) is often time-consuming, expensive, and prone to mistakes.
The question we try to address is: can we evaluate the sensitivity of deep
learning models to arbitrary visual attributes without an annotated test set?

This paper argues the case that Zero-shot Model Diagnosis (ZOOM) is possible
without the need for a test set nor labeling. To avoid the need for test sets,
our system relies on a generative model and CLIP. The key idea is enabling the
user to select a set of prompts (relevant to the problem) and our system will
automatically search for semantic counterfactual images (i.e., synthesized
images that flip the prediction in the case of a binary classifier) using the
generative model. We evaluate several visual tasks (classification, key-point
detection, and segmentation) in multiple visual domains to demonstrate the
viability of our methodology. Extensive experiments demonstrate that our
method is capable of producing counterfactual images and offering sensitivity
analysis for model diagnosis without the need for a test set.

22footnotetext: The code is publicly available at the project page:
https://zero-shot-model-diagnosis.github.io/.  Figure 1: Given a
differentiable deep learning model (e.g., a cat/dog classifier) and user-
defined text attributes, how can we determine the model’s sensitivity to
specific attributes without using labeled test data? Our system generates
counterfactual images (bottom right) based on the textual directions provided
by the user, while also computing the sensitivity histogram (top right).

##  1 Introduction

Deep learning models inherit data biases, which can be accentuated or
downplayed depending on the model’s architecture and optimization strategy.
Deploying a computer vision deep learning model requires extensive testing and
evaluation, with a particular focus on features with potentially dire social
consequences (e.g., non-uniform behavior across gender or ethnicity). Given
the importance of the problem, it is common to collect and label large-scale
datasets to evaluate the behavior of these models across attributes of
interest. Unfortunately, collecting these test datasets is extremely time-
consuming, error-prone, and expensive. Moreover, a balanced dataset, that is
uniformly distributed across all attributes of interest, is also typically
impractical to acquire due to its combinatorial nature. Even with careful
metric analysis in this test set, no robustness nor fairness can be guaranteed
since there can be a mismatch between the real and test distributions [25].
This research will explore model diagnosis without relying on a test set in an
effort to democratize model diagnosis and lower the associated cost.

Counterfactual explainability as a means of model diagnosis is drawing the
community’s attention [20, 5]. Counterfactual images visualize the sensitive
factors of an input image that can influence the model’s outputs. In other
words, counterfactuals answer the question: “How can we modify the input image
$\mathbf{x}$ (while fixing the ground truth) so that the model prediction
would diverge from $\mathbf{y}$ to $\hat{\mathbf{y}}$?”. The parameterization
of such counterfactuals will provide insights into identifying key factors of
where the model fails. Unlike existing image-space adversary techniques [4,
18], counterfactuals provide semantic perturbations that are interpretable by
humans. However, existing counterfactual studies require the user to either
collect uniform test sets [10], annotate discovered bias [15], or train a
model-specific explanation every time the user wants to diagnose a new model
[13].

On the other hand, recent advances in Contrastive Language-Image Pretraining
(CLIP) [24] can help to overcome the above challenges. CLIP enables text-
driven applications that map user text representations to visual manifolds for
downstream tasks such as avatar generation [7], motion generation [37] or
neural rendering [22, 30]. In the domain of image synthesis, StyleCLIP [21]
reveals that text-conditioned optimization in the StyleGAN [12] latent space
can decompose latent directions for image editing, allowing for the mutation
of a specific attribute without disturbing others. With such capability, users
can freely edit semantic attributes conditioned on text inputs. This paper
further explores its use in the scope of model diagnosis.

The central concept of the paper is depicted in Fig. 1. Consider a user
interested in evaluating which factors contribute to the lack of robustness in
a cat/dog classifier (target model). By selecting a list of keyword
attributes, the user is able to (1) see counterfactual images where semantic
variations flip the target model predictions (see the classifier score in the
top-right corner of the counterfactual images) and (2) quantify the
sensitivity of each attribute for the target model (see sensitivity histogram
on the top). Instead of using a test set, we propose using a StyleGAN
generator as the picture engine for sampling counterfactual images. CLIP
transforms user’s text input, and enables model diagnosis in an open-
vocabulary setting. This is a major advantage since there is no need for
collecting and annotating images and minimal user expert knowledge. In
addition, we are not tied to a particular annotation from datasets (e.g.,
specific attributes in CelebA [16]).

To summarize, our proposed work offers three major improvements over earlier
efforts:

  * •

The user requires neither a labeled, balanced test dataset, and minimal expert
knowledge in order to evaluate where a model fails (i.e., model diagnosis). In
addition, the method provides a sensitivity histogram across the attributes of
interest.

  * •

When a different target model or a new user-defined attribute space is
introduced, it is not necessary to re-train our system, allowing for practical
use.

  * •

The target model fine-tuned with counterfactual images not only slightly
improves the classification performance, but also greatly increases the
distributional robustness against counterfactual images.

Figure 2: The ZOOM framework. Black solid lines stand for forward passes, red
dashed lines stand for backpropagation, and purple dashed lines stand for
inference after the optimization converges. The user inputs single or multiple
attributes, and we map them into edit directions with the method in Sec. 3.2.
Then we assign to each edit direction (attribute) a weight, which represents
how much we are adding/removing this attribute. We iteratively perform
adversarial learning on the attribute space to maximize the counterfactual
effectiveness.

##  2 Related Work

This section reviews prior work on attribute editing with generative models
and recent efforts on model diagnosis.

###  2.1 Attribute Editing with Generative Models

With recent progress in generative models, GANs supports high-quality image
synthesis, as well as semantic attributes editing [35]. [6, 1] edit the images
by perturbing the intermediate latent space encoded from the original images.
These methods rely on images to be encoded to latent vectors to perform
attribute editing. On the contrary, StyleGAN [12] can produce images by
sampling the latent space. Many works have explored ways to edit attributes in
the latent space of StyleGAN, either by relying on image annotations [27] or
in an unsupervised manner [28, 8]. StyleSpace [34] further disentangles the
latent space of StyleGAN and can perform specific attribute edits by
disentangled style vectors. Based upon StyleSpace, StyleCLIP [21] builds the
connection between the CLIP language space and StyleGAN latent space to enable
arbitrary edits specified by the text. Our work adopts this concept for fine-
grained attribute editing.

###  2.2 Model Diagnosis

To the best of our knowledge, model diagnosis without a test set is a
relatively unexplored problem. In the adversarial learning literature, it is
common to find methods that show how image-space perturbations [4, 18] flip
the model prediction; however, such perturbations lack visual
interpretability. [36] pioneers in synthesizing adversaries by GANs. More
recently, [9, 26, 23] propose generative methods to synthesize semantically
perturbed images to visualize where the target model fails. However, their
attribute editing is limited within the dataset’s annotated labels. Instead,
our framework allows users to easily customize their own attribute space, in
which we visualize and quantify the biased factors that affect the model
prediction. On the bias detection track, [13] co-trains a model-specific
StyleGAN with each target model, and requires human annotators to name
attribute coordinates in the Stylespace. [15, 3, 14] synthesize counterfactual
images by either optimally traversing the latent space or learning an
attribute hyperplane, after which the user will inspect the represented bias.
Unlike previous work, we propose to diagnose a deep learning model without any
model-specific re-training, new test sets, or manual annotations/inspections.

##  3 Method

This section firstly describes our method to generate counterfactual images
guided by CLIP in a zero-shot manner. We then introduce how we perform the
sensitivity analysis across attributes of interest. Fig. 2 shows the overview
of our framework.

###  3.1 Notation and Problem Definition

Let $f_{\theta}$, parameterized by $\theta$, be the target model that we want
to diagnose. In this paper, $f_{\theta}$ denotes two types of deep nets:
binary attribute classifiers and face keypoint detectors. Note that our
approach is extendable to any end-to-end differentiable target deep models.
Let $\mathcal{G}_{\phi}$, parameterized by $\phi$, be the style generator that
synthesizes images by $\mathbf{x}=\mathcal{G}_{\phi}(\mathbf{s})$ where
$\mathbf{s}$ is the style vector in Style Space $\mathcal{S}$ [34]. We denote
a counterfactual image as $\mathbf{\hat{x}}$, which is a synthesized image
that misleads the target model $f_{\theta}$, and denote the original reference
image as $\mathbf{x}$. $a$ is defined as a single user input text-based
attribute, with its domain $\mathcal{A}=\\{a_{i}\\}_{i=1}^{N}$ for $N$ input
attributes. $\mathbf{\hat{x}}$ and $\mathbf{x}$ differs only along attribute
directions $\mathcal{A}$. Given a set of
$\\{f_{\theta},\mathcal{G}_{\phi},\mathcal{A}\\}$, our goal is to perform
counterfactual-based diagnosis to interpret where the model fails without
manually collecting nor labeling any test set. Unlike traditional approaches
of image-space noises which lack explainability to users, our method
adversarially searches the counterfactual in the user-designed semantic space.
To this end, our diagnosis will have three outputs, namely counterfactual
images (Sec. 3.3), sensitivity histograms (Sec. 3.4), and distributionally
robust models (Sec. 3.5).

###  3.2 Extracting Edit Directions

This section examines the terminologies, method, and modification we adopt in
ZOOM to extract suitable global directions for attribute editing. Since CLIP
has shown strong capability in disentangling visual representation [19], we
incorporate style channel relevance from StyleCLIP [21] to find edit
directions for each attribute.

Given the user’s input strings of attributes, we want to find an image
manipulation direction $\Delta\mathbf{\mathbf{s}}$ for any
$\mathbf{s}\sim\mathcal{S}$, such that the generated image
$\mathcal{G}_{\phi}(\mathbf{s}+\Delta{\mathbf{s})}$ only varies in the input
attributes. Recall that CLIP maps strings into a text embedding
$\mathbf{t}\in\mathcal{T}$, the text embedding space. For a string attribute
description $a$ and a neutral prefix $p$, we obtain the CLIP text embedding
difference $\Delta\mathbf{t}$ by:

$\displaystyle\Delta\mathbf{t}=\operatorname{CLIP_{text}}(p\oplus
a)-\operatorname{CLIP_{text}}(p)$ (1)

where $\oplus$ is the string concatenation operator. To take ‘Eyeglasses’ as
an example, we can get
$\Delta\mathbf{t}=\operatorname{CLIP_{text}}(\textnormal{`a face with
Eyeglasses'})-\operatorname{CLIP_{text}}(\textnormal{`a face'})$.

To get the edit direction, $\Delta\mathbf{s}$, we need to utilize a style
relevance mapper $\mathbf{M}\in\mathbbm{R}^{c_{\mathcal{S}}\times
c_{\mathcal{T}}}$ to map between the CLIP text embedding vectors of length
$c_{\mathcal{T}}$ and the Style space vector of length $c_{\mathcal{S}}$.
StyleCLIP optimizes $\mathbf{M}$ by iteratively searching meaningful style
channels: mutating each channel in $\mathcal{S}$ and encoding the mutated
images by CLIP to assess whether there is a significant change in
$\mathcal{T}$ space. To prevent undesired edits that are irrelevant to the
user prompt, the edit direction $\Delta\mathbf{s}$ will filter out channels
that the style value change is insignificant:

$\displaystyle\Delta\mathbf{s}$
$\displaystyle=(\mathbf{M}\cdot\Delta\mathbf{t})\odot\mathbbm{1}((\mathbf{M}\cdot\Delta\mathbf{t})>\lambda),$
(2)

where $\lambda$ is the hyper-parameter for the threshold value.
$\mathbbm{1}(\cdot)$ is the indicator function, and $\odot$ is the element-
wise product operator. Since the success of attribute editing by the extracted
edit directions will be the key to our approach, Appendix A will show the
capability of CLIP by visualizing the global edit direction on multiple
sampled images, conducting the user study, and analyzing the effect of
$\lambda$.

###  3.3 Style Counterfactual Synthesis

Identifying semantic counterfactuals necessitates a manageable parametrization
of the semantic space for effective exploration. For ease of notation, we
denote $(\Delta\mathbf{s})_{i}$ as the global edit direction for $i^{th}$
attribute $a_{i}\in\mathcal{A}$ from the user input. After these $N$
attributes are provided and the edit directions are calculated, we initialize
the control vectors $\mathbf{w}$ of length $N$ where the $i^{th}$ element
$w_{i}$ controls the strength of the $i^{th}$ edit direction. Our
counterfactual edit will be a linear combination of normalized edit
directions:
$\mathbf{s}_{edit}=\sum_{i=1}^{N}w_{i}\frac{(\Delta\mathbf{s})_{i}}{||(\Delta\mathbf{s})_{i}||}$.

The black arrows in Fig. 2 show the forward inference to synthesize
counterfactual images. Given the parametrization of attribute editing
strengths and the final loss value, our framework searches for counterfactual
examples in the optimizable edit weight space. The original sampled image is
$\mathbf{x}=G_{\phi}(\mathbf{s})$, and the counterfactual image is

$\displaystyle\mathbf{\hat{x}}=G_{\phi}(\mathbf{s}+\mathbf{s}_{edit})=G_{\phi}\left(\mathbf{s}+\sum_{i=1}^{N}w_{i}\frac{(\Delta\mathbf{s})_{i}}{||(\Delta\mathbf{s})_{i}||}\right),$
(3)

which is obtained by minimizing the following loss, $\mathcal{L}$, that is the
weighted sum of three terms:

$\displaystyle\mathcal{L}(\mathbf{s},\mathbf{w})$
$\displaystyle=\alpha\mathcal{L}_{target}(\mathbf{\hat{x}})+\beta\mathcal{L}_{struct}(\mathbf{\hat{x}})+\gamma\mathcal{L}_{attr}(\mathbf{\hat{x}}).$
(4)

We back-propagate to optimize $\mathcal{L}$ w.r.t the weights of the edit
directions $\mathbf{w}$, shown as the red pipeline in Fig. 2.

The targeted adversarial loss $\mathcal{L}_{target}$ for binary attribute
classifiers minimizes the distance between the current model prediction
$f_{\theta}(\mathbf{\hat{x}})$ with the flip of original prediction
$\hat{p}_{cls}=1-f_{\theta}(\mathbf{x})$. In the case of an eyeglass
classifier on a person wearing eyeglasses, $\mathcal{L}_{target}$ will guide
the optimization to search $\mathbf{w}$ such that the model predicts no
eyeglasses. For a keypoint detector, the adversarial loss will minimize the
distance between the model keypoint prediction with a set of random points
$\hat{p}_{kp}\sim\mathcal{N}$:

$\displaystyle\text{(binary classifier)
}\mathcal{L}_{target}(\mathbf{\hat{x}})={L}_{CE}(f_{\theta}(\mathbf{\hat{x}}),\hat{p}_{cls}),$
(5) $\displaystyle\text{(keypoint detector)
}\mathcal{L}_{target}(\mathbf{\hat{x}})={L}_{MSE}(f_{\theta}(\mathbf{\hat{x}}),\hat{p}_{kp}).$
(6)

If we only optimize $\mathcal{L}_{target}$ w.r.t the global edit directions,
it is possible that the method will not preserve image statistics of the
original image and can include the particular attribute that we are
diagnosing. To constrain the optimization, we added a structural loss
$\mathcal{L}_{struct}$ and an attribute consistency loss $\mathcal{L}_{attr}$
to avoid generation collapse. $\mathcal{L}_{struct}$ [32] aims to preserve
global image statistics of the original image $\mathbf{x}$ including image
contrasts, background, or shape identity during the adversarial editing. While
$\mathcal{L}_{attr}$ enforces that the target attribute (perceived ground
truth) be consistent on the style edits. For example, when diagnosing the
eyeglasses classifier, ZOOM preserves the original status of eyeglasses and
precludes direct eyeglasses addition/removal.

$\displaystyle\mathcal{L}_{struct}(\mathbf{\hat{x}})$
$\displaystyle=L_{SSIM}(\mathbf{\hat{x}},\mathbf{x})$ (7)
$\displaystyle\mathcal{L}_{attr}(\mathbf{\hat{x}})$
$\displaystyle={L}_{CE}(\operatorname{CLIP}(\mathbf{\hat{x}}),\operatorname{CLIP}(\mathbf{x}))$
(8)

Given a pretrained target model $f_{\theta}$, a domain-specific style
generator $G_{\phi}$, and a text-driven attribute space $\mathcal{A}$, our
goal is to sample an original style vector $\mathbf{s}$ for each image and
search its counterfactual edit strength $\mathbf{\hat{w}}$:

$\displaystyle\mathbf{\hat{w}}$
$\displaystyle=\operatorname*{argmin}_{\mathbf{w}}\mathcal{L}(\mathbf{s},\mathbf{w}).$
(9)

Unless otherwise stated, we iteratively update $\mathbf{w}$ as:

$\displaystyle\mathbf{w}$
$\displaystyle=\mbox{clamp}_{\mathrm{[-\epsilon,\epsilon]}}(\mathbf{w}-\eta\nabla_{\mathbf{w}}\mathcal{L}),$
(10)

where $\eta$ is the step size and $\epsilon$ is the clamp bound to avoid
synthesis collapse caused by exaggerated edit. Note that the maximum
counterfactual effectiveness does not indicate the maximum edit strength
(i.e., $w_{i}=\epsilon$), since the attribute edit direction does not
necessarily overlap with the target classifier direction. The attribute change
is bi-directional, as the $w_{i}$ can be negative in Eq. 3. Details of using
other optimization approaches (e.g., linear approximation [18]) will be
discussed in Appendix C.

###  3.4 Attribute Sensitivity Analysis

Single-attribute counterfactual reflects the sensitivity of target model on
the individual attribute. By optimizing independently along the edit direction
for a single attribute and averaging the model probability changes over
images, our model generates independent sensitivity score $h_{i}$ for each
attribute $a_{i}$:

$\displaystyle
h_{i}=\mathbb{E}_{\mathbf{x}\sim\mathcal{P}(\mathbf{x}),\mathbf{\hat{x}}=\text{ZOOM}(\mathbf{x},a_{i})}|f_{\theta}(\mathbf{x})-f_{\theta}(\mathbf{\hat{x}})|.$
(11)

The sensitivity score $h_{i}$ is the probability difference between the
original image $\mathbf{x}$ and generated image $\mathbf{\hat{x}}$, at the
most counterfactual point when changing attribute $a_{i}$.

We synthesize a number of images from $\mathcal{G}_{\phi}$, then iteratively
compute the sensitivity for each given attribute, and finally normalize all
sensitivities to draw the histogram as shown in Fig. 4. The histogram
indicates the sensitivity of the evaluated model $f_{\theta}$ on each of the
user-defined attributes. Higher sensitivity of one attribute means that the
model is more easily affected by that attribute.

Figure 3: Effect of progressively generating counterfactual images on (left)
cat/dog classifier (0-Cat / 1-Dog), and (right) perceived age classifier
(0-Senior / 1-Young). Model probability prediction during the process is
attached at the top right corner.

(a) Model diagnosis histograms generated by ZOOM on four facial attribute
classifiers.

(b) Model diagnosis histograms generated by ZOOM on four classifiers trained
on manually-crafted imbalance data.

Figure 4: Model diagnosis histograms generated by ZOOM. The vertical axis
values reflect the attribute sensitivities calculated by averaging the model
probability change over all sampled images. The horizontal axis is the
attribute space input by user.

###  3.5 Counterfactual Training

The multi-attribute counterfactual approach visualizes semantic combinations
that cause the model to falter, providing valuable insights for enhancing the
model’s robustness. We naturally adopt the concept of iterative adversarial
training [18] to robustify the target model. For each iteration, ZOOM receives
the target model parameter and returns a batch of mutated counterfactual
images with the model’s original predictions as labels. Then the target model
will be trained on the counterfactually-augmented images to achieve the robust
goal:

$\small\theta^{*}=\operatorname*{argmin}_{\theta}\mathbb{E}_{\mathbf{x}\sim\mathcal{P}(\mathbf{x}),\mathbf{\hat{x}}=\text{ZOOM}(\mathbf{x},\mathcal{A})}{L}_{CE}(f_{\theta}(\mathbf{\hat{x}}),f_{\theta}(\mathbf{x}))$
(12)

where batches of $\mathbf{x}$ are randomly sampled from the StyleGAN generator
$\mathcal{G}_{\phi}$. In the following, we abbreviate the process as
Counterfactual Training (CT). Note that, although not explicitly expressed in
Eq. 12, the CT process is a min-max game. ZOOM synthesizes counterfactuals to
maximize the variation of model prediction (while persevering the perceived
ground truth), and the target model is learned with the counterfactual images
to minimize the variation.

##  4 Experimental Results

This section describes the experimental validations on the effectiveness and
reliability of ZOOM. First, we describe the model setup in Sec. 4.1. Sec. 4.2
and Sec. 4.3 visualize and validate the model diagnosis results for the
single-attribute setting. In Sec. 4.4, we show results on synthesized
multiple-attribute counterfactual images and apply them to counterfactual
training.

###  4.1 Model Setup

Pre-trained models: We used Stylegan2-ADA [11] pretrained on FFHQ [12] and
AFHQ [1] as our base generative networks, and the pre-trained CLIP model [24]
which is parameterized by ViT-B/32. We followed StyleCLIP [21] setups to
compute the channel relevance matrices $\mathcal{M}$.

Target models: Our classifier models are ResNet50 with single fully-connected
head initialized by TorchVision111https://pytorch.org/blog/how-to-train-state-
of-the-art-models-using-torchvision-latest-primitives/. In training the binary
classifiers, we use the Adam optimizer with learning rate 0.001 and batch size
128. We train binary classifiers for Eyeglasses, Perceived Gender, Mustache,
Perceived Age attributes on CelebA and for cat/dog classification on AFHQ. For
the 98-keypoint detectors, we used the HR-Net architecture [31] on WFLW [33].

###  4.2 Visual Model Diagnosis: Single-Attribute

Understanding where deep learning model fails is an essential step towards
building trustworthy models. Our proposed work allows us to generate
counterfactual images (Sec. 3.3) and provide insights on sensitivities of the
target model (Sec. 3.4). This section visualizes the counterfactual images in
which only one attribute is modified at a time.

Fig. 3 shows the single-attribute counterfactual images. Interestingly (but
not unexpectedly), we can see that reducing the hair length or joyfulness
causes the age classifier more likely to label the face to an older person.
Note that our approach is extendable to multiple domains, as we change the
cat-like pupil to dog-like, the dog-cat classification tends towards the dog.
Using the counterfactual images, we can conduct model diagnosis with the
method mentioned in Sec. 3.4, on which attributes the model is sensitive to.
In the histogram generated in model diagnosis, a higher bar means the model is
more sensitive toward the corresponding attribute.

Fig. 4(a) shows the model diagnosis histograms on regularly-trained
classifiers. For instance, the cat/dog classifier histogram shows outstanding
sensitivity to green eyes and vertical pupil. The analysis is intuitive since
these are cat-biased traits rarely observed in dog photos. Moreover, the
histogram of eyeglasses classifier shows that the mutation on bushy eyebrows
is more influential for flipping the model prediction. It potentially reveals
the positional correlation between eyeglasses and bushy eyebrows. The
advantage of single-attribute model diagnosis is that the score of each
attribute in the histogram are independent from other attributes, enabling
unambiguous understanding of exact semantics that make the model fail.
Diagnosis results for additional target models can be found in Appendix B.

###  4.3 Validation of Visual Model Diagnosis

Evaluating whether our zero-shot sensitivity histograms (Fig. 4) explain the
true vulnerability is a difficult task, since we do not have access to a
sufficiently large and balanced test set fully annotated in an open-vocabulary
setting. To verify the performance, we create synthetically imbalanced cases
where the model bias is known. We then compare our results with a supervised
diagnosis setting [17]. In addition, we will validate the decoupling of the
attributes by CLIP.

####  4.3.1 Creating imbalanced classifiers

In order to evaluate whether our sensitivity histogram is correct, we train
classifiers that are highly imbalanced towards a known attribute and see
whether ZOOM can detect the sensitivity w.r.t the attribute. For instance,
when training the perceived-age classifier (binarized as Young in CelebA), we
created a dataset on which the trained classifier is strongly sensitive to
Bangs (hair over forehead). The custom dataset is a CelebA training subset
that consists of $20,200$ images. More specifically, there are $10,000$ images
that have both young people that have bangs, represented as $(1,1)$, and
$10,000$ images of people that are not young and have no bangs, represented as
$(0,0)$. The remaining combinations of $(1,0)$ and $(0,1)$ have only 100
images. With this imbalanced dataset, bangs is the attribute that dominantly
correlates with whether the person is young, and hence the perceived-age
classifier would be highly sensitive towards bangs. See Fig. 5 (the right
histograms) for an illustration of the sensitivity histogram computed by our
method for the case of an age classifier with bangs (top) and lipstick
(bottom) being imbalanced.

Figure 5:  The sensitivity of the age classifier is evaluated with ZOOM
(right) and AttGAN (left), achieving comparable results.

We trained multiple imbalanced classifiers with this methodology, and
visualize the model diagnosis histograms of these imbalanced classifiers in
Fig. 4(b). We can observe that the ZOOM histograms successfully detect the
synthetically-made bias, which are shown as the highest bars in histograms.
See the caption for more information.

(a) Mustache classifier

(b) Perceived age classifier

Figure 6: Confusion matrix of CLIP score variation (vertical axis) when
perturbing attributes (horizontal axis). This shows that attributes in ZOOM
are highly decoupled.  Figure 7: Multi-attribute counterfactual in faces. The
model probability is documented in the upper right corner of each image.

####  4.3.2 Comparison with supervised diagnosis

We also validated our histogram by comparing it with the case in which we have
access to a generative model that has been explicitly trained to disentangle
attributes. We follow the work on [17] and used AttGAN [6] trained on the
CelebA training set over $15$ attributes222Bald, Bangs, Black_Hair,
Blond_Hair, Brown_Hair, Bushy_Eyebrows, Eyeglasses, Male, Mouth_Slightly_Open,
Mustache, No_Beard, Pale_Skin, Young, Smiling, Wearing_Lipstick.. After the
training converged, we performed adversarial learning in the attribute space
of AttGAN and create a sensitivity histogram using the same approach as Sec.
3.4. Fig. 5 shows the result of this method on the perceived-age classifier
which is made biased towards bangs. As anticipated, the AttGAN histogram
(left) corroborates the histogram derived from our method (right).
Interestingly, unlike ZOOM, AttGAN show less sensitivity to remaining
attributes. This is likely because AttGAN has a latent space learned in a
supervised manner and hence attributes are better disentangled than with
StyleGAN. Note that AttGAN is trained with a fixed set of attributes; if a new
attribute of interest is introduced, the dataset needs to be re-labeled and
AttGAN retrained. ZOOM, however, merely calls for the addition of a new text
prompt. More results in Appendix B.

####  4.3.3 Measuring disentanglement of attributes

Previous works demonstrated that the StyleGAN’s latent space can be entangled
[27, 2], adding undesired dependencies when searching single-attribute
counterfactuals. This section verifies that our framework can disentangle the
attributes and mostly edit the desirable attributes.

We use CLIP as a super annotator to measure attribute changes during single-
attribute modifications. For $1,000$ images, we record the attribute change
after performing adversarial learning in each attribute, and average the
attribute score change. Fig. 6 shows the confusion matrix during single-
attribute counterfactual synthesis. The horizontal axis is the attribute being
edited during the optimization, and the vertical axis represents the CLIP
prediction changed by the process. For instance, the first column of Fig. 6(a)
is generated when we optimize over bangs for the mustache classifier. We
record the CLIP prediction variation. It clearly shows that bangs is the
dominant attribute changing during the optimization. From the main diagonal of
matrices, it is evident that the ZOOM mostly perturbs the attribute of
interest. The results indicate reasonable disentanglement among attributes.

###  4.4 Visual Model Diagnosis: Multi-Attributes

In the previous sections, we have visualized and validated single-attribute
model diagnosis histograms and counterfactual images. In this section, we will
assess ZOOM’s ability to produce counterfactual images by concurrently
exploring multiple attributes within $\mathcal{A}$, the domain of user-defined
attributes. The approach conducts multi-attribute counterfactual searches
across various edit directions, identifying distinct semantic combinations
that result in the target model’s failure. By doing so, we can effectively
create more powerful counterfactuals images (see Fig. 9).

Figure 8: Multi-attribute counterfactual on Cat/Dog classifier. The number in
each image is the predicted probability of being a dog.

Figure 9:  Multiple-Attribute Counterfactual (MAC, Sec. 4.4) compared with
Single-Attribute Counterfactual (SAC, Sec. 4.2). We can see that optimization
along multiple directions enable the generation of more powerful
counterfactuals.

Fig. 7 and Fig. 8 show examples of multi-attribute counterfactual images
generated by ZOOM, against human and animal face classifiers. It can be
observed that multiple face attributes such as lipsticks or hair color are
edited in Fig. 7, and various cat/dog attributes like nose pinkness, eye
shape, and fur patterns are edited in Fig. 8. These attribute edits are
blended to affect the target model prediction. Appendix B further illustrates
ZOOM counterfactual images for semantic segmentation, multi-class
classification, and a church classifier. By mutating semantic representations,
ZOOM reveals semantic combinations as outliers where the target model
underfits.

In the following sections, we will use the Flip Rate (the percentage of
counterfactuals that flipped the model prediction) and Flip Resistance (the
percentage of counterfactuals for which the model successfully withheld its
prediction) to evaluate the multi-attribute setting.

(a) Sensitivity histograms generated by ZOOM on attribute combinations.

(b) Model diagnosis by ZOOM over $19$ attributes. Our framework is
generalizable to analyze facial attributes of various domains.

Figure 10: Customizing attribute space for ZOOM.

####  4.4.1 Customizing attribute space

In some circumstances, users may finish one round of model diagnosis and
proceed to another round by adding new attributes, or trying a new attribute
space. The linear nature of attribute editing (Eq. 3) in ZOOM makes it
possible to easily add or remove attributes. Table 1 shows the flip rates
results when adding new attributes into $\mathcal{A}$ for perceived age
classifier and big lips classifier. We can observe that a different attribute
space will results in different effectiveness of counterfactual images. Also,
increasing the search iteration will make counterfactual more effective (see
last row). Note that neither re-training the StyleGAN nor user-
collection/labeling of data is required at any point in this procedure.
Moreover, Fig. 10(a) shows the model diagnosis histograms generated with
combinations of two attributes. Fig. 10(b) demonstrates the capability of ZOOM
in a rich vocabulary setting where we can analyze attributes that are not
labeled in existing datasets [16, 29].

####  4.4.2 Counterfactual training results

This section evaluates regular classifiers trained on CelebA [16] and
counterfactually-trained (CT) classifiers on a mix of CelebA data and
counterfactual images as described in Sec. 3.5. Table 2 presents accuracy and
flip resistance (FR) results. CT outperforms the regular classifier. FR is
assessed over 10,000 counterfactual images, with FR-25 and FR-100 denoting
Flip Resistance after 25 and 100 optimization iterations, respectively. Both
use $\eta=0.2$ and $\epsilon=30$. We can observe that the classifiers after CT
are way less likely to be flipped by counterfactual images while maintaining a
decent accuracy on the CalebA testset. Our approach robustifies the model by
increasing the tolerance toward counterfactuals. Note that CT slightly
improves the CelebA classifier when trained on a mixture of CelebA images
(original images) and the counterfactual images generated with a generative
model trained in the FFHQ [12] images (different domain).

Method |  AC Flip Rate (%) |  BC Flip Rate (%)
---|---|---
Initialize ZOOM by $\mathcal{A}$ | 61.95 | 83.47
\+ Attribute: Beard | 72.08 | 90.07
\+ Attribute: Smiling | 87.47 | 96.27
\+ Attribute: Lipstick | 90.96 | 94.07
\+ Iterations increased to 200 | 92.91 | 94.87
Table 1:  Model flip rate study. The initial attribute space $\mathcal{A}=$ {Bangs, Blond Hair, Bushy Eyebrows, Pale Skin, Pointy Nose}. AC is the perceived age classifier and BC is the big lips classifier. Attribute |  Metric |  Regular (%) |  CT (Ours) (%)
---|---|---|---
Perceived Age | CelebA Accuracy | 86.10 | 86.29
ZOOM FR-25 | 19.54 | 97.36
ZOOM FR-100 | 9.04 | 95.65
Big Lips | CelebA Accuracy | 74.36 | 75.39
ZOOM FR-25 | 14.12 | 99.19
ZOOM FR-100 | 5.93 | 88.91
Table 2:  Results of network inference on CelebA original images and ZOOM-
generated counterfactual. The CT classifier is significantly less prone to be
flipped by counterfactual images, while test accuracy on CelebA remains
performant.

##  5 Conclusion and Discussion

In this paper, we present ZOOM, a zero-shot model diagnosis framework that
generates sensitivity histograms based on user’s input of natural language
attributes. ZOOM assesses failures and generates corresponding sensitivity
histograms for each attribute. A significant advantage of our technique is its
ability to analyze the failures of a target deep model without the need for
laborious collection and annotation of test sets. ZOOM effectively visualizes
the correlation between attributes and model outputs, elucidating model
behaviors and intrinsic biases.

Our work has three primary limitations. First, users should possess domain
knowledge as their input (text of attributes of interest) should be relevant
to the target domain. Recall that it is a small price to pay for model
evaluation without an annotated test set. Second, StyleGAN2-ADA struggles with
generating out-of-domain samples. Nevertheless, our adversarial learning
framework can be adapted to other generative models (e.g., stable diffusion),
and the generator can be improved by training on more images. We have
rigorously tested our generator with various user inputs, confirming its
effectiveness for regular diagnosis requests. Currently, we are exploring
stable diffusion models to generate a broader range of classes while
maintaining the core concept. Finally, we rely on a pre-trained model like
CLIP which we presume to be free of bias and capable of encompassing all
relevant attributes.

Acknowledgements: We would like to thank George Cazenavette, Tianyuan Zhang,
Yinong Wang, Hanzhe Hu, Bharath Raj for suggestions in the presentation and
experiments. We sincerely thank Ken Ziyu Liu, Jiashun Wang, Bowen Li, and Ce
Zheng for revisions to improve this work.

## References

  * [1] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  StarGAN v2: Diverse Image Synthesis for Multiple Domains.  In CVPR, 2020.
  * [2] Edo Collins, Raja Bala, Bob Price, and Sabine Süsstrunk.  Editing in Style: Uncovering the Local Semantics of GANs.  In CVPR, 2020.
  * [3] Emily Denton and Ben Hutchinson and Margaret Mitchell and Timnit Gebru and Andrew Zaldivar.  Image counterfactual sensitivity analysis for detecting unintended bias.  arXiv preprint arXiv:1906.06439, 2019.
  * [4] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and Harnessing Adversarial Examples.  2014\.
  * [5] Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee.  Counterfactual Visual Explanations.  In ICML, 2019.
  * [6] Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen.  AttGAN: Facial Attribute Editing by Only Changing What You Want.  In IEEE TIP, 2019.
  * [7] Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu.  AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars.  In ACM TOG, 2022.
  * [8] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  GANSpace: Discovering Interpretable GAN Controls.  In NeurIPS, 2020.
  * [9] Ameya Joshi, Amitangshu Mukherjee, Soumik Sarkar, and Chinmay Hegde.  Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers.  In ICCV, 2019.
  * [10] Kimmo Karkkainen and Jungseock Joo.  FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation.  In WACV, 2021.
  * [11] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training Generative Adversarial Networks with Limited Data.  In NeurIPS, 2020.
  * [12] Tero Karras, Samuli Laine, and Timo Aila.  A Style-Based Generator Architecture for Generative Adversarial Networks.  In CVPR, 2019.
  * [13] Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William T. Freeman, Phillip Isola, Amir Globerson, Michal Irani, and Inbar Mosseri.  Explaining in Style: Training a GAN To Explain a Classifier in StyleSpace.  In ICCV, 2021.
  * [14] Bo Li, Qiulin Wang, Jiquan Pei, Yu Yang, and Xiangyang Ji.  Which Style Makes Me Attractive? Interpretable Control Discovery and Counterfactual Explanation on StyleGAN.  arXiv preprint arXiv:2201.09689, 2022.
  * [15] Zhiheng Li and Chenliang Xu.  Discover the Unknown Biased Attribute of an Image Classifier.  In ICCV, 2021.
  * [16] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep Learning Face Attributes in the Wild.  In ICCV, 2015.
  * [17] Jinqi Luo, Zhaoning Wang, Chen Henry Wu, Dong Huang, and Fernando De la Torre.  Semantic image attack for visual model diagnosis.  arXiv preprint arXiv:2303.13010, 2023.
  * [18] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards Deep Learning Models Resistant to Adversarial Attacks.  In ICLR, 2018.
  * [19] Joanna Materzynska, Antonio Torralba, and David Bau.  Disentangling Visual and Written Concepts in CLIP.  In CVPR, 2022.
  * [20] Ramaravind K. Mothilal, Amit Sharma, and Chenhao Tan.  Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations.  In ACM FAccT, 2020.
  * [21] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery.  In ICCV, 2021.
  * [22] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  DreamFusion: Text-to-3D using 2D Diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [23] Haonan Qiu, Chaowei Xiao, Lei Yang, Xinchen Yan, Honglak Lee, and Bo Li.  SemanticAdv: Generating Adversarial Examples via Attribute-conditioned Image Editing.  In ECCV, 2020.
  * [24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  In ICML, 2021.
  * [25] Vikram V. Ramaswamy, Sunnie S. Y. Kim, and Olga Russakovsky.  Fair Attribute Classification Through Latent Space De-Biasing.  In CVPR, 2021.
  * [26] Axel Sauer and Andreas Geiger.  Counterfactual Generative Networks.  In ICLR, 2021.
  * [27] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs.  In IEEE TPAMI, 2020.
  * [28] Yujun Shen and Bolei Zhou.  Closed-Form Factorization of Latent Semantics in GANs.  In CVPR, 2021.
  * [29] Philipp Terhörst, Daniel Fährmann, Jan Niklas Kolf, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper.  MAAD-Face: A Massively Annotated Attribute Dataset for Face Images.  In IEEE TIFS, 2021.
  * [30] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.  CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields.  In CVPR, 2022.
  * [31] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao.  Deep High-Resolution Representation Learning for Visual Recognition.  In IEEE TPAMI, 2019.
  * [32] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image Quality Assessment: from Error Visibility to Structural Similarity.  In IEEE TIP, 2004.
  * [33] Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou.  Look at Boundary: A Boundary-Aware Face Alignment Algorithm.  In CVPR, 2018.
  * [34] Zongze Wu, Dani Lischinski, and Eli Shechtman.  StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation.  In CVPR, 2021.
  * [35] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang.  GAN Inversion: A Survey.  In IEEE TPAMI, 2022.
  * [36] Chaowei Xiao, Bo Li, Jun-yan Zhu, Warren He, Mingyan Liu, and Dawn Song.  Generating Adversarial Examples with Adversarial Networks.  In IJCAI, 2018.
  * [37] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model.  arXiv preprint arXiv:2208.15001, 2022.

##  Appendix A Validation for CLIP-guided Editing

Our methodology relies on CLIP-guided fine-grained image editing to provide
adequate model diagnostics. It is critical to verify CLIP’s ability to link
language and visual representations. This section introduces two techniques
for validating CLIP’s capabilities.

###  A.1 Visualization for edited images

In this section, we analyze the decoupling of attribute editing used in
StyleCLIP [21] in our domain.

Effect of $\lambda$. Fig. 17 shows the effect of $\lambda$ in Equation 2 of
the main text [21] . Originally in StyleCLIP, this filter parameter (denoted
as $\beta$ in [21]) helps the style disentanglement for editing. As we have
normalized the edit vectors, which contributes to disentanglement in our
framework, the impact of $\lambda$ on style disentanglement is reduced.
Consequently, $\lambda$ primarily influences intensity control and denoising.

Single-attribute editing. Fig. 18 and Fig. 20 show a set of images of
different object categories by editing different attributes extracted with the
global edit directions method (as described in Section 3.2 of the main text).
By analyzing the user’s input attribute string, we can see that the modified
image only alters in the attribute direction while maintaining the other
attributes.

Multiple-attribute editing. We demonstrate the validity of our method for
editing multiple attributes through linear combination (as outlined in
Equation 3 of the main text) by presenting illustrations of combined edits in
Figure 21.

###  A.2 User study for edited images

To validate that our counterfactual image synthesis preserve fine-grained
details and authenticity, we conducted a user study validating two aspects:
synthesis fidelity and attribute consistency.

User study for synthesis fidelity. The classification of the counterfactual
synthesis image vs real images by the user is employed to confirm that no
unrealistic artifacts are introduced throughout the process of our model Fig.
15(a) shows sample questions of this study. In theory, the worst-case scenario
is that users can accurately identify the semantic modification and achieve a
user recognition rate of $100\%$. Conversely, the best-case scenario would be
that users are unable to identify any counterfactual synthesis and make random
guesses, resulting in a user recognition rate of $50\%$.

User study for attribute consistency. We ask users whether they agree that the
counterfactual and original images are consistent on the ground truth w.r.t.
the target classifier. For example, during the counterfactual synthesis for
the cat/dog classifier, a counterfactual cat image should stay consistent as a
cat. Fig. 15(b) shows another sample questions. The worst case is that the
counterfactual changes the ground truth label to affect the target model,
which makes the user agreement rate very low (even to zero).

The user study statistics are presented in Table 3. The study involved $34$
participants with at least an undergraduate level of education, who were
divided into two groups using separate collector links. The participants
themselves randomly selected their group (i.e., the link clicked), and their
responses were collected.

Figure 11: Effect of progressively generating counterfactual images on the
Cat/Dog classifier (0-Cat / 1-Dog), and the Perceived Gender classifier
(0-Female / 1-Male). Model probability prediction during the process is
attached at the top right corner.

The production of high-quality counterfactual images is supported by the
difficulty users had in differentiating them. Additionally, the majority of
users concurred that the counterfactual images do not change the ground truth
concerning the target classifier, confirming that our methodology generates
meaningful counterfactuals. However, it should be noted that due to the nature
of our recognition system, human volunteers are somewhat more responsive to
human faces. As a result, we observed a slightly higher recognition rate in
the human face (FFHQ) domain than in the animal face (AFHQ) domain.

###  A.3 Stability across CLIP phrasing/wording:

It is worth noting that the resulting counterfactual image is affected by the
wording of the prompt used. In our framework, we subtract the neutral phrase
(such as ”a face”) after encoding in CLIP space to ensure that the attribute
edit direction is unambiguous enough. Our experimentation has shown that as
long as the prompt accurately describes the object, comparable outcomes can be
achieved. For instance, we obtained similar sensitivity results on the
perceived-age classifier using prompts like ”a picture of a person with X,” ”a
portrait of a person with X,” or other synonyms. Examples of this are
presented in Figure 13.

Figure 12: ZOOM counterfactuals on more tasks (segmentation, multi-class classifier) and additional visual domains (cars, churches). Zoom in for better visibility. Name of Study | Domain | Group 1 | Group 2
---|---|---|---
Synthesis Fidelity ( Recognition Rate $\downarrow$, %) | FFHQ | 62.12 | 71.79
AFHQ | 51.30 | 50.55
Attribute Consistency ( Agreement Rate $\uparrow$, %) | FFHQ | 94.12 | 90.76
AFHQ | 89.92 | 88.26
Table 3: User study results. We can see from the table that our counterfactual
synthesis preserves the visual quality and maintains the ground truth labels
from the user’s perspective.

##  Appendix B Additional Results of Model Diagnosis

###  B.1 Additional counterfactual images

Fig. 11 shows more examples of single-attribute counterfactual images on the
Cat/Dog and Perceived Gender classifiers. The output prediction is shown in
the top-right corner. It shows that the model prediction is flipped without
changing the actual target attribute. In addition to binary classification and
key-point detection in our manuscript, we further illustrate the extension of
ZOOM counterfactuals on semantic segmentation, multi-class classification, and
binary church classifier (BCC) in Fig. 12. Fig. 16 shows more examples of
multiple-attribute counterfactual images.

###  B.2 Additional histograms

Fig. 14 shows more histograms on the classifiers trained on CelebA (top) and
the classifiers that are intentionally biased (bottom). The models and
datasets are created using the same method described in Section 4 of the main
text.

##  Appendix C Ablation of the Adversarial Optimization Method

When there are multiple attributes (i.e., $N>1$) to optimize, linearizing the
cost function as grid in high dimensional space will help to efficiently
approximate convergence in limited epochs. Specifically, we have the option to
adopt PGD [18] (i.e., update using
$\eta\cdot\operatorname{sign}(\nabla_{\mathbf{w}}\mathcal{L})$) for efficient
optimization. We compared generating counterfactuals with and without
projected gradients. Table 4 shows the visual quality and flip rate of the
generated counterfactuals. We can observe that ZOOM-PGD image quality is finer
under Structured Similarity Indexing Method (SSIM) [32], while ZOOM-SGD has a
higher flip rate. The images from ZOOM-PGD is finer since the signed method
stabilizes the optimization by eliminating problems of gradient vanishing and
exploding.

Optimization | Classifier |  SSIM ($\uparrow$) |  Flip Rate (%, $\uparrow$)
---|---|---|---
SGD | Perceived Age | 0.5732 | 67.24
Perceived Gender | 0.5815 | 49.40
Mustache | 0.5971 | 36.33
PGD | Perceived Age | 0.8065 | 50.19
Perceived Gender | 0.7035 | 42.84
Mustache | 0.7613 | 25.10
Table 4: The comparison of counterfactuals generated with stochastic gradient
descent (SGD) and projected gradient descent (PGD) method. We can observe that
ZOOM-PGD image quality is finer under SSIM (Structured Similarity Indexing
Method) [32] metrics, while ZOOM-SGD has a higher flip rate.

Our empirical observation during the experiment is that ZOOM-PGD frequently
oscillates around a local minima of edit weights and fails to reach an optimal
counterfactual. We hypothesize that the reason of lower flip rates from the
signed method is that the edit weight search is constrained on nodes of a grid
space (the grid unit length is step-size $\eta$), which loses precision and
underperforms during counterfactual search.

Figure 13:  Sensitivity histograms when using four instances of phrases with a
similar concept. Zoom in for better visibility.

Figure 14: The above histograms show ZOOM on three regularly trained
classifiers on CelebA, and the bottom histograms show ZOOM successfully
detects the bias in the manually-crafted imbalanced classifiers.

(a) Evaluating visual fidelity. We show two images and let users choose the
one that they think is edited. The counterfactuals are generated on Eyeglasses
classifier and Cat/Dog classifier.

(b) Evaluating attribute consistency. The user classifies whether the ground
truth is flipped. Example of counterfactual images on Cat/Dog classifier and
Eyeglasses classifier is shown above.

Figure 15: Sample questions in the user study. Each user answers $10$
questions for each of the two user studies.

(a) Multiple-attribute counterfactual for cat/dog classifier.

(b) Multiple-attribute counterfactual for eyeglasses classifier.

(c) Multiple-attribute counterfactual for perceived gender classifier.

(d) Multiple-attribute counterfactual for mustache classifier.

(e) Multiple-attribute counterfactual for perceived age classifier.

Figure 16: Multi-attribute counterfactual in the human face and animal face
domain. The right-up corner of each image records the model output prediction.

(a) Effect of $\lambda$ values for editing beard.

(b) Effect of $\lambda$ values for editing pale skin.

Figure 17: Visualization of the effect of different $\lambda$ values.

(a) Attribute editing: a cat with green eyes.

(b) Attribute editing: a cute cat.

(c) Attribute editing: a dog with round face.

(d) Attribute editing: a cute dog.

(e) Attribute editing: a cat with round face.

(f) Attribute editing: a cat with pointed ears.

(g) Attribute editing: a dog with open mouth.

(h) Attribute editing: a black dog.

Figure 18: Visualization of global edit directions by utilizing the StyleCLIP
channel relevance matrix. Images are sampled from the AFHQ domain using
StyleGAN2-ADA. Every column demonstrates an edited image from edit weight
$w=-30$ to $w=30$. Weights of five images are linearly interpolated as
$\\{-30,-15,0,15,30\\}$. We can see that global edit directions are
generalizable on multiple images.

(a) Attribute editing: an angry face.

(b) Attribute editing: a face with eyeglasses.

(c) Attribute editing: a cute face.

(d) Attribute editing: a face with blond hair.

(e) Attribute editing: a face with bangs.

(f) Attribute editing: a smiling face.

(g) Attribute editing: a happy face.

(h) Attribute editing: a face with curly hair.

(a) Attribute editing: a face with beard.

(b) Attribute editing: a face with lipstick.

(c) Attribute editing: a tired face.

(d) Attribute editing: a skinny face.

(e) Attribute editing: a male face.

(f) Attribute editing: a surprised face.

(g) Attribute editing: a face with long hair.

(h) Attribute editing: a face with pale skin.

Figure 20: Visualization of global edit directions by utilizing the StyleCLIP
channel relevance matrix. Images are sampled from the FFHQ domain using
StyleGAN2-ADA. Every column demonstrates an edited image from edit weight
$w=-30$ to $w=30$. Weights of five images are linearly interpolated as
$\\{-30,-15,0,15,30\\}$. We can see that global edit directions are
generalizable on multiple images.

(a) Combination of smiling ($w_{1}$) and lipstick ($w_{2}$).

(b) Combination of pale skin ($w_{1}$) and blond hair ($w_{2}$).

Figure 21: Visualization of traversing on directional (attribute) style
vectors to validate the effectiveness of multiple attribute editing.


# The Cubic Vortical Whitham Equation

John D. Carter<EMAIL_ADDRESS>Mathematics Department, Seattle
University, USA  Henrik Kalisch  Department of Mathematics, University of
Bergen, Norway  Christian Kharif  Malek Abid  Aix Marseille Université, CNRS,
Centrale Marseille, IRPHE UMR 7342, F-13384, Marseille, France

###### Abstract

The cubic-vortical Whitham equation is a two-dimensional model for wave motion
on a vertically sheared current of constant vorticity in a shallow inviscid
fluid. It generalizes the classical Whitham equation by allowing constant
vorticity and by adding a cubic nonlinear term. The inclusion of this extra
nonlinear term allows the equation to admit periodic, traveling-wave solutions
with larger amplitude than the Whitham equation. Increasing vorticity leads to
solutions with larger amplitude as well. The stability of these solutions is
examined numerically. All moderate- and large-amplitude solutions, regardless
of wavelength, are found to be unstable. A formula for a stability cutoff as a
function of vorticity and wavelength for small-amplitude solutions is
presented. In the case with zero vorticity, small-amplitude solutions are
unstable with respect to the modulational instability if $kh>1.252$, where $k$
is the wavenumber and $h$ is the mean fluid depth.

##  1 Introduction

The dimensionless Korteweg-deVries (KdV) equation,

$u_{t}+u_{x}+\frac{1}{6}u_{xxx}+\frac{3}{2}uu_{x}=0,$ (1)

is a model for the evolution of small-amplitude long waves on a shallow fluid
with a horizontal, impermeable bed. The KdV equation is one of the most well-
known nonlinear partial differential equations (PDEs). It has been extensively
studied both mathematically, see for example [24, 30, 1, 22], and
experimentally, see for example [26, 31, 14, 15]).

The KdV equation only reproduces the linear phase velocity of the water wave
problem in the long-wave limit. In order to address this deficiency, Whitham
[29, 30] proposed the nonlocal equation that now bears his name

$u_{t}+K*u_{x}+\frac{3}{2}uu_{x}=0,$ (2)

where $K$ is the Fourier multiplier defined by

$K=\sqrt{\frac{\tanh(k)}{k}}.$ (3)

Here, $h$ and $\sqrt{h/g}$ are taken as the implicit length and temporal
scales respectively where $h$ is the mean fluid depth and $g$ represents the
acceleration due to gravity. The Whitham equation reproduces the
unidirectional phase velocity of the full water wave problem. Moldabayev [25],
Trillo et al. [28], and Carter [7] showed that the Whitham equation is a more
accurate model for the evolution of waves of depression than the KdV and Serre
equations. Ehrnström & Kalisch [12] proved that the Whitham equation admits
small-amplitude, periodic traveling-wave solutions and numerically computed
periodic, travelling-wave solutions with a variety of amplitudes including
those close to the highest wave. Ehrnström & Wahlén [13] proved Whitham’s
conjecture that (2) admits a highest, cusped traveling-wave solution. Hur &
Johnson [16] proved that small but finite amplitude periodic solutions of the
Whitham equation are stable with respect to long wavelength perturbations when
$kh<1.145$ and are unstable with respect to long wavelength perturbations when
$kh>1.145$. Here, $k$ is the wavenumber of the solution and $h$ is the fluid
depth. This result is qualitatively different than the Bottman & Deconinck [6]
proof that _all_ traveling-wave solutions of KdV are stable. However, it is
qualitatively similar to the Benjamin & Feir [4] result that establishes the
stability cut-off as $kh=1.363$. Sanford et al. [27] numerically corroborated
the Hur & Johnson [16] result and numerically showed that all large-amplitude
periodic traveling-wave solutions of the Whitham equation are unstable
regardless of their wavelength.

In recent years, generalizations of the Whitham equation have gained
significant interest. Aceves-Sánchez et al. [2], Deconinck & Trichtchenko
[10], and Hur & Pandey [18] all proposed different generalizations of the
Whitham equation that match the _bidirectional_ phase velocity of the water-
wave problem. Hur & Johnson [17] studied a generalization of the Whitham
equation that includes surface tension. Dinvay et al. [11] showed that this
capillary-Whitham equation gives a more accurate reproduction of the water-
wave problem than the KdV and Kawahara (fifth-order KdV) equations. Carter &
Rozman [8] presented results from a numerical study of solutions to the
capillary-Whitham equation. Herein, we focus on a generalization of the
(unidirectional) Whitham equation that includes constant vorticity and cubic
nonlinearity.

Kharif et al. [21] and Kharif & Abid [19] proposed an asymptotic model for
fully nonlinear water waves propagating on a vertically sheared current of
constant vorticity in shallow water that satisfies the unidirectional, linear
dispersion relation. From this model they derived, within the framework of
weakly nonlinear waves, a generalization of the Whitham equation which they
named the vor-Whitham equation. As a generalization of this, they proposed the
following higher-order nonlinear model

$u_{t}+\mathcal{K}*u_{x}+\alpha uu_{x}+\beta u^{2}u_{x}=0,$ (4)

where $\mathcal{K}$ is the Fourier multiplier defined by

$\mathcal{K}=-\frac{\Omega\tanh(k)}{2k}+\sqrt{\frac{\tanh(k)}{k}+\frac{\Omega^{2}\tanh^{2}(k)}{4k^{2}}},$
(5)

and the real constants $\alpha$ and $\beta$ are given by

$\alpha=\frac{\Omega^{2}+3}{\sqrt{\Omega^{2}+4}},$ (6a)
$\beta=-\frac{6+\Omega^{2}}{2(4+\Omega^{2})^{3/2}}.$ (6b)

As above, the implicit length and temporal scales are defined by $h$ and
$\sqrt{h/g}$. The parameter $\Omega$ represents the opposite sign of the
constant vorticity. Plots of $\alpha$ and $\beta$ are included in Figure 1. We
refer to equation (4) as the cubic vortical Whitham or CV-Whitham equation.
Note that the Fourier multiplier $K$, see equation (3), is the $\Omega=0$
version of the Fourier multiplier $\mathcal{K}$. The classical Whitham
equation is obtained from (4) and (5) by setting $\alpha=\frac{3}{2}$ and
$\beta=\Omega=0$.

Figure 1: Plots of $\alpha$ and $\beta$ as functions of $\Omega$.

The CV-Whitham equation admits the following conserved quantities.

$\mathcal{Q}_{1}=\int_{0}^{L}u~{}dx,$ (7a)
$\mathcal{Q}_{2}=\int_{0}^{L}u^{2}dx,$ (7b)
$\mathcal{Q}_{3}=\frac{1}{2}\int_{0}^{L}\left(u\mathcal{K}*u+\frac{\alpha}{3}u^{3}+\frac{\beta}{6}u^{4}\right)dx.$
(7c)

The conserved quantity $\mathcal{Q}_{3}$ is the Hamiltonian. These quantities
are used as checks on the accuracy of our numerical results.

The remainder of this paper is organized as follows. Section 2 contains a
description of the method we use to compute periodic, traveling-wave solutions
to the CV-Whitham equation. It also contains plots and comparisons of these
solutions. Section 3 contains the linear stability analysis along with
numerical stability results. Section 4 contains a summary of the work.

##  2 Traveling-wave solutions

Consider periodic, traveling-wave solutions of the form

$u(x,t)=f(x-ct)=f(z),$ (8)

where $c$ is a real constant and $f$ is a smooth, real-valued function of
$z=x-ct$ with period $L$. Substituting (8) into (4) and integrating with
respect to $z$ once gives

$-cf+\mathcal{K}*f+\frac{\alpha}{2}f^{2}+\frac{\beta}{3}f^{3}=B,$ (9)

where $B$ is the constant of integration. Unlike the corresponding equation in
the Whitham case, equation (9) does not possess an invariant that allows the
mean of the solution $f$ or the constant $B$ to be set to zero without loss of
generality. Nevertheless, we assume that the mean of the solution is zero
because such solutions are most physically relevant. Equation (9) can be
solved approximately by assuming $f$ has a Fourier expansion of the form

$f(z)=\sum_{k=-N}^{N}\hat{f}(k)\exp\left(\frac{2\pi ikz}{L}\right),$ (10)

where $N$ is a large positive integer and the $\hat{f}$ are complex constants.
We assume that the solutions are real and even. Therefore, all $\hat{f}$ are
real valued and $\hat{f}(-k)=\hat{f}(k)$ for $k=1,\dots,N$. Additionally, we
impose $\hat{f}(0)=0$ so that the solutions have zero mean. Substituting (10)
into (9) gives the following system of equations

$\begin{split}-c\hat{f}(k)&+\mathcal{K}\left(\frac{2\pi
k}{L}\right)\hat{f}(k)+\frac{\alpha}{2}\left(\sum_{l=\max(-N,-N+k)}^{\min(N,N+k)}\hat{f}(k-l)\hat{f}(l)\right)\\\
&+\frac{\beta}{3}\left(\sum_{j=\max(-2N,-N+k)}^{\min(2N,N+k)}\sum_{l=\max(-N,-N+j)}^{\min(N,N+j)}\hat{f}(k-j)\hat{f}(j-l)\hat{f}(l)\right)=0,\hskip
14.22636pt\text{for $k=1,\dots,N$}.\end{split}$ (11)

The case $k=0$ is not included in this system because it is used to determine
the integration constant $B$ and is otherwise unimportant. Equation (11) is a
system of $N$ equations for the $N+1$ unknowns $c$ and $\hat{f}(k)$ for
$k=1,\dots,N$. A solution to this system is found using linear theory and
Newton’s method. Given values for $L$ and $\Omega$, choose a value of $c$
slightly larger than $c=\mathcal{K}\left(\frac{2\pi}{L}\right)$ and then
solving (11) using Newton’s method with $\hat{f}(1)=0.1$ and $\hat{f}(k)=0$
for $k=2,\dots,N$ as the initial estimate for the solution. This gives the
Fourier coefficients of a small-amplitude, $L$-periodic solution to the CV-
Whitham equation. Fourier coefficients of solutions with larger amplitudes are
found by incrementing $c$ and solving (11) via Newton’s method using the
$\hat{f}$ values from the previous solution as the initial estimate.

This procedure can be used to find families of solutions to the CV-Whitham
equation for any $\Omega$ and (positive) $L$. Representative plots of
solutions are included in Figures 2 and 3. Figure 2 contains plots of
solutions to the CV-Whitham equation when $L=\pi$ and $\Omega=0.0,\pm 0.2,\pm
0.4,\pm 0.6,\pm 0.8,\pm 1.0$ and Figure 3 contains plots of solutions when
$L=2\pi$ for the same values of $\Omega$. These plots (and others omitted for
conciseness) show that the families of solutions are qualitatively similar
regardless of the values of $L$ and $\Omega$. Small-amplitude solutions appear
very trigonometric and solution steepness increases with wave height. For each
$L$ and $\Omega$, there appears to be a solution with maximal wave height.
This solution is very steep at its peak and may be cusped. However since our
numerics are based on Fourier series, it is difficult to accurately resolve
the precise shape of the solution with maximal wave height. Figures 2 and 3
also show that for a given period, as $\Omega$ increases, the wave height of
the solution with maximal wave height decreases in magnitude.

Figure 2: Traveling-wave solutions to the CV-Whitham equation with period
$L=\pi$ for eleven different values of $\Omega$. Figure 3: Traveling-wave
solutions to the CV-Whitham equation with period $L=2\pi$ for eleven different
values of $\Omega$.

The CV-Whitham equation examined herein is similar to the vor-Whitham equation
examined in Kharif et al. [20]. The only difference is the addition of the
cubic nonlinear term in (4). In essence, the CV-Whitham equation is “more
nonlinear” than the vor-Whitham equation. For given values of $L$ and
$\Omega$, the difference between periodic traveling-wave solutions of the CV-
Whitham and vor-Whitham equations is small. As $\Omega$ increases in magnitude
this difference decreases. (Note that the magnitude of $\beta$, the
coefficient of the cubic nonlinear term, is maximized when $\Omega=0$.) For
given values of $L$ and $\Omega$, the solution of maximal height of the CV-
Whitham equation appears to have a larger wave height than the wave of maximal
height of the vor-Whitham equation. This means that the added nonlinearity
allows solutions of larger amplitude.

Figures 4 and 5 include plots of the bifurcation curves showing wave height,
$H$, as a function of wave speed, $c$. These plots show that for given values
of $L$ and $\Omega$, as wave speed increases, so does wave height. For a fixed
value of $L$, the maximal wave height increases as $\Omega$ decreases.
However, as $\Omega$ decreases below $-1.0$, the increase in maximal wave
height is small. For a fixed value of $\Omega$, the maximal wave height
increases as the period increases. The difference between the bifurcation
plots of the CV- and vor-Whitham equations is small. There is not a simple
relationship between the phase speeds of the two equations, though the
difference increases as $\Omega$ decreases.

Figure 4: Plot of wave height, $H$, versus wave speed, $c$, for solutions with
period $L=2\pi$ and nine values of $\Omega$. The curves from left to right are
ordered by decreasing values of $\Omega$. Figure 5: Plots of wave height, $H$,
versus wave speed, $c$, for solutions with $\Omega=\pm 1.0$ and four different
periods. The four curves on the left correspond to $\Omega=1.0$ and the four
on the right correspond to $\Omega=1.0$.

Figure 6 includes plots of solutions with period $L=2\pi$ and wave height of
$H=0.25$ for five different values of $\Omega$. The plots show that as the
values of $\Omega$ increase, the wave steepness also increases. The same trend
is observed in the vor-Whitham equation, see Kharif et al. [20].

Figure 6: Plots of traveling-wave solutions to the CV-Whitham equation with
period, $L=2\pi$, and wave height, $H=0.25$, for five values of $\Omega$. The
solution steepness increases with $\Omega$.

##  3 Stability

In order to study the stability of traveling-wave solutions to the CV-Whitham
equation, we use the Fourier-Floquet-Hill method of Deconinck & Kutz [9]. In
order to do this, first make the change of variables defined by

$z=x-ct,\hskip 56.9055pt\tau=t.$ (12)

This converts the CV-Whitham equation to

$u_{\tau}-cu_{z}+\mathcal{K}*u_{z}+\alpha uu_{z}+\beta u^{2}u_{z}=0,$ (13)

and converts the traveling-wave solutions of (4) into stationary solutions of
(13). In order to study the stability of these solutions, consider perturbed
solutions of the form

$u_{pert}(z,\tau)=u(z)+\epsilon v(z,\tau)+\mathcal{O}(\epsilon^{2}),$ (14)

where $u$ is one of the solutions introduced in the previous section and
$\epsilon v$ is the leading-order part of the perturbation. Substituting (14)
into (13) and linearizing gives

$v_{\tau}-cv_{z}+\mathcal{K}*v_{z}+(\alpha u^{\prime}+2\beta
uu^{\prime})v+(\alpha u+\beta u^{2})v_{z}=0,$ (15)

where “prime” means derivative with respect to $z$. Since this nonlocal PDE is
constant coefficient in $\tau$, without loss of generality assume

$v(z,\tau)=V(z)\mbox{e}^{\lambda\tau}+c.c.,$ (16)

where $V$ is a complex-valued function, $\lambda$ is a complex constant, and
$c.c.$ stands for complex conjugate. Substituting (16) into (15) and
rearranging gives

$\mathcal{L}V=\lambda V,$ (17)

where $\mathcal{L}$ is the nonlocal differential operator defined by

$\mathcal{L}=c\partial_{z}-\mathcal{K}*\partial_{z}-(\alpha u^{\prime}+2\beta
uu^{\prime})-(\alpha u+\beta u^{2})\partial_{z}.$ (18)

Equation (17) is a differential eigenvalue problem with eigenvalue $\lambda$
and eigenfunction $V$. Floquet’s theorem [9] states that all bounded solutions
to this equation have the form

$V(z)=\mbox{e}^{i\mu z}\sum_{k=-N}^{N}\hat{V}(k)\mbox{e}^{2\pi ikz/L}$ (19)

where the $\hat{V}(k)$ are complex numbers and
$\mu\in[-\frac{\pi}{L},\frac{\pi}{L}]$ is a real constant known as the Floquet
parameter. If any $\lambda$ has a positive real part, then the corresponding
perturbation grows exponentially in $\tau$ and the solution, $u(z)$, is said
to be unstable. If all $\lambda$ are purely imaginary, then the solution is
said to be spectrally stable because there are no perturbations that grow
exponentially.

Substituting (19) into (17) and truncating gives

$\hat{\mathcal{L}}\hat{\mathcal{V}}=\lambda\hat{\mathcal{V}},$ (20)

where $\hat{\mathcal{V}}=(\hat{V}(-N),\hat{V}(-N+1),\dots,\hat{V}(N))^{T}$ and
the elements of the matrix $\hat{\mathcal{L}}$ are given by

$\hat{\mathcal{L}}_{mn}=\begin{cases}i\left(\mu+\frac{2\pi
m}{L}\right)\left(c-\mathcal{K}\left(\mu+\frac{2\pi
m}{L}\right)\right),&m=n,\\\ i\left(\mu+\frac{2\pi
m}{L}\right)\left(-\alpha\hat{u}(n-m)-\beta\hat{w}(n-m)\right),&m\neq
n,\end{cases}$ (21)

where $\hat{w}$ are the Fourier coefficients of $u^{2}$. Given values of $L$
and $\Omega$ along with a solution, we choose an equally-spaced sampling of
$\mu$ values and find the corresponding eigenvalues and eigenvectors using a
standard eigensolver (“eigen” in the Julia programming language).

Figure 7 contains plots of the stability spectra (in the left column) and the
growth rate versus the Floquet parameter (in the right column) for solutions
with $L=2\pi$, $\Omega=0.0$, and a number of different wave heights. The plots
corresponding to solutions of the CV-Whitham equation corresponding to other
values of $L$ and $\Omega$ are qualitatively similar to those shown in Figure
7. In the $L=2\pi$ and $\Omega=0.0$ case, all solutions with wave height less
than $H=0.5426$ or speed less than $c=0.9524$ are spectrally stable. Solutions
with larger wave heights or speeds are unstable. This cutoff is close to the
classical result for the full water wave problem, see Figure 3 of McLean [23].
(A more precise comparison cannot be made because [23] includes a plot instead
of a table of values.) We note that our numerical exploration of
$(L,\Omega)$-space showed that all large-amplitude solutions of the CV-Whitham
equation are unstable regardless of wavelength.

The plots in Figure 7 show that as the wave height or speed of the solution
increases, so does the maximum instability growth rate. This means that
solutions with larger wave height are more unstable than solutions with
smaller wave height. The plots also show that if a solution with “moderate”
wave height is unstable, then there is a “figure eight” centered at the origin
of the spectrum. For solutions with “large” wave height, there is a “figure
infinity” centered at the origin of the spectrum surrounded by a larger oval.
As the wave height continues to increase, the size of the figure infinity
decreases until it appears to collapse to the origin (i.e. becomes too small
to resolve using double precision computations). Simultaneously, the outer
oval pinches off along the vertical axis and becomes two smaller ovals. For
solutions with very large wave height, there are two oval-like shapes centered
on the real axis. As wave height increases, these ovals decrease in size while
their centers trend away from the origin while remaining centered on the real
axis. It is unclear if the centers of these ovals head to
$\Re(\lambda)=\pm\infty$ or to finite values as wave height increases.

The plots of the growth rate versus the Floquet parameter show that solutions
with large enough wave height are unstable with respect to a perturbation with
$\mu=0.0$. This means that such solutions are unstable with respect to
perturbations that have the same wavelength as the solution. This is quite
different than the standard modulational (long-wave) instability that has been
proven to exist in the small-amplitude limit of the Whitham equation, see
[16]. If $\Omega=0.0$ and $L=2\pi$, solutions with $H\geq 0.78976$ or $c\geq
0.98735$ are unstable with respect to perturbations with period $2\pi$. We
were unable to find a simple relationship between $(\Omega,L)$ and the $(H,c)$
value of this cutoff.

Figure 7: Plots of the stability spectra (left) and growth rate versus $\mu$
(right) for a number of solutions to the CV-Whitham equation with $L=2\pi$ and
$\Omega=0.0$. The second row of plots correspond to solutions with wave
heights that are larger than those in the first row.

As an independent check on the results shown in Figure 7, we selected the
traveling-wave solution to the CV-Whitham equation with $\Omega=0.0$,
$H=0.8065$, and $c=0.9880$, see Figure 8(a), and an unstable perturbation with
$\mu=0.5$ and $\lambda=0.18905$, see Figure 8(b). Two periods of the solution
are shown because the period of the perturbation is twice that of the solution
when $\mu=0.5$. Then we used

$u_{pert}(x,0)=u(x)+10^{-7}v(x,0)$ (22)

as an initial condition in our CV-Whitham solver while monitoring the growth
of the perturbation. The perturbation grew exponentially with rate of
approximately $r=0.18905$ until its amplitude was large enough that nonlinear
effects began to play a role. This result corroborates the value of one point
in Figure 7.

Figure 8: Plots of (a) two periods of a solution to the CV-Whitham equation
with $L=2\pi$, $\Omega=0.0$, $H=0.8065$, and $c=0.9880$ and (b) the
corresponding most unstable perturbation, $v(x,0)$, when $\mu=0.5$.

For comparative purposes, we let $\Omega=0.4$ and selected the traveling-wave
solution to the CV-Whitham equation with $H=0.5969$ and $c=0.8272$, see Figure
9(a). The stability analysis described in Section 3 with $\mu=0.0$, i.e. only
considering perturbations with the same as the unperturbed solution, led to a
single unstable perturbation with $\lambda=0.1540$, see Figure 9(b). The
perturbations shown in Figures 8(b) and 9(b) are similar, but the perturbation
in the $\mu=0.0$ case effects all wave peaks in the exactly same manner. Using
equation (22) as an initial condition in this case led to the perturbation
growing exponentially with a rate of approximately $r=0.1530$. Similar checks
were made for a handful of other points including some with complex $\lambda$
values. The agreement between $\lambda$ (growth rate computed via the
stability analysis) and $r$ (growth rate measured by computing the time
evolution of the perturbed solution) was typically in the range of three to
five decimal places.

Figure 9: Plots of (a) one period of the solution to the CV-Whitham equation
with $L=2\pi$, $\Omega=0.40$, $H=0.5969$, and $c=0.827233$ and (b) the
corresponding most unstable perturbation, $v(x,0)$, when $\mu=0.0$.

Benjamin [3] showed that small-amplitude, periodic traveling waves are
unstable with respect to long-wavelength perturbations (i.e. the modulational
instability) if the wave number is greater than $k=1.363$ and are stable with
respect to long-wavelength perturbations otherwise. Hur & Johnson [16] proved
that this cutoff occurs at $k=1.146$ for the Whitham equation. Hur & Johnson
[17] derived a formula for the boundary between small-amplitude solutions of
the vor-Whitham equation that are stable and unstable with respect to the
modulational instability. This region represented by the gray area in Figure
10. Binswanger et al. [5] derived a general formula that can be used for a
wide variety of Whitham-type equations. This formula states that if

$n(0,k)\Psi^{\prime\prime}(k)<0,$ (23)

where

$n(0,k)=-k\left(\frac{(g^{\prime\prime}(0))^{2}}{\Psi^{\prime}(k)}+\frac{k(g^{\prime\prime}(0))^{2}}{2\Psi(k)-\Psi(2k)}+\frac{1}{2}g^{\prime\prime\prime}(0)\right),$
(24a)
$\Psi(k)=k\mathcal{K}-k\left(-\frac{\Omega}{2}+\sqrt{1+\frac{\Omega^{2}}{4}}\right),$
(24b) $g(y)=\frac{\alpha}{2}y^{2}+\frac{\beta}{3}y^{3},$ (24c)

then small-amplitude solutions of the CV-Whitham equation are stable with
respect to the modulational instability. Plots of the stability (with respect
to the modulational instability) regions in the $(\Omega,k)$-planes for the
CV-Whitham and vor-Whitham equations are included in Figure 10. The plots show
that the stability region for the CV-Whitham equation is slightly larger than
that of the vor-Whitham equation. Thus, the addition of the cubic nonlinear
term provides a small stabilizing effect. We numerically corroborated these
results by computing the stability of solutions near the cutoff boundaries.

Equations (23) and (24c) establish that when $\Omega=0$ in the CV-Whitham
equation, the stability cutoff occurs at $k=1.252$. This means that the CV-
Whitham cutoff is closer to the “exact” cutoff of $k=1.363$ than the $k=1.146$
cutoff obtained from the Whitham equation. Recall that the CV-Whitham equation
is obtained from a Taylor series expansion of equation (20) of Kharif & Abid
[19]. Unfortunately, a cutoff for that equation cannot be determined because
it does not meet the requirements of the Binswanger et al. [5] methodology.
However, if one were to generalize the Binswanger et al. work to higher-order
Taylor series approximations, we hypothesize that a cutoff limit of $k=1.363$
would be approached as the order of the Taylor series approximation is
increased.

Figure 10: Plots of the stability region in the $(\Omega,k)$-plane for the CV-
Whitham and vor-Whitham equations. The black region corresponds to
$(\Omega,k)$ values for which small-amplitude, periodic, traveling-wave
solutions to the CV-Whitham equation are stable with respect to the
modulational instability. The gray region shows where small-amplitude,
periodic, traveling-wave solutions to both the CV-Whitham and vor-Whitham
equations are stable.

Finally, we examine peaking by considering an initial condition of the form

$u(x,0)=a\cos\left(x+\frac{\pi}{2}\right)+\frac{3-\tanh^{2}(1)}{4\tanh^{3}(1)}a^{2}\cos\left(2x+\frac{\pi}{2}\right),$
(25)

where $a=0.20$ in the CV-Whitham equation with $\Omega=1.0$. Figure 11 shows
the evolution of this initial condition for a variety of $t$ values. Peaking
occurs shortly after $t=1.95$. As the peaking time is approached, both the
asymmetry and wave height of the solution increase. For $t$ values close to
peaking, the numerics show the incipient formation of a vertical jet at the
crest of the front of the wave. Increasing the value of $a$ in equation (25)
leads to earlier peaking times. Decreasing the value of $a$ leads to either
later peaking times or no peaking at all. Increasing the value of $\Omega$
leads to earlier peaking times, while decreasing $\Omega$ leads to either
later peaking times or no peaking at all.

Figure 11: Plots of $u(x,t)$ versus $x$ for a variety of $t$ values for the
initial condition given in equation (25) with $a=0.20$.

##  4 Summary

We have examined periodic traveling-wave solutions of the cubic-vortical
Whitham equation. We showed numerically that the addition of a cubic nonlinear
term allows the solutions to achieve larger wave height. We showed that
increasing vorticity (decreasing $\Omega$) also leads to solutions with larger
amplitudes. We showed that all moderate-amplitude solutions are unstable and
that some large-amplitude solutions are unstable with respect to perturbations
with the same wavelength as the solution. We showed that small-amplitude
solutions of the CV-Whitham equation are stable with respect to the
modulational instability for larger ranges of vorticity and wavelength than in
the vortical Whitham equation. Finally, we showed that the CV-Whitham equation
more accurately reproduces the $kh=1.363$ modulational instability cutoff than
does the Whitham equation.

##  5 Acknowledgements

We thank Bernard Deconinck for helpful discussions. This material is based
upon work supported by the National Science Foundation under Grant No.
DMS-1716120 (JDC). Malek Abid and Christian Kharif acknowledge the funding by
the Excellence Initiative of Aix-Marseille University—A*Midex, a French
“Investissements d’Avenir programme” AMX-19-IET-010. Additionally, this work
was supported by a CIRM Research in Pairs Grant that allowed JDC and HK to
visit CK and MA in Luminy, France.

## References

  * [1] M.J. Ablowitz and H. Segur.  Solitons and the Inverse Scattering Transform.  SIAM, Philadelphia, 1981.
  * [2] P. Aceves-Sánchez, A.A. Minzoni, and P. Panayotaros.  Numerical study of a nonlocal model for water-waves with variable depth.  Wave Motion, 50:80–93, 2013.
  * [3] T.B. Benjamin.  Instability of periodic wavetrains in nonlinear dispersive systems.  Proceedings of the Royal Society of London, Series A, 299(1456):59–76, 1967.
  * [4] T.B. Benjamin and J.E. Feir.  The disintegration of wave trains on deep water: Part I. Theory.  Journal of Fluid Mechanics, 27:417–430, 1967.
  * [5] A.L. Binswanger, M.A. Hoefer, B. Ilan, and P. Sprenger.  Whitham modulation theory for generalized Whitham equations and a general criterion for modulational instability.  Studies in Applied Mathematics, 2021.
  * [6] N. Bottman and B. Deconinck.  KdV cnoidal waves are linearly stable.  Discrete and Continuous Dynamical Systems A, 25:1163–1180, 2009\.
  * [7] J.D. Carter.  Bidirectional Whitham equations as models of waves on shallow water.  Wave Motion, 82:51–61, 2018.
  * [8] J.D. Carter and M. Rozman.  Stability of periodic, traveling-wave solutions to the capillary-Whitham equation.  Fluids, 4:58, 2019.
  * [9] B. Deconinck and J.N. Kutz.  Computing spectra of linear operators using Hill’s method.  Journal of Computational Physics, 219:296–321, 2006.
  * [10] B. Deconinck and O. Trichtchenko.  High-frequency instabilities of small-amplitude Hamiltonian PDEs.  Discrete and Continuous Dynamical Systems, 37(3):1323–1358, 2015\.
  * [11] E. Dinvay, D. Moldabayev, D. Dutykh, and H. Kalisch.  The Whitham equation with surface tension.  Nonlinear Dynamics, 88:1125–1138, 2017.
  * [12] M. Ehrnström and H. Kalisch.  Traveling waves for the Whitham equation.  Differential and Integral Equations, 22:1193–1210, 2009.
  * [13] M. Ehrnström and E. Wahlén.  On Whitham’s conjecture of a highest cusped wave for a nonlocal dispersive equation.  Annales de l’Institut Henri Poincare. Analyse non linéar, 36:769–799, 2019.
  * [14] J. Hammack.  A note on tsunamis: their generation and propagation in an ocean of uniform depth.  Journal of Fluid Mechanics, 60:769–799, 1973.
  * [15] J.L. Hammack and H. Segur.  The Korteweg-de Vries equation and water waves. Part 2. Comparison with experiments.  Journal of Fluid Mechanics, 65:289–314, 1974.
  * [16] V.M. Hur and M.A. Johnson.  Modulational instability in the Whitham equation for water waves.  Studies in Applied Mathematics, 134(1):120–143, 2015.
  * [17] V.M. Hur and M.A. Johnson.  Modulational instability in the Whitham equation with surface tension and vorticity.  Nonlinear Analysis, 129:104–118, 2015.
  * [18] V.M. Hur and A.K. Pandey.  Modulational instability in a full-dispersion shallow water model.  Studies in Applied Mathematics, 142:3–47, 2019.
  * [19] C. Kharif and M. Abid.  Nonlinear water waves in shallow water in the presence of constant vorticity: A Whitham approach.  European Journal of Mechanics B/ Fluids, 72:12–22, 2018.
  * [20] C. Kharif, M. Abid, J.D. Carter, and H. Kalisch.  Stability of periodic progressive gravity wave solutions of the Whitham equation in the presence of vorticity.  Physics Letters A, 384:126060, 2020.
  * [21] C. Kharif, M. Abid, and J. Touboul.  Rogue waves in shallow water in the presence of a vertically sheared current.  Journal of Ocean Engineering and Marine Energy, 3:301–308, 2017\.
  * [22] D. Lannes.  The Water Waves Problem: Mathematical Analysis and Asymptotics.  American Mathematical Society, 2013.
  * [23] J.W. McLean.  Instabilities of finite-amplitude gravity waves on water of finite depth.  Journal of Fluid Mechanics, 114:331–341, 1982.
  * [24] J.W. Miles.  The Korteweg-de Vries equation, a historical essay.  Journal of Fluid Mechanics, 106:131–147, 1981.
  * [25] D. Moldabayev, H. Kalisch, and D. Dutykh.  The Whitham equation as a model for surface water waves.  Physica D, 309:99–107, 2015.
  * [26] J.S. Russell.  Report on waves.  Report of the fourteenth meeting of the British Association for the Advancement of Science, York, pages 311–390, 1844.
  * [27] N. Sanford, K. Kodama, J.D. Carter, and H. Kalisch.  Stability of traveling wave solutions to the Whitham equation.  Physics Letters A, 378:2100–2107, 2014.
  * [28] S. Trillo, M. Klein, G.F. Clauss, and M. Onorato.  Observation of dispersive shock waves developing from initial depressions in shallow water.  Physica D, 333:276–284, 2016.
  * [29] G.B. Whitham.  Variational methods and applications to water waves.  Proceedings of the Royal Society of London, A, 299:6–25, 1967.
  * [30] G.B Whitham.  Linear and Nonlinear Waves.  John Wiley & Sons, Inc., New York, 1974.
  * [31] N.J. Zabusky and C.J. Galvin.  Shallow-water waves, the Korteweg-de Vries equation and solitons.  Journal of Fluid Mechanics, 47:811–824, 1971.