# DE-Net: Dynamic Text-guided Image Editing Adversarial Networks

Ming Tao1 Bing-Kun Bao1 Hao Tang2 Fei Wu1 Longhui Wei3 Qi Tian3
1Nanjing University of Posts and Telecommunications 2CVL, ETH Zürich 3Huawei
Inc.

###### Abstract

Text-guided image editing models have shown remarkable results. However, there
remain two problems. First, they employ fixed manipulation modules for various
editing requirements (e.g., color changing, texture changing, content adding
and removing), which results in over-editing or insufficient editing. Second,
they do not clearly distinguish between text-required and text-irrelevant
parts, which leads to inaccurate editing. To solve these limitations, we
propose: (i) a Dynamic Editing Block (DEBlock) which composes different
editing modules dynamically for various editing requirements. (ii) a
Composition Predictor (Comp-Pred) which predicts the composition weights for
DEBlock according to the inference on target texts and source images. (iii) a
Dynamic text-adaptive Convolution Block (DCBlock) which queries source image
features to distinguish text-required parts and text-irrelevant parts.
Extensive experiments demonstrate that our DE-Net achieves excellent
performance and manipulates source images more correctly and accurately. Code
is available at https://github.com/tobran/DE-Net.

##  1 Introduction

The last few years have witnessed the great success of Generative Adversarial
Networks (GANs) [5] for a variety of applications [23, 33, 8, 9, 24]. Among
them, image editing is one of the important applications of GANs. Recently,
some works tried to bridge the gap between the natural language and image
editing [4, 16, 15, 7, 11, 12, 17, 32]. Different from other image editing
tasks, text-guided image editing aims to translate source images to target
images according to given text descriptions. Due to the advantages of natural
language, the text guidance makes the image editing process more controllable
and direct. Moreover, owing to its promising applications, this task has
attracted many researchers and achieved many significant progresses [4, 17,
11, 12].

Figure 1: Benefiting from dynamic editing design, our DE-Net can deal with
various editing tasks (e.g., color changing, content editing). Furthermore,
the text-adaptive convolution in DCBlock enables more accurate manipulations.

However, current text-guided image editing models have two problems. First,
they tackle various editing requirements equally with fixed manipulation
modules. Different from other image-to-image tasks (e.g., image colorization,
style transfer, and image inpainting), which design effective models for
specific tasks, text-guided image editing is an open-target task. It contains
various editing tasks (e.g., color changing, texture changing, content adding,
and removing) prompted by various text guidance. But current models ignore the
difference between varying editing requirements and employ fixed manipulation
modules for all editing tasks. It leads the whole network tends to make a
trade-off for different editing tasks. These compromises limit the
manipulation ability and result in over-editing or insufficient editing. As
the result shown in Figure 1, previous models [11, 12] fail to deal with
different editing requirements.

Second, the editing preciseness is deficient, as current manipulation modules
cannot clearly distinguish between text-required and text-irrelevant parts.
Text-visual feature concatenations [4, 17] and ACM module [11, 12] are widely
used to manipulate image features. However, naive concatenation treats text-
required and text-irrelevant parts equally, resulting in text information
overwriting the source image features. And the ACM module predicts editing
parameters from the encoded source images directly, without introducing text
information, which makes it unable to accurately edit text-required parts and
preserve text-irrelevant parts. As the result shown in Figure 1, the previous
model [11, 12] diffuses the text information to the whole image.

To address these problems, we propose a novel Dynamic text-guided image
Editing adversarial Network (i.e., DE-Net). Different from previous works [4,
17, 11, 12] which employ fixed editing blocks, our DE-Net employs Dynamic
Editing Block (DEBlock). The DEBlock combines spatial- and channel-wise
manipulations dynamically for different editing requirements. Furthermore, we
propose the Composition Predictor (Comp-Pred) to predict the composition
weights of DEBlock. The DEBlock and Comp-Pred enable our network can deal with
various editing tasks through suitable combinations of spatial- and channel-
wise manipulations. In addition, to improve the editing preciseness of the
manipulation module, we propose a novel Dynamic text-adaptive Convolution
Block (DCBlock). The DCBlock employs a dynamic text-adaptive convolutional
layer that can automatically adjust the kernel weights according to the given
text guidance. This enables our model to distinguish text-required and text-
irrelevant image features through querying the visual features according to
text guidance. Moreover, we add a lightweight semantic decoder that provides
global-local visual features for DCBlock to obtain a more accurate prediction
of editing parameters.

Overall, our contributions can be summarized as follows:

  * •

We propose a novel Dynamic text-guided Editing Block (DEBlock) to enable our
model can deal with a variety of editing tasks adaptively through the dynamic
composition of different editing modules.

  * •

We propose a novel Composition Predictor (Comp-Pred), which predicts the
composition weights for DEBlock according to the inference on text and visual
features.

  * •

We propose a novel Dynamic text-adaptive Convolution Block (DCBlock) which can
distinguish text-required parts and text-irrelevant parts of source images.

  * •

Compared with the current state-of-the-art methods, our DE-Net achieves much
better performance on commonly used public datasets.

##  2 Related Work

Text-to-Image Synthesis bridges the text information and image generation.
Based on Generative Adversarial Network (GAN) [5], some researchers tried to
bridge the text and image modality. [21] first employed the conditional
Generative Adversarial Network (cGAN) to synthesize plausible images from
given text descriptions. [34, 35] proposed the stacked architecture, which
stacks multiple generators and discriminators to synthesize high-resolution
images. [31] introduced the attention mechanism to fuse the word information
and image features. This text-image attention mechanism has been widely
applied in many text-related generative works [36, 2]. [25] proposed the DF-
GAN with deep fusion block and matching-aware gradient penalty, which enables
one-stage high-quality generation. Recently, some large transformer-based
text-to-image methods [20, 13, 3] show excellent performance on complex image
synthesis. They tokenize the images and take the image tokens and word tokens
to make auto-regressive training by a unidirectional transformer [19, 1].

Text-Guided Image Editing aims to manipulate source images according to given
text descriptions. For example, [4] first proposed SISGAN based on an encoder-
decoder architecture. It concatenates the encoded representations with text
semantics to manipulate source images. The TAGAN [17] introduced a word-level
attention mechanism in a discriminator to classify fine-grained attributes
independently. However, both SISGAN [4] and TAGAN [17] only preserve coarse
visual features from source images. [11] proposed the ManiGAN, which is a two-
stage architecture with the ACM and DCM modules. The ACM module synthesizes
text-matching images while preserving a rough shape from source images. And
the DCM module rectifies mismatched image features and completes missing
details. Then, [12] proposed a more efficient word-level discriminator to
facilitate training a lightweight generator. Recently, the ManiTrans [28]
employ the extra knowledge of pre-training autoregressive transformer for
text-guided image editing. These methods have shown significant progress in
text-guided image editing tasks. Recently, some text-guided image editing
works focus on face editing [29, 30, 18]. They employ a pretrained StyleGAN
[8] and manipulate its latent according to face descriptions.

In this work, we focus on the main research direction of the text-guided image
editing task, which deals with open-target image editing [4, 17, 11, 12].
Unlike previous models, our proposed DE-Net employs a Dynamic Editing Block
(DEBlock) with a Composition Predictor (Comp-Pred) to combine the editing
processes adaptively for various editing requirements. Moreover, we equip our
DE-Net with a Dynamic text-adaptive Convolution Block (DCBlock) which can
distinguish between text-required and text-irrelevant image features. Compared
with previous works, our model edits the text-required image features more
efficiently and preserves text-irrelevant source image features more
accurately.

##  3 The Proposed DE-Net

Figure 2: The architecture of the proposed DE-Net. The DE-Net comprises a
source image encoder, a target image decoder, a semantic decoder, a
Composition Predictor (Comp-Pred), and a pretrained text encoder [31]. The
DEBlock and DCBlock are introduced in the target image decoder to enable
effective and accurate manipulations, respectively.

In this paper, we propose a novel GAN model for text-guided image editing
named Dynamic text-guided Image Editing adversarial Networks (DE-Net) (see
Figure 2). To manipulate source images according to text guidance more
effectively and accurately, we propose: (i) a novel manipulation module called
Dynamic Editing Block (DEBlock), which manipulates the image features
adaptively through the dynamic combination between spatial and channel-wise
editing. (ii) a novel Composition Predictor (Comp-Pred) that can predict the
combination weights according to the inference on text and visual features.
(iii) a novel Dynamic text-adaptive Convolution Block (DCBlock) that improves
the editing preciseness through querying the visual features according to text
guidance. In the following of this section, we first present the overall
structure of our DE-Net and then introduce the Comp-Pred, DEBlock, and DCBlock
in detail.

###  3.1 Dynamic Text-guided Image Editing Framework

Different from previous models [4, 17, 11, 12, 28], our DE-Net is based on a
novel dynamic text-guided image editing framework. First, it can compose
suitable editing processes based on input images and texts dynamically.
Second, it can distinguish between text-required and text-irrelevant image
features through dynamic text-adaptive convolution.

As shown in Figure 2, the whole editing framework is composed of a source
image encoder, a target image decoder, a Composition Predictor (Comp-Pred), a
semantic decoder, and a pre-trained text encoder [11, 12]. To introduce multi-
scale source visual features for manipulation and preservation, we add full
skip connections between the source image encoder and two decoders. In DE-Net,
the source image is first encoded to $4{\times}4$ resolution by the source
image encoder. Then, the encoded source image feature is fed into the semantic
decoder and target image decoder, respectively. The semantic decoder comprises
several upsampling and convolutional layers (UPBlock). It can capture long-
range visual information and decode the global-local semantic features for the
DCBlock through the growing size of semantic features. The target image
decoder is composed of multiple DEBlocks and DCBlocks. The DEBlock manipulates
source image features both from spatial and channel dimensions and dynamically
combines these two manipulations at different image scales. And the Comp-Pred
predicts the combination weights for DEBlock according to the inference on
text and visual features. The DCBlock adjusts its candidate convolution
kernels and applies text-adaptive convolution on global-local semantic
features provided by the semantic decoder. It enables our DE-Net to
distinguish between text-required and text-irrelevant parts.

To stabilize the training process of adversarial learning, we further
introduce the target-aware discriminator [25] in DE-Net. Finally, our DE-Net
can be formulated as:

$\begin{split}L_{D}=&-\mathbb{E}[min(0,-1+D(x,e))]\\\
&-(1/2)\mathbb{E}[min(0,-1-D(G(x,\hat{e}),\hat{e}))]\\\
&-(1/2)\mathbb{E}[min(0,-1-D(x,\hat{e}))]\\\
&+k\mathbb{E}[(\|\nabla_{x}D(x,e)\|+\|\nabla_{e}D(x,e)\|)^{p}],\\\
L_{G}=&-\mathbb{E}[D(G(x,\hat{e}),\hat{e})]+\lambda_{1}\mathbb{E}[\|G(x,\hat{e})-x\|_{2}]\\\
&-\lambda_{2}\mathbb{E}[S(G(x,\hat{e}),\hat{e})],\\\ \end{split}$ (1)

where $x$ denotes the source images; $e$ denotes the source sentence
embeddings, $\hat{e}$ denotes the target sentence embeddings, $G$ denotes the
generator network, $D$ denotes the discriminator network, $k$ and $p$ are two
hyperparameters of discriminator to balance the effectiveness of gradient
penalty. $S$ represents the cosine similarity between the encoded visual and
text features predicted by pre-trained DAMSM network [31, 11, 12].
$\lambda_{1}$ and $\lambda_{2}$ are two balancing coefficients of the
generator.

###  3.2 Composition Predictor

Figure 3: Illustration of the proposed Comp-Pred and DCBlock. (a) The Comp-
Pred predicts the combination weights for DEBlock. (b) The DEBlock composes
suitable editing processes for each input pair according to the combined
weights of C-Affine and S-Affine.

To enable our DE-Net to compose suitable editing processes based on current
source images and text guidance, we propose the Composition Predictor (Comp-
Pred). As shown in Figure 2 and Figure 3, for visual information, it takes the
encoded $4{\times}4$ image feature and maps it to the visual vector $\bm{V}$
through two convolution layers. For text information, it takes the sentence
embedding provided by the text encoder, and maps it to a text vector $\bm{T}$
through one-hidden-layer MLP. Then we concatenate the $\bm{V}$ and $\bm{T}$
and get the $\bm{V^{\prime}}$ and $\bm{T^{\prime}}$ through an MLP to close
the gap between the visual and text domain. The difference
$\bm{V^{\prime}}-\bm{T^{\prime}}$ and the element-wise product
$\bm{V^{\prime}}*\bm{T^{\prime}}$ are calculated and concatenated with the
original vectors $\bm{V}$ and $\bm{T}$ to enhance the distance between text
and visual information. Lastly, we adopt an MLP and a Sigmoid function to
predict the combination weights $\bm{\alpha}$ of spatial-wise and channel-wise
editing for $N$ DEBlocks in the target image decoder. The $\bm{\alpha}_{n}$ is
a vector and its length is the channel size of the visual feature after
convolution layer in each DEBlock.

###  3.3 Dynamic Text-guided Image Editing Block

As mentioned before, the existing naive concatenation [4, 17] or ACM module
[17, 12] cannot make an efficient interaction between text guidance and source
image features. Moreover, these manipulation modules treat various editing
requirements equally, ignoring the different model requirements. To address
these problems, we first decompose the visual feature editing process into
spatial and channel-wise editing. Then, we propose a novel DEBlock to re-
compose suitable editing processes for each input pair through the dynamic
combination of spatial and channel-wise affine transformation.

As shown in Figure 3(a), the DEBlock is composed of upsampling layer,
convolution layer, and two kinds of affine layers. One is Channel-wise Affine
layer (C-Affine), the other is Spatial Affine layer (S-Affine). To fully
manipulate source image features, our DEBlock manipulates both spatial and
channel dimensions and combines these two manipulations dynamically through
combination weights $\bm{\alpha}_{n}$ predicted by Comp-Pred. Specifically, in
C-Affine, the text sentence vector $\bm{t}$ is sent to two different one-
hidden-layer MLPs to predict the scaling parameters $\bm{\gamma}_{c}$ and
shifting parameters $\bm{\theta}_{c}$ for each channel:

$\bm{\gamma}_{c}={\rm MLP}_{1}(\bm{t}),\qquad\bm{\theta}_{c}={\rm
MLP}_{2}(\bm{t}).$ (2)

After obtaining $\bm{\gamma}_{c}$ and $\bm{\theta}_{c}$ parameters from MLPs,
the C-Affine transformation can be formally expressed as follows:

${\rm
C{-}Aff}(\bm{x}_{c,h,w}|\bm{t})=\bm{\gamma}_{c}\cdot{\bm{x}_{c,h,w}}+\bm{\theta}_{c}.$
(3)

where $\bm{x}$ is the image feature; $c$, $h$, and $w$ denotes the channel
size, feature height, and feature width, respectively.

Unlike the C-Affine, the S-Affine applies affine transformations on the
spatial dimension of image features. The spatial scaling parameters
$\bm{\gamma}_{h,w}$ and shifting parameters $\bm{\theta}_{h,w}$ are predicted
by the DCBlock and two convolution layers for each pixel in image feature. As
shown in Figure 4, the DCBlock employs the text-adaptive convolution layer to
get the text-attended visual features $\bm{f}$. Then, the spatial scaling and
shifting parameters are predicted by two convolution layers on the predicted
text-attended visual features:

$\bm{\gamma}_{h,w}={\rm Conv}_{1}(\bm{f}),\qquad\bm{\theta}_{h,w}={\rm
Conv}_{2}(\bm{f}).$ (4)

Thus, the S-Affine can be formally expressed as follows:

${\rm
S{-}Aff}(\bm{x}_{c,h,w}|\bm{t})=\bm{\gamma}_{h,w}\cdot{\bm{x}_{c,h,w}}+\bm{\theta}_{h,w}.$
(5)

To fully manipulate source image features, our model integrates the C-Affine
and S-Affine in DEBlock. And the $n$th DEBlock in the target image decoder
combines these two manipulations dynamically through the combination weights
$\bm{\alpha}_{n}$ predicted by Comp-Pred:

$\displaystyle{\rm{DEBLK}}(\bm{x}|\bm{t})=$ $\displaystyle{\rm
C{-}Aff}(\bm{x}|\bm{t})\cdot{\bm{\alpha}_{n}}$ (6) $\displaystyle+{\rm
S{-}Aff}(\bm{x}|\bm{f})\cdot({1-\bm{\alpha}_{n}}).$

Compared with previous manipulation modules, our DEBlock fully manipulates
source image features both from spatial and channel dimensions and considers
different contributions of these two dimensions at different image scales.
Moreover, our experiments and ablation studies also demonstrate the
effectiveness of the DEBlock.

###  3.4 Dynamic Text-adaptive Convolution Block

Figure 4: Illustration of the DCBlock. The text-attended features are
predicted by the text-conditioned convolution layer.

To improve the editing preciseness of the manipulation module, we propose the
DCBlock as an editing positioning module. As shown in Figure 4, the weights of
convolution kernels in DCBlock are adjusted according to the given text
guidance. However, directly generating all parameters of the convolution
kernel not only requires a high computational cost but also results in
overfitting to train text descriptions. To address this problem, we apply
scaling and shifting operations for each candidate convolution kernel. First,
we employ two MLPs to predict the scaling parameters $\bm{\varphi}_{c\\_in}$
and shifting parameters $\bm{\omega}_{c\\_in}$ for the convolution kernel. The
modulation on one convolution kernel can be formally expressed as follows:

${\rm
Mod}(\bm{k})=\bm{\varphi}_{c\\_in}\cdot{\bm{k}_{c\\_in,k\\_h,k\\_w}}+\bm{\omega}_{c\\_in},$
(7)

where $\bm{k}_{c\\_in,k\\_h,k\\_w}$ is the convolution kernel; $c\\_in$,
$k\\_h$, and $k\\_w$ denotes the input channel size, kernel height, and kernel
width, respectively.

With this text-guided adjustment for candidate convolution kernels, the
dynamic convolutional (Dy-Conv) operation acts as a querying process according
to text descriptions. However, as the semantic information in one convolution
kernel is limited, it is hard to incorporate the whole text semantic
information into one convolution kernel. Inspired by the multi-head attention
mechanism in Transformer [26], we employ multiple dynamic convolution kernels
as different attention heads. In our work, we set 8 dynamic convolution
kernels in DCBlock. It allows our model to jointly attend to different image
features from different text representation subspaces. As shown in Figure 4,
each dynamic convolution kernel predicts an attended feature map according to
partial text semantics. Then, we stack these attention features along the
channel direction as a whole attended feature map for predicting the spatial
affine parameters for DEBlock.

However, directly applying Dy-Conv operations on image features from the
source image encoder is inefficient. Each pixel of image features in the
source image encoder only contains the semantic information of its local
neighborhood through downsampling layers. It cannot capture long-range
contextual information, resulting in low accuracy and efficiency of Dy-Conv.
To cope with this problem, we employ a semantic decoder to decode global-local
semantics, as shown in Figure 2. The source image is first downsampled through
the image encoder and then upsampled by the semantic decoder with full skip
connections from the source image encoder. Through this encoder-decoder
structure with full skip-connections, the semantic decoder can capture long-
range visual information, and each pixel of decoded semantic features contains
global-local semantic information. We apply Dy-Conv operations on the global-
local semantic features to achieve more accurate queried results. After that,
we employ two convolutional layers to predict the spatial scaling and shifting
parameters for DEBlock.

The proposed DCBlock enables our DE-Net to distinguish between text-required
and text-irrelevant parts. Furthermore, the semantic decoder provides the
global-local semantic features, and the multi-head dynamic convolutional
operations query the global-local semantic features from different text
representation subspaces. Armed with the DCBlock, the target image decoder can
manipulate source image features more accurately.

##  4 Experiments

Figure 5: Qualitative comparison between different methods on the test set of
CUB and COCO.

In this section, we introduce the datasets, training details, and evaluation
metrics used in our experiments. Then we compare the text-guided image editing
performance with previous models quantitatively and qualitatively.

Datasets. We conduct experiments on two challenging datasets: CUB bird [27]
and COCO [14]. For the CUB bird dataset, there are 11,788 images belonging to
200 bird species, with each image corresponding to ten language descriptions.
For the COCO dataset, it contains 80k images for training and 40k images for
testing. Each image corresponds to 5 language descriptions.

Training and Evaluation Details. We employ the Adam optimizer [10] with
$\beta_{1}{=}0.0$ and $\beta_{2}{=}0.9$ to train our model. According to the
Two Timescale Update Rule (TTUR) [6], the learning rate is set to 0.0001 for
the generator and 0.0004 for the discriminator. The hyper-parameters of the
discriminator $k$ and $p$ are set to 2 and 6 as [25]. The hyper-parameters of
the generator $\lambda_{1}$ and $\lambda_{2}$ are set to 40 and 4 for all the
datasets.

Following previous works [28, 11], we employ the Inception Score (IS) [22],
CLIP-score, L2-error, and Manipulation Precision (MP) to evaluate the
performance. Higher IS means higher fidelity of the edited images. CLIP-score
is the text-image cosine similarity calculated by a pretrained CLIP network
[31] to measure the quality of text-guided manipulation. L2-error is the
$L_{2}$ pixel difference between edited images and source images to measure
the quality of source feature preserving. Lastly, MP simultaneously measures
the quality of text-guided manipulation and source feature preserving. It can
be formulated as:

${\rm MP}=(1-{\rm L2\\_error})\times{\rm(CLIP\\_score)}.$ (8)

It must be pointed out that the Fréchet Inception Distance (FID) [6] widely
adopted in text-to-image synthesis is not suitable for text-guided image
editing. The FID calculates the Fréchet distance between synthetic images and
ground truth in the feature space of a pre-trained Inception v3 network. But
there is no ground truth (human edited images) in this task, and calculating
with source images makes the FID prefer lazy models which do not edit input
images. So we adopt the IS to evaluate the image fidelity as previous works
[28, 11].

Model | CUB | COCO
---|---|---
IS $\uparrow$ |  CLIP-score $\uparrow$ |  L2-error $\downarrow$ |  MP $\uparrow$ |  IS $\uparrow$ |  CLIP-score $\uparrow$ |  L2-error $\downarrow$ |  MP $\uparrow$
TAGAN | 3.72 | 0.202 | 0.185 | 0.164 | 9.33 | 0.125 | 0.102 | 0.112
ManiGAN | 4.19 | 0.213 | 0.051 | 0.202 | 22.60 | 0.119 | 0.031 | 0.114
DF-GAN | 4.54 | 0.221 | 0.046 | 0.210 | 20.10 | 0.129 | 0.027 | 0.125
Lightweight-GAN | 4.66 | 0.188 | 0.132 | 0.163 | 24.80 | 0.136 | 0.025 | 0.132
ManiTrans | 5.02 | 0.235 | 0.013 | 0.231 | 21.45 | 0.131 | 0.017 | 0.128
DE-Net (Ours) | 5.08 | 0.240 | 0.010 | 0.237 | 25.81 | 0.192 | 0.015 | 0.189

Table 1: Quantitative comparison between different methods on the test set of
CUB and COCO.

###  4.1 Quantitative Comparisons

We compare the proposed method with several state-of-the-art text-guided image
editing models, including TAGAN [17], ManiGAN [11], ManiTrans [28]. We also
modify the state-of-the-art text-to-image model DF-GAN [25] and compare our
model with it. As shown in Table 1, our proposed DE-Net achieves the highest
IS, CLIP-score, MP, and the lowest L2-error on the CUB and COCO test dataset.
The highest IS shows that our method can produce more realistic manipulated
results. And the highest CLIP-score, MP, and lowest L2-error prove that our
method can manipulate text-required image parts and preserve text-irrelevant
parts more correctly and accurately. The extensive quantitative evaluation
results on CUB and COCO demonstrate the superiority and effectiveness of our
proposed DE-Net. Furthermore, the advantages of our model are more obvious in
the complex COCO images. It demonstrates the significant improvements of our
DE-Net when dealing with various editing requirements, e.g., textual changing,
background changing, object adding and removing.

###  4.2 Qualitative Comparisons

In this subsection, we compare the images synthesized by ManiGAN [11],
Lightweight GAN [12], and our DE-Net. We evaluate the quality of the edited
images from two aspects, text-related source feature manipulation and text-
irrelevant source feature preservation. First, we compare the quality of the
edited images on the CUB dataset, then compare the results on the more
challenging COCO dataset.

As the results shown in 1st, 2nd, and 4th columns, both ManiGAN [11] and
Lightweight GAN [12] cannot accurately synthesize the “black wings”, “yellow
throat”, and “white belly”, respectively. Moreover, they tend to change the
color of twigs and backgrounds (see 1st, 3rd, 4th, and 5th columns). While our
DE-Net can manipulate these features successfully and preserve the text-
irrelevant image features like twigs and backgrounds more accurately. The
superiority is more obvious on the challenging COCO dataset, which contains
various editing tasks. As the results shown in the 6th, 7th, and 10th columns,
both ManiGAN [11] and Lightweight GAN [12] cannot synthesize the “a herd of
sheep”, “grassy field”, and “some pictures”, respectively. These results show
that they only tend to edit the colors of source images (see 8th, 9th, and
10th columns). However, our DE-Net can both edit the color (see 6th, and 7th
columns), texture (see 7nd and 8th columns) of source images, and synthesize
new contents in target images (see 6th and 10th columns). Moreover, both
ManiGAN [11] and Lightweight GAN [12] tend to edit the text-irrelevant
regions. For example, the color of the sky is also painted with the color of
“grassy field” (see 7th column). While our DE-Net can preserve these text-
irrelevant features or manipulate them to adapt to the text description (see
6th, and 7th columns).

The qualitative comparison on both CUB and COCO datasets shows that our DE-Net
can manipulate the source images according to the text guidance while
preserving text-irrelevant image features more effectively and accurately.

###  4.3 Ablation Study

To verify the effectiveness of different components in DE-Net, we conduct
ablation studies on the COCO test dataset. The components include Dynamic
Text-guided Image Editing Block (DE), Composition Predictor (CP), Dynamic
Text-adaptive Convolution Block (DC), and the Semantic Decoder (SD). The
results are provided in Table 2.

Baseline.  Our baseline is based on an encoder-decoder architecture that
employs the target-way discriminator [25]. The baseline concatenates the text
features with the encoded image features.

Effect of DEBlock (DE).  The experimental results show that the DEBlock
achieves better performances than only employing channel-wise (B w/CAFF) or
spatial-wise affine transformation (B w/SAFF). The results demonstrate that
our Dynamic Editing strategy, which composes suitable editing processes
dynamically based on input text-image pairs, is more effective than fixed
editing modules.

Effect of Comp-Pred (CP).  Removing the Comp-Pred (B w/ DE w/o CP) reduces the
editing performance. To further evaluate the effectiveness of the difference
and element-wise product in Comp-Pred, we remove them from CP and name the new
module CP*. Compared with CP*, the Comp-Pred (B w/ DE w/ CP) achieves better
performances. The comparison results prove the effectiveness of Comp-Pred.

Method |  IS $\uparrow$ |  CLIP-score $\uparrow$ |  L2 $\downarrow$ |  MP $\uparrow$
---|---|---|---|---
Baseline (B) | 11.08 | 0.115 | 0.057 | 0.108
B w/ CAFF | 12.16 | 0.124 | 0.051 | 0.117
B w/ SAFF | 13.33 | 0.131 | 0.047 | 0.124
B w/ DE w/ CP | 17.21 | 0.155 | 0.031 | 0.150
B w/ DE w/o CP | 15.36 | 0.141 | 0.040 | 0.135
B w/ DE w/ CP* | 14.13 | 0.145 | 0.039 | 0.139
B w/ DE w/ CP w/OC | 18.47 | 0.160 | 0.030 | 0.155
B w/ DE w/ CP w/DC1 | 20.21 | 0.171 | 0.025 | 0.166
B w/ DE w/ CP w/DC4 | 22.18 | 0.176 | 0.021 | 0.172
B w/ DE w/ CP w/DC16 | 21.01 | 0.179 | 0.021 | 0.175
B w/ DE w/ CP w/DC | 23.11 | 0.180 | 0.018 | 0.176
B w/ DE w/ CP w/DC w/SD (DE-Net) | 25.81 | 0.192 | 0.015 | 0.189

Table 2: Ablation Study of DE-Net on the test set of COCO.

Effect of DCBlock (DC).  To verify the effectiveness of DCBlock, we replace it
with an Ordinary Convolution layer (OC). The results show that the DCBlock
enables our model to manipulate more accurately (lower DIFF, higher MP). We
further evaluate the multi-head mechanism of DCBlock by changing the number of
text-adaptive convolution kernels to 1, 4, 16 (DC1, DC4, DC16). The results
show that more convolution kernels do not mean better performance. The reason
may be that more convolution kernels reduce the convergence efficiency of the
DE-Net. Therefore, we set 8 text-adaptive convolution kernels in DCBlock.

Effect of Semantic Decoder (SD).  Without the Semantic Decoder (B w/ DE w/ CP
w/DC), the model predicts the spatial editing parameters from the feature
provided by the source image encoder. Armed with the Semantic Decoder, the
model (B w/ DE w/ CP w/DC w/SD ) improves the IS, SIM, MP, and decreases the
DIFF. The results prove that the semantic decoder can promote the DCBlock to
make more effective and accurate manipulations.

###  4.4 Visualisation results of DCBlock

To further demonstrate the effectiveness of DCBlock, we visualize the text-
conditioned queried results at the last DCBlock in the target image decoder.
As shown in Figure 6, the kernel attends to different semantics in source
image features. In addition, we find that the convolution kernel can be
divided into two types, one focusing on text-related image parts and the other
focusing on text-irrelevant image parts. For the CUB example, kernel 8
segments the whole background, which is text-irrelevant image parts. The other
kernel attends on text-related image parts, e.g., “wings”, “head”, “belly”.
For the COCO example, kernel 8 segments the sky, which is text-irrelevant
image parts. The other kernel attends to text-related image parts, e.g.,
ground, human, and mountain, which need to be edited. It demonstrates that our
DCBlock can attend to different semantic parts and distinguish between text-
related and text-irrelevant parts.

##  5 Conclusion

Figure 6: Visualization of the queried results by different text-adaptive
convolution kernels.

In this paper, we propose a novel DE-Net for the text-guided image editing
task. Compared with previous models, our DE-Net can manipulate the source
images according to text guidance more correctly and accurately. Via DE-Net,
we propose a Dynamic text-guided Editing Block (DEBlock) to enable our model
can deal with a variety of editing tasks adaptively through the dynamic
composition of different editing modules. We also propose a novel Composition
Predictor (Comp-Pred) that compares the source image features and given text
guidance, and predict the combination weights for DEBlock. Furthermore, we
propose a new Dynamic text-adaptive Convolution Block (DCBlock) with a
semantic decoder to help the target image decoder distinguish between text-
required and text-irrelevant parts. Extensive experiment results demonstrate
that the proposed DE-Net significantly outperforms state-of-the-art models on
both CUB and COCO datasets.

## References

  * [1] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165, 2020.
  * [2] Jun Cheng, Fuxiang Wu, Yanling Tian, Lei Wang, and Dapeng Tao.  Rifegan: Rich feature generation for text-to-image synthesis from prior knowledge.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10911–10920, 2020.
  * [3] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  Advances in Neural Information Processing Systems, 34, 2021.
  * [4] Hao Dong, Simiao Yu, Chao Wu, and Yike Guo.  Semantic image synthesis via adversarial learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5706–5714, 2017.
  * [5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, pages 2672–2680, 2014.
  * [6] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in neural information processing systems, pages 6626–6637, 2017.
  * [7] Wentao Jiang, Ning Xu, Jiayun Wang, Chen Gao, Jing Shi, Zhe Lin, and Si Liu.  Language-guided global image editing via cross-modal cyclic mechanism.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2115–2124, 2021.
  * [8] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [9] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [10] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations, 2015.
  * [11] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr.  Manigan: Text-guided image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7880–7889, 2020.
  * [12] Bowen Li, Xiaojuan Qi, Philip HS Torr, and Thomas Lukasiewicz.  Lightweight generative adversarial networks for text-guided image manipulation.  arXiv preprint arXiv:2010.12136, 2020.
  * [13] Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al.  M6: A chinese multimodal pretrainer.  arXiv preprint arXiv:2103.00823, 2021.
  * [14] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [15] Xihui Liu, Zhe Lin, Jianming Zhang, Handong Zhao, Quan Tran, Xiaogang Wang, and Hongsheng Li.  Open-edit: Open-domain image manipulation with open-vocabulary instructions.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pages 89–106. Springer, 2020.
  * [16] Yahui Liu, Marco De Nadai, Deng Cai, Huayang Li, Xavier Alameda-Pineda, Nicu Sebe, and Bruno Lepri.  Describe what to change: A text-guided unsupervised image-to-image translation approach.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 1357–1365, 2020.
  * [17] Seonghyeon Nam, Yunji Kim, and Seon Joo Kim.  Text-adaptive generative adversarial networks: Manipulating images with natural language.  In Advances in neural information processing systems, pages 42–51, 2018.
  * [18] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085–2094, 2021.
  * [19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [20] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  arXiv preprint arXiv:2102.12092, 2021.
  * [21] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In Proceedings of the International Conference on Machine Learning, pages 1060–1069, 2016.
  * [22] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In Advances in neural information processing systems, pages 2234–2242, 2016.
  * [23] Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J Corso, and Yan Yan.  Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation.  In CVPR, 2019.
  * [24] Hao Tang, Dan Xu, Yan Yan, Philip HS Torr, and Nicu Sebe.  Local class-specific and global image-level generative adversarial networks for semantic-guided scene generation.  In CVPR, 2020.
  * [25] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu.  Df-gan: A simple and effective baseline for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16515–16525, 2022.
  * [26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [27] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [28] Jianan Wang, Guansong Lu, Hang Xu, Zhenguo Li, Chunjing Xu, and Yanwei Fu.  Manitrans: Entity-level text-guided image manipulation via token-wise semantic alignment and generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10707–10717, 2022.
  * [29] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.  Tedigan: Text-guided diverse face image generation and manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2256–2265, 2021.
  * [30] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.  Towards open-world text-guided face image generation and manipulation.  arXiv preprint arXiv:2104.08910, 2021.
  * [31] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [32] Zipeng Xu, Tianwei Lin, Hao Tang, Fu Li, Dongliang He, Nicu Sebe, Radu Timofte, Luc Van Gool, and Errui Ding.  Predict, prevent, and evaluate: Disentangled text-driven image manipulation empowered by pre-trained vision-language model.  In CVPR, 2022.
  * [33] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In International Conference on Machine Learning, pages 7354–7363, 2019.
  * [34] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 5907–5915, 2017.
  * [35] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas.  Stackgan++: Realistic image synthesis with stacked generative adversarial networks.  IEEE TPAMI, 41(8):1947–1962, 2018.
  * [36] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.  Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5802–5810, 2019.


# Jupiter’s Polar Vortex Crystals Explored using the Shallow Water Equations

Sihe Chen<EMAIL_ADDRESS>Planetary Science Department, California
Institute of Technology.  Andrew P. Ingersoll  Planetary Science Department,
California Institute of Technology.  Cheng Li  Department of Climate and Space
Sciences and Engineering, University of Michigan, Ann Arbor.

###### Abstract

At the poles of Jupiter, cyclonic vortices are clustered together in patterns
made up of equilateral triangles called vortex crystals. Such patterns are
seen in laboratory flows but never before in a planetary atmosphere, where the
planet’s rotation and gravity add new physics. Here we use the shallow water
(SW) equations at the pole of a rotating planet to study the emergence and
evolution of vortices starting from an initial random pattern of small-scale
turbulence. The flow is in a single layer with a free surface whose slope
produces the horizontal pressure gradient force. We explored three parameters
in the problem: the mean kinetic energy of the initial turbulence, the
horizontal scale of the initial turbulence, and the radius of deformation of
the undisturbed fluid layer. We find that some regions of this parameter space
lead to vortex crystals and others lead to chaotic behavior and mergers. Our
results identified that the relative change of the layer thickness is the key
quantity that determines whether the vortex crystal or chaotic patterns
emerge. In doing so we are learning about Jupiter’s atmosphere – the small-
scale turbulence and the vertical structure of the layer in which the cyclones
exist. We are also learning about basic physics since vortex crystals are a
fundamental phenomenon in fluid mechanics.

††preprint: AIP/123-QED

##  I Introduction

At each pole of Jupiter, a central cyclonic vortex is surrounded by a ring of
cyclonic vortices at a latitude of 83°, corresponding to a circle 8700 km in
radius. The individual cyclones have radii, as defined by their azimuthal
velocity maxima, of 1000 km. In the north, the ring has 8 circumpolar
cyclones, and in the south, the ring has 5 circumpolar cyclones. Thus the
central cyclone and the cyclones around it resemble approximately a two-
dimensional structure made up of equilateral triangles. Such patterns have
been studied in the laboratory and are appropriately called vortex crystals
Fine  _et al._ (1995); Schecter  _et al._ (1999); Siegelman, Young, and
Ingersoll (2022).

Here we use the shallow water (SW) equations at the pole of a rotating planet
to study the emergence and evolution of vortices starting from an initial
random pattern of small-scale turbulence. The flow is in a single layer, of
thickness h, with a free surface that is capable of propagating gravity waves.
We use a reduced-gravity modelChassignet and Cushman-Roisin (1991), where the
active layer is floating hydrostatically on a much deeper fluid layer at rest.
We use the reduced gravity $g_{r}=g\Delta\rho/\rho$, a constant, where $g$ is
the gravitational acceleration at the planet’s surface and $\Delta\rho/\rho$
is the fractional density difference between the two layers. Then the
geopotential, $\Phi=g_{r}h$, is one of the variables of the SW system, the
other being the velocity $\mathbf{u}$. The upper boundary of the active layer
is a free surface with zero pressure, so $-\nabla\Phi$ is the horizontal
pressure gradient force per unit mass within the layer. Since $\Phi$ is
proportional to the thickness, it obeys the horizontal continuity equation.
This, along with the horizontal momentum equation, the full set of equations
of the SW system is constituted:

$\displaystyle\frac{\partial\Phi}{\partial
t}+(\mathbf{u}\cdot\nabla)\Phi+\Phi\nabla\cdot\mathbf{u}=0$ (1)
$\displaystyle\frac{\partial\mathbf{u}}{\partial
t}+(\mathbf{u}\cdot\nabla)\mathbf{u}+f\hat{\mathbf{k}}\times\mathbf{u}=-\nabla\Phi$
(2)

We are modeling a thin horizontal layer on a rotating sphere, so the vertical
component of the Coriolis acceleration is what matters. Thus
$\hat{\mathbf{k}}$ is the vertical unit vector and $f=2\Omega\sin\phi$ is the
Coriolis parameter, where $\Omega$ is the planetary rotation rate and $\phi$
is latitude. In this paper, we use 3 parameters – the root mean square
velocity $U$ of the initial turbulence, the horizontal wavelength $L_{i}$ of
the initial turbulence, and the radius of deformation $L_{d}$ of the
undisturbed fluid layer. The planetary radius $a$ is 71,492 km, and the period
of rotation is 9h 55m 29s. These two numbers do not change, and serve to
define the units of length and time, respectively. Our results hold for other
planets when the units are re-scaled accordingly. Other approaches use planet
size as an additional parameterBrueshaber, Sayanagi, and Dowling (2019), but
our approach keeps the number of parameters to a minimum. We explore this 3D
parameter space and find that some regions lead to vortex crystals and others
lead to chaotic behavior and mergers.

One might ask if this single-layer SW model is realistic enough to capture all
the important parameters of Jupiter’s atmosphere: the nature of the small-
scale turbulence, e.g., whether it arises from convection, baroclinic
instability, breaking gravity waves, or breaking planetary waves; the
thickness and vertical structure of the layer where the cyclones exist; the
nature of dissipation and whether there are upscale and downscale cascades of
energy and so on. The answer is no, but we cannot achieve perfect realism.
There is no sufficient observational data for the poles, especially about
vertical structure. The current knowledge is related to small-scale turbulence
and inverse cascades, at least near cloud-top altitudesIngersoll  _et al._
(2022); Siegelman  _et al._ (2022). In the future, we are gaining information
from the Juno mission about the roots of the vortices and stable layers at
depth. This paper explores the range of parameters that produce vortex
crystals, providing insights to such measurements, and constraints to the
future modeling work of the polar region.

##  II Dimensionless numbers and length scales

Dimensionless numbers are useful in estimating the relative importance of
different terms in the equations. The Rossby number $U/(fL_{i})$, where
$f=2\Omega$ is the Coriolis parameter at the poles. It gives the importance of
fluid accelerations relative to the Coriolis acceleration, and the former
arises from motions on the scale of the initial disturbance. The individual
cyclones at Jupiter’s poles have peak azimuthal velocities of 80 m/s and peak
relative vorticities
$\zeta=\hat{\mathbf{k}}\cdot(\nabla\times\mathbf{u})\approx 3.0\times
10-4$s-1, the latter being almost equal to the planetary vorticity $f$ at the
pole of $3.5\times 10^{-4}$ s-1 Ingersoll  _et al._ (2022). The alternative
definition for the Rossby number is $\zeta/f$, which is approximately 1 for
the large cyclones and may be larger than 1 for the initial turbulence.

The reduced gravity $g_{r}$ and layer thickness $h$ appear in the equations
only as a single product, which is the variable $\Phi=g_{r}h$ and has
dimensions of velocity squared. The global mean value of $\Phi$ divided by
$f^{2}$ gives $L_{d}^{2}$, the square of the radius of deformation. It
measures the mean thickness and vertical structure of the fluid layer and is
the second free parameter that we vary. Large $L_{d}$ is related to vertical
stability. For a given velocity, large $L_{d}$ means that pressure surfaces
are nearly horizontal and the horizontal flow is nearly incompressible, as
seen in Eq. (1).

The third parameter that we vary is the horizontal scale $L_{i}$ of the
initial small-scale turbulence. As detailed in the next section, this
parameter determines the wavelengths of the initial turbulence, which is the
lengthscale of which the energy cascade starts. The Rossby number $U/(fL_{i})$
of the initial field in our experiments then ranges from 0.1 to 2.3.

The initial $dh/h$ is a combination of the three basic initial parameters $U$,
$L_{d}$, and $L_{i}$. It is the fractional departure of the mean surface
height from equilibrium, and when the value becomes of order 1, the shallow
water thickness can potentially become negative. This causes the sound speed
and hence the Courant–Friedrichs–Lewy (CFL) number to become undefined, and we
stop the simulation when this happens. Physically, this does not mean a
complete failure of the shallow water system: it could be trying to represent
breaking gravity waves in the 3D physical system. The quasigeostrophic (QG)
equations have gravity waves filtered out and are based on the assumptions of
small perturbation to the thickness and small Rossby number: $dh/h\ll 1$ and
$U/(fL_{i})\ll 1$. So the SW equations are more general than QG. To estimate
the value of $dh/h$ in the SW system, we use a scaling based on cyclostrophic
balance:

$\frac{u^{2}}{r}+fu=-g_{r}dhdr,$ (3)

Taking $v\sim U$, $dr\sim L_{i}$, and $L_{d}^{2}\sim g_{r}h$, we obtain

$dh/h\sim\frac{U^{2}+fUL_{i}}{(fL_{d})^{2}}.$ (4)

This covers both small and large values of the Rossby number $U/(fL_{i})$.
When the Rossby number is small, the initial scale of the turbulence matters,
which gives

$dh/h\sim\frac{fUL_{i}}{(fL_{d})^{2}};$ (5)

When the Rossby number becomes large, the velocity dominates, and the initial
scale of turbulence has minimal influence on the thickness perturbation:

$dh/h\sim\frac{U^{2}}{(fL_{d})^{2}}.$ (6)

In this paper, we use the expression presented in Eq. (4) as to calculate the
values of $dh/h$.

The increase of $f$ with latitude – the beta effect, works differently at the
poles than at mid latitudes. Here $\beta=df/\textfractionsolidus
dy=2\Omega\cos\phi/a$, where $y$ is the poleward coordinate and $a$ is the
radius of the planet. At the mid-latitudes, the quantity $\sqrt{U/\beta}$ is a
length scale below which eddies dominate and above which zonal jets dominate
Rhines (1975). The turbulence is assumed to be cascading from small scales to
large scales, but when it reaches the scale of $\sqrt{U/\beta}$, the cascade
stops. However, since $\beta$ goes to zero at the poles, the cascade can never
reach that scale at the poles, and eddies should always dominate.

The question then arises: at what latitude does the transition, with zonal
jets below that latitude and eddies above it, take place? If we expand $\beta$
at radial distance $r$ away from the pole using its derivative at the pole:
$\beta\approx 2\Omega r/a^{2}$, and assume that the eddy scale inside $r$ is
$r$ itself, as on Jupiter. Then

$\sqrt{\frac{U}{\beta}}\approx\sqrt{\frac{Ua^{2}}{2\Omega r}}\approx r.$ (7)

Solving for $r$ gives

$r\approx\left(\frac{Ua^{2}}{2\Omega}\right)^{\frac{1}{3}}.$ (8)

The answer has been given before Ingersoll  _et al._ (2022); Siegelman, Young,
and Ingersoll (2022), but here we applied an assumption about the eddies to
provide the physical intuition of the lengthscale. With U = 80 m/s, the
expression gives 10,500 km, which is a fairly good match to the 8700 km radius
of the circumpolar ring. Note that this critical latitude varies as $U^{1/3}$,
and it is only a scaling estimate, not a quantitative theory.

##  III Model description

We solve shallow-water equations using an Athena++-based shallow-water model
Li and Chen (2019); Stone  _et al._ (2020) to study the genesis of the
vortices and their subsequent evolution. The setup of the initial condition is
similar to Siegelman et al. (2022)Siegelman, Young, and Ingersoll (2022). We
assume that the initial streamfunction is a superposition of sinusoidal waves.
For each wave numbered $n$, it has an argument of
$k(x\cos\theta_{n}+y\sin\theta_{n}+\alpha_{n})$. Here k is the wavenumber,
within a $\pm 5\%$ range from the central wavenumber specified by
$2\pi/L_{i}$. $\theta_{n}$ and $\alpha_{n}$ are random numbers between $0$ and
$2\pi$, meaning that the waves are from a narrow ring in wavenumber space and
uniformly distributed in direction. The initial velocity field is
geostrophically balanced and is populated everywhere within a region extending
from the pole to a distance 5$\times$104 km away from the pole. This region
extends down to 50° of latitude, making it much larger than the vortices
themselves. The amplitude of the noise is chosen so that the root mean square
velocity in the field between 60 degrees and the pole is $U$. The choice of 60
degrees is made as most of the energy in the final patterns is always confined
within this region. The final magnitudes of velocities observed in the final
patterns, corresponding to each of the initial U selections, are shown in
Figure 1.

The jovian vortices were measured to have velocity profiles that reach 80 to
110 m/s at approximately 1000 km from their centers Grassi  _et al._ . This
implies that, for $U$ values of 24 m/s and 70 m/s, the velocity patterns are
closer to the observations. Hence, we chose U values of 24, 70, 120, and 170
m/s, bracketing and exploring beyond the observed values. The value of $L_{d}$
in the polar region is not measured, but the mid-latitude value is estimated
to be 1000 km WON (2011). Various estimates for $L_{d}$ in the polar region
span the range from 300 km to 1300 km. For the central vortex in the north,
the requirement that the thickness cannot be negative leads to the smallest
value of $L_{d}$ being at least 750 km Ingersoll  _et al._ (2022). Hence, we
choose $L_{d}$ values to be well larger than this lower bound, ranging from
1750 to 8400 km. For $L_{i}$, we choose turbulence scales that are smaller
than the scale of the vortices, i.e., 210, 350, and 700 km. For the choice of
the parameters, we focused on regions of transition between an unstructured
pattern and vortex crystals, which leads to different $L_{d}$ choices with
different $L_{i}$ values.

Figure 1:  Velocity magnitudes along a central chord of the computational
domain for four different values of $U$, all with $L_{i}=$350 km and
$L_{d}=$2100 km.

The flow takes place in a plane tangent to the surface of the planet at the
pole. For most of the scenarios, we choose a 2048$\times$2048 grid, with
$x_{max}=y_{max}=6\times 10^{4}$ km, and $x_{min}=-x_{max},y_{min}=-y_{max}$.
This domain covers from the pole to about 40 degrees of latitude. With a grid
convergence test, we found that $\Delta x\leq L_{i}/2$ is necessary to resolve
the initial turbulence. For an initial forcing scale of 105 km, a model with
doubled resolution has to be used, increasing the computational burden by 8
times. Hence, we start our exploration from the minimum length scale Li=210
km.

Additionally, we impose a sponge layer that starts at radius $r_{*}=5\times
10^{4}$ km from the pole, corresponding to 50 degrees of latitude. The
velocities are dissipated in the sponge layer by Rayleigh friction. The
dissipation rate increases linearly from $0$ at $r=r_{*}$ to $0.5$ day-1 at
$r=1.1r_{*}$.

##  IV Results

Fig. 2 shows flow undergoing an inverse energy cascade. The small-scale energy
contained in the initial turbulence cascades into larger scales, forming
structured vorticity. The energy thus flows from kinetic energy to potential
energy, while the scale at which the energy cascade stops varies by case. In
the first 10 days, distinct behavior starts to be present. Both scenarios
develop vortices from the initial turbulence within days after the start of
simulation. However, even though Case 1 has a smaller initial forcing scale,
it ends up with larger vortices, which means more extensive merging takes
place. In comparison, Case 2 shows a regular pattern of similarly sized
vortices. The following development of the patterns are also different: Case 1
continues to merging and forming larger vortices, while Case 2 stop merging
and settles in to a vortex crystal configuration, not showing significant
changes in pattern for over a thousand days.

Figure 2: The vorticity contours of two cases: an example where the vortices
do not form a regular pattern, and an example where the vortices organize to a
vortex crystal. The patterns at days 10, 100, 500, and 1000 are presented. The
circles are at latitudes of 80, 70, 60, and 50 degrees. Parameter choices for
each of the two cases: Case 1 (upper row): $L_{i}=$210 km, $U=$120 m/s, and
$L_{d}=$2100 km; Case 2 (lower row): $L_{i}=$350 km, $U$=70 m/s, and
$L_{d}$=7000 km. These values give large $dh/h=0.043$ for Case 1 and small
$dh/h=0.002$ for Case 2, respectively. Figure 3: The vortex patterns at day
600. The full dataset simulated is shown, and values of $dh/h$ are presented
as the number on top of each frame. The numbers placed within the frames in
the top-left corners mark the level of irregularity based on the $dA/A$ metric
described in the text. Note that the $L_{d}$ values along the y-axis are
different for the three panels. This change is to focus on the transition from
irregular patterns with relatively small $L_{d}$ values to vortex crystal
patterns with large $L_{d}$ values. The value of this transition, between
$dA/A\leq 1$ and $dA/A>1$, also varies with U and Li, which is why the axes
are different.

In Fig. 3, we organize 48 images at day 600 and label them by $dh/h$. With a
glance, it becomes apparent that cases with small $dh/h$ show a more regular
pattern, closer to a vortex crystal, as compared to cases with large $dh/h$.
Each image has a pattern, and we want a simple but quantitative way to
classify it. We start by locating all the vortices in the image - patches of
fluid whose vorticity is larger than $0.1f$, regardless of the sign. Then we
calculate the area of each patch and the mean area $A$ and standard deviation
$dA$ of all the patches in the image. The metric measuring the irregularity of
the image is then $dA/A$. For aid in graphical depiction, we put the measured
values of all the $dA/A$ values, one for each image, into 9 bins from low to
high. We number the bins with single digits from 0 to 8.

The levels of irregularity defined as the single-digit numbers in Fig. 3 agree
well with our subjective impression. Numbers less than 2 look like regular
vortex crystals. Numbers 2 to 5 look somewhat irregular, and 6 to 8 look
chaotic and irregular. It is not a perfect metric, and the connection to
$L_{d}$ and other input parameters is not perfect, perhaps because there are
random elements in the merging process. Blank spaces in the left panel are
where $dh/h$ was so large that the program crashed before reaching 600 days.
Crashing was always due to the geopotential $\Phi$ becoming negative.

If one scans across any of the 4 images with constant $L_{d}$ and $L_{i}$, it
is clear that irregularity increases with $U$. Similarly, if one scans down
any of the 4 images with constant $U$ and $L_{i}$, it is clear that
irregularity decreases as $L_{d}$ increases. Both of these observations are
consistent with irregularity increasing with $dh/h$, which has
$fUL_{i}+U^{2}2$ in the numerator and $L_{d}^{2}$ in the denominator. As shown
in Fig. 2, for smaller $dh/h$ values, the patterns stop evolving after the
initial turbulence is organized into vortices. However, for larger $dh/h$
values, the pattern remains evolving for a long time, allowing the vortices to
merge, which does not favor a vortex crystal structure.

Figure 4: Levels of irregularity for each image in Fig. 3 as a function of
input parameters. The irregularity of a case is its dA/A value expressed as a
single-digit number from 0 to 8 as described in the text. The input parameters
$U$, $L_{d}$, and $L_{i}$ define a 3D space, and the variable $dh/h$ is
computed from those parameters. There are no double-digit numbers. Digits that
share the same locations are offset by a small amount to avoid overlapping.

Fig. 4 is a regime diagram in the 3D space defined by the input parameters
$U$, $L_{d}$, and $L_{i}$. The output variable is $dA/A$, which measures the
computed patterns’ irregularity. Vortex crystals are the most regular and have
the lowest values of $dA/A$. Chaotic behavior and mergers have the highest
values. In principle, $dA/A$ fills parameter space and summarizes the result
of our investigation, and it is useful to study how the low, intermediate, and
high values of $dA/A$ are distributed throughout parameter space after 600
days, after which the patterns have settled down. We do this by treating the
parameter space as a cube and projecting the $dA/A$ values along the 3 axes.
We added another input parameter, $dh/h$, which is a combination of the
original three. It is found that the combined variable $dh/h$ is more closely
related to vertical structure than the others. $L_{d}$, on the other hand, is
more closely related to horizontal structure: it is the e-folding distance
over which vortices’ interaction with each other drops.

In the upper right panel of Fig. 4, irregularity bin numbers of 3 or more are
confined to the lower right corner of the figure, where $U$ is large and
$L_{d}$ is small, specifically $L_{d}\leq 4000$ km for $U=$170 m/s and
$L_{d}\leq$2000 km for $U=70$ m/s. This suggests that
$dh/h=(U^{2}+fUL_{i})/(fL_{d})^{2}$ is the controlling variable. This is borne
out by the two left panels of Fig. 4, which always show large $dA/A$ bins when
$dh/h$ is large. Both large $U$ and small $L_{d}$ seem to be necessary for
merging and chaos. Otherwise, vortex crystals are favored. From the definition
of $dh/h$, it appears that $L_{i}$ has an effect only when the initial Rossby
number is less than 1.

The upper left panel of Fig. 4 has the integers representing $dA/A$ varying
almost monotonically along a line in the projected parameter space. The small
integers are in the upper left corner, and the large integers are in the lower
right corner. The variables along the two axes are highly correlated and are
responsible for most of the variations from regularity to irregularity from
one run of the model to the next. This is an expected observation because the
formula for $dh/h$ has $U$ and $L_{i}$ in the numerator and $L_{d}^{2}$ in the
denominator. $L_{i}$ has a relatively small effect on the level of
irregularity.

A detailed view of the metric of irregularities is shown in Fig. 5. While we
did the parameter search by using various values of $L_{d}$, $L_{i}$, and $U$.
We used $dh/h$ as a combined parameter that contains the information of all
three variables. We obtain more insights on the effect of each parameter: the
smaller radius of deformation $L_{d}$ shows a general trend resulting in a
more irregular pattern. Larger characteristic velocity $U$ results in a
slightly more irregular pattern.

While Fig. 4 shows 2D landscapes of levels of irregularities, Fig. 5 shows the
$dA/A$ bin numbers as scatter plots, with $L_{d}$, $U$, $L_{i}$, and $dh/h$ as
the independent variables placed in the x-axis. As seen in the lower right
panel, $dh/h$ does the best job of predicting when the flow will evolve to
large $dA/A$, signifying an irregular pattern, as contrasted with small
$dA/A$, which signifies a vortex crystal pattern. This is consistent with
$dh/h$ being the controlling variable.

Figure 5: Scatter plots of variables versus the irregularity measure $dA/A$.

It has been noticed in Fig. 5 that there is a strong and linear correlation
between the value of $dh/h$ to the level of irregularities, which is not
obtained with the three original variables. This finding bridges the earlier
results and discussions related to Jupiter vortices: models of shallow water
did not create results with vortex crystal patterns; however, with a QG model,
the generation of vortex crystal was obtained Siegelman, Young, and Ingersoll
(2022). Hence, we suggest that having small $dh/h$, which is obtained when the
atmosphere has low velocity and large vertical stability, helps the formation
of stable vortex crystals, and that may be the case with Jupiter.

##  V Discussions & Conclusions

The perturbation to the layer thickness, $dh/h$, emerged as a dominant factor
influencing the subsequent phases of vortex evolution. This phenomenon depicts
the interplay between two-dimensional turbulence and the Coriolis effect in a
rotating frame, which poses a limitation on the inverse energy cascade to
larger scales.

We provide a qualitative argument that explains this phenomenon here. In the
context of a non-rotating frame of reference, two-dimensional turbulence
exhibits an inverse cascade behavior, transporting energy towards larger
scales. In the absence of the Coriolis force, the inherent tendency of the
vortex is to merge, culminating in the largest scale the system permits.

However, the narrative shifts in a rotating frame, especially at the poles. In
the rotating spheres, a beta effect is present that drives the cyclones to
move along the positive potential vorticity gradient and anti-cyclones along
the negative. Surrounding the geostrophically balanced cyclones, there are
rings of negative vorticity, which form a negative potential vorticity
gradient, shielding other vortices from moving in and merging. This is well
observed, and such a structure is identified as the shielding of the vortex Li
_et al._ (2020); Ingersoll  _et al._ (2022). The shielding effects change the
dynamic behaviors, stopping further inverse cascades by inhibiting the merging
of vortices into larger scales. This results in a cut-off, keeping the inverse
energy cascade at relatively small scales.

This dynamic is encapsulated in Fig. 3, vortices with a larger $dh/h$ persist
in channeling energy to larger scales, while their counterparts with a smaller
$dh/h$ stopped this process, preserving a nearly constant scale, which
eventually forms a vortex crystal configuration.

Another observation is associated with the radius of deformation. In previous
publications, the vortex crystal is produced in an infinite-radius-of-
deformation QG modelSiegelman, Young, and Ingersoll (2022). In previous work
that produced a single polar vortex instead of vortex crystals, the values of
$dh/h$ are relatively large due to small values of radius of deformation (e.g.
1700 kmIngersoll  _et al._ (2015), and 1300 km Brueshaber, Sayanagi, and
Dowling (2019)). The shallow-water case where the vortex crystal stably
exists, on the other hand, explored a radius of deformation up to 10,000 km.
Hence, the underestimation of the polar radius of deformation may be the major
reason why a vortex crystal was not predicted before being observed. The polar
atmosphere of Jupiter may have a higher buoyancy frequency, i.e. may be more
stably stratified than previously estimated.

Our results offer a tangible linkage from the strength and scale of forcing to
the resulting vortex patterns: weak and small-scale perturbations result in
small values of $dh/h$, more QG dynamics, and more regular vortex lattice
patterns. On the other hand, when the perturbation is strong and large-scale,
the values of $dh/h$ are large, and the vortices tend to merge and form larger
structures, resulting in more Saturn-like polar patterns.

###### Acknowledgements.

This research was carried out at the California Institute of Technology under
a contract with the National Aeronautics and Space Administration (NASA),
Grant/Cooperative Agreement Number 80NSSC20K0555, and a contract with the Juno
mission, which is administered for NASA by the Southwest Research Institute.
CL was supported by the NASA Juno Program, under NASA Contract NNM06AA75C from
the Marshall Space Flight Center, through subcontract 699056KC to the
University of Michigan from the Southwest Research Institute. We also thank
Dr. Huazhi Ge for his suggestions in developing the simulations presented in
this paper.

## Data Availability Statement

The model program is made public on GitHub under the repository
https://github.com/chengcli/canoe. The simulation output data that support the
findings of this study are available from the corresponding author upon
reasonable request.

## bibliography

## References

  * Fine  _et al._ (1995) K. S. Fine, A. C. Cass, W. G. Flynn, and C. F. Driscoll, “Relaxation of 2d turbulence to vortex crystals,” Phys. Rev. Lett. 75, 3277–3280 (1995).
  * Schecter  _et al._ (1999) D. A. Schecter, D. H. E. Dubin, K. S. Fine, and C. F. Driscoll, “Vortex crystals from 2D Euler flow: Experiment and simulation,” Physics of Fluids 11, 905–914 (1999).
  * Siegelman, Young, and Ingersoll (2022) L. Siegelman, W. R. Young, and A. P. Ingersoll, “Polar vortex crystals: Emergence and structure,” Proceedings of the National Academy of Sciences 119, e2120486119 (2022).
  * Chassignet and Cushman-Roisin (1991) E. P. Chassignet and B. Cushman-Roisin, “On the influence of a lower layer on the propagation of nonlinear oceanic eddies,” Journal of Physical Oceanography 21, 939 – 957 (1991).
  * Brueshaber, Sayanagi, and Dowling (2019) S. R. Brueshaber, K. M. Sayanagi, and T. E. Dowling, “Dynamical regimes of giant planet polar vortices,” Icarus 323, 46–61 (2019).
  * Ingersoll  _et al._ (2022) A. P. Ingersoll, S. P. Ewald, F. Tosi, _et al._ , “Vorticity and divergence at scales down to 200 km within and around the polar cyclones of jupiter,” Nat. Astron. 6, 1280–1286 (2022).
  * Siegelman  _et al._ (2022) L. Siegelman, P. Klein, A. P. Ingersoll, _et al._ , “Moist convection drives an upscale energy transfer at jovian high latitudes,” Nat. Phys. 18, 357–361 (2022).
  * Rhines (1975) P. B. Rhines, “Waves and turbulence on a beta-plane,” Journal of Fluid Mechanics 69, 417–443 (1975).
  * Li and Chen (2019) C. Li and X. Chen, “Simulating nonhydrostatic atmospheres on planets (snap): Formulation, validation, and application to the jovian atmosphere,” ApJS 240 (2019), 10.3847/1538-4365/aafdaa.
  * Stone  _et al._ (2020) J. M. Stone, K. Tomida, C. J. White, _et al._ , “The athena++ adaptive mesh refinement framework: Design and magnetohydrodynamic solvers,” ApJS 249 (2020), 10.3847/1538-4365/ab929b.
  * (11) D. Grassi, A. Adriani, M. L. Moriconi, A. Mura, F. Tabataba-Vakili, A. Ingersoll, G. Orton, C. Hansen, F. Altieri, G. Filacchione, G. Sindoni, B. M. Dinelli, F. Fabiano, S. J. Bolton, S. Levin, S. K. Atreya, J. I. Lunine, T. Momary, F. Tosi, A. Migliorini, G. Piccioni, R. Noschese, A. Cicchetti, C. Plainaki, A. Olivieri, D. Turrini, S. Stefani, R. Sordini, and M. Amoroso, “First estimate of wind fields in the jupiter polar regions from jiram-juno images,” Journal of Geophysical Research: Planets 123, 1511–1524.
  * WON (2011) “Vertical structure of jupiter’s oval ba before and after it reddened: What changed?” Icarus 215, 211–225 (2011).
  * Li  _et al._ (2020) C. Li, A. P. Ingersoll, A. P. Klipfel, and H. Brettle, “Modeling the stability of polygonal patterns of vortices at the poles of jupiter as revealed by the juno spacecraft,” Proceedings of the National Academy of Sciences 117, 24082–24087 (2020).
  * Ingersoll  _et al._ (2015) A. P. Ingersoll, S. P. Ewald, F. Tosi, _et al._ , “Polar vortex formation in giant-planet atmospheres due to moist convection,” Nat. Geosci. 8, 523–526 (2015).