# Dual-Domain Image Synthesis using Segmentation-Guided GAN

Dena Bazazian Andrew Calway Dima Damen
Department of Computer Science, University of Bristol
{dena.bazazian, andrew.calway<EMAIL_ADDRESS>
###### Abstract

We introduce a segmentation-guided approach to synthesise images that
integrate features from two distinct domains. Images synthesised by our dual-
domain model belong to one domain within the semantic-mask, and to another in
the rest of the image - smoothly integrated. We build on the successes of few-
shot StyleGAN and single-shot semantic segmentation to minimise the amount of
training required in utilising two domains.

The method combines few-shot cross-domain StyleGAN with a latent optimiser to
achieve images containing features of two distinct domains. We use a
segmentation-guided perceptual loss, which compares both pixel-level and
activations between domain-specific and dual-domain synthetic images. Results
demonstrate qualitatively and quantitatively that our model is capable of
synthesising dual-domain images on a variety of objects (faces, horses, cats,
cars), domains (natural, caricature, sketches) and part-based masks (eyes,
nose, mouth, hair, car bonnet). The code is publicly available111
https://github.com/denabazazian/Dual-Domain-Synthesis.

|  |
---|---|---
|  |
|  |
Figure 1: Dual-Domain Synthesis combines segmentation-guided features from the
source domain (left), and the target domain (centre) - corresponding
segmentation masks are incorporated. Synthesised images (right) – top&bottom:
caricature face with natural eyes/nose/mouth, middle: natural face with
caricature features. The first two rows are from paired examples, i.e. the
same latent code, while the third row shows an unpaired example.

##  1 Introduction

Automated image manipulation, particularly portrait editing, is of significant
interest to researchers and practitioners alike with applications in
animation, gaming and social media entertainment. Advances in Generative
Adversarial Networks (GANs) [11] have led to tremendous progress and
innovation, enabling the synthesis of realistic images [18, 19], and leading
to techniques for image-to-image translation [24] and domain translation [25],
able to synthesise a corresponding image of one domain using a reference image
from another, transferring the style.

In this paper, we focus on segmentation-guided dual-domain feature integration
– combining segmentation parts from one domain with an image from another in a
way that is visually convincing. For example, replacing the eyes and mouth of
a natural face image with those from a caricature, so that the result is a
dual-domain image. Examples are shown in Fig. 1. Note the distinction from
previous mask-guided transfer methods which combine styles and/or features
from the same domain [40, 26, 12].

We utilise the few-shotGAN [25] technique to generate related images of two
domains from a common latent space. To create the segmentation masks
efficiently, we employ the one-shot segmentation approach of RepurposingGAN
[31]. We leverage these two StyleGAN-based approaches [25, 31] to obtain input
images and masks, and propose a segmentation-guided weighted combination with
perceptual and pixel-wise losses to synthesise the dual-domain image.

To the best of our knowledge, this is the first attempt to synthesise images
based on the features from two domains. The contributions can be summarised as
follows. First, we introduce Dual-Domain Image Synthesis, a technique for
generating images by integrating the features of two domains, guided by a
semantic segmentation model. Second, we demonstrate that the approach works
for images of faces, horses, cats and cars, synthesising dual-domain images
from natural, caricature and sketch images. We test a variety of part-based
segmentations for guidance, such as hair in faces, eyes in cats, heads of
horses and car bonnets.

##  2 Related work

Generative Models: Deep generative models have demonstrated robust results in
generating realistic images for various tasks such as image segmentation [31,
10, 8, 28], image inpainting [23], inverse graphics [39], conditional image
synthesising [40, 26] and domain translations [25]. State-of-the-art GANs,
such as StyleGAN [18] and StyleGAN2 [19], are capable of generating incredibly
realistic images at a high resolution, making these popular models in works
such as [31, 39, 25, 7, 6, 27]. The manifolds of changes in spatially
localised regions for the images generated by StyleGAN are explored in [35].
Cross-domain correspondence techniques based on StyleGAN are proposed in [25,
27] to generate images from two domains using a handful of examples. We make
use of [25] since it demonstrates better performance on far-domain adaptation.
In this work we instead generate images which integrate the two domains into a
single image.

Latent Optimisation: One of the powerful aspects of StyleGAN-based methods is
the ability to generate realistic images from random latent codes and hence
the option of seeking optimal codes w.r.t chosen loss criteria. A number of
recent papers have adopted this approach to generate images based on
perceptual loss, by comparing activation layers from convolutional networks
[17, 38, 6, 7]. Optimising for the perceptual loss in [17] allows transferring
the style from one image to another. Furthermore, perceptual loss is used in
[6, 7] to fit a projected synthetic image into the distribution of images of a
particular domain.

Image Completion: We here group works that attempt mask-guided image synthesis
and inpainting. Mask-guided synthesise techniques combine source and reference
images based on a given mask to generate a realistic image [40, 26, 12], while
image inpainting techniques fill an arbitrary hole of a given image to
generate a natural image [23, 22]. Part-based editing in synthetic images was
proposed in [9] to transfer the appearance of a specific object part from a
reference image to a target image. K-means is applied to the hidden layer
activations of the StyleGAN generator to reveal a decomposition of the
generated output into semantic objects and object-parts. However, note that
all these previous part-based methods combine images from the same domain,
contrasting with our approach which seeks to integrate features from distinct
domains.

Image Blending: Image blending aims to seamlessly blend an object from a
source image onto a target image with lightly mask adjustment [37]. [34]
combines synthesis with Gaussian-Poisson blending, and is trained using
blended ground-truth. [37] combines a Poisson gradient loss, style loss, and
content loss, without the need for training data. Similarly, image
Harmonisation adjusts the illumination, colour, and texture of the foreground
mask. All blending approaches aim to general in order to apply to any pair of
images. In contrast, our proposed approach integrates semantic parts from one
domain, importantly modifying the full image to ensure it belongs to a
specific target domain.

##  3 Proposed Approach

We aim to generate a visually convincing image that integrates features of two
domains, guided by part-based segmentations. Our pipeline utilises two
generative models for the source and target domains along with a segmentation
model for localising the corresponding mask to integrate the features of two
domains. The masks indicate the semantic parts of the image that should be
integrated from one domain into the other. We start by revisiting works we
build on, then detail our approach.

###  3.1 Revisiting Few-shotGAN

The work of few-shotGAN [25] considers a main domain, for which a large-scale
dataset is readily available, and trains a StyleGAN model on this data. Given
few-shot examples (e.g. ten samples) from a second domain, few-shotGAN [25]
utilises the StyleGAN trained on the large dataset for pretraining and
transfers the diversity information from this model to a second model. The
training preserves the relative similarities and differences between instances
via a cross-domain distance consistency loss.

We use [25] to train two generative models, one with many-shot examples and
the other adapted from the first with few-shot examples. Importantly, the two
generative models $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$ share the same
latent space $\mathcal{Z}$ during adaptation. This is of particular importance
to our method, as we base the ability to integrate features from both domains
on sampling from this common latent space. Given the same random vector
$z^{\star}\subset\mathcal{Z}$, two images are synthesised, one from each
generator, and thus from two domains. We refer to these as _paired_ images as
they share the same latent code, albeit from two domains.

###  3.2 Revisiting RepurposingGAN

We train a part-based semantic segmentation model, using the method from
RepurposingGAN [31]. Each synthesised image is passed through a series of
spatial convolutions, and a binary pixel-level segmentation is computed from a
unique generative computation which can be traced back through each
convolutional layer down to the initial latent code. The strength of [31] is
based on the capability to train this segmentation model with one-shot, i.e.
by manually segmenting a single synthetic image.

We use RepurposingGAN [31] to train our segmentation model from a randomly
synthesised image of one domain. The same segmentation model is used for
semantically segmenting synthetic images from both domains. This is only
plausible thanks to the adaptation method in Sec. 3.1. We demonstrate
experimentally that this method is robust across domains for a variety of
semantic masks.

###  3.3 Dual-Domain Synthesis (DDS)

We introduce Dual-Domain Synthesis (DDS) framework in this section. We
consider as input to our network two generative models from two domains (Sec.
3.1). We select one of these two domains to be the ‘source’ domain
($\mathcal{D}_{s}$), and the other as the ‘target’ domain
($\mathcal{D}_{t}$)222Note that the source domain may or may not correspond to
the few-shot domain in the adaptation, and similarly for the target domain.,
and accordingly label the two trained generative models as $\mathcal{G}_{s}$
and $\mathcal{G}_{t}$. Given one latent vector
$z^{\star}\in\mathcal{N}(\mathbf{0},\mathbf{1})$, we generate two synthetic
images such that $x_{s}\in\mathbb{R}^{n\times n\times 3}$, where $x_{s}$ is an
image from the source domain ($\mathcal{D}_{s}$), and
$x_{t}\in\mathbb{R}^{n\times n\times 3}$, where $x_{t}$ is an image from the
target domain ($\mathcal{D}_{t}$). We pass both images to the trained semantic
segmentation model (Sec. 3.2). Corresponding spatial masks denoting the
semantic part in the source and target images are defined as $y_{s}$ and
$y_{t}$. Our goal is to synthesise one image, that is perceptually close to
the _target_ domain, but contains features from the _source_ domain. The
spatial masks are used for guiding the regions that should be transferred from
source to target.

A naive approach to achieve dual-domain synthesis is simply to use additive
‘cut and paste’ to combine the relevant masked regions. We denote this as the
crossover image ($x_{c}$). The source image $x_{s}$ is masked by the
corresponding regions in $y_{s}$ indicating the parts we wish to integrate
from the source domain. Additionally, we wish to remove/subtract the
corresponding segmentation parts from the target image. We thus use the binary
complement of the target mask as $\bar{y}_{t}$. Fig. 2 shows examples of the
generated source and target images and their corresponding masks. The naive
crossover image $x_{c}$ is thus computed as:

$x_{c}=(x_{s}\otimes y_{s})+(x_{t}\otimes\bar{y}_{t}),$ (1)

where $\otimes$ is pixel-wise multiplication/masking.

Importantly, while this image contains the features from both domains, it is
not a visually convincing dual-domain image. We thus aim to search the latent
space $\mathcal{Z}$ for the the optimal latent vector $\hat{z}$ that can
generate our dual-domain image such that
${\hat{x}_{t}\leftarrow\mathcal{G}_{t}(\hat{z})}$. We use a gradient-based
optimisation that iteratively updates a latent code
${\hat{z}\in\mathcal{N}(\mathbf{0},\mathbf{1})}$ using a loss function that is
based on a combination of perceptual loss [17] and pixel-level Mean Square
Error (MSE).

We initialise $\hat{z}$ randomly333Initial experimental evidence showed that
starting from $z^{\star}$ produces inferior results with the optimisation
stuck in local minima.. We base our perceptual loss on that proposed in [6],
which compares the activations of a pre-trained convolutional network from two
images, a synthesised image and a reference image. In DDS, we do not have a
reference image in the dual-domain, and thus our perceptual loss considers
both the source and target domains.

Figure 2: Top:  A random latent code $z^{\star}$ is fed into two different
generators to create two corresponding images from two different domains,
source image $x_{s}$ and target image $x_{t}$. Masks of the generated images,
$y_{s}$ and $y_{t}$, are then computed from a one-shot segmentation model.
Bottom: Naive crossover image ($x_{c}$), where $\bar{y}_{t}$ denotes the
binary complement of $y_{t}$.  Figure 3: Perceptual latent loss optimisation.
$\odot,\otimes$ and $\oplus$ indicate MSE loss, multiplication and summation,
respectively. The colours of each arrow depict distinct data paths.
$x_{s},y_{s},x_{t}$ and $\bar{y}_{t}$ match those illustrated in Fig.2.
$z^{\star}$ is fixed while $\hat{z}$ is a random latent vector that is to be
optimised. $\hat{x}_{s}$ and $\hat{x}_{t}$ are generated iteratively based on
the optimisation of $\hat{z}$. Examples of $\hat{x}_{s}$ and $\hat{x}_{t}$ are
shown for different iterations during the optimisation
($i_{1},i_{10},i_{50},i_{1000}$).

We use activations from the layers $conv_{1\\_1}$, $conv_{1\\_2}$,
$conv_{2\\_2}$ and $conv_{3\\_3}$ of _VGG-16_ [29] for the perceptual loss as
proposed in [6]. We downsample the segmentation binary masks at each layer to
match the resolutions of the corresponding activation layer, and duplicate the
mask across the channel dimension, to match the number of channels in the
layer. To measure the similarity between the generated image and the expected
image during optimisation, we apply a loss function that is a weighted
combination of perceptual loss and the pixel-wise MSE loss:

$\begin{split}&\mathcal{L}_{s}=\mathcal{L}_{p}(\mathcal{G}_{s}(\hat{z}),x_{s},y_{s})+\left\|\mathcal{G}_{s}(\hat{z})\otimes
y_{s}-x_{s}\otimes y_{s}\right\|^{2}\\\
&\mathcal{L}_{t}=\mathcal{L}_{p}(\mathcal{G}_{t}(\hat{z}),x_{t},\bar{y}_{t})+\left\|\mathcal{G}_{t}(\hat{z})\otimes\bar{y}_{t}-x_{t}\otimes\bar{y}_{t}\right\|^{2}\end{split}$
(2)

where $x_{s},x_{t}$ are source and target images, $\bar{y}_{t}$ is a binary
complement of the target mask $y_{t}$.
$\mathcal{G}_{s}(\cdot),\mathcal{G}_{t}(\cdot)$ are the generators for source
and target domains respectively. The perceptual loss $\mathcal{L}_{p}(\cdot)$
in Eq. 2 is computed as :

$\mathcal{L}_{p}(I_{1},I_{2},\lambda)=\sum_{j=1}^{J}\frac{1}{N_{j}}\left\|\mathcal{F}_{j}(I_{1})\times\lambda_{j}-\mathcal{F}_{j}(I_{2})\times\lambda_{j}\right\|$
(3)

where $I_{1},I_{2}$ are two images, $\lambda$ is the mask, $\lambda_{j}$ is
the downsampled segmentation mask according to the size of each $conv$ layer
$j$. $\mathcal{F}_{j}$ is the activations of the $conv$ layers and $N_{j}$ is
the number of scalars in the $j^{th}$ layer output, and $J$ is the number of
convoultional layers. Fig. 3 shows an illustration of the process of the
perceptual loss.

The perceptual loss is combined with the pixel-wise MSE loss in Eq. 2 for
integrating the features from the two domains. In order to increase the
quality of the generated image, and ensure its segmentation-guided similarity
to the source and target images, we use an additional MSE loss based on the
naive cross over image as:

$\mathcal{L}_{c}=\left\|s_{c}-x_{c}\right\|$ (4)

where $x_{c}$ is the naive crossover image from Eq. 1, and $s_{c}$ is a naive
crossover synthetic image computed each iteration $i$ as:

$s_{c}=(\mathcal{G}_{s}(\hat{z})\otimes
y_{s})+(\mathcal{G}_{t}(\hat{z})\otimes\bar{y}_{t}).$ (5)

The overall loss is then defined as:

$\mathcal{L}oss=\alpha\mathcal{L}_{s}+\beta\mathcal{L}_{t}+\gamma\mathcal{L}_{c},$
(6)

Note that we only optimise $\hat{z}$ and do not update the weights of the
convolutional network, generative networks or the domain-specific synthetic
images $x_{s},x_{t}$. We then use the target domain generator to produce the
dual-domain image, such as: $\hat{x}\leftarrow\mathcal{G}_{t}(\hat{z})$. The
complete process is summarised in Algorithm 1.

Input: $x_{s},x_{t},y_{s},\bar{y}_{t}$

Model : $\mathcal{G}_{s},\mathcal{G}_{t}$

Parameter : $\alpha,\beta,\gamma$

Result: $\hat{x}$

$x_{c}\leftarrow x_{s}\otimes y_{s}+x_{t}\otimes\bar{y}_{t}$;

$\hat{z}\leftarrow\mathcal{N}(\mathbf{0},\mathbf{1})$;

$\nabla\leftarrow Optimiser$;

while  _i $<$ maxIterations_ do

$\mathcal{L}_{s},\mathcal{L}_{t}\leftarrow Eq.\ref{eq:l}$;

$\mathcal{L}_{c}\leftarrow Eq.\ref{eq:naive_crossover_LOSS}$;

$\mathcal{L}oss\leftarrow\alpha\mathcal{L}_{s}+\beta\mathcal{L}_{t}+\gamma\mathcal{L}_{c}$;

$\hat{z}\leftarrow\nabla(\hat{z})$

end while

$\hat{x}\leftarrow\mathcal{G}_{t}(\hat{z})$;

Algorithm 1 Dual-Domain Synthesis algorithm $\hat{x}_{t_{1}}$ | $\hat{x}_{t_{5}}$ | $\hat{x}_{t_{10}}$ | $\hat{x}_{t_{15}}$ | $\hat{x}_{t_{20}}$ | $\hat{x}_{t_{25}}$ | $\hat{x}_{t_{30}}$ | $\dots$ | $\hat{x}_{t_{500}}$ | $\dots$ | $\hat{x}_{t_{1000}}$ | $x_{t}$ | $x_{s}$
---|---|---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  | $\dots$ |  | $\dots$ |  |  |
|  |  |  |  |  |  | $\dots$ |  | $\dots$ |  |  |
|  |  |  |  |  |  | $\dots$ |  | $\dots$ |  |  |
|  |  |  |  |  |  | $\dots$ |  | $\dots$ |  |  |
Figure 4: Intermediate DDS results for $DDS_{car.\rightarrow nat.}$ with a
semantic mask of the eye/nose/mouth regions. Source and target domains are
caricature and natural faces respectively. In the first row, note the distinct
caricature eyes, nose and mouth in $i\geq 500$ iterations.

##  4 Implementation Details

We used the StyleGAN2 and few-shotGAN implementations provided in [3] and [4],
and 10 images for few-shot far-domain adaptation in each case. Segmentation
models are based on the implementation of RepurposingGAN provided in [5]. We
manually label semantic masks in each case as follows. We first synthesise an
image by feeding a random latent code to a StyleGAN model [19]. We then
manually label the mask of the part(s) of interest, and train the segmentation
model as in [5].

In all experiments, we optimise DDS for $1000$ iterations, using a learning
rate of $0.01$ with the Adam optimiser [20]. $J$ in Eq. 3 is set to 4, given
the layers chosen from _VGG-16_. Based on empirical experiments we set the
${\alpha,\beta,\gamma}$ in Eq. 6 as ${0.9,1,0.5}$, respectively. Robustness to
these hyperparameters was evaluated using an ablation study in Sec. 5.1.

##  5 Experiments

In this section, we present and discuss quantitative and qualitative
experimental results of DDS on the domains of face, horse, car and cat images.

###  5.1 DDS on Face Domains

We perform experiments on human face images based on a StyleGAN model [3]
trained on Flickr-Faces-HQ (FFHQ) [19] to generate natural human face images.
Then, we use few-shotGAN model trained on Face-Caricature [25] and Face-Sketch
[32] images (models from [4]). We experiment with the domain pairs natural
$\leftrightarrow$ caricature and natural $\leftrightarrow$ sketch and two
part-based segmentation masks, one indicating the regions of eyes, nose and
mouth444We combine three face parts into one mask (nose, eyes and mouth) to
show more experimental results in one image. and the other indicating the
hair.

Qualitative results: Fig. 4 first shows the intermediate results of DDS on the
experiments for integrating the features of caricature domain into the natural
face domain based on the eye/nose/mouth segmentation model. We illustrate the
evolution of $\hat{x}_{t_{i}}$ while the latent code $\hat{z}$ is being
iteratively optimised to generate the dual-domain image. We compare these to
the target and source domains. Note that the optimisation seems to first focus
on the identity and pose in the target domain, with the last iterations
focusing on the dual-domain image synthesis.

Our qualitative results on the face domains are shown in Fig. 5. We
demonstrate natural face images and their corresponding images from either
caricature domain or sketch domain along with the semantic masks. We present
our images in pairs, where we switch the source and target domains. For
example, in the first pair, DDS incorporated the eye/nose/mouth from the
caricature into the natural face. Notice how the eye colour and mouth were
adapted but are distinct from that of the natural image. The nose has also
been copied from the caricature image, with the skin colour integrated. In the
second column, we swap the source and target domains integrating the eyes,
nose and mouth from the natural image into the caricature. The natural nose is
unmissable but properly integrated.

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Figure 5: DDS results on human faces. Paired images are generated from the same latent code $z^{\star}$. The odd columns refer to natural human face images. The second and fourth columns represent corresponding images from caricature domain. The sixth and eighth columns depict corresponding images from sketch face domain. The corresponded masks shown along with the images. The second row shows the DDS results based on the eye/nose/mouth segmentation model, while the forth row shows the DDS results based on the hair segmentation model.  $Domain_{nat.}$ | $\textstyle{x_{c_{car.\rightarrow nat.}}}$ | $\scriptstyle{DDS_{car.\rightarrow nat.}}$ | $Domain_{car.}$ | $\textstyle{x_{c_{nat.\rightarrow car.}}}$ | $\scriptstyle{DDS_{nat.\rightarrow car.}}$
---|---|---|---|---|---
|  |  |  |  |
Figure 6: Comparing naive crossover to $DDS_{car.\leftrightarrow nat.}$.
“$nat.$” and “$car.$” denote the natural face and caricature domains
respectively. We show the naive crossover $x_{c_{car.\rightarrow nat.}}$ as
well as our DDS results $DDS_{car.\rightarrow nat.}$. Analogous results for
$nat.\rightarrow car.$ are also shown.

Evaluation metrics: We use standard metrics to evaluate our proposed
technique. In each experiment, we consider the synthesised images as $\hat{x}$
and images in the domain we compare to as $x$. Note that there is no ground-
truth or examples of the target dual-domain images. We thus compare the
distribution of all synthesised images to three domains: source, target and
the naive crossover images. Generally, we aim for DDS images to be close to
the target domain while maintaining the source features guided by the masks.
We use the following metrics in our quantitative results:

  * •

Fréchet Inception Distance (FID) [14] defined as
$FID(\hat{x},x)~{}=~{}\left\|\mu_{\hat{x}}-\mu_{x}\right\|^{2}+\Gamma(\sigma_{\hat{x}}+\sigma_{x}-2\sqrt{\sigma_{\hat{x}}\sigma_{x}})$,
where $\mu_{x}$, $\sigma_{x}$, $\mu_{\hat{x}}$, $\sigma_{\hat{x}}$ are,
respectively, the mean and covariance of $x$ and $\hat{x}$, and
$\Gamma(\cdot)$ is the trace operation.

  * •

Structural Similarity Index Measure (SSIM) [33] defined as
$SSIM(\hat{x},x)~{}=~{}\frac{(2\mu_{\hat{x}}\mu_{x}+c_{1})(2\sigma_{\hat{x}x}+c_{2})}{(\mu_{\hat{x}}^{2}+\mu_{x}^{2}+c_{1})(\sigma_{\hat{x}}^{2}+\sigma_{x}^{2}+c_{2})}$,
where $c_{1}$ and $c_{2}$ stabilise the division with a weak denominator. We
used $c_{1}=10^{-4}$ and $c_{2}=9\times 10^{-4}$ in all experiments as
implemented in [1].

  * •

Peak Signal-to-Noise Ratio (PSNR) [15] defined as
$PSNR(\hat{x},x)=~{}10\log_{10}(R^{2}/MSE(\hat{x},x))$, where
$\scriptsize{MSE(\hat{x},x)}=~{}\frac{1}{n}\sum_{i=0}^{n}\left[\hat{x}_{i}-x_{i}\right]^{2}$,
and $R$ is the maximal in the image data.

For each DDS experiment, we use the notation
${\mathcal{D}_{s}\rightarrow\mathcal{D}_{t}}$ to identify the source
($\mathcal{D}_{s}$) and target ($\mathcal{D}_{t}$) domains, as we integrate
features from the source domain into an image from the target domain, as
illustrated in Fig. 6.

Quantitative results: Table 1 presents our results. In each reported measure,
we average results from 100 randomly sampled latent codes. Note in particular
that the scores confirm that our results in almost all the experiments have
the better $FID$, $SSIM$ and $PSNR$ scores in the target domain, demonstrating
the expected behaviour of the synthetic images being closer to the target
domain.

Table 1: Metric comparisons on the face domains. $DDS_{\mathcal{D}_{s}\rightarrow\mathcal{D}_{t}}$ integrating features from the source domain into the target domain in every case. $\\{x_{c}\\}$ denotes the set of all naive crossover images given by $x_{c_{\mathcal{D}_{s}\rightarrow\mathcal{D}_{t}}}$. $\mathcal{M}_{1}$ denotes the mask of eyes/nose/mouth while $\mathcal{M}_{2}$ denotes hair mask. FID$\downarrow$, SSIM$\uparrow$ and PSNR$\uparrow$ are the metrics, where $\downarrow$ indicates lower is better and $\uparrow$ indicates higher is better. |  | $DDS_{caricature\rightarrow natural}$ |  $DDS_{natural\rightarrow caricature}$ |  $DDS_{sketch\rightarrow natural}$ |  $DDS_{natural\rightarrow sketch}$
---|---|---|---|---|---
|  | FID$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | FID$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | FID$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | FID$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$
$\mathcal{M}_{1}$ | $\mathcal{D}_{s}$ | 281.61 | 0.39 | 27.98 | 259.12 | 0.38 | 27.95 | 195.80 | 0.31 | 27.87 | 225.31 | 0.33 | 27.81
$\\{x_{c}\\}$ | 114.03 | 0.69 | 29.45 | 121.21 | 0.75 | 28.75 | 200.35 | 0.71 | 29.81 | 207.41 | 0.65 | 29.28
$\mathcal{D}_{t}$ | 73.51 | 0.70 | 29.51 | 74.51 | 0.76 | 28.76 | 79.87 | 0.73 | 29.84 | 97.65 | 0.66 | 29.29
$\mathcal{M}_{2}$ | $\mathcal{D}_{s}$ | 296.61 | 0.47 | 27.99 | 290.63 | 0.42 | 27.92 | 245.61 | 0.38 | 27.87 | 262.90 | 0.36 | 27.83
$\\{x_{c}\\}$ | 155.91 | 0.73 | 29.63 | 95.30 | 0.71 | 28.43 | 177.03 | 0.66 | 29.25 | 152.41 | 0.61 | 28.54
$\mathcal{D}_{t}$ | 87.28 | 0.74 | 29.65 | 76.96 | 0.72 | 28.45 | 90.58 | 0.64 | 29.31 | 94.83 | 0.62 | 28.49

Monitoring latent code optimisation: Fig. 7 plots the FID metric at each
iteration of the latent optimising process. We consider $DDS_{car.\rightarrow
nat.}$ and $DDS_{sketch\rightarrow nat.}$, thus using the caricature and
sketch images as source and the natural faces as target with the two
segmentation models $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$. We observe FID
during optimisation in each case. Fig. 7 showcases how the dual-domain
synthesised image gets closer to both domains but importantly significantly,
and more quickly, closer to the target domain, and remains distinct from
either domain at the end of the optimisation.

Figure 7: FID during latent code optimisation iterations, comparing the dual-
domain images to source (dashed) and target (solid) domains for 4 experiments.
Dual-domain images get closer to both domains during optimisation but
significantly closer to target.

Robustness to hyperparameters: We next assess the robustness of our results to
the parametric weights of the loss function in Eq. 6. We performed this
experiment on the domains of natural human faces and caricature, and the hair
segmentation model, and we consider FID over the target domain
($\mathcal{D}_{t}$) as a metric to evaluate the results of several
experiments. Fig. 8 demonstrates that by considering ${\alpha,\beta,\gamma}$
as ${0.9,1,0.5}$ respectively we can have the best FID in this experiment. It
can be perceived from Fig. 8 that $\beta$ is the most sensitive hyperparameter
in Eq. 6. The reason for this sensitivity is that $\beta$ weights the target
loss ($\mathcal{L}_{t}$) and these FID values are computed over the target
domain. Therefore, when $\beta$ is equal to zero, we do not consider the
target loss ($\mathcal{L}_{t}$) in our optimisation.

Figure 8: Impact of $\alpha$, $\beta$ and $\gamma$ in loss function (Eq. 6) on natural face (target) and caricature domains (source) with the hair mask. FID values are computed over the target domain ($\mathcal{D}_{t}$). |  |  |
---|---|---|---
|  |  |
Figure 9: DDS results on horse domains. Odd columns are from natural horse domain, and even columns are from sketch horse domain. Segmentation masks are incorporated. Second row shows the corresponding dual-domain images.  |  |  |
---|---|---|---
|  |  |
Figure 10: DDS results on car domains. Odd columns are standard car images, and even columns are abandoned cars. Segmentation masks are incorporated. Second row shows the DDS results. |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
Figure 11: The DDS results on images from unpaired latent codes ($z^{\star}_{1},z^{\star}_{2}$). We show the segmentation mask in each case. Successful examples (col 1-4) show that DDS can bridge small pose differences. Failures (col 5-8) are due to larger pose variations. |  |  |
---|---|---|---
|  |  |
Figure 12: DDS results on unpaired human and cat images based on the
segmentation model on eyes. First row shows the unpaired images with their
segmentation mask. Second row shows human faces with the cat eyes and cats
with the human eyes.

###  5.2 DDS on Horse and Car Domains

We performed further experiments on horse and car images. We used the
StyleGAN2 [19] model to generate natural horse images from [2] which is
trained on LSUN Horses [36]. We collected 10 sketch horses for the purpose of
this experiment through a web search. We scale these images to $256\times 256$
and use these 10 images for the few-shotGAN far-domain adaptation [25]. We
train our one-shot segmentation model to segment the head of the horse. Fig. 9
shows qualitative results.

For car images, we used the StyleGAN2 [19] model to generate the standard car
images from [2], which is trained on LSUN Cars [36], and used 10 abandoned car
images from the few-shotGAN implementation [4] for few-shot adaptation. Fig.
10 shows the qualitative results. The one-shot segmentation model is trained
to segment the bonnet and lights of a car. Note that for both cars and horses,
we use the $conv$ layers of AlexNet [21] for the perceptual loss. This
performs better than VGG [29] (see appendix).

###  5.3 Unpaired latents, different objects, limitations

Failures in the domain adaptation (Sec. 3.1) or in the segmentation masks
(Sec. 3.2) will negatively impact DDS. In all previous results, we pass the
same latent code $z^{\star}$ to the two generative models. Thanks to the
adaptation algorithm, this guarantees the same pose in the corresponding
images.

In Fig. 11 , we show that DDS can work with unpaired latent codes with small
pose variations. In all the domains and examples, DDS assumes the
corresponding source and target images $x_{s},x_{t}$ are roughly aligned in
pose. Fig. 11 also shows that DDS cannot recover from large pose variations in
the unpaired images where the segmentation masks are not aligned.

We performed further unpaired experiments on cat and natural face images. We
used the StyleGAN2 [19] model to generate cat images from [2] which is trained
on LSUN Cats [36]. We used a segmentation model on eyes with the purpose of
generating dual-domain images of human face with the cat eyes and vice versa.
Fig. 12 shows successful dual-domain synthesis. This demonstrates that DDS is
not limited to examples when one object types, and can be used with two
different many-shot styleGANs and are not adapted from one another, when masks
are aligned. We present further results on this in appendix where both the
source and target domains are from the few-shotGAN. The results in Fig. 12
demonstrate the potential of DDS on two unpaired examples from unrelated GANs
and distinct segmentation models. However, successful examples are challenging
to find due to the variation in pose when randomly sampling latent variables.
This forms a limitation that we wish to address in future work, by utilising
segmentation alignment approaches.

We also noted that in contrast with prior assumption [38], the choice of
backbone for the perceptual loss impacts DDS results and is domain specific.
We present further results on this in appendix. Further exploration is needed
to ensure robustness to backbone choice in the perceptual loss.

##  6 Conclusion

We presented Dual-Domain Synthesis framework, an approach to generate visually
convincing images that contain features from different domains. Our generated
images are obtained by combining two correspondence images guided by part-
based segmentation from a one-shot segmentation model. We demonstrated the
approach on several image classes (face, car, horse, cat), multiple domains
(natural, sketch, caricature, abandoned) and various part-based segmentation
models (eye, nose, mouth, hair, car bonnet).

Extensive experiments illustrate the potential of the approach, both in terms
of qualitative and quantitative assessment. While the majority of the results
focus on paired latent codes and object (e.g. faces), the approach can be
extended to distinct GANs trained independently and unpaired latent codes,
assuming part-based alignment.

## References

  * [1] SSIM Implementation.  ”https://github.com/Po-Hsun-Su/pytorch-ssim/”, 2017.  [Online].
  * [2] StyleGAN2 Models.  ”https://nvlabs-fi-cdn.nvidia.com/stylegan2/networks/”, 2019.  [Online].
  * [3] PyTorch Implementation of Analyzing and Improving the Image Quality of StyleGAN.  ”https://github.com/rosinality/stylegan2-pytorch/”, 2020.  [Online].
  * [4] Implementation of Few-shot Image Generation via Cross-domain Correspondence.  ”https://github.com/utkarshojha/few-shot-gan-adaptation/”, 2021\.  [Online].
  * [5] Implementation of Repurposing GANs for One-shot Semantic Part Segmentation.  ”https://github.com/bryandlee/repurpose-gan/”, 2021.  [Online].
  * [6] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2StyleGAN: How to Embed Images into the StyleGAN Latent Space?  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432–4441, 2019.
  * [7] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2StyleGAN++: How to Edit the Embedded Images?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8296–8305, 2020.
  * [8] Mohammed Al-Rawi, Dena Bazazian, and Ernest Valveny.  Can Generative Adversarial Networks Teach Themselves Text Segmentation?  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019.
  * [9] Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk.  Editing in Style: Uncovering the Local Semantics of GANs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5771–5780, 2020.
  * [10] Wolfgang Fuhl, David Geisler, Wolfgang Rosenstiel, and Enkelejda Kasneci.  The Applicability of Cycle GANs for Pupil and Eyelid Segmentation, Data Generation and Image Refinement.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019.
  * [11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative Adversarial Nets.  Advances in Neural Information Processing Systems, 27, 2014.
  * [12] Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, and Lu Yuan.  Mask-guided Portrait Editing with Conditional GANs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3436–3445, 2019.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  Advances in Neural Information Processing Systems, 30, 2017.
  * [15] Quan Huynh-Thu and Mohammed Ghanbari.  Scope of Validity of PSNR in Image/Video Quality Assessment.  Electronics Letters, 44(13):800–801, 2008.
  * [16] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  SqueezeNetX: AlexNet-level Accuracy with 50x Fewer Parameters and $<$ 0.5MB Model Size.  arXiv preprint arXiv:1602.07360, 2016.
  * [17] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution.  In European Conference on Computer Vision, pages 694–711. Springer, 2016.
  * [18] Tero Karras, Samuli Laine, and Timo Aila.  A Style-Based Generator Architecture for Generative Adversarial Networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and Improving the Image Quality of StyleGAN.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Alex Krizhevsky.  One Weird Trick for Parallelizing Convolutional Neural Networks.  arXiv preprint arXiv:1404.5997, 2014.
  * [22] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro.  Image Inpainting for Irregular Holes using Partial Convolutions.  In Proceedings of the European Conference on Computer Vision, pages 85–100, 2018.
  * [23] Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, and Jing Liao.  PD-GAN: Probabilistic Diverse GAN for Image Inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9371–9381, 2021.
  * [24] Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised Image-to-Image Translation Networks.  In Advances in Neural Information Processing Systems, pages 700–708, 2017.
  * [25] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot Image Generation via Cross-domain Correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10743–10752, 2021.
  * [26] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic Image Synthesis with Spatially-Adaptive Normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2337–2346, 2019.
  * [27] Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang.  Few-shot Adaptation of Generative Adversarial Networks.  arXiv preprint arXiv:2010.11943, 2020.
  * [28] Dimitrios Sakkos, Edmond SL Ho, and Hubert PH Shum.  Illumination-aware Multi-Task GANs for Foreground Segmentation.  IEEE Access, 7:10976–10986, 2019.
  * [29] Karen Simonyan and Andrew Zisserman.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [30] Konstantin Sofiiuk, Polina Popenova, and Anton Konushin.  Foreground-aware semantic representations for image harmonization.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1620–1629, 2021.
  * [31] Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakorn.  Repurposing GANs for One-shot Semantic Part Segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4475–4485, 2021.
  * [32] Xiaogang Wang and Xiaoou Tang.  Face Photo-Sketch Synthesis and Recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(11):1955–1967, 2008.
  * [33] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image Quality Assessment: From Error Visibility to Structural Similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [34] Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang.  GP-GAN: Towards realistic high-resolution image blending.  In Proceedings of the 27th ACM international conference on multimedia, pages 2487–2495, 2019.
  * [35] Mengyu Yang, David Rokeby, and Xavier Snelgrove.  Mask-Guided Discovery of Semantic Manifolds in Generative Models.  4th Workshop on Machine Learning for Creativity and Design at NeurIPS 2020, 2020.
  * [36] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop.  arXiv preprint arXiv: 1506.03365, 2015.
  * [37] Lingzhi Zhang, Tarmily Wen, and Jianbo Shi.  Deep image blending.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 231–240, 2020.
  * [38] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.
  * [39] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, A. Torralba, and S. Fidler.  Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering.  arXiv preprint arXiv: abs/2010.09125, 2021.
  * [40] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka.  SEAN: Image Synthesis with Semantic Region-Adaptive Normalization.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5103–5112, 2020.

## Appendix

In addition to the appendix sections below, we provide our code publicly with
the relevant instructions 555 https://github.com/denabazazian/Dual-Domain-
Synthesis.

##  Appendix A Impact of pre-trained networks on perceptual loss

We evaluate the impact of using the activations from different pre-trained
convolutional backbones on our model. We assess four backbone models namely
VGG [29], AlexNet [21], SqueezeNet [16], ResNet [13], all pretrained on
ImageNet. The shape of the layers to calculate the losses in each backbone are
included in Tables 7 \- 7 respectively.

Fig. 13 shows the qualitative results of the human face and caricature
domains. Table 2 shows the numerical results of this experiment on the hair
segmentation model; all the metrics of this table are computed over the target
domain. These experiments demonstrate that the pre-trained VGG [29] model
offers the best performance for the perceptual loss in faces. Previous work
[6, 7] only applied pretrained VGG [29] model for perceptual loss, and in [38]
the assumption for the perceptual loss was that the results from VGG [29] and
AlexNet [21] are similar. We show qualitatively and quantitatively that some
network backbones perform better than others.

We also demonstrate that this is domain specific in Fig. 14 and Fig. 15. For
both the horse and the car domains, AelxNet offers significantly better
results than VGG. This is an interesting observation given that the backbones
are trained on the same data. We stipulate that the architecture design has
been motivated by different tasks. Further exploration of the role of backbone
in perceptual loss calculation is needed.

Images |  |  |  |
---|---|---|---|---
ResNet |  |  |  |
SqueezeNet |  |  |  |
AlexNet |  |  |  |
VGG |  |  |  |
Figure 13: The impact of four pre-trained backbones for computing perceptual loss. The first two columns uses the eye/nose/mouth segmentation model, while the last two columns uses the hair segmentation model. Table 2: Comparison of the performance of four backbones for perceptual loss. The result are on the natural faces and caricature domains, where the segmentation model is trained to segment hair. |  | $DDS_{car.\rightarrow nat.}$ | $DDS_{nat.\rightarrow car.}$
---|---|---|---
|  | FID$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | FID$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$
ResNet | $\mathcal{D}_{t}$ | 95.66 | 0.55 | 28.89 | 98.71 | 0.47 | 27.93
SqueezNet | $\mathcal{D}_{t}$ | 89.48 | 0.62 | 29.15 | 94.29 | 0.52 | 28.05
AlexNet | $\mathcal{D}_{t}$ | 88.63 | 0.66 | 29.36 | 93.27 | 0.71 | 28.11
VGG | $\mathcal{D}_{t}$ | 87.28 | 0.71 | 29.58 | 76.96 | 0.72 | 28.39
|  |  |
---|---|---|---
|  |  |
|  |  |
Figure 14: DDS results on horse domains. Odd columns are from natural horse domain, and even columns are from sketch horse domain. Segmentation masks are Incorporated. Second and third rows show the corresponding dual-domain images using the perceptual loss from the $conv$ layers of VGG [29] and AlexNet [21] respectively.  |  |  |
---|---|---|---
|  |  |
|  |  |
Figure 15: DDS results on car domains. Odd columns are standard car images,
and even columns are abandoned cars. Segmentation masks are incorporated.
Second and third row show the DDS results using the perceptual loss from the
$conv$ layers of VGG [29] and AlexNet [21] respectively.

##  Appendix B Dual-Domain Synthesis vs. Image Blending

To demonstrate the difference of our DDS with image blending approaches, we
provide a comparison in Fig. 16 between Dual-Domain and recent Image Blending
works (using the public codes of [37, 30]). Blending does not achieve better
or even comparable results.

|  |  |  |
---|---|---|---|---
|  |  |  |
Figure 16: Comparing [30] (col3) and [37] (col4) to Dual-Domain (col5), for
paired (top) and unpaired (bottom) examples.

##  Appendix C DDS from few-shotGAN domains

We performed further experiments on synthesising images which contain features
from both the caricature and sketch domains. Note that both GANs are trained
using few-shots, adapted independently from the natural face StyleGAN. Fig. 17
shows dual-domain images based on the integrating caricature and sketch
features.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
Figure 17: DDS results on caricature and sketch domains. The first row show a
pair of images from the domains of caricature and sketch. The eyes/nose/mouth
segmentation model used in these experiments, and all the correspondences
masks are shown in the first row. The second row shows the dual-domain images
containing the features of caricature and sketch based on the segmentation
model. The changes in the shape of the nose and mouth and colour of the eyes
can be observed from the dual-domain images.

##  Appendix D Final vs. intermediate latent space editing

In our results, we used the synthesised image from the last layer of StyleGAN.
Instead, intermediate representations can be used to accommodate unpaired
images of varying poses. We experiment with using the concatenation of hidden
intermediate layer activations. In this experiment for the images with
resolution $256\times 256$, we get the $13$ layers of features as described in
Table 7. Hence, for computing the perceptual loss instead of passing the
images to $conv$ layers of _VGG_ , we directly use the activations of
intermediate latent space of styleGAN from both the target and source domains.
Fig. 18 demonstrates a comparison of the results when we apply DDS framework
and when we use the intermediate latent activations.

While intermediate activations can accommodate variations in pose, they cannot
maintain the features from two domains.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
Figure 18: Comparison of the results of using the intermediate latent
activations. First row shows unpaired images of two domains, masks are
incorporated. Second row shows the DDS results, where each image maintain
distinct features from the two domains. Third row shows the results when we
use the intermediate latent space features for computing the perceptual loss.
While image are integrated, they do not maintain the distinctness of the two
domains. For example, in the first image (row 3, col 1) the output is purely a
natural face. Table 3: Shape of the intermediate StyleGAN features.

$\mathbf{w}\in\mathcal{W}$ | $shape$
---|---
$\mathbf{w}_{0}$ | $[512\times 4\times 4]$
$\mathbf{w}_{1}$ | $[512\times 8\times 8]$
$\mathbf{w}_{2}$ | $[512\times 8\times 8]$
$\mathbf{w}_{3}$ | $[512\times 16\times 16]$
$\mathbf{w}_{4}$ | $[512\times 16\times 16]$
$\mathbf{w}_{5}$ | $[512\times 32\times 32]$
$\mathbf{w}_{6}$ | $[512\times 32\times 32]$
$\mathbf{w}_{7}$ | $[512\times 64\times 64]$
$\mathbf{w}_{8}$ | $[512\times 64\times 64]$
$\mathbf{w}_{9}$ | $[256\times 128\times 128]$
$\mathbf{w}_{10}$ | $[256\times 128\times 128]$
$\mathbf{w}_{11}$ | $[128\times 256\times 256]$
$\mathbf{w}_{12}$ | $[128\times 256\times 256]$

Table 4: Shape of the _VGG-16_ layers that used for perceptual loss.

$Layer$ | $shape$
---|---
$conv_{1\\_1}$ | $[64\times 256\times 256]$
$conv_{1\\_2}$ | $[64\times 256\times 256]$
$conv_{2\\_2}$ | $[256\times 64\times 64]$
$conv_{3\\_3}$ | $[512\times 32\times 32]$

Table 5: Shape of the _AlexNet_ layers that used for perceptual loss.

$Layer$ | $shape$
---|---
$relu_{1}$ | $[64\times 55\times 55]$
$relu_{2}$ | $[192\times 27\times 27]$
$relu_{3}$ | $[384\times 13\times 13]$
$relu_{4}$ | $[256\times 13\times 13]$
$relu_{5}$ | $[256\times 13\times 13]$

Table 6: Shape of the _SqueezeNet_ layers that used for perceptual loss.

$Layer$ | $shape$
---|---
$relu_{1}$ | $[64\times 127\times 127]$
$relu_{2}$ | $[128\times 63\times 63]$
$relu_{3}$ | $[256\times 31\times 31]$
$relu_{4}$ | $[384\times 15\times 15]$
$relu_{5}$ | $[384\times 15\times 15]$
$relu_{6}$ | $[512\times 15\times 15]$
$relu_{7}$ | $[512\times 15\times 15]$

Table 7: Shape of the _ResNet-18_ layers that used for perceptual loss.

$Layer$ | $shape$
---|---
$relu_{1}$ | $[64\times 128\times 128]$
$conv_{2}$ | $[64\times 64\times 64]$
$conv_{3}$ | $[128\times 32\times 32]$
$conv_{4}$ | $[256\times 16\times 16]$
$conv_{5}$ | $[512\times 8\times 8]$

##  Appendix E Sketch horse data

Fig. 19 shows the images of sketch horses that used for training few-shotGAN.

|  |  |  |
---|---|---|---|---
|  |  |  |
Figure 19: Training samples for generating sketch horse images.


# Narrow and ultra-narrow transitions in highly charged Xe ions as probes of
fifth forces

Nils-Holger Rehbehn ID<EMAIL_ADDRESS>Max-Planck-Institut für
Kernphysik, D–69117 Heidelberg, Germany  Michael K. Rosner ID Max-Planck-
Institut für Kernphysik, D–69117 Heidelberg, Germany  Julian C. Berengut ID
School of Physics, University of New South Wales, Sydney, New South Wales
2052, Australia  Max-Planck-Institut für Kernphysik, D–69117 Heidelberg,
Germany  Piet O. Schmidt ID Physikalisch–Technische Bundesanstalt, D–38116
Braunschweig, Germany  Leibniz Universität Hannover, D–30167 Hannover, Germany
Thomas Pfeifer ID Max-Planck-Institut für Kernphysik, D–69117 Heidelberg,
Germany  Ming Feng Gu  Space Science Laboratory, University of California,
Berkeley, CA 94720, USA  José R. Crespo López-Urrutia ID crespojr@mpi-
hd.mpg.de Max-Planck-Institut für Kernphysik, D–69117 Heidelberg, Germany

###### Abstract

Optical frequency metrology in atoms and ions can probe hypothetical fifth-
forces between electrons and neutrons by sensing minute perturbations of the
electronic wave function induced by them. A generalized King plot has been
proposed to distinguish them from possible Standard Model effects arising
from, e.g., finite nuclear size and electronic correlations. Additional
isotopes and transitions are required for this approach. Xenon is an excellent
candidate, with seven stable isotopes with zero nuclear spin, however it has
no known visible ground-state transitions for high resolution spectroscopy. To
address this, we have found and measured twelve magnetic-dipole lines in its
highly charged ions and theoretically studied their sensitivity to fifth-
forces as well as the suppression of spurious higher-order Standard Model
effects. Moreover, we identified at 764.8753(16) nm a E2-type ground-state
transition with 500 s excited state lifetime as a potential clock candidate
further enhancing our proposed scheme.

Indirect evidence from galactic rotation, gravitational lensing and
cosmological evolution suggests the existence of dark matter (DM) [1, 2]. Its
constituents, by coupling to Standard Model (SM) particles, could also
influence the neutrino-mass hierarchy [3] and explain open physics questions.
Additional fields could cause a fifth-force coupling of electrons with
neutrons, inducing small but measurable effects [4, 5, 6] in atomic systems.
In an electronic transition, sensitivity to a fifth-force arises because the
overlap with the nucleus changes between the ground and excited state,
reflecting interactions as small as a fraction of the linewidth.

However, since the transition energies cannot be calculated accurately,
isotope shift spectroscopy is employed. The classical method of the King plot
(KP) uses two transitions in at least three different isotope pairs are used.
Plotting the isotope shifts scaled by the nuclear-mass parameter eliminates
the charge radius, typically leading to a linear behavior [7] from which
atomic constants characterising atomic recoil (mass shift) and the overlap of
the electronic wavefunction with the nucleus (field shift) can be derived.
Optical frequency metrology has reduced uncertainties in the determination of
transition energies of forbidden transitions by orders of magnitude, making KP
methods far more sensitive. On top of such isotopic shifts (IS), hypothetical
fifth forces would add minute perturbations causing a deviation from linearity
[6, 5, 8, 9, 10, 11]. However, unknown SM effects of higher order [6, 8]
(sometimes dubbed ‘spurions’) could also induce sizable non-linearities that
are hard to distinguish from those of the hypothetical forces.

A recently devised generalized King plot (GKP) [12, 13] overcomes this by
using more transitions and isotope pairs to build a set of linear equations
determining the higher-order effects, and even disposing of the need of exact
nuclear masses (‘no-mass GKP’). Recent dedicated experiments in ytterbium
measured a deviation from the King linearity [14], which has since been
confirmed using other transitions [15, 16, 17]. The most likely reason for the
deviation is changes in the nuclear deformation of Yb between isotopes [18],
however the analysis show that a second, yet unidentified source of
nonlinearity is also present. In contrast, a measurement in calcium [19],
where higher-order effects are expected to be smaller, was consistent with
King linearity.

To further enhance the GKP sensitivity, as many transitions and even-isotopes
as possible are sought after. An ideal candidate is xenon ($Z=54$). It has
seven natural zero-nuclear-spin isotopes (124, 126, 128, 130, 132, 134, 136),
and four more radio-isotopes (118, 120, 122 and 138) with lifetimes longer
than minutes. Its mass reduces Doppler shifts in comparison with lighter
elements, and its nuclear charge magnifies relativistic and QED effects [20],
which is in itself a key field of research [21].

Figure 1: Measured optical ground-state transitions of Xe9+ through Xe17+. The
Zeeman structure (arrows) was fitted based on line identification with fac
calculations. For Xe10+ the Grotrian diagram has been expanded. Dotted line:
calculated E2 transition; dashed line: calculated M1 transition (see
Supplemental Materials [URL will be inserted by publisher] for details and
further Grotrian diagrams).

In this Letter, we find twelve xenon ground-state, magnetic dipole (M1)
transitions in the optical region, with wavelength uncertainties in the order
of $0.1\,\mathrm{pm}$, and one optical electric quadrupole (E2) clock
transition in charge states Xe9+ through Xe17+. We have calculated and
evaluated theoretical King plots to find which combination of transitions
leads to the highest possible sensitivity to a hypothetical fifth-force
between neutrons and electrons.

Highly charged ions (HCI) such as those studied here are very well suited for
high-precision experiments [22], since their very low polarizability
suppresses effects of external electromagnetic perturbations. Moreover, the
reduced number of bound electrons reduces their theoretical complexity. Recent
experimental developments like sympathetic cooling [23], application of
quantum logic spectroscopy [24] to HCI [25], and algorithmic cooling [26] have
made clocks based on HCI with a sub-Hz uncertainty possible [27]. They will
help extending GKP applications into beyond-the-SM (BSM) parameter regions not
yet constrained by scattering experiments [28, 29, 30, 31, 32] and fifth-force
studies [33, 34, 35, 36, 37, 38].

Table 1: Transitions in highly charged Xe ions: Measured energies, wavelengths ($\lambda_{\mathrm{vac}}$, vacuum) and g-factors of upper and lower energy levels obtained through fitting of the Zeeman-structure with their corresponding uncertainties. Theoretical transition probabilities $A_{ki}$, ab-intio wavelengths and g-factors were calculated with ambit [39]. |  | Observed values |  ambit calculations |
---|---|---|---|---
Ion | Transition | Energy (eV) |  $\lambda_{\mathrm{vac}}$ (nm) | $g_{\mathrm{upper}}$ | $g_{\mathrm{lower}}$ |  $A_{ki}$ (s-1) |  $\lambda$ (nm) | $g_{\mathrm{upper}}$ | $g_{\mathrm{lower}}$ | Type |
Xe9+ | $4d^{9}$ | ${}^{2}\\!D_{3/2}$ | - | ${}^{2}\\!\\!D_{5/2}$ | 2.07151156(28) | 598.520419(90) | $0.792(2)$ | $1.189(1)$ | 67.5 | 595.4 | 0.8 | 1.2 | M1 |
Xe${}^{10+}_{a}$ | $4d^{8}$ | ${}^{3}\\!F_{3}$ | - | ${}^{3}\\!\\!F_{4}$ | 1.8808627(14) | 659.18789(56) | $1.082(4)$ | $1.238(3)$ | 88.4 | 665.1 | 1.0833 | 1.2426 | M1 |
Xe${}^{10+}_{b}$ | $4d^{8}$ | ${}^{3}\\!F_{2}$ | - | ${}^{3}\\!\\!F_{4}$ | 1.6209726(34) | 764.8753(16)(Ritz) |  |  | 0.002 | 735.6 | 0.9792 | 1.2426 | E2 |
Xe${}^{10+}_{c}$ | $4d^{8}$ | ${}^{1}\\!G_{4}$ | - | ${}^{3}\\!\\!F_{4}$ | 5.0567231(73) | 245.18684(36)(Ritz) |  |  | 67.1 | 242.9 | 1.0074 | 1.2426 | M1 |
Xe${}^{11+}_{a}$ | $4d^{7}$ | ${}^{2}\\!G_{9/2}$ | - | ${}^{4}\\!\\!F_{9/2}$ | 3.9339517(50) | 315.16451(45) | $1.105(37)$ | $1.321(35)$ | 107.7 | 313.3 | 1.0823 | 1.3054 | M1 |
Xe${}^{11+}_{b}$ | $4d^{7}$ | ${}^{4}\\!F_{7/2}$ | - | ${}^{4}\\!\\!F_{9/2}$ | 1.71358555(87) | 723.53667(37) | $1.242(9)$ | $1.306(7)$ | 82.8 | 727.4 | 1.2276 | 1.3054 | M1 |
Xe12+ | $4d^{6}$ | ${}^{3}\\!H_{4}$ | - | ${}^{5}\\!\\!D_{4}$ | 3.7252183(78) | 332.82398(79) | $1.058(32)$ | $1.455(31)$ | 110.8 | 328.6 | 1.0438 | 1.462 | M1 |
Xe${}^{15+}_{a}$ | $4d^{3}$ | ${}^{4}\\!P_{1/2}$ | - | ${}^{4}\\!\\!F_{3/2}$ | 4.2747781(74) | 290.03657(50) | $2.17(20)$ | $0.78(9)$ | 16.8 | 297.7 | 2.2137 | 0.47304 | M1 |
Xe${}^{15+}_{b}$ | $4d^{3}$ | ${}^{4}\\!P_{3/2}$ | - | ${}^{4}\\!\\!F_{3/2}$ | 4.0448609(24) | 306.52277(20) | $0.856(57)$ | $0.799(28)$ | 101.7 | 337.1 | 1.4357 | 0.47304 | M1 |
Xe${}^{16+}_{a}$ | $4d^{2}$ | ${}^{1}\\!D_{2}$ | - | ${}^{3}\\!\\!F_{2}$ | 4.4207651(39) | 280.45869(25) | $1.197(11)$ | $0.704(15)$ | 84.4 | 271.3 | 1.2027 | 0.70681 | M1 |
Xe${}^{16+}_{b}$ | $4d^{2}$ | ${}^{3}\\!F_{3}$ | - | ${}^{3}\\!\\!F_{2}$ | 2.24105833(38) | 553.23949(11) | $1.073(1)$ | $0.702(2)$ | 137.6 | 555.4 | 1.0833 | 0.70681 | M1 |
Xe${}^{16+}_{c}$ | $4d^{2}$ | ${}^{3}\\!P_{1}$ | - | ${}^{3}\\!\\!F_{2}$ | 4.9243614(65) | 251.77721(33)(Ritz) |  |  | 10.3 | 244.3 | 1.5 | 0.70681 | M1 |
Xe17+ | $4d$ | ${}^{2}\\!D_{5/2}$ | - | ${}^{2}\\!\\!D_{3/2}$ | 2.84178291(47) | 436.290179(73) | $1.184(2)$ | $0.785(3)$ | 123.7 | 433.8 | 1.2 | 0.8 | M1 |

We produced the ions of interest with an electron beam ion trap (EBIT) [40,
41], the Heidelberg EBIT (HD-EBIT) [42], from a differentially pumped atomic
beam of Xe interacting with electrons at selected energies. For producing the
HCI of interest Xe9+ through Xe33+, we scanned the electron beam energy in the
range 100-2500 eV in 10-eV increments. Every time the ionization potential of
a given charge state is surpassed, the next higher one appears in the trap,
and with it a different spectrum. In each cycle, we kept these ions trapped
for 60 s, and dumped them at the end by briefly inverting the axial trapping
potential set by trap electrodes. This removes impurity ions that slowly
accumulate due to evaporation of barium and tungsten from the electron-gun
cathode.

Figure 2: (a) Isotopic shift from a hypothetical fifth-force for the forbidden
ground-state transitions in Xe9+ through Xe17+ for a fixed coupling constant
$y_{e}y_{n}=1\times 10^{-13}$ and varying mediator mass. (b) Theoretical King
plot for the six possible Xe-isotope pairs using the Xe${}^{11+}_{b}$, Xe12+
pair as example. We set the coupling constant to $y_{e}y_{n}=1\times 10^{-13}$
and the mediator mass to $m_{\Phi}=1\times
10^{5}\,\frac{\mathrm{eV}}{\mathrm{c}^{2}}$. The error bars represent 100 mHz
measurement uncertainty modified by the mass parameter $\mu_{a}$.

Electron-impact excitation populates the upper levels of the observed
transitions both directly and through cascades. The electron and ion density
conditions in the EBIT are such that optical magnetic-dipole transitions with
Einstein coefficients higher than $\approx$ 10 s-1 can be measured [43, 44,
45, 46, 47]. For this purpose, a set of in-vacuo lenses projects an image of
the cylindrical ion cloud through a quartz vacuum window. This intermediate
image is rotated by 90 degrees by a periscope and relayed by two lenses to the
entrance slit of an optical spectrometer, as in Refs. [43, 44]. In the present
work, we used a Czerny-Turner spectrometer with 2 m focal length [45, 46, 47]
to record the wavelength range 250-800 nm, and calibrated it with hollow-
cathode lamps of different elements. In its focal plane, a CCD-camera, cooled
to $-80^{\circ}\mathrm{C}$, took several images for averaging with an exposure
time of 60 minutes each. Pixels showing high signals due to cosmic muons were
identified and removed from the images. Stray-light background was also
subtracted. After obtaining overview spectra with a 150 grooves/mm grating, we
performed measurements at higher resolution using two gratings with 1800 and
3600 grooves/mm, respectively. The results are shown in Figure 1.

For identification, we calculated for each ion the electronic structure and
transition rates with the Flexible Atomic Code (fac) [48] and ambit [39]. The
advantage of fac is its calculation speed, while ambit is used for its more
precise results. Since the 8-T field of the EBIT separates the Zeeman
components to a resolvable extent, we fitted for each line its centroid,
experimental linewidth, the g-factors of the upper and lower state, $\pi$ and
$\sigma$ amplitudes, and compare the results with theory. We corrected the
identification of the 436.2 nm line in Ref. [49], from Xe18+ to Xe17+ based on
the Zeeman splitting. Table 1 presents the key parameters of the thirteen
discovered ground-state transitions. Wavelength uncertainties are the
quadratic average of those from the Zeeman fits and the spectrometer
dispersion. Results of ambit calculations are also given there: ab-initio
wavelength $\lambda$, Einstein coefficient $A_{ki}$, and expected g-factors.
Furthermore, we tabulate in the Supplemental Material [URL will be inserted by
publisher] many other identified lines not involving the electronic ground-
state.

A key result of our search is the identification of a ground-state clock
transition of E2 electric quadrupole in Xe10+ from the lowest excited state.
By means of Ritz-Rydberg combinations (see Supplemental Materials [URL will be
inserted by publisher]), we determine for the $4d^{8}$
${}^{3}\\!F_{2}\rightarrow^{3}\\!\\!\\!F_{4}$ transition a vacuum wavelength
of 764.8753(16) nm. We compare this with an ambit calculation yielding 735.6
nm and an E2 transition rate of $A_{ki}=0.002\,\mathrm{s}^{-1}$, which is
within the expected calculation uncertainty. The low transition rate suggests
that this is a suitable candidate for an optical clock with a sensitivity to
fifth forces that can be fully exploited due to its narrow linewidth of 0.3
mHz. We also have preliminary assignments for a few additional E2 candidates
in other charge states in the Supplemental Materials [URL will be inserted by
publisher], but a conclusive identification will require complementary
measurements.

Following an established approach, we checked the suitability of the found
ground-state transitions for GKP studies. For this, we added to the
electromagnetic interaction potentials used for the fac calculations an
additional term for the hypothetical fifth-force as a Yukawa potential [12]:

$V_{\Phi}(r)=y_{\mathrm{e}}y_{\mathrm{n}}(A-Z)\frac{\hbar c}{4\pi
r}\exp\left(-\frac{c}{\hbar}m_{\Phi}r\right),$ (1)

where $y_{\mathrm{e}}y_{\mathrm{n}}$ is the coupling strength between
electrons and neutrons; $A$ the nuclear mass, $Z$ the nuclear charge; $\hbar$
the reduced Planck constant, and $c$ the speed of light. The range of the
force is defined by the mass parameter $m_{\Phi}$. Using an automatized
script, we performed fac calculations varying the force range, its strength,
as well as the nuclear charge radius and mass, and calculated the
corresponding perturbation on the isotope-shifts.

Between two isotopes, the total isotope shift (IS)
$\delta\nu_{i}^{a}=\nu_{A_{\mathrm{ref.}}}-\nu_{A}$ is expressed by the
following equation [12] for the transition $i$ and the isotope pair
$a=(A_{\mathrm{ref.}},A)$:

$\delta\nu_{i}^{a}=K_{i}\mu_{a}+F_{i}\delta\langle
r^{2}\rangle_{a}+y_{\mathrm{e}}y_{\mathrm{n}}X_{i}\gamma_{a}.$ (2)

Here, the first two terms represent the mass shift and the field shift,
respectively. The former depends on the difference of the inverse of the
isotope masses $\mu_{a}=1/m_{A_{\mathrm{ref.}}}-1/m_{A}$, and the latter on
the mean-square charge radii difference $\delta\langle
r^{2}\rangle_{a}=\langle r^{2}\rangle_{A_{\mathrm{ref.}}}-\langle
r^{2}\rangle_{A}$. The third term is caused by the fifth force, and depends on
the coupling strength $y_{\mathrm{e}}y_{\mathrm{n}}$ between electrons and
neutrons, the electronic constant $X_{i}=X_{i}(V_{\Phi})$ due to the Yukawa
potential, and a factor $\gamma_{a}=(A_{\mathrm{ref.}}-Z)-(A-Z)$ depending on
the difference in number of neutrons. We plot the fifth-force shift versus
varying mediator masses $m_{\Phi}$ in Fig. 2(a).

To study the effect of a fifth-force, we used a King plot [7], where the
isotope shift in Eq. 2 is divided by the mass parameter $\mu_{a}$, which then
yields a common nuclear parameter $\delta\langle r^{2}\rangle_{a}/\mu_{a}$ in
the SM part. Between two modified transitions, the SM parts lead to a linear
behavior, but a fifth-force would break it. For the six Xe-isotope pairs this
is shown as theoretical prediction in Fig. 2(b). The mediator mass
$m_{\Phi}=1\times 10^{5}\frac{\mathrm{eV}}{\mathrm{c}^{2}}$ and the coupling
parameter $y_{\mathrm{e}}y_{\mathrm{n}}=10^{-13}$ are fixed at arbitrary, but
theoretically possible values [6]. Mass uncertainties are neglected. The
fifth-force shift in Fig. 2(a) is on the order of hundreds of Hz, but only a
small fraction remains as a non-linearity due to the alignment of the SM
linearity with the fifth-force contributions [6].

Although each pairing of transitions experiences a different non-linearity,
similar fine-structure transitions share common-mode shifts, reducing the
total sensitivity as in Refs. [13, 47]. Following these works, we quantified
non-linearities in the King plot with the area $\mathrm{NL}$ spanned by the
isotope pairs. Error propagation of the assumed measurement uncertainty on the
isotope-shifts yielded its uncertainty $\Delta\mathrm{NL}$. With this, we
defined the resolution $R$ as $R=\mathrm{NL}/\Delta\mathrm{NL}$. A value $R=1$
sets the lower bound of the coupling parameter $y_{\mathrm{e}}y_{\mathrm{n}}$
that can be resolved, as shown for given transition pairs in Fig. 3.

Figure 3: Sensitivity limit for $y_{\mathrm{e}}y_{\mathrm{n}}$ for selected
transition pairs with assumed frequency uncertainty of 100 mHz. Dashed lines:
Classical King plot (KP) neglecting mass uncertainty. Calcium King plot from
Solaro  _et al._ [19] with 100 mHz uncertainty included for comparison. Dash-
dotted line: Inclusion of a second-order field shift of $1\,\mathrm{kHz}$ and
mass uncertainty. Solid lines: Predicted no-mass GKP for Yb [13] and for the
present Xe, using four transitions.

We assume a frequency uncertainty of $\Delta\nu=100\,\mathrm{mHz}$ as recently
achieved [25, 27] and discussed in Ref. [47]. Dashed lines in Fig. 3 depict
King plots neglecting mass uncertainty and higher-order SM effects. Inclusion
of the former in one of these pairs, e.g., Xe12+, Xe${}^{11+}_{b}$, for
isotopes 130 to 136 (relative mass uncertainties $\approx 10^{-10}$) and 124
to 128 (between $10^{-8}$ and $10^{-7}$) [50] diminishes sensitivity by three
orders of magnitude. Another order of magnitude is lost if nuclear deformation
causes a second-order field shift of $1\,\mathrm{kHz}$ (dash-dotted line) [51,
18] estimated by evaluating calculated higher-order shifts in other elements
[8]. We can overcome these losses with a no-mass generalized King plot [13]
(shown as solid line) by adding more transitions to expand the King plot into
a higher dimension. This restores the sensitivity by three orders of
magnitude, as shown in Fig. 3 and brings it to the level of the ytterbium GKP
with an assumed frequency uncertainty of $100\,\mathrm{mHz}$ from Ref. [13]. A
different xenon pairing would improve the sensitivity around $10^{5}$ eV. Full
sensitivity is not recovered due to the error propagation of the four
transitions needed. Note that either a reduction of the $\nu$-uncertainty down
to 5 mHz, as already achieved by optical clocks [52], or a separate,
sufficiently accurate mass measurement would lead to better sensitivity.

If more higher-order SM contributions are expected, more transitions can be
used to suppress their spurious effects. The seven stable even isotopes of Xe
can tackle up to three such ‘spurions’ by expanding the KP into five
dimensions. With thirteen ground-state transitions, one can use various
pairings to optimize sensitivity to NP. By contrast, other candidates such as
calcium or ytterbium only have enough even isotopes to suppress one ‘spurion’.

In the future, we plan to extend our search towards the EUV region to include
ground-state transitions of even higher-charged xenon ions. This increases the
number of transitions and their GKP sensitivity, as electronic wave functions
with stronger overlap with the nucleus are involved. Currently emerging and
continuously developing EUV frequency combs [53, 54, 55, 56] should in the
future allow frequency metrology on HCI in this spectral regime. Moreover, the
presently assumed uncertainty of $100\,\mathrm{mHz}$ is due to that of the SI
second reference, which is a Cs transition [57, 58]. By performing direct
frequency comparisons with optical clocks [59, 60, 61], or by measuring the
isotope shift of simultaneously trapped ions [62], one could enhance
sensitivity even by a few orders of magnitude.

In summary, we found and identified thirteen optical ground-state transitions
in highly charged xenon ions, theoretically analyzed them, and evaluated pairs
with high sensitivity to Yukawa-type fifth-forces. Such pairings overcome
limitations caused by both known and unknown SM effects by enabling the method
of no-mass generalized King plots. We found for xenon a sensitivity already
comparable to ytterbium, but with the clear advantage of its more numerous
even isotopes and reduced nuclear deformation for future frequency metrology
with sub-Hz uncertainty. Moreover, we identified a ground-state ultra-narrow
clock (E2) transition with 0.3 mHz linewidth complementing the other twelve M1
lines. The analysis of this set of transitions can also be used to conduct
further tests of nuclear deformation [63].

## Acknowledgements

Financial support was provided by the Max-Planck-Gesellschaft. We acknowledge
support from the Max Planck-Riken-PTB Center for Time, Constants and
Fundamental Symmetries, the collaborative research center “SFB 1225
(ISOQUANT)” and the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under Germany’s Excellence Strategy – EXC-2123 QuantumFrontiers –
390837967. JCB was supported in this work by the Alexander von Humboldt
Foundation and the Australian Research Council (DP190100974). This work has
been funded by the “European Metrology Program for Innovation and Research”
(EMPIR) project 20FUN01 TSCAC. This project has received funding from the
EMPIR programme co-financed by the participating states and from the European
Union’s Horizon 2020 research and innovation programme. This project has
received funding from the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation programme (grant agreement No
101019987).

## References

  * Bertone  _et al._ [2005] G. Bertone, D. Hooper, and J. Silk, Particle dark matter: evidence, candidates and constraints, Physics Reports 405, 279 (2005).
  * Bertone and Hooper [2018] G. Bertone and D. Hooper, History of dark matter, Rev. Mod. Phys. 90, 045002 (2018).
  * Alvey and Fairbairn [2019] J. Alvey and M. Fairbairn, Linking scalar dark matter and neutrino masses with icecube 170922a, Journal of Cosmology and Astroparticle Physics 2019 (07), 041.
  * Safronova  _et al._ [2018] M. S. Safronova, D. Budker, D. DeMille, D. F. J. Kimball, A. Derevianko, and C. W. Clark, Search for new physics with atoms and molecules, Rev. Mod. Phys. 90, 025008 (2018).
  * Frugiuele  _et al._ [2017] C. Frugiuele, E. Fuchs, G. Perez, and M. Schlaffer, Constraining new physics models with isotope shift spectroscopy, Phys. Rev. D 96, 015011 (2017).
  * Berengut  _et al._ [2018] J. C. Berengut, D. Budker, C. Delaunay, V. V. Flambaum, C. Frugiuele, E. Fuchs, C. Grojean, R. Harnik, R. Ozeri, G. Perez, and Y. Soreq, Probing new long-range interactions by isotope shift spectroscopy, Phys. Rev. Lett. 120, 091801 (2018).
  * King [1963] W. H. King, Comments on the article “peculiarities of the isotope shift in the samarium spectrum”, J. Opt. Soc. Am. 53, 638 (1963).
  * Flambaum  _et al._ [2018] V. V. Flambaum, A. J. Geddes, and A. V. Viatkina, Isotope shift, nonlinearity of king plots, and the search for new particles, Phys. Rev. A 97, 032510 (2018).
  * Fichet [2018] S. Fichet, Quantum forces from dark matter and where to find them, Phys. Rev. Lett. 120, 131801 (2018).
  * Drake  _et al._ [2021] G. W. F. Drake, H. S. Dhindsa, and V. J. Marton, King and second-king plots with optimized sensitivity for lithium ions, Phys. Rev. A 104, L060801 (2021).
  * Dhindsa  _et al._ [2022] H. S. Dhindsa, V. J. Marton, and G. W. F. Drake, Search for light bosons with king and second-king plots optimized for lithium ions, Physics of Particles and Nuclei 53, 800 (2022).
  * Mikami  _et al._ [2017] K. Mikami, M. Tanaka, and Y. Yamamoto, Probing new intra-atomic force with isotope shifts, The European Physical Journal C 77, 896 (2017).
  * Berengut  _et al._ [2020] J. C. Berengut, C. Delaunay, A. Geddes, and Y. Soreq, Generalized king linearity and new physics searches with isotope shifts, Phys. Rev. Research 2, 043444 (2020).
  * Counts  _et al._ [2020] I. Counts, J. Hur, D. P. L. Aude Craik, H. Jeon, C. Leung, J. C. Berengut, A. Geddes, A. Kawasaki, W. Jhe, and V. Vuletić, Evidence for nonlinear isotope shift in ${\mathrm{Yb}}^{+}$ search for new boson, Phys. Rev. Lett. 125, 123002 (2020).
  * Figueroa  _et al._ [2022] N. L. Figueroa, J. C. Berengut, V. A. Dzuba, V. V. Flambaum, D. Budker, and D. Antypas, Precision determination of isotope shifts in ytterbium and implications for new physics, Phys. Rev. Lett. 128, 073001 (2022).
  * Ono  _et al._ [2022] K. Ono, Y. Saito, T. Ishiyama, T. Higomoto, T. Takano, Y. Takasu, Y. Yamamoto, M. Tanaka, and Y. Takahashi, Observation of nonlinearity of generalized king plot in the search for new boson, Phys. Rev. X 12, 021033 (2022).
  * Hur  _et al._ [2022] J. Hur, D. P. L. Aude Craik, I. Counts, E. Knyazev, L. Caldwell, C. Leung, S. Pandey, J. C. Berengut, A. Geddes, W. Nazarewicz, P.-G. Reinhard, A. Kawasaki, H. Jeon, W. Jhe, and V. Vuletić, Evidence of two-source king plot nonlinearity in spectroscopic search for new boson, Phys. Rev. Lett. 128, 163201 (2022).
  * Allehabi  _et al._ [2021] S. O. Allehabi, V. A. Dzuba, V. V. Flambaum, and A. V. Afanasjev, Nuclear deformation as a source of the nonlinearity of the king plot in the ${\mathrm{Yb}}^{+}$ ion, Phys. Rev. A 103, L030801 (2021).
  * Solaro  _et al._ [2020] C. Solaro, S. Meyer, K. Fisher, J. C. Berengut, E. Fuchs, and M. Drewsen, Improved isotope-shift-based bounds on bosons beyond the standard model through measurements of the ${{}^{2}\mathrm{D}}_{3/2}-{{}^{2}\mathrm{D}}_{5/2}$ interval in ${\mathrm{Ca}}^{+}$, Phys. Rev. Lett. 125, 123003 (2020).
  * Shabaev  _et al._ [2018] V. M. Shabaev, A. I. Bondarev, D. A. Glazov, M. Y. Kaygorodov, Y. S. Kozhedub, I. A. Maltsev, A. V. Malyshev, R. V. Popov, I. I. Tupitsyn, and N. A. Zubova, Stringent tests of QED using highly charged ions, Hyperfine Interactions 239, 60 (2018).
  * Indelicato [2019] P. Indelicato, QED tests with highly charged ions, Journal of Physics B: Atomic, Molecular and Optical Physics 52, 232001 (2019).
  * Kozlov  _et al._ [2018] M. G. Kozlov, M. S. Safronova, J. R. Crespo López-Urrutia, and P. O. Schmidt, Highly charged ions: Optical clocks and applications in fundamental physics, Rev. Mod. Phys. 90, 045005 (2018).
  * Schmöger  _et al._ [2015] L. Schmöger, O. O. Versolato, M. Schwarz, M. Kohnen, A. Windberger, B. Piest, S. Feuchtenbeiner, J. Pedregosa-Gutierrez, T. Leopold, P. Micke, A. K. Hansen, T. M. Baumann, M. Drewsen, J. Ullrich, P. O. Schmidt, and J. R. C. López-Urrutia, Coulomb crystallization of highly charged ions, Science 347, 1233 (2015).
  * Schmidt  _et al._ [2005] P. O. Schmidt, T. Rosenband, C. Langer, W. M. Itano, J. C. Bergquist, and D. J. Wineland, Spectroscopy using quantum logic, Science 309, 749 (2005).
  * Micke  _et al._ [2020] P. Micke, T. Leopold, S. A. King, E. Benkler, L. J. Spieß, L. Schmöger, M. Schwarz, J. R. Crespo López-Urrutia, and P. O. Schmidt, Coherent laser spectroscopy of highly charged ions using quantum logic, Nature 578, 60 (2020).
  * King  _et al._ [2021] S. A. King, L. J. Spieß, P. Micke, A. Wilzewski, T. Leopold, J. R. Crespo López-Urrutia, and P. O. Schmidt, Algorithmic Ground-State Cooling of Weakly Coupled Oscillators Using Quantum Logic, Phys. Rev. X 11, 041049 (2021).
  * King  _et al._ [2022] S. A. King, L. J. Spieß, P. Micke, A. Wilzewski, T. Leopold, E. Benkler, R. Lange, N. Huntemann, A. Surzhykov, V. A. Yerokhin, J. R. Crespo López-Urrutia, and P. O. Schmidt, An optical atomic clock based on a highly charged ion, Nature 611, 43 (2022).
  * Barbieri and Ericson [1975] R. Barbieri and T. Ericson, Evidence against the existence of a low mass scalar boson from neutron-nucleus scattering, Physics Letters B 57, 270 (1975).
  * Leeb and Schmiedmayer [1992] H. Leeb and J. Schmiedmayer, Constraint on hypothetical light interacting bosons from low-energy neutron experiments, Phys. Rev. Lett. 68, 1472 (1992).
  * Nesvizhevsky  _et al._ [2008] V. V. Nesvizhevsky, G. Pignol, and K. V. Protasov, Neutron scattering and extra-short-range interactions, Phys. Rev. D 77, 034020 (2008).
  * Pokotilovski [2006] Y. N. Pokotilovski, Constraints on new interactions from neutron scattering experiments, Physics of Atomic Nuclei 69, 924 (2006).
  * Adler  _et al._ [1974] S. L. Adler, R. F. Dashen, and S. B. Treiman, Comments on proposed explanations for the muonic-atom x-ray discrepancy, Phys. Rev. D 10, 3728 (1974).
  * Bordag  _et al._ [2001] M. Bordag, U. Mohideen, and V. Mostepanenko, New developments in the casimir effect, Physics Reports 353, 1 (2001).
  * Bordag  _et al._ [2009] M. Bordag, G. L. Klimchitskaya, U. Mohideen, and V. M. Mostepanenko,  _Advances in the Casimir effect_ , Vol. 145 (OUP Oxford, 2009).
  * Germann  _et al._ [2021] M. Germann, S. Patra, J.-P. Karr, L. Hilico, V. I. Korobov, E. J. Salumbides, K. S. E. Eikema, W. Ubachs, and J. C. J. Koelemeij, Three-body QED test and fifth-force constraint from vibrations and rotations of ${\mathrm{}}HD{}{}{̂+}$, Phys. Rev. Research 3, L022028 (2021).
  * Salumbides  _et al._ [2013] E. J. Salumbides, J. C. J. Koelemeij, J. Komasa, K. Pachucki, K. S. E. Eikema, and W. Ubachs, Bounds on fifth forces from precision measurements on molecules, Phys. Rev. D 87, 112008 (2013).
  * Jaffe  _et al._ [2017] M. Jaffe, P. Haslinger, V. Xu, P. Hamilton, A. Upadhye, B. Elder, J. Khoury, and H. Müller, Testing sub-gravitational forces on atoms from a miniature in-vacuum source mass, Nat Phys 13, 938 (2017).
  * Biesheuvel  _et al._ [2016] J. Biesheuvel, J.-P. Karr, L. Hilico, K. S. E. Eikema, W. Ubachs, and J. C. J. Koelemeij, Probing QED and fundamental constants through laser spectroscopy of vibrational transitions in HD+, Nat Commun 7, 10385 (2016).
  * Kahl and Berengut [2019] E. Kahl and J. Berengut, AMBiT: A programme for high-precision relativistic atomic structure calculations, Computer Physics Communications 238, 232 (2019).
  * Levine  _et al._ [1988] M. A. Levine, R. E. Marrs, J. R. Henderson, D. A. Knapp, and M. B. Schneider, The Electron Beam Ion Trap: A New Instrument for Atomic Physics Measurements, Physica Scripta 1988, 157 (1988).
  * Levine  _et al._ [1989] M. A. Levine, R. E. Marrs, J. N. Bardsley, P. Beiersdorfer, C. L. Bennett, M. H. Chen, T. Cowan, D. Dietrich, J. R. Henderson, D. A. Knapp, A. Osterheld, B. M. Penetrante, M. B. Schneider, and J. H. Scofield, The use of an electron beam ion trap in the study of highly charged ions, Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms 43, 431 (1989).
  * Crespo López-Urrutia  _et al._ [1999a] J. R. Crespo López-Urrutia, A. Dorn, R. Moshammer, and J. Ullrich, The freiburg electron beam ion trap/source project FreEBIT, Physica Scripta T80, 502 (1999a).
  * Draganić  _et al._ [2003] I. Draganić, J. R. Crespo López-Urrutia, R. DuBois, S. Fritzsche, V. M. Shabaev, R. S. Orts, I. I. Tupitsyn, Y. Zou, and J. Ullrich, High precision wavelength measurements of QED-sensitive forbidden transitions in highly charged argon ions, Phys. Rev. Lett. 91, 183001 (2003).
  * Orts  _et al._ [2006] R. S. Orts, Z. Harman, J. R. C. López-Urrutia, A. N. Artemyev, H. Bruhns, A. J. G. Martínez, U. D. Jentschura, C. H. Keitel, A. Lapierre, V. Mironov, V. M. Shabaev, H. Tawara, I. I. Tupitsyn, J. Ullrich, and A. V. Volotka, Exploring relativistic many-body recoil effects in highly charged ions, Phys. Rev. Lett. 97, 103002 (2006).
  * Bekker  _et al._ [2018] H. Bekker, C. Hensel, A. Daniel, A. Windberger, T. Pfeifer, and J. R. Crespo López-Urrutia, Laboratory precision measurements of optical emissions from coronal iron, Phys. Rev. A 98, 062514 (2018).
  * Bekker  _et al._ [2019] H. Bekker, A. Borschevsky, Z. Harman, C. H. Keitel, T. Pfeifer, P. O. Schmidt, J. R. Crespo López-Urrutia, and J. C. Berengut, Detection of the 5p – 4f orbital crossing and its optical clock transition in Pr9+, Nature Communications 10, 5651 (2019).
  * Rehbehn  _et al._ [2021] N.-H. Rehbehn, M. K. Rosner, H. Bekker, J. C. Berengut, P. O. Schmidt, S. A. King, P. Micke, M. F. Gu, R. Müller, A. Surzhykov, and J. R. C. López-Urrutia, Sensitivity to new physics of isotope-shift studies using the coronal lines of highly charged calcium ions, Phys. Rev. A 103, L040801 (2021).
  * Gu [2008] M. F. Gu, The flexible atomic code, Canadian Journal of Physics 86, 675 (2008).
  * Crespo López-Urrutia  _et al._ [1999b] J. R. Crespo López-Urrutia, P. Beiersdorfer, K. Widmann, and V. Decaux, Visible spectrum of highly charged ions: the forbidden optical lines of Kr, Xe, and Ba ions in the Ar I to Ni I isoelectronic sequence, Physica Scripta T80, 448 (1999b).
  * Commission on Isotopic Abundances and Atomic Weights (2003) [CIAAW] Commission on Isotopic Abundances and Atomic Weights (CIAAW), Atomic weight of xenon (2003), (accessed: 2021-08-15).
  * Allehabi  _et al._ [2020] S. O. Allehabi, V. A. Dzuba, V. V. Flambaum, A. V. Afanasjev, and S. E. Agbemava, Using isotope shift for testing nuclear theory: The case of nobelium isotopes, Phys. Rev. C 102, 024326 (2020).
  * Bloom  _et al._ [2014] B. J. Bloom, T. L. Nicholson, J. R. Williams, S. L. Campbell, M. Bishof, X. Zhang, W. Zhang, S. L. Bromley, and J. Ye, An optical lattice clock with accuracy and stability at the $10^{-18}$ level, Nature 506, 71 (2014).
  * Cingöz  _et al._ [2012] A. Cingöz, D. C. Yost, T. K. Allison, A. Ruehl, M. E. Fermann, I. Hartl, and J. Ye, Direct frequency comb spectroscopy in the extreme ultraviolet, Nature 482, 68 (2012).
  * Lyu  _et al._ [2020] C. Lyu, S. M. Cavaletto, C. H. Keitel, and Z. Harman, Interrogating the temporal coherence of EUV frequency combs with highly charged ions, Phys. Rev. Lett. 125, 093201 (2020).
  * Nauta  _et al._ [2017] J. Nauta, A. Borodin, H. B. Ledwa, J. Stark, M. Schwarz, L. Schmöger, P. Micke, J. R. Crespo López-Urrutia, and T. Pfeifer, Towards precision measurements on highly charged ions using a high harmonic generation frequency comb, Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms 408, 285 (2017), proceedings of the 18th International Conference on the Physics of Highly Charged Ions (HCI-2016), Kielce, Poland, 11-16 September 2016.
  * Nauta  _et al._ [2021] J. Nauta, J.-H. Oelmann, A. Borodin, A. Ackermann, P. Knauer, I. S. Muhammad, R. Pappenberger, T. Pfeifer, and J. R. C. López-Urrutia, XUV frequency comb production with an astigmatism-compensated enhancement cavity, Opt. Express 29, 2624 (2021).
  * Weyers  _et al._ [2018] S. Weyers, V. Gerginov, M. Kazda, J. Rahm, B. Lipphardt, G. Dobrev, and K. Gibble, Advances in the accuracy, stability, and reliability of the PTB primary fountain clocks, Metrologia 55, 789 (2018).
  * Guéna  _et al._ [2017] J. Guéna, S. Weyers, M. Abgrall, C. Grebing, V. Gerginov, P. Rosenbusch, S. Bize, B. Lipphardt, H. Denker, N. Quintin, S. M. F. Raupach, D. Nicolodi, F. Stefani, N. Chiodo, S. Koke, A. Kuhl, F. Wiotte, F. Meynadier, E. Camisard, C Chardonnet, Y. L. Coq, M. Lours, G. Santarelli, A. Amy-Klein, R. L. Targat, O. Lopez, P. E. Pottie, and G. Grosche, First international comparison of fountain primary frequency standards via a long distance optical fiber link, Metrologia 54, 348 (2017).
  * Rosenband  _et al._ [2008] T. Rosenband, D. B. Hume, P. O. Schmidt, C. W. Chou, A. Brusch, L. Lorini, W. H. Oskay, R. E. Drullinger, T. M. Fortier, J. E. Stalnaker, S. A. Diddams, W. C. Swann, N. R. Newbury, W. M. Itano, D. J. Wineland, and J. C. Bergquist, Frequency ratio of Al+ and Hg+ single-ion optical clocks; metrology at the 17th decimal place, Science 319, 1808 (2008).
  * Godun  _et al._ [2014] R. M. Godun, P. B. R. Nisbet-Jones, J. M. Jones, S. A. King, L. A. M. Johnson, H. S. Margolis, K. Szymaniec, S. N. Lea, K. Bongs, and P. Gill, Frequency ratio of two optical clock transitions in ${}^{171}{\mathrm{Yb}}^{+}$ and constraints on the time variation of fundamental constants, Phys. Rev. Lett. 113, 210801 (2014).
  * Beloy  _et al._ [2021] K. Beloy, M. I. Bodine, T. Bothwell, S. M. Brewer, S. L. Bromley, J.-S. Chen, J.-D. Deschênes, S. A. Diddams, R. J. Fasano, T. M. Fortier, Y. S. Hassan, D. B. Hume, D. Kedar, C. J. Kennedy, I. Khader, A. Koepke, D. R. Leibrandt, H. Leopardi, A. D. Ludlow, W. F. McGrew, W. R. Milner, N. R. Newbury, D. Nicolodi, E. Oelker, T. E. Parker, J. M. Robinson, S. Romisch, S. A. Schäffer, J. A. Sherman, L. C. Sinclair, L. Sonderhouse, W. C. Swann, J. Yao, J. Ye, X. Zhang, and Boulder Atomic Clock Optical Network (BACON) Collaboration, Frequency ratio measurements at 18-digit accuracy using an optical clock network, Nature 591, 564 (2021).
  * Manovitz  _et al._ [2019] T. Manovitz, R. Shaniv, Y. Shapira, R. Ozeri, and N. Akerman, Precision measurement of atomic isotope shifts using a two-isotope entangled state, Phys. Rev. Lett. 123, 203001 (2019).
  * Zyriliou, A.  _et al._ [2022] Zyriliou, A., Mertzimekis, T. J., Chalil, A., Vasileiou, P., Mavrommatis, E., Bonatsos, Dennis, Martinou, Andriana, Peroulis, S., and Minkov, N., A study of some aspects of the nuclear structure in the even-even Yb isotopes, Eur. Phys. J. Plus 137, 352 (2022).