# Robust GAN inversion

Egor Sevriugov
Skolkovo institute of science and technology
Moscow, Russia 121205
<EMAIL_ADDRESS>Ivan Oseledets
Skolkovo institute of science and technology
Moscow, Russia 121205
<EMAIL_ADDRESS>
###### Abstract

Recent advancements in real image editing have been attributed to the
exploration of Generative Adversarial Networks (GANs) latent space. However,
the main challenge of this procedure is GAN inversion, which aims to map the
image to the latent space accurately. Existing methods that work on extended
latent space $W+$ are unable to achieve low distortion and high editability
simultaneously. To address this issue, we propose an approach which works in
native latent space $W$ and tunes the generator network to restore missing
image details. We introduce a novel regularization strategy with learnable
coefficients obtained by training randomized StyleGAN 2 model - WRanGAN. This
method outperforms traditional approaches in terms of reconstruction quality
and computational efficiency, achieving the lowest distortion with 4 times
fewer parameters. Furthermore, we observe a slight improvement in the quality
of constructing hyperplanes corresponding to binary image attributes. We
demonstrate the effectiveness of our approach on two complex datasets: Flickr-
Faces-HQ and LSUN Church.

##  1 Introduction

The emergence of generative adversarial neural networks (GANs) has made a
great contribution to high quality image synthesis. A well-known model in this
field is StyleGAN, which has achieved remarkable results. Moreover, several
works [19, 14, 17, 25, 23, 10] have demonstrated that GANs possess a wide
range of interpretable semantics, providing the basis for image editing. This
property enables the alteration of certain attributes while preserving the
identity of the image relative to others. However, the application of this
property to real images has been limited due to the need to accurately map
them into the latent space. This task, known as GAN inversion, initially
focused on mapping images into the native latent space $W$. Yet, authors in
[1] have shown that this approach leads to significant differences between the
original and generated images. Subsequent work has shifted focus to the
extended latent space $W+$[2, 20, 3, 24, 26, 9], which improves the quality of
image reconstruction but degrades editability. This issue, called the
distortion-editability tradeoff [24], limits the possibility of using codes
obtained in the $W+$ space.

Another way to solve the problem was proposed in [21], which includes a small
change in the generator parameters when working with the latent space $W$ \-
pivotal tuning inversion (PTI). We improve their idea by using adaptive
regularization instead of one number, since each parameter has different
contribution to the model performance.

In this paper, we present a novel approach for learning regularization that
allows for high-quality image reconstructions while preserving the ability of
the model to generate realistic images. This approach is based on a randomized
version of StyleGAN 2 called WRanGAN, in which part of the model weights are
assumed to be normally distributed with trainable mean and variance. To apply
different regularization coefficients, we use the reparameterization trick
[15] during the inversion procedure. The effectiveness of our technique was
evaluated on two complex datasets, the Flickr-Faces-HQ Dataset (FFHQ) [12] and
LSUN Churches [28]. Our contributions are summarized as follows:

  * •

We present a novel adaptive regularization scheme based on an investigation of
different regularization strategies and their effect on reconstruction quality
and model corruption.

  * •

We introduce WRanGAN, a model that learns appropriate regularization
coefficients via a randomization of the StyleGAN model.

  * •

We evaluate WRanGAN in terms of generation, reconstruction, binary attributes
extraction and computational cost, and compare it to several baselines in a
qualitative and quantitative manner.

##  2 Problem setting

###  2.1 Latent Space Manipulation

GANs allow the generation of images that are controlled by semantic directions
[19, 14, 17, 25, 23, 10]. In particular, in the work [10] authors proposed
estimating the subspaces that are invariant under random-walk diffusion for
identification. Supervision in the form of facial attribute labels was used in
[23] to find meaningful linear directions in the latent space. The
identification of latent directions based on the principal component analysis
(PCA) was proposed in [10].

###  2.2 GAN inversion

Recent research has focused on the improving reconstruction quality of GAN
inversion task, which involves finding the latent code that accurately
reproduce real image. This task can be divided into two main groups:
optimization methods that directly modify the latent code to minimize a loss
function [1, 18], and encoder-based methods that use a trained encoder to
generate an image [7, 3, 20, 24]. Generally, methods operate in the native
latent space $W$, which can lead to significant visual differences compared to
the original image [1]. On the other hand, the extended latent space $W+$ is
much more expressive and allows for the reproduction of more unique image
details. However, this approach is limited by the fixed generator parameters.
To address this issue, some approaches have proposed to modify the generator
network to fix visual artifacts, as demonstrated in [21]. Others have used
hypernetworks to predict the change of generator parameters in order to
minimize distortion and preserve the realism of the generated image, as seen
in [4] and [6].

###  2.3 Distortion-editability tradeoff

The GAN inversion in the extended latent space $W+$ significantly improves
reconstruction of real images, but at the same time it leads to degradation of
editability called distortion-editability trade-off [24]. There are works [30]
and [24], where the authors proposed to search for editable latent codes in an
extended latent space $W+$. A completely different way to solve this problem
was proposed in the works [4] and [21]. Instead of trying to find a balance
between editability and distortion, the authors suggest using the advantage of
projection into the latent space $W$ and updating the generator parameters to
minimize distortion. In this paper we also used projection to native latent
space to reach high editability.

Figure 1: Comparison of different regularization strategies of GAN inversion.
Simple Weight Tune represents optimization of model parameters with low
regularization coefficient. PTI represents pivotal tuning approach - high
regularization coefficient. And the last one is proposed WRanGAN. For each
approach metrics were calculated: MSE (lower values is better) measuring
distortion, FID (lower values is better) evaluated over images generated by
shifting latent code, measuring model corruption. For each approach we
presented result of latent code shifting in the 4 orthogonal directions
corresponding to the maximal change in the image.

###  2.4 Generator tuning

Model tuning significantly improves ability to reproduce real image [21, 4,
6]. But changing the parameters of the model can damage its quality. In order
to improve the realism of generated images after modification of the generator
weights, non-saturating GAN loss was used to train hypernetworks [4, 6]
(encoders predicting the necessary weight shift) . Despite the significant
improvement in the quality of reproduction, these methods are still inferior
to the PTI approach [21] based on direct weight optimization. But optimization
of model parameters without any additional constraints requires imposing a
regularization with a high coefficient in order not to damage the realism of
the generated images and forces to optimize all the parameters of the model to
reach low distortion. As a result, it leads to a significant increase in the
computational costs.

##  3 Method

In general, the proposed method solves the general problem by applying non-
equal learnable regularization. This allows to set appropriate regularization
coefficient for each parameter depending on its effect on model performance
(realism of generated images). In general approach, the first stage learns
appropriate regularization coefficients for the inversion task by adversarial
training of a generator with partially randomized parameters. The second
stage, then, uses these coefficients for an inversion procedure consisting of
encoder projection and regularized optimization minimizing particular loss
function.

Figure 2: Model corruption evaluation for two regularization strategies:
Simple Weight Tune and WRanGAN. Calculations performed over 30 randomly taken
images from FFHQ dataset. Corresponding mean values are also presented.

###  3.1 Model corruption assessment

To measure how well the model after tuning is, we use a common technique known
as Fréchet Inception Distance (FID) [8] and Kernel Inception Distance (KID)
[5]. In general case, evaluation is performed on a large set of images
produced by the generator network. However, since our main focus is on edited
images, we slightly change this tool by performing calculations over 1000
images obtained by shifting latent code (editing) in random directions. The
shift norm is taken accordingly to the characteristic size of the style space
(variance of style features).

###  3.2 Regularized inversion

In order to avoid degradation of realism in generated images, regularization
term is often added to optimization procedure. The general task formulation
looks like this:

$\hat{w},\hat{\theta}_{G}=\arg\min\limits_{w,\theta_{G}}\mathcal{L}(G(w,\theta_{G}),\hat{x})+\alpha_{\mathrm{reg}}\|\theta_{G}-\theta_{G,0}\|^{2}_{2}$

Here, $\hat{x}$ some real image,
$\alpha_{\mathrm{reg}}\|\theta_{G}-\theta_{G,0}\|^{2}_{2}$ the regularization
term, $\theta_{G,0}$ the initial values of the generator weights. For WRanGAN
inversion we used $\mathcal{L}=2\mathcal{L}_{2}+\mathcal{L}_{\mathrm{LPIPS}}$
and initialize intermediate latent code $w$ by mapping the output of the
trained encoder $E$ to intermediate latent space $W$: $w=f(E(\hat{x}))$.
$\alpha_{\mathrm{reg}}$ \- regularization coefficient, which choice is the
balance between reconstruction quality and model corruption. We considered
three strategies of regularization to illustrate this paradigm:

  * •

low regularization value (Simple Weight Tune)

  * •

high regularization value (PTI)

  * •

appropriate regularization coefficients (WRanGAN)

The results of our experiments, presented in Figure 1, demonstrate that a low
regularization value can impair visual quality and significantly reduce the
variability of the model (the ability to manipulate certain image attributes
suffers significantly). This is reflected in the FID metric used for model
corruption evaluation, which shows the highest value for Simple Weight Tune
strategy. Furthermore, applying a high regularization coefficient does not
reach the lowest distortion, as evidenced by the mean squared error (MSE).
Finally, the proposed WRanGAN model shows a good balance between both aspects:
reconstruction quality and model corruption.

A more detailed statistical view is presented in Figure 2. Here, we evaluated
the model corruption for the Simple Weight Tune and WRanGAN strategies in
comparison with the PTI approach - the difference for each image between the
FID metric value for the particular strategy and the corresponding value for
PTI. The results for the appropriate regularization strategy are much better
than those for Simple Weight Tune. The proposed method does not corrupt the
model.

###  3.3 WRanGAN inversion

In this part we have discussed how to apply regularization to randomized model
parameters $\theta_{G}\sim N(\mu_{\theta},\sigma_{\theta})$. To this end, we
employ the reparameterization trick, which states that
$\theta_{G}^{i}=\mu_{\theta}^{i}+\epsilon^{i}\sigma_{\theta}^{i}$ where
$\epsilon\sim N(0,1)$ and $i$ is the index of a particular parameter. By
regularizing the parameter $\epsilon$, we get the following equation:

$\alpha_{\mathrm{reg}}\|\epsilon\|^{2}_{2}=\sum\limits_{i}\frac{\alpha_{\mathrm{reg}}}{\sigma^{i}_{\theta}}(\theta^{i}_{G}-\mu^{i}_{\theta})^{2}=\sum\limits_{i}\alpha_{\mathrm{reg}}^{i}(\theta_{G}^{i}-\theta^{i}_{G,0})^{2}$

Here, we have used the notations
$\alpha_{\mathrm{reg}}^{i}=\frac{\alpha_{\mathrm{reg}}}{\sigma^{i}_{\theta}}$
and $\mu^{i}_{\theta}=\theta^{i}_{G,0}$, and have obtained a standard
regularization formulation with different regularization coefficients for each
randomized model parameter. The tips discussed in this part are summarized in
Algorithm 1.

Algorithm 1 Algorithm of WRanGAN inversion

Input: real image $\hat{x}$, generator parameters
$\mu_{\theta},\sigma_{\theta}$
Parameter: regularization coefficient $\alpha_{\mathrm{reg}}$
Output: latent code $w$ and parameterized randomization $\epsilon$

Initialize $w=E(x)$ by the output of encoder network

Initialize $\epsilon$ with small value ($10^{-4}$)

for number of iterations do

Set generator weights
$\theta_{G}\leftarrow\mu_{\theta}+\sigma_{\theta}\epsilon$

Update parameters $w,\epsilon$ minimizing:
$\mathcal{L}(G(w,\theta_{G}),\hat{x})+\alpha_{\mathrm{reg}}\|\epsilon\|_{2}^{2}$

end for

return $w$,$\epsilon$

Figure 3: Dependence of MSE on number of randomized layers $N$ versus
regularization coefficient. The lower the curve the better chosen number of
layers. Algorithm 2 WRanGAN training algorithm.

Input: pretrained StyleGAN 2 weights $\theta_{G,0}$, dataset $\hat{x}$
Parameter: batch size $m$
Output: $\mu_{\theta}$, $\sigma_{\theta}$

Initialize $\mu_{\theta}=\theta_{G,0}$

Initialize $\sigma_{\theta}=1$ for randomized parameters

for number of training iterations do

Sample ${z^{(1)},...,z^{(m)}}\sim N(0,1)$

Map to intermediate latent space $w^{(i)}=f(z^{(i)})$

Sample ${\hat{x}^{(1)},...,\hat{x}^{(m)}}$ from training dataset $\hat{x}$

Sample $\epsilon\sim N(0,1)$ and calculate
$\theta_{G}=\mu_{\theta}+\epsilon\sigma_{\theta}$

Update discriminator weights $\theta_{D}$ minimizing:
$\frac{1}{m}\sum\limits_{i=1}^{m}\mathcal{L}_{D}(D(\hat{x}^{(i)}),D(G(w^{(i)},\theta_{G})))$

Sample ${z^{(1)},...,z^{(m)}}\sim N(0,1)$

Map to intermediate latent space $w^{(i)}=f(z^{(i)})$

Sample $\epsilon\sim N(0,1)$ and calculate
$\theta_{G}=\mu_{\theta}+\epsilon\sigma_{\theta}$

Update parameters $(\mu_{\theta},\sigma_{\theta})$ minimizing:
$\frac{1}{m}\sum\limits_{i=1}^{m}\mathcal{L}_{g}(D(G(w^{(i)},\theta_{G})))$

end for

return $\mu_{\theta}$,$\sigma_{\theta}$

| Number of
---
randomized layers
| Relative increase in
---
amount of parameters
4 | 7%
6 | 23%
8 | 39%
Table 1: Quantitative comparison of memory cost on the number of randomized layers |  |  |  | LPIPS$\downarrow$ |  |  |
---|---|---|---|---|---|---|---
Domain | Model | Method | MSE$\downarrow$ | VGG | Alex | MS-SSIM$\uparrow$ |  GPU usage (MB)$\downarrow$ |  Time (s)$\downarrow$
|  | E4E | 0.062 | 0.389 | 0.235 | 0.605 | 2499 | 1.64
|  | Restyle | 0.035 | 0.335 | 0.154 | 0.72 | 2483 | 0.28
|  | SG2 W+ | 0.04 | 0.14 | 0.138 | 0.783 | 4295 | 97.9
|  | HyperStyle | 0.026 | 0.288 | 0.105 | 0.788 | 3583 | 0.31
| StyleGAN 2 | PTI | 0.024 | 0.293 | 0.06 | 0.776 | 3133 | 35.46
FFHQ | WRanGAN |  | WRanGAN
---
inversion
0.007 | 0.085 | 0.083 | 0.929 | 2557 | 23.27
|  | E4E | 0.142 | 0.506 | 0.418 | 0.263 | 2499 | 1.64
|  | Restyle | 0.087 | 0.411 | 0.25 | 0.489 | 2483 | 0.28
|  | SG2 W+ | 0.107 | 0.225 | 0.235 | 0.543 | 4295 | 97.9
| StyleGAN 2 | PTI | 0.053 | 0.411 | 0.065 | 0.643 | 3133 | 47
LSUN Church | WRanGAN |  | WRanGAN
---
inversion
0.033 | 0.177 | 0.224 | 0.782 | 2557 | 23.27
Table 2: Quantitative reconstruction results of WRanGAN model with corresponding method compared to StyleGAN 2 inversion approaches including encoder and optimization based. Assessment performed over several standard metrics, for each of them, the arrow identifies which values are better (lower $\downarrow$ / higher $\uparrow$). The best results for each evaluated metric are highlighted in bold. Values in blue outline the cases that we outperform PTI. Domain | Model | FID | Precision | Recall
---|---|---|---|---
Human Faces | StyleGAN 2 | 4.27 | 0.7 | 0.42
WRanGAN | 5.61 | 0.65 | 0.45
LSUN Church | StyleGAN 2 | 4.3 | 0.61 | 0.37
WRanGAN | 3.57 | 0.55 | 0.42
Table 3: WRanGAN model quality evaluation performed using FID, Precision, and
Recall metrics for two domains, FFHQ and LSUN Church. The best results for
each domain and metric are highlighted in bold.

###  3.4 Weight randomization

The idea of randomizing the model was inspired by Bayesian GAN [22], where the
generator and discriminator networks both assumed to have some distribution
over their internal parameters. However, randomizing the entire network is
computationally expensive due to the increased number of parameters required
for both training and generator parameters tuning during the inversion step.
We have already illustrated how to perform inversion using appropriate
regularization coefficients, and here, we present how to obtain such
coefficients.

#### How many parameters to randomize?

In [4], experiments were conducted to determine the most effective parameters
to be changed in the generator. It was decided to limit the randomization to
the last few convolutional layers, excluding the toRGB layers, and to keep the
discriminator architecture unchanged. To determine the appropriate number of
layers for randomization, a grid search was conducted over $N=4,6,8$ and
different equal regularization coefficients for the Simple Weight Tune method.
The results of this search are presented in Figure 3, and the computational
costs are shown in Table 1. It was determined that randomizing only the last
$N=6$ convolutional layers yielded the best results with a minimal increase in
computational costs.

#### How to train?

To train the WRanGAN model, a pre-trained model was used to initialize the
mean value of the model parameters $\mu_{\theta}=\theta_{G,0}$. Standard
deviation was then added to each randomized parameter with value equal one.
The generator and discriminator were trained together to reach the global
optima, as outlined in Algorithm 2.

##  4 Experiments

This section presents the results of the evaluation of the proposed WRanGAN
model. Below are presented the details of conducted experiments: datasets,
baselines, and hyperparameters.

Figure 4: Qualitative evaluation of WRanGAN inversion results compared to ones
produced by StyleGAN 2 using various approaches for FFHQ domain. For each
reconstruction provided zoomed version (interesting regions were cropped) to
see the difference in details completely. Figure 5: Qualitative evaluation of
WRanGAN inversion results compared to ones produced by StyleGAN 2 using
various approaches for LSUN Church domain. For each reconstruction provided
zoomed version (interesting regions were cropped) to see the difference in
details completely.

#### Technical details.

  * •

Models: We used the StyleGAN 2 [13] model as a basis, with pre-trained models
and base code for implementation taken from an open
resource111https://github.com/rosinality/stylegan2-pytorch.

  * •

Datasets: We trained using the Flickr-Faces-HQ Dataset (FFHQ) [12] with
pictures resized to resolution $256$x$256$, and LSUN Churches [28] with
pictures center-cropped and resized to $256$x$256$. We randomly sampled 1000
images from both datasets for testing.

  * •

WRanGAN training details: We used standard parameters for StyleGAN 2, and
trained on 2 GPUs with a batch size of $8$ for $200k$ iterations.

  * •

WRanGAN inversion details: For the encoder $E$ in Algorithm 1, we used the
architecture proposed by [24] and trained with default parameters. We used the
Adam optimizer with a learning rate of $lr=10^{-3}$, and the number of
iterations needed for convergence was set to $500$. The randomization
parameter was initialized with the value $\epsilon=10^{-4}$, and we used a
regularization coefficient of $\alpha_{\mathrm{reg}}=10^{-4}$.

Experiments were conducted on 4 Tesla V100-SXM2 GPUs with 16 GB of memory.

###  4.1 WRanGAN model evaluation

We evaluated the performance of our WRanGAN model by running several metrics
such as FID, Precision, and Recall [16] and comparing our results with those
produced by the StyleGAN 2 model (see Table 3). WRanGAN showed an improvement
in the Recall metric for both data domains, which suggests that the generator
is more likely to reproduce particular real images. However, we observed a
slight decrease in the Precision metric. For further details on the randomized
parameters of the model and their effect on the generated images, please refer
to Appendix A.

###  4.2 Inversion quality assessment

Evaluating the quality of inversion, various encoder based approaches such as
e4e [24], ReStyle [3], and HyperStyle[4], as well as optimization based
approaches like SG2 W+ [13] and PTI [21], were taken into consideration.
Standard metrics including mean squared error (MSE), LPIPS[29] with the VGG
and Alex feature network, and MS-SSIM [27] were used for assessment. As
summarized in Table 2, the WRanGAN model proposed surpasses all the other
methods applied to StyleGAN 2 on most metrics. Not only does it achieve the
lowest distortion, its computational efficiency is also much higher compared
to PTI, as the tuning procedure requires optimizing 4 times fewer parameters.
This translates to 600 megabytes less GPU memory required during image
inversion, making it more practical to use. Additionally, calculations are 1.5
and 2 times faster for FFHQ and LSUN Church domain respectively. Visualization
of the results in Figure 4 and Figure 5 further illustrate how the improvement
in reconstruction affects the image, with the WRanGAN approach being able to
reproduce unique details such as bangs, outline of the eyes, and small church
windows. More detailed visualizations and an investigation of the distribution
of randomized parameters for real mapped images can be found in Appendices B
and C respectively.

Domain | Model | Method | FID | KID
---|---|---|---|---
FFHQ | StyleGAN 2 | Restyle | 244 | 0.25
PTI | 222 | 0.225
WRanGAN |  | WRanGAN
---
inversion
225 | 0.232
LSUN Church | StyleGAN 2 | Restyle | 139 | 0.128
PTI | 133 | 0.134
WRanGAN |  | WRanGAN
---
inversion
144 | 0.144
Table 4: Model corruption evaluation. The two best methods for StyleGAN 2 were
taken for comparison:PTI and Restyle. The lower values are better for each
metric. Figure 6: Qualitative evaluation of WRanGAN editing quality compared
to PTI approach applied over StyleGAN 2 model. Figure 7: Qualitative
evaluation of WRanGAN interpolation quality compared to PTI approach applied
over StyleGAN 2 model. Here $\alpha$ denotes interpolation step.

###  4.3 Model corruption evaluation

Initially, we mentioned the effect of tuning the model parameters on the
ability to generate realistic images. In this part, we performed an assessment
of model corruption of tuned models using FID and KID metrics. The comparison
performed with two most efficient encoder and optimization based approaches:
Restyle and PTI. The results for both domains are presented in Table 4. The
difference in FID and KID values between the proposed WRanGAN and StyleGAN 2
inversion approaches does not exceed $11$ for both domains, which is much
smaller than $24$ (quality drop demonstrated by the Simple Weight Tune
regularization strategy). As a result, we concluded that WRanGAN preserves
ability generate realistic images after tuning.

Attribute | StyleGAN 2 | WRanGAN
---|---|---
Gender | 73.9 | 75.0
Eyeglasses | 99.8 | 99.9
Smile | 99.5 | 99.8
Age | 99.5 | 99.4
Mouth open | 98.2 | 98.5
Table 5: Classification accuracy (%) on separation boundaries in latent space
with respect to different face attributes. The best results are highlighted in
bold.

###  4.4 Editing and interpolation quality assessment

We conducted an experiment to confirm that the WRanGAN model possesses the
same excellent property as the StyleGAN 2 model - for any binary attribute,
there exists a hyperplane in latent space such that all samples from the same
side have the same attribute [23]. To do this, we trained a classifier
predicting the following attributes: Gender, Eyeglasses, Smile, Age, Open
Mouth. We then constructed hyperplanes in the latent space corresponding to
the selected attributes and evaluated their correctness, as shown in Table 5.
The results of our experiment demonstrate that the WRanGAN model has slightly
superior performance compared to the basic StyleGAN 2 model. This is also
evident in the visualization presented in Figure 6, where it is easy to notice
that the presence of glasses in the original image significantly affected the
editing of attributes in the case of the PTI method. However, WRanGAN
demonstrates an excellent performance. It’s also noticeable from interpolation
comparison presented in Figure 7. For more examples please refer to Appendix
D.

##  5 Conclusion

We have proposed a randomized version of the StyleGAN 2 model, dubbed WRanGAN,
which is able to learn the appropriate scaling (standard deviation) for each
parameter defining the corresponding regularization coefficient. Our approach
to GAN tuning using non-equal regularization coefficients demonstrated
superior results in terms of distortion and computational efficiency compared
to the most successful approach, pivotal tuning inversion. Moreover, it did
not corrupt the model, allowing for image editing. We also showed that in the
latent space of a randomized model, it is easy to construct a hyperplane
corresponding to the standard image attributes for the FFHQ domain.

Our method requires less memory per image in the inversion process, making it
easy to parallelize calculations. Additionally, the method is slightly
dependent on the network architecture enabling to transfer to other structures
such as StyleGAN 3 [11].

## Ethical Statement

The considered approach allows one to reproduce a photo of a real person with
high accuracy. So, the user can edit photos of real people, which can lead to
illegal actions.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4431–4440, 2019.
  * [2] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [3] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Restyle: A residual-based stylegan encoder via iterative refinement, 2021\.
  * [4] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H. Bermano.  Hyperstyle: Stylegan inversion with hypernetworks for real image editing, 2021.
  * [5] Mikołaj Bińkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying mmd gans, 2018.
  * [6] Tan M. Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua.  Hyperinverter: Improving stylegan inversion via hypernetwork.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [7] Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, and Xiaokang Yang.  Collaborative learning for faster stylegan embedding, 2020.
  * [8] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2018.
  * [9] Xueqi Hu, Qiusheng Huang, Zhengyi Shi, Siyuan Li, Changxin Gao, Li Sun, and Qingli Li.  Style transformer for image inversion and editing.  arXiv preprint arXiv:2203.07932, 2022.
  * [10] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  In Proc. NeurIPS, 2020.
  * [11] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In Proc. NeurIPS, 2021.
  * [12] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4396–4405, 2019.
  * [13] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8107–8116, 2020.
  * [14] Valentin Khrulkov, Leyla Mirvakhabova, Ivan Oseledets, and Artem Babenko.  Disentangled representations from non-disentangled models, 2021.
  * [15] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  CoRR, abs/1312.6114, 2014.
  * [16] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models.  CoRR, abs/1904.06991, 2019.
  * [17] William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, and Antonio Torralba.  The hessian penalty: A weak prior for unsupervised disentanglement.  In Proceedings of European Conference on Computer Vision (ECCV), 2020.
  * [18] Martin Pernuš, Vitomir Štruc, and Simon Dobrišek.  High resolution face editing with masked gan latent code optimization, 2021.
  * [19] David Pfau, Irina Higgins, Aleksandar Botev, and Sébastian Racanière.  Disentangling by subspace diffusion.  Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [20] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: A stylegan encoder for image-to-image translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2287–2296, June 2021.
  * [21] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  arXiv preprint arXiv:2106.05744, 2021.
  * [22] Yunus Saatci and Andrew G Wilson.  Bayesian gan.  In Advances in neural information processing systems, pages 3622–3631, 2017.
  * [23] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In CVPR, 2020.
  * [24] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  In ACM Transactions on Graphics, volume 40, pages 1–14, 2021.
  * [25] Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the gan latent space.  In International Conference on Machine Learning, pages 9786–9796. PMLR, 2020.
  * [26] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen.  High-fidelity gan inversion for image attribute editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [27] Z. Wang, Eero Simoncelli, and Alan Bovik.  Multiscale structural similarity for image quality assessment.  volume 2, pages 1398 – 1402 Vol.2, 12 2003.
  * [28] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [29] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [30] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  In Proceedings of European Conference on Computer Vision (ECCV), 2020.

.

##  Appendix A Investigation of WRanGAN model randomization

Figure 8: Dependence of distribution of randomized parameters variance with respect to index of convolutional layer for model trained on FFHQ dataset. Figure 9: Dependence of distribution of randomized parameters variance with respect to index of convolutional layer for model trained on LSUN Church dataset. Layer index | Percentage
---|---
7 | 0.13 %
8 | 0.16 %
9 | 0.18 %
10 | 0.29 %
11 | 0.40 %
12 | 0.52 %
Table 6: Percentage of small variances ($\sigma_{\theta}<10^{-3}$) for each randomized convolutional layer.  Figure 10: Qualitative reconstruction examples for images taken from FFHQ dataset. Figure 11: Qualitative reconstruction examples for images taken from LSUN Church dataset. Layer index | MSE
---|---
7 | 0.017
8 | 0.017
9 | 0.015
10 | 0.015
11 | 0.014
12 | 0.014
Table 7: Influence of each randomized layer on generator output.

The WRanGAN model is a type of generative model that optimizes two parameters
- the mean and variance - in its learning process. It is hypothesized that a
larger variance value allows for more changes to be made, which can be
observed by examining the distributions of the variance values of the trained
WRanGAN model. Figures 8 and 9 present the distributions for the FFHQ and LSUN
Church domains respectively. It can be seen that the distribution is shifted
towards zero as the number of randomized layers increases. In order to verify
this idea, a test was conducted to measure the number of parameters that have
a variance close to zero ($\sigma_{\theta}<10^{-3}$) in each layer. The
results, as presented in Table 6, indicate that the last layers have a greater
effect on the model performance. To further investigate this idea, a procedure
was conducted in which the parameters of different layers were changed and the
impact of these changes on the output image was assessed by calculating the
MSE metric (Table 7). The results confirm the hypothesis that the last layers
have the greatest influence.

Figure 12: Qualitative reconstruction examples for FFHQ dataset with
difference map, which represents pixel wise difference between reconstructed
and original images. Map colors range from purple to yellow - from exact match
to maximal difference. Figure 13: Qualitative reconstruction examples for LSUN
Church dataset with difference map, which represents pixel wise difference
between reconstructed and original images. Map colors range from purple to
yellow - from exact match to maximal difference.

##  Appendix B Additional qualitative results on reconstruction

The proposed approach of WRanGAN demonstrates a clear advantage in terms of
reproducing unique details, as demonstrated in the additional examples of
Figure 10 and Figure 11. Difference maps in Figure 12 and Figure 13 more
clearly show the areas of the image that differ from the original, where the
WRanGAN model has more accurately reproduced the unique details of facial skin
tones, wrinkles, clothing elements, complex hairstyles, and background
elements, such as the windows, friezes, pilasters, and clock in the case of
churches. However, some unique details are not completely restored even with
the use of the new approach.

Figure 14: Distribution of parameter $\epsilon$ among randomized layers for
image taken from FFHQ. Figure 15: Distribution of parameter $\epsilon$ among
randomized layers for image taken from FFHQ. Figure 16: Distribution of
parameter $\epsilon$ among randomized layers for image taken from FFHQ. Figure
17: Distribution of parameter $\epsilon$ among randomized layers for image
taken from FFHQ. Figure 18: Distribution of parameter $\epsilon$ among
randomized layers for image taken from FFHQ. Figure 19: Distribution of
parameter $\epsilon$ among randomized layers for image taken from LSUN Church.
Figure 20: Distribution of parameter $\epsilon$ among randomized layers for
image taken from LSUN Church. Figure 21: Distribution of parameter $\epsilon$
among randomized layers for image taken from LSUN Church. Figure 22:
Distribution of parameter $\epsilon$ among randomized layers for image taken
from LSUN Church. Figure 23: Distribution of parameter $\epsilon$ among
randomized layers for image taken from LSUN Church.

##  Appendix C Investigation of inversion procedure results

The proposed methodology utilizes a reparametrization trick to work with
randomized parameters. Specifically, the generator parameter $\theta_{g}$ is
represented as $\theta_{g}=\mu_{\theta}+\epsilon\sigma_{\theta}$, where
$\epsilon$ is sampled from a normal distribution. Figures 14, 15, 16, 17, 18,
and Figures 19, 20, 21, 22, 23, provide examples of the resulting distribution
of $\epsilon$ for images taken from FFHQ and LSUN Church respectively. In
general, the parameter is close to normally distributed, albeit with a
variance that is 10 times lower than that which was specified at the Wrangan
training stage for FFHQ, and 7 times lower for LSUN Church. Despite this
reduction, the proposed methodology is still successful.

Figure 24: Qualitative editing comparisons for FFHQ dataset Figure 25:
Qualitative editing comparisons for FFHQ dataset Figure 26: Qualitative
editing comparisons for FFHQ dataset Figure 27: Qualitative editing
comparisons for LSUN Church Figure 28: Qualitative editing comparisons for
LSUN Church

##  Appendix D Additional qualitative comparisons for editing

We conducted additional comparative experiments of the proposed WRanGAN
approach and the PTI inversion method for the StyleGAN 2 model in two domains:
the FFHQ domain, with semantic directions corresponding to binary image
attributes [23], and the LSUN Church domain, with the first 4 vectors obtained
by PCA approach [10]. The results were captured in Figures 24, 25, 26, 27, and
28.