# ParGAN: Learning Real Parametrizable Transformations

Diego Martin
Google Switzerland
<EMAIL_ADDRESS>
&Alessio Tonioni
Google Switzerland
<EMAIL_ADDRESS>
&Federico Tombari
Google Switzerland
Technical University of Munich
<EMAIL_ADDRESS>

###### Abstract

Current methods for image-to-image translation produce compelling results,
however, the applied transformation is difficult to control, since existing
mechanisms are often limited and non-intuitive. We propose ParGAN, a
generalization of the cycle-consistent GAN framework to learn image
transformations with simple and intuitive controls. The proposed generator
takes as input both an image and a parametrization of the transformation. We
train this network to preserve the content of the input image while ensuring
that the result is consistent with the given parametrization. Our approach
does not require paired data and can learn transformations across several
tasks and datasets. We show how, with disjoint image domains with no annotated
parametrization, our framework can create smooth interpolations as well as
learn multiple transformations simultaneously.

##  1 Introduction

|
---
|  day$\rightarrow$night |  |  |  |  |
---|---|---|---|---|---
summer$\rightarrow$sunset |  |  |  |  |
summer$\rightarrow$winter |  |  |  |  |
| $p=0.00$ | $p=0.25$ | $p=0.50$ | $p=0.75$ | $p=1.00$
Figure 1:  Left: Unimodal transformations [1, 2] (a) generate a unique image.
Multimodal ones (b) often require sampling and lack explicit control. The
output $y$ of our proposal (c) is driven by both the input $x$ and an
_intuitive parametrization_ $p$. Right: results of our method on different
tasks parametrized by a scalar. Each row is generated from the same input $x$
and increasing values of $p$.

Recent developments in Generative Adversarial Networks (GANs) have introduced
a general framework to learn image-to-image mapping functions between two
disjoint input and output domains (_e.g_., two sets of images) [2, 1]. While
some computer vision tasks can be directly addressed by these models (_e.g_.,
colorization, image to segmentation mask, style transfer…), others cannot. For
example, transforming an outdoor image of a sunny day into the same scene
under different weather conditions is certainly a useful application as a data
augmentation tool for training autonomous driving systems. However, solving
this task would require to learn a one-to-many mapping function or, even
better, a continuous parametrizable transformation, since weather conditions
can hardly be categorized as discrete classes [3]. Additional examples of
problems with similar continuous parametrizations might be learning the
illumination changes caused by a moving light source on a certain object or
scene, changing the focus point between foreground and background, etc.

The most common GAN-based architectures for image-to-image translation [2, 1]
unfortunately cannot handle these tasks, as they learn a unimodal mapping
function and do not provide any mechanism to explicitly control the
transformation process. Recent efforts have extended these formulations to
learn multimodal outputs [4, 5, 6] by relying on latent space sampling to
produce multiple images with the same content but different styles.
Unfortunately, the sampling process is often not directly interpretable and
hardly scales to practical applications. For example, given the sunny image
from the aforementioned use case, it is difficult to control the generator to
synthesize a range of images under different rainy conditions, or to
explicitly control the amount of “raininess”.

To address these issues we have developed a generalization of the GAN-based
unpaired image-to-image translation framework that allows it to learn a
parametrizable transformation between two or more domains without the need of
explicit supervision. We have designed our framework, which we named ParGAN,
for explicit control over the changes in the transformation: our generator
network accepts as input a parameterization of the transformation encoded in
real-world magnitudes (_e.g_., position in pixels, intensity change as a
percentage…). These values are easy to understand and control by a human user.
The discriminator also receives this parametrization in addition to the
images, in order to judge whether the generated image belongs to the target
domain and whether it is consistent with the value of $p$. Both components of
our GAN are therefore forced to explicitly take into account the
parametrization of the transformation when generating or discriminating
samples.

Feeding a parameterization of the transformation provides an easy and tunable
solution to generate both multi-modal outputs (_i.e_., same input image with
different parametrizations) as well as continuous and human interpretable
transformations. Our framework can be applied when there exist annotations for
a continuous parameterization of the transformation (_i.e_., supervised data),
as well as in the case of only domain-level categorical parameterization
(_i.e_., soft parameterization). For example, given two disjoint image
domains, we can use this information as soft supervision and define a
categorical parameterization of the transformation between them (_i.e_., a
single parameter with a value of either 0 or 1 according to the domain). Our
framework can learn to map images from one domain to the other, as well as to
smoothly interpolate between them according to the provided parameterization
without the need for samples from the intermediate domains at training time.
The same principle can be applied to transformations between a source domain
and multiple targets. In this scenario a single generator is able to smoothly
interpolate between the source and each one of the target domains
independently (disentanglement), while also obtaining promising results when
combining multiple transformations (mixing).

In the left part of Fig. 1 we summarize our novel problem formulation (c) with
respect to uni-modal (a) and multi-modal (b) image-to-image translation
networks. On the right part of Fig. 1 we show results on several tasks
obtained by our architecture using the soft parameterization described above.
To sum up the novel contributions of our work:

  * •

We propose a generalization of the unsupervised image-to-image translation
framework to learn explicitly parametrizable transformations. Existing works
either do not provide any control [1, 2, 7] or rely on hardly interpretable
sampling from a latent space [8, 9, 10, 11].

  * •

Our framework is general and can be applied to multiple use cases either with
supervision on the transformation parameters or only with soft categorical
labels. Other works have addressed only specific tasks with strong prior
knowledge [12, 13].

  * •

Our generator not only learns to replicate the transformation observed at
training time, but also to combine and smoothly interpolate multiple
transformations.

##  2 Related Works

##### Generative Adversarial Networks

The original GAN formulation [14] creates samples that resemble those of a
target distribution from a random noise vector. Since this vector is not
interpretable, several approaches have been proposed to improve this aspect.
Chen et al. [9] map independent dimensions of the input space to different
attributes in the output, while [15] uses known labels as additional inputs to
control the generation process. StyleGAN [8] proposes to disentangle content
and style by using the input noise vector to generate a set of learned affine
transformations that control different aspects of the image style at different
stages of the network. In all those works, however, the style depends on a
highly dimensional latent space that cannot be easily interpreted. Thus,
generating an image with a specific set of features can only be achieved by
either brute-force sampling or tuning some preexisting configuration.

##### Image-to-image translation

Works in the image-to-image translation literature attempt to learn a mapping
from an image domain to another (day to night, summer to winter…). GAN-based
works like Pix2Pix [2] or Pix2PixHD [7] learn this transformation using paired
images in the two domains. This setup allows for explicit supervision by
directly comparing the generated and expected images at a pixel or feature
level, achieving very realistic results. However, obtaining paired datasets is
often difficult and for certain tasks outright impossible. Cyclic-consistent
architectures such as CycleGAN [1], DiscoGAN [16] or CyCADA [17] remove this
constraint, enabling training with unpaired data in two different domains.
Since these networks learn a uni-modal mapping by design, it is difficult to
add variability or control the output. The common approach of feeding random
noise does not work for these methods, as the networks tend to ignore it [2].
Multimodal image-to-image translation approaches, such as [10, 5, 4, 6], have
been recently proposed to obtain multiple outputs from a single input. They
implement variability by sampling from a latent space, a parameterization not
easily controllable or interpretable without starting from some preexisting
configuration. StarGAN [18] learns multiple image transformations using a
single model by using attribute labels for guidance. Contrary to our approach,
this work does not explore the interpolation between source and target
domains. Additionally, it only focuses on discrete attribute transformations,
whereas we also explore parametrized transformations over a continuous domain.

##### Interpretable Parameterizations

Recent approaches have attempted to shed light on the interpretability of the
latent spaces used to condition generative models. [19] generates images
disentangling content and style by conditioning the former on a semantic
segmentation mask and the latter on a reference image. Approaches such as [20,
21] learn a controllable transformation for the face-aging task. However, they
still rely on a categorical parameterization (a discrete set of
transformations) while our model smoothly interpolates along a valid range.
HoloGAN [12] generates multiple viewpoints of the same image content produced
by a single random latent vector. The network is specifically designed for the
3D rotation task, as it explicitly uses a rigid-body motion layer, thus it
cannot be extended to non-geometric transformations. Recently [13] proposes a
network that, given an image of a face, learns to estimate the
parameterization required to generate a similar 3D avatar model using a
rendering engine. This task specific formulation does not generalize well
either. [22] modifies CycleGAN to increase the resolution of faces conditioned
by a one-hot encoded attribute vector (skin color, hair, glasses…). This
approach is also task-specific and discrete. A similar strategy is followed by
[23], where the conditioning parameters are intended as a “guidance” mechanism
to improve the reconstruction, although realistic outcomes can be obtained by
changing the parameters. This approach needs paired data, a requirement we
wish to avoid.

Recent works focused on exploring the learned latent space and determine which
dimensions alter a characteristic. [24] attempts to find the decision boundary
between binary labels, and [25] learns trajectories that conform to a smooth
transformation of some feature (zoom, viewpoint…). [26] produces images of
high quality and learns a style interpolation, however, it still needs to
extract the style code from a given image.

##  3 Method

Given a source image domain $X{}$, a target image domain $Y{}$ and a set of
possible parametrizations for a transformation $P{}\subset\mathbb{R}^{n}$, we
aim to implement a _parametrizable_ mapping $G^{P}{}:X{},P{}\rightarrow Y{}$.
We learn this by means of a neural network that generates a valid sample
$y{}\in Y{}$ by taking as input a source image $x{}\in X{}$ and a
parametrization for the transformation to apply $p{}\in P{}$. We rely on two
unpaired sets of training samples, $\\{x{{}_{i}}\\}_{i=1}^{N}$ and
$\\{y{{}_{j}},p{{}_{j}}\\}_{j=1}^{M}$, and on adversarial training by means of
a parametrization-aware discriminator $D^{P}_{Y}{}:Y{},P{}\rightarrow[0,1]$ to
provide the training signal for $G^{P}{}$.

###  3.1 Preliminaries: CycleGAN

Our work builds on top of the cyclic-consistent formulation first introduced
by _CycleGAN_ [1], a GAN architecture designed to learn a mapping function
$G{}:X{}\rightarrow Y{}$ between unpaired image domains $X{}$ and $Y{}$. To
achieve this objective the authors leverage an additional inverse mapping
function $F{}:Y{}\rightarrow X{}$ and one discriminator for each image domain,
_i.e_., $D_{X}{}$ and $D_{Y}{}$. The optimization objective is expressed as:
expressed as:

$\displaystyle\begin{split}\mathcal{L}_{gan}(G{},D_{Y}{},X{},Y{})&=\mathbb{E}_{y{}\sim
Y{}}[logD_{Y}{(}y{})]+\mathbb{E}_{x{}\sim
X{}}[log(1-D_{Y}{}(G{(}x{})))]\end{split}$ (1)
$\displaystyle\begin{split}\mathcal{L}_{cyc}(G{},F{})&=\mathbb{E}_{x{}\sim
X{}}[||F{}(G{}(x{}))-x{}||_{1}]+\mathbb{E}_{y{}\sim
Y{}}[||G{}(F{}(y{}))-y{}||_{1}]\end{split}$ (2)
$\displaystyle\begin{split}\mathcal{L}(G{},F{},D_{X}{},D_{Y}{})&=\mathcal{L}_{gan}(G{},D_{Y}{},X{},Y{})+\mathcal{L}_{gan}(F{},D_{X}{},Y{},X{})+\lambda\mathcal{L}_{cyc}(G{},F{})\end{split}$
(3)

where $G{}$ and $F{}$ are trained to minimize $\mathcal{L}$, while $D_{X}{}$
and $D_{Y}{}$ to maximize it. Our work starts from a similar formulation and
extends it to produce controllable transformations by means of
parametrization-aware generators and discriminators.

###  3.2 Learning parametrizable transformations

Figure 2:  Training of the ParGAN architecture starting with a sample $x{}\in
X{}$. A complete train iteration includes a mirrored cycle starting with a
sample $y{}\in Y{}$ being first fed to $F^{P}{}$, then to $G^{P}{}$.

Since our objective is learning _parametrizable_ transformations, we extend
the generators and discriminators of a cyclic architecture to have as
additional input a conditioning vector denoted as $p$. We refer to those
networks adding a P superscript: $G^{P}{}$, $F^{P}{}$, $D^{P}_{X}{}$,
$D^{P}_{Y}{}$. To maximize the generality of the proposed solution we encode
the parameters of the transformation in a $n$-dimensional vector of floats:
$p{}\in P{}\subset\mathbb{R}^{n}$. In Fig. 2 we report a schematic
representation of half of a training iteration for the framework. A full train
iteration will include a mirrored version of the architecture swapping $x{}$
and $y{}$ and inverting the order of the generators and discriminators.

##### Generators.

The generator architecture is based on a classic hourglass architecture, where
a set of convolutional layers encode the image into a compressed
representation that is fed to a series of residual blocks. In parallel, a
stack of fully connected layers learn to transform the raw values of $p$ to a
higher dimensionality representation. At the bottleneck layer of the generator
the image representation and the transformed $p$ are mixed by concatenation to
every spatial location along the feature dimension. Then a series of
convolutional layers decode the combined data into the final image. Instance
normalization is used after each convolution with ReLU as activation function.

To compute $\mathcal{L}_{cyc}$, at training time, we provide both generators
with the same conditioning vector $p{}$. Therefore, $G^{P}{}$ learns to apply
the transformation parametrized by $p{}$ while $F^{P}{}$ learns to undo it.

##### Discriminators.

Similarly to the generators, we extend the discriminator formulation of
CycleGAN to take into account the transformation of the parameterization to
learn. We denote these modified components as $D^{P}_{X}{}$ and $D^{P}_{Y}{}$
respectively. An initial convolutional layer operates solely on the image to
extract a suitable deep representation. In parallel $p{}$ is transformed
through a series of fully connected layers in a similar fashion as in the
generator. The learned representations of the image and of $p{}$ are mixed by
concatenation to every spatial location in the same way as the generators.
Afterwards, a series of convolutional layers operate on the mixed
representation to produce a real/fake classification score for every image
patch. The final prediction of the network is obtained with a mean average
pooling over all patches. LeakyReLU is used as activation function, with
instance normalization layers after every convolution.

##### Optimization objective.

The overall optimization goal of ParGAN extends Eq. 3 to take into account the
new parametric components and to rely on mean squared classification [27]. We
denote with $(x{},y{},p{})\sim(X{},Y{},P{})$ the distribution of training
examples. Each training sample is composed of a random image from $X{}$ and an
image from $Y$ with its corresponding parametrization $p$. We formalize our
objective as:

$\begin{split}&\mathcal{L}_{gan}^{P}{}(G^{P}{},D^{P}_{Y}{},X{},Y{},P{})=\mathbb{E}_{(x{},y{},p{})\sim(X{},Y{},P{})}\frac{1}{2}\left[(1-D^{P}_{Y}{(}y{},p{}))^{2}+D^{P}_{Y}{}(G{(}x{},p{}),p{})^{2}\right]\end{split}$
(4)
$\begin{split}\mathcal{L}_{cyc}^{P}{}(G^{P}{},F^{P}{})&=\mathbb{E}_{(x{},y{},p{})\sim(X{},Y{},P{})}\frac{1}{2}\big{[}||F{}^{P}{}(G{}^{P}{}(x{},p{}),p{})-x{}||_{1}+||G{}^{P}{}(F{}^{P}{}(y{},p{}),p{})-y{}||_{1}\big{]}\end{split}$
(5)
$\begin{split}\mathcal{L}(G^{P}{},F^{P}{},D^{P}_{X}{},D^{P}_{Y}{})&=\mathcal{L}_{gan}^{P}{}(G^{P}{},D^{P}_{Y}{},X{},Y{},P{})+\mathcal{L}_{gan}^{P{}}(F^{P}{},D^{P}_{X}{},X{},Y{},P{})+\lambda\mathcal{L}_{cyc}^{P}{}(G^{P}{},F^{P}{})\\\
\end{split}$ (6)

Eq. 6 defines the optimization objective composed of two GAN losses (defined
in Eq. 4) and a cyclic consistency loss (defined Eq. 5) weighted by a scalar
$\lambda$. The generators ($G^{P}{}$ and $F^{P}{}$) are optimized to minimize
the objective, while the discriminators ($D^{P}_{X}{}$ and $D^{P}_{Y}{}$) to
maximize it.

##### Soft Parametrization:

As we will show in Sec. 4.3 our formulation can be used even when the training
set does not provide an explicit parametrization but only a source $(X{})$ and
a target domain $(Y)$. In this setup, we can still use our framework to
produce not only target images, but also to smoothly interpolate between the
two domains. First, we define a new target domain composed of $X{}\cup Y{}$
and associate to each sample a one-dimensional parametrization vector $p{}$.
We use as parametrization the value $0$ for samples from $X{}$ and $1$ for
samples from $Y{}$. Thus our target domain becomes
$\\{(x_{1},0)\dots(x_{m},0),(y_{1},1)\dots(y_{n},1)\\}$.

Once we have created this softly parametrized target dataset, we train the GAN
system according to Eq. 6. Once $G^{P}{}$ is trained, we can use as
parametrization any real value between $[0,1]$ to generate transformations
that smoothly interpolate between the source and the target domains.

When provided with many target domains $(Y{{}_{0}},\dots,Y{{}_{n}})$, we can
apply the same technique using an $n$-dimensional vector for $p{}$. The
parametrization corresponds to a vector of zeros for samples originated from
$X{}$, while samples originated from the kth target domain $Y{{}_{k}}$ have
the kth entry set to one (one-hot encoding). In Sec. 4.4 we show that the
generator not only learns to smoothly interpolate between the source and any
of the target domains independently, but also, to some extent, to mix the
different transformations without having direct access to references for this
kind of transformations during training.

##  4 Experimental Results

Sec. 4.1 present our experimental setup, In Sec. 4.2 we show how we learn to
replicate explicitly parametrized transformations. Sec. 4.3 shows the use of a
soft parametrization approach to learn a smooth interpolable transformation.
Finally Sec. 4.4 shows how to combine multiple transformations in an
unsupervised fashion. Due to space constraint more experimental results and a
comparison to other methods are reported in the supplementary.

###  4.1 Experimental Setup

We consider various datasets for experimental evaluation. Although some
include paired samples (each image has a correspondence in the transformed
domain), we always assume unpaired data. All results reported in this paper
have been generated using unseen images from the validation set.
Book covers. To experiment with the idea of learning complex data
augmentations we have generated an ad-hoc dataset of book covers randomly
illuminated by a single beam of light. We start from clean and perfectly lit
images from [28], which we use as source domain ($X{}$). Then we simulate the
effect of a beam of light illuminating the cover with the Blender rendering
engine111blender.org and use these images as target domain ($Y{}$). Each image
is parametrized by a three dimensional float vector, with the first two
dimensions encoding the $(x,y)$ coordinates of the center of the light in the
image reference system, and the last the normalized intensity of the light.
SYNTHIA. In order to experiment with a soft parameterization, we learn a
transformation between different subsets of the SYNTHIA [29] dataset. The
dataset has 5 sequences, out of which we use 01, 02, 04 and 05 for training
and 06 for evaluation.
INIT. This dataset [30] consists of real-world dashcam images of driving
scenes subdivided by weather conditions. We have used the sunny and night
categories, with an 80:20 training/test split.

To measure quantitatively the impact of our transformation with respect to the
source and target domains, we use the Fréchet Inception Distance (FID)
[31]222We use the Inception v3 weights from https://github.com/fchollet/deep-
learning-models. and the Learned Perceptual Image Patch Similarity (LPIPS)
[32]333Implementation from github.com/alexlee-gk/lpips-tensorflow using the
AlexNet lin (v0.1) configuration.. FID compares statistics of Inception-v3
activations to measure the distance between two image distributions, while
LPIPS explicitly compares pairs of images and evaluates the perceptual
distance between image patches.

###  4.2 Explicit Parametrization

We first investigate if our model can learn an explicitly parametrized
transformation when provided with a target dataset of annotated samples such
as Book-Covers.

In Fig. 3 we show different images generated by ParGAN starting from the same
source image but with different parametrization vectors as input. The
generator correctly learns to generate realistic images augmented with a beam
of light according to the requested parametrization. For example in the last
row the position of the light beam is fixed and only the beam intensity
varies. The picture also shows how the generator learns a _smooth_ and
_disentangled_ transformation function even if trained with just sparse
samples of the space of possible transformations parametrized by $P{}$. Our
formulation is the first to provide this level of _explicit_ and _human
interpretable_ control over the transformation process.

In Tab. 1 we report the FID and the LPIPS metrics computed between different
sets of real and generated samples parametrized by different $p{}$. We keep
two axes of $p$ fixed and modify the remaining entry.

The tables show that when both distributions are parametrized by the same
conditioning vector, the corresponding distances are minimal (bold value along
the diagonal of the tables). In case of a parameterization mismatch, instead,
the distance between the two distributions tends to increase proportionally to
the distance between the two parametrizations. These results show that, for
the Book-Covers dataset, our network can correctly learn the transformation
parametrized by $p{}$.

Input Image |  | Lowest |  |  |  |  | Highest
---|---|---|---|---|---|---|---
|  Change x |  |  |  |  |  |
|  Change y |  |  |  |  |  |
|  Illumination |  |  |  |  |  |
Figure 3: Results on the Book-Covers dataset. Different components of the
parameterization (x-axis, y-axis, illumination) are correctly applied
independently by a single generator, as shown in each row.

|  |  | Gen. images
---|---|---
| y | 0.1 | 0.3 | 0.5 | 0.7 | 0.9
Real images | 0.1 | 44.653 | 51.138 | 56.646 | 57.139 | 59.616
0.3 | 45.999 | 39.51 | 44.004 | 51.397 | 56.183
0.5 | 58.377 | 49.87 | 36.199 | 43.358 | 51.292
0.7 | 52.312 | 50.086 | 39.629 | 33.255 | 38.819
0.9 | 59.292 | 56.471 | 46.54 | 32.878 | 27.206
|  |  |  | Gen. images
---|---|---
| y | 0.1 | 0.3 | 0.5 | 0.7 | 0.9
Real images | 0.1 | 0.064 | 0.084 | 0.115 | 0.134 | 0.147
0.3 | 0.077 | 0.056 | 0.079 | 0.109 | 0.132
0.5 | 0.109 | 0.082 | 0.055 | 0.083 | 0.108
0.7 | 0.125 | 0.106 | 0.071 | 0.051 | 0.073
0.9 | 0.134 | 0.120 | 0.095 | 0.059 | 0.041
(a) FID |  | (b) LPIPS

Table 1: Distances between real and generated images for Book-Covers. We fix
the x and illumination components of $p$ and sample different values for y.
The generated images are closest (bold) to the real images with the same y
value and the distance increases with it. The scores average the values for
100 samples. For the results on the x and illumination axes, we refer to the
supplement.

###  4.3 Soft Parametrization

We now test ParGAN for the case when no explicit parametrization is provided
at training time using the soft parametrization technique described in Sec.
3.2. We train a mapping between two domains and evaluate the impact of the
parametrization by sampling different values for $p$ in $[0,1]$. In this
scenario the input parametrization acts as a knob to control the intensity of
the transformation being applied, with $0$ not modifying anything (source
$\rightarrow$ source mapping) and $1$ applying the complete transformation
(source $\rightarrow$ target). Interestingly, the generator learns to smoothly
interpolate between the two states even if during training no explicit
supervision for this task was provided. In Fig. 4 and Fig. 1 we show images
generated using this configuration.

Source |  | 0.0 | 0.25 | 0.5 | 0.75 | 1.0
---|---|---|---|---|---|---
|  INIT
day$\rightarrow$night |  |  |  |  |
Figure 4: Evaluation using a soft parametrization. We refer to the supplement
for more samples.

Ideally, the distance between the source and transformed images should
increase with $p$, as the GAN progressively introduces more changes. Likewise,
the distance to the target domain should decrease. We verify this
experimentally and in Tab. 2 we report the FID and LPIPS scores for the
SYNTHIA summer$\rightarrow$winter and INIT day$\rightarrow$night tasks
comparing the result images with randomly picked sets of source and target
samples.

|  | Vs. source images | Vs. target images
---|---|---
p | FID | LPIPS | FID | LPIPS
0 | 76.361 | 0.586 | 150.662 | 0.625
0.25 | 79.533 | 0.591 | 147.997 | 0.616
0.5 | 92.919 | 0.611 | 140.181 | 0.605
0.75 | 101.385 | 0.618 | 137.599 | 0.601
1 | 100.634 | 0.613 | 132.208 | 0.590
|  | Vs. source images | Vs. target images
---|---|---
p | FID | LPIPS | FID | LPIPS
0. | 95.625 | 0.650 | 188.851 | 0.734
0.25 | 96.735 | 0.654 | 183.410 | 0.731
0.5 | 101.213 | 0.659 | 177.468 | 0.721
0.75 | 108.562 | 0.668 | 166.436 | 0.700
1 | 115.793 | 0.683 | 149.868 | 0.679
a) SYNTHIA summer$\rightarrow$winter | b) INIT day$\rightarrow$night

Table 2: Average FID and LPIPS scores. Dark color in the denotes lower values
(more similar images). The distance w.r.t. the source domain increases with
$p$ and decreases for the target.

The distance to the target images is often much larger than for the source
domain, since in one case we are applying a complex image transformation and
in the other a simple identity mapping.

###  4.4 Simultaneous learning of transformations

We test the behavior of our model when using a single source domain and
multiple targets. Following the strategy outlined in Sec. 3.2 we use a soft-
parameterization associating to each target domain a one-hot encoded
categorical vector $p$.

Thus, for a two-target transformation, the source domain has label $p=[0,0]$,
and the two target domains have $p=[1,0]$ and $p=[0,1]$ respectively.

We train in the same fashion as the previous experiments for the
summer$\rightarrow$winter+night task on the SYNTHIA dataset and report the
result in Fig. 5. Each transformation is learned independently and does not
conflict with the others (the generator learns a disentangled mapping). We
refer to the supplementary material for a qualitative and quantitative
comparison between the transformations learned jointly and independently as
well as for experimental results.

\begin{overpic}[width=56.37271pt]{images/datasets/summer2winter_night-4/06-000432-original} \put(17.0,6.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize$\text{Source}$}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=56.37271pt]{images/datasets/summer2winter_night-4/06-000432-0.00-0.25} \put(17.0,6.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize$p=[0,0.25]$}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=56.37271pt]{images/datasets/summer2winter_night-4/06-000432-0.00-0.50} \put(17.0,6.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize$p=[0,0.50]$}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=56.37271pt]{images/datasets/summer2winter_night-4/06-000432-0.00-0.75} \put(17.0,6.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize$p=[0,0.75]$}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=56.37271pt]{images/datasets/summer2winter_night-4/06-000432-0.00-1.00} \put(17.0,6.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize$p=[0,1]$}}}\end{flushright}}} \end{overpic}
---|---|---|---|---
\begin{overpic}[width=56.37271pt]{images/datasets/summer2winter_night-4/06-000432-0.25-0.00} \put(17.0,6.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize$p=[0.25,0]$}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=56.37271pt]{images/datasets/summer2winter_night-4/06-000432-0.50-0.00} \put(17.0,6.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize$p=[0.50,0]$}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=56.37271pt]{images/datasets/summer2winter_night-4/06-000432-0.75-0.00} \put(17.0,6.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize$p=[0.75,0]$}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=56.37271pt]{images/datasets/summer2winter_night-4/06-000432-1.00-0.00} \put(17.0,6.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize$p=[1,0]$}}}\end{flushright}}} \end{overpic}
Figure 5: summer$\rightarrow$winter+night transformations (Top: Night, Bottom:
Winter). The network learns disentangled transformations that can be applied
without mixing.

#### Mixing Transformations

ParGAN can learn to _mix_ the two transformations to some extent, producing
images that display features from different domains (e.g. dark sky and snow).
We show some results in Fig. 6.

\begin{overpic}[width=73.7146pt]{images/datasets/summer2winter_night_softrain/06-000137-0.00-0.25-0.75} \put(34.0,20.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize 0\% {winter}} {\scriptsize 25\% {night}} {\scriptsize 75\% {rain}}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=73.7146pt]{images/datasets/summer2winter_night_softrain/06-000137-0.00-1.00-1.00} \put(34.0,20.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize 0\% {winter}} {\scriptsize 100\% {night}} {\scriptsize 100\% {rain}}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=73.7146pt]{images/datasets/summer2winter_night_softrain/06-000137-0.25-0.00-0.50} \put(34.0,20.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize 25\% {winter}} {\scriptsize 0\% {night}} {\scriptsize 50\% {rain}}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=73.7146pt]{images/datasets/summer2winter_night_softrain/06-000137-0.50-0.75-0.50} \put(34.0,20.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize 50\% {winter}} {\scriptsize 75\% {night}} {\scriptsize 50\% {rain}}}}\end{flushright}}} \end{overpic} | \begin{overpic}[width=73.7146pt]{images/datasets/summer2winter_night_softrain/06-000137-0.50-0.75-0.00} \put(34.0,20.0){\parbox{42.67912pt}{\begin{flushright}{{\color[rgb]{1,1,1}{\scriptsize 50\% {winter}} {\scriptsize 75\% {night}} {\scriptsize 0\% {rain}}}}\end{flushright}}} \end{overpic}
---|---|---|---|---
Figure 6: Arbitrary mixings on the summer$\rightarrow$winter+night+softrain
task.

##  5 Conclusions

We have proposed ParGAN, a new model for image-to-image translation using
intuitive parameterizations to control the outcome. Our formulation is able to
learn not only how to replicate complex parametric transformations, but also
how to smoothly interpolate between disjoint domains using only a soft
categorical parameterization. Additionally, our method does not rely on prior
knowledge or ad-hoc architectures. While those approaches would have likely
lead to faster convergence and higher reconstruction quality, it would
constrain the versatility of the method to a small range of tasks, requiring
fundamental changes in order to learn different transformations. Since human
interpretability is at the core of our proposal, our network always takes a
parameterization of the transformation encoded in real-world magnitudes.
Finally, we have shown some promising results on learning how to mix different
transformations in a completely unsupervised fashion.

In the future, we wish to test our method on more parametric transformations
to further verify the generality of the proposed solutions. Moreover, our
method still generates some artifacts while trying to mix different
transformations. We believe that this issue might be mitigated by explicitly
adding an adversarial component in the loss function to target these use
cases. We will explore this direction in future works.

## Broader Impact

Despite their recent development, learning-based approaches for image
transformation have already found applications in various fields, such as
entertainment or content generation. We believe that tasks such as object
detection or semantic segmentation can benefit from our method by increasing
the amount of available training data. Specially, our method can be of
particular interest to reduce bias for unbalanced categories, such as day and
night in the context of autonomous driving. This can help to increase accuracy
on underrepresented modalities. Parametrizable methods like ours can also
determine the kind of data to generate more precisely based on particular
requirements. Examples are constraining the location of an augmentation to a
particular area, or limiting the percentage of a global transformation to
apply (such as 50% snow). In addition to data augmentation techniques, the
network can also be deployed as part of image editing tools to allow end users
to edit images directly.

While the quality of GAN-generated imagery keeps improving, some generated
images may still present some artifacts. This is not only aesthetically
unpleasant but it can also impact the success of downstream tasks inducing
unwanted biases into the generated datasets. A recent work [33] also points
out that images generated by cyclic architectures might contain invisible
artifacts not present in the original image distribution, again a source of
bias in the data set. The main way to fight these unwanted artifacts is to
increase the quality of the generated images (_i.e_., better architectures and
better adversarial loss functions) as well as to add explicit regularization
to limit this type of misbehavior.

## References

  * [1] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.:  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In: Computer Vision (ICCV), 2017 IEEE International Conference on. (2017)
  * [2] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.:  Image-to-image translation with conditional adversarial networks.  arxiv (2016)
  * [3] Hoffman, J., Darrell, T., Saenko, K.:  Continuous manifold based adaptation for evolving visual domains.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2014) 867–874
  * [4] Zhu, J.Y., Zhang, R., Pathak, D., Darrell, T., Efros, A.A., Wang, O., Shechtman, E.:  Toward multimodal image-to-image translation.  In: Advances in Neural Information Processing Systems. (2017) 465–476
  * [5] Huang, X., Liu, M.Y., Belongie, S., Kautz, J.:  Multimodal unsupervised image-to-image translation.  In: Proceedings of the European Conference on Computer Vision (ECCV). (2018) 172–189
  * [6] Lee, H., Tseng, H., Huang, J., Singh, M.K., Yang, M.:  Diverse image-to-image translation via disentangled representations.  CoRR abs/1808.00948 (2018)
  * [7] Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.:  High-resolution image synthesis and semantic manipulation with conditional gans.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2018)
  * [8] Karras, T., Laine, S., Aila, T.:  A style-based generator architecture for generative adversarial networks.  CoRR abs/1812.04948 (2018)
  * [9] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.:  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  CoRR abs/1606.03657 (2016)
  * [10] Zhu, J., Zhang, R., Pathak, D., Darrell, T., Efros, A.A., Wang, O., Shechtman, E.:  Toward multimodal image-to-image translation.  In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R., eds.: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA. (2017) 465–476
  * [11] Huang, X., Liu, M.Y., Belongie, S., Kautz, J.:  Multimodal unsupervised image-to-image translation.  In: ECCV. (2018)
  * [12] Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C., Yang, Y.L.:  Hologan: Unsupervised learning of 3d representations from natural images (2019)
  * [13] Shi, T., Yuan, Y., Fan, C., Zou, Z., Shi, Z., Liu, Y.:  Face-to-parameter translation for game character auto-creation (2019)
  * [14] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.:  Generative adversarial nets.  In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q., eds.: Advances in Neural Information Processing Systems 27.  Curran Associates, Inc. (2014) 2672–2680
  * [15] Mirza, M., Osindero, S.:  Conditional generative adversarial nets.  CoRR abs/1411.1784 (2014)
  * [16] Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J.:  Learning to discover cross-domain relations with generative adversarial networks.  In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org (2017) 1857–1865
  * [17] Hoffman, J., Tzeng, E., Park, T., Zhu, J., Isola, P., Saenko, K., Efros, A.A., Darrell, T.:  Cycada: Cycle-consistent adversarial domain adaptation.  CoRR abs/1711.03213 (2017)
  * [18] Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.:  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2018)
  * [19] Park, T., Liu, M., Wang, T., Zhu, J.:  Semantic image synthesis with spatially-adaptive normalization.  CoRR abs/1903.07291 (2019)
  * [20] Antipov, G., Baccouche, M., Dugelay, J.:  Face aging with conditional generative adversarial networks.  CoRR abs/1702.01983 (2017)
  * [21] Wang, Z., Tang, X., Luo, W., Gao, S.:  Face aging with identity-preserved conditional generative adversarial networks.  In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, IEEE Computer Society (2018) 7939–7947
  * [22] Lu, Y., Tai, Y., Tang, C.:  Conditional cyclegan for attribute guided face image generation.  CoRR abs/1705.09966 (2017)
  * [23] Yu, X., Porikli, F., Fernando, B., Hartley, R.:  Hallucinating unaligned face images by multiscale transformative discriminative networks.  International Journal of Computer Vision (2019)
  * [24] Shen, Y., Gu, J., Tang, X., Zhou, B.:  Interpreting the latent space of gans for semantic face editing.  CoRR abs/1907.10786 (2019)
  * [25] Jahanian, A., Chai, L., Isola, P.:  On the "steerability" of generative adversarial networks.  CoRR abs/1907.07171 (2019)
  * [26] Anokhin, I., Solovev, P., Korzhenkov, D., Kharlamov, A., Khakhulin, T., Silvestrov, A., Nikolenko, S., Lempitsky, V., Sterkin, G.:  High-resolution daytime translation without domain labels.  In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (June 2020)
  * [27] Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.:  Least squares generative adversarial networks.  In: Proceedings of the IEEE International Conference on Computer Vision. (2017) 2794–2802
  * [28] Iwana, B.K., Raza Rizvi, S.T., Ahmed, S., Dengel, A., Uchida, S.:  Judging a book by its cover.  arXiv preprint arXiv:1610.09204 (2016)
  * [29] Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.:  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (June 2016) 3234–3243
  * [30] Shen, Z., Huang, M., Shi, J., Xue, X., Huang, T.:  Towards instance-level image-to-image translation.  In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. (2019)
  * [31] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.:  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In: Advances in Neural Information Processing Systems. (2017) 6626–6637
  * [32] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.:  The unreasonable effectiveness of deep features as a perceptual metric.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2018) 586–595
  * [33] Zhang, K.A., Cuesta-Infante, A., Xu, L., Veeramachaneni, K.:  Steganogan: High capacity image steganography with gans.  CoRR abs/1901.03892 (2019)
  * [34] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.:  TensorFlow: Large-scale machine learning on heterogeneous systems (2015) Software available from tensorflow.org.
  * [35] Kingma, D.P., Ba, J.:  Adam: A method for stochastic optimization.  In Bengio, Y., LeCun, Y., eds.: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. (2015)

## Supplementary material for

In the main body of this paper we have proposed our method for controllable
image-to-image translation and showed its capabilities on different datasets.
In this supplement, we show additional experiments and qualitative results in
order to further analyze the properties of our method and how it compares to
existing approaches.

In Sec. 1.1 we describe the datasets used to evaluate our method. Sections
1.2,1.3 and 1.4 report additional quantitative and qualitative results. In
Sec. 1.5 we compare our model to a standard CycleGAN [1] to show that the
addition of the parametrization does not affect the generated image quality.
In Sec. 1.6 we show how our models learns to some extent to interpolate
between different domain even without having ever received explicit
supervision for it In Sec. 1.7 and Sec. 1.8 we provide some intuition on how
the network learns parametric transformations. In Sec. 1.9 we show some
interesting domain generalization result on image acquired by a phone. Finally
we conclude with some implementation details, Sec. 1.10, and ablation studies
on the better way of injecting $p$ in the generator and discriminator networks
Sec. 1.11.

###  1.1 Datasets

|
---|---

\begin{overpic}[width=69.38078pt]{images/datasets/synthia-original/summer/02-000385} \put(2.0,2.0){{{\color[rgb]{1,1,1}Summer}}} \end{overpic} | \begin{overpic}[width=69.38078pt]{images/datasets/synthia-original/winter/04-000408} \put(2.0,2.0){{{\color[rgb]{1,1,1}Winter}}} \end{overpic} | \begin{overpic}[width=69.38078pt]{images/datasets/synthia-original/night/02-000385} \put(2.0,2.0){{{\color[rgb]{1,1,1}Night}}} \end{overpic} | \begin{overpic}[width=69.38078pt]{images/datasets/synthia-original/softrain/02-000385} \put(2.0,2.0){{{\color[rgb]{1,1,1}Softrain}}} \end{overpic} | \begin{overpic}[width=69.38078pt]{images/datasets/synthia-original/fog/02-000385} \put(2.0,2.0){{{\color[rgb]{1,1,1}Fog}}} \end{overpic} | \begin{overpic}[width=69.38078pt]{images/datasets/synthia-original/sunset/02-000385} \put(2.0,2.0){{{\color[rgb]{1,1,1}Sunset}}} \end{overpic}
---|---|---|---|---|---

|  |  |  |  |
---|---|---|---|---|---

Figure 1: Examples from different datasets, one per row: (1) Book-Covers [28]
(left for original images and right for image rendered with our light
transformations), (2) SYNTHIA [29], and (3) INIT [30] (left for day and right
for night).

We explore the capabilities of our model in two different tasks: learning a
transformation guided by an explicit parametrization and domain translation
using soft labels. We use both real and synthetic data. In Fig. 1 we show
several samples from the datasets that we used for training.

###  1.2 Quantitative analysis of explicit parameterization

In order to measure the quality of our learned transformations, we measure the
similarity (in terms of FID and LPIPS between a ground truth image with known
transformation properties and the same book cover with varying values in $p$.
In Tables 1 and 2 we report the results when changing the values of the x and
illumination axes for Book-Covers.

|  |  | Gen. images
---|---|---
| x | 0.1 | 0.3 | 0.5 | 0.7 | 0.9
Real images | 0.1 | 41.739 | 47.339 | 52.822 | 48.888 | 46.95
0.3 | 50.613 | 48.411 | 48.506 | 52.25 | 53.345
0.5 | 52.887 | 51.408 | 46.358 | 50.455 | 54.313
0.7 | 52.763 | 54.878 | 49.369 | 41.152 | 42.421
0.9 | 51.103 | 55.345 | 53.384 | 45.096 | 38.200
|  |  |  | Gen. images
---|---|---
| x | 0.1 | 0.3 | 0.5 | 0.7 | 0.9
Real images | 0.1 | 0.062 | 0.076 | 0.102 | 0.120 | 0.125
0.3 | 0.080 | 0.061 | 0.080 | 0.103 | 0.115
0.5 | 0.106 | 0.085 | 0.065 | 0.087 | 0.104
0.7 | 0.114 | 0.101 | 0.076 | 0.060 | 0.074
0.9 | 0.117 | 0.107 | 0.090 | 0.066 | 0.054
(a) FID |  | (b) LPIPS

Table 1: Comparison between real and generated images for the Book-Covers
dataset, keeping y and illumination fixed and varying the x component of $p$.
The more dissimilar the values of x, the higher the distance. In bold the
smallest value column-wise.

|  |  | Gen. images
---|---|---
| ill. | 0.1 | 0.3 | 0.5 | 0.7 | 0.9
Real images | 0.1 | 38.434 | 43.924 | 54.309 | 60.664 | 64.754
0.3 | 53.696 | 42.526 | 45.665 | 49.593 | 53.514
0.5 | 50.360 | 35.291 | 34.303 | 36.321 | 38.541
0.7 | 58.330 | 42.100 | 38.004 | 39.777 | 42.197
0.9 | 69.816 | 51.095 | 46.816 | 45.315 | 46.511
|  |  |  | Gen. images
---|---|---
| ill. | 0.1 | 0.3 | 0.5 | 0.7 | 0.9
Real images | 0.1 | 0.063 | 0.08 | 0.121 | 0.15 | 0.168
0.3 | 0.107 | 0.065 | 0.073 | 0.09 | 0.105
0.5 | 0.139 | 0.069 | 0.053 | 0.058 | 0.067
0.7 | 0.171 | 0.089 | 0.06 | 0.055 | 0.059
0.9 | 0.196 | 0.11 | 0.074 | 0.063 | 0.061
(a) FID |  | (b) LPIPS

Table 2: Comparison between real and generated images for the Book-Covers
dataset on the illumination component of $p$.

These results complement Tab. 1 in the main paper and show how our generator
can correctly apply the desired transformation in most cases. This is
testified by the distances between generated and real images being usually
lower when the parametrization between the two sets matches (_i.e_., values
along the diagonal).

###  1.3 Disentanglement

In Fig. 2 we show the disentangling capabilities of our model applied to the
Book-Covers dataset. Each axis can be modified independently with no effect
over the others, and arbitrary combinations are also feasible.

Change x |  |  |  |  |  |
---|---|---|---|---|---|---
Change y |  |  |  |  |  |
Change illumination |  |  |  |  |  |
Change all |  |  |  |  |  |
Figure 2:  Additional results on the Book-Covers dataset. The first three rows
show disentangled transformations, where two out of three axes are kept fixed.
The last row contains entangled transformations, where all axes are modified
simultaneously.

###  1.4 Results for the soft-parametrization

Source |  | 0.0 | 0.25 | 0.5 | 0.75 | 1.0
---|---|---|---|---|---|---
|  summer$\rightarrow$winter |  |  |  |  |
|  summer$\rightarrow$fog |  |  |  |  |
|  summer$\rightarrow$sunset |  |  |  |  |
|  summer$\rightarrow$softrain |  |  |  |  |
Figure 3: Evaluation on different tasks for SYNTHIA.

We show in Fig. 3 generated images for the SYNTHIA dataset. We use the summer
subset as source domain and learn multiple transformations
(summer$\rightarrow$winter, summer$\rightarrow$fog, summer$\rightarrow$sunset,
summer$\rightarrow$softrain).

In Fig. 4 we show qualitative results using the INIT dataset for the
day$\rightarrow$night task. The network learns to interpolate between the two
domains, progressively darkening the image in the necessary areas (sky, road)
and adding elements proper of the target domain (car lights, street lights).

Source | 0.0 | 0.25 | 0.5 | 0.75 | 1.0
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Figure 4: Results on the INIT dataset for the day$\rightarrow$night case Source |  | 0.0 | 0.25 | 0.5 | 0.75 | 1.0
---|---|---|---|---|---|---
|  summer$\rightarrow$dawn |  |  |  |  |
|  summer$\rightarrow$sunset |  |  |  |  |
|  summer$\rightarrow$night |  |  |  |  |
|  summer$\rightarrow$dawn |  |  |  |  |
|  summer$\rightarrow$sunset |  |  |  |  |
|  summer$\rightarrow$night |  |  |  |  |
Figure 5: Disentangled evaluation of the summer$\rightarrow$dawn+sunset+night
task.

Similarly to the Book-Covers dataset, the network can learn multiple domain
transformations simultaneously. We show this in Fig. 5 for the
summer$\rightarrow$dawn+sunset+night task. A _single_ model is trained to
learn three different domain mappings. No conflicts between the domains (such
as artifacts) are noticeable.

We measure the quality of these transformations numerically using the FID [31]
and LPIPS [32] metrics. We evaluate the model with different values of $p$ on
a random set of images from the source domain and measure the distance to
different random subsets of both source and target images. Ideally, images
transformed with $p=0$ have the lowest metric value when compared against the
source domain, as the network simply performs an identity map.

The opposite happens when comparing with the target images. The most similar
image set is that with $p=1$. We report our findings on Tab. 3 for individual
transformations and Tab. 4 for entangled ones. The results show how the
network is able to progressively make the image more dissimilar from the
source domain and more similar to the target one.

|  | Vs. source | Vs. target
---|---|---
p | FID | LPIPS | FID | LPIPS
0 | 77.560 | 0.583 | 258.961 | 0.807
0.25 | 84.117 | 0.585 | 253.026 | 0.789
0.5 | 99.404 | 0.614 | 240.897 | 0.749
0.75 | 123.534 | 0.665 | 228.889 | 0.720
1 | 185.267 | 0.703 | 210.790 | 0.701
|  | Vs. source | Vs. target
---|---|---
p | FID | LPIPS | FID | LPIPS
0 | 76.225 | 0.584 | 179.929 | 0.641
0.25 | 85.276 | 0.583 | 178.206 | 0.614
0.5 | 108.043 | 0.596 | 169.597 | 0.594
0.75 | 125.122 | 0.611 | 160.732 | 0.584
1 | 164.958 | 0.637 | 151.933 | 0.566
a) summer$\rightarrow$softrain | b) summer$\rightarrow$fog
|  | Vs. source | Vs. target
---|---|---
p | FID | LPIPS | FID | LPIPS
0 | 77.359 | 0.585 | 162.797 | 0.633
0.25 | 80.647 | 0.586 | 162.495 | 0.626
0.5 | 87.051 | 0.589 | 161.036 | 0.621
0.75 | 95.664 | 0.597 | 155.675 | 0.611
1 | 113.402 | 0.608 | 155.786 | 0.608
|  | Vs. source | Vs. target
---|---|---
p | FID | LPIPS | FID | LPIPS
0 | 76.282 | 0.584 | 149.165 | 0.635
0.25 | 77.612 | 0.584 | 148.333 | 0.628
0.5 | 81.391 | 0.587 | 150.723 | 0.622
0.75 | 85.515 | 0.590 | 148.327 | 0.617
1 | 95.609 | 0.604 | 144.940 | 0.615
c) summer$\rightarrow$fall | d) summer$\rightarrow$spring
|  | Vs. source | Vs. target
---|---|---
p | FID | LPIPS | FID | LPIPS
0 | 74.068 | 0.584 | 163.253 | 0.697
0.25 | 75.020 | 0.589 | 162.684 | 0.685
0.5 | 81.797 | 0.642 | 161.785 | 0.656
0.75 | 92.5742 | 0.657 | 157.313 | 0.645
1 | 121.722 | 0.664 | 155.643 | 0.631
|  | Vs. source | Vs. target
---|---|---
p | FID | LPIPS | FID | LPIPS
0 | 74.363 | 0.583 | 159.036 | 0.663
0.25 | 76.965 | 0.583 | 157.723 | 0.657
0.5 | 83.311 | 0.592 | 149.511 | 0.645
0.75 | 103.049 | 0.614 | 142.353 | 0.627
1 | 134.259 | 0.649 | 146.279 | 0.612
e) summer$\rightarrow$dawn | f) summer$\rightarrow$sunset
Table 3: Average FID and LPIPS scores for multiple tasks using subsets of SYNTHIA comparing the result image with randomly picked source and target images. A darker background color denotes a lower distance between the two sets of images. |  0% Winter 0% Night |  100% Winter 0% Night |  0% Winter 100% Night |  50% Winter 0% Night
---|---|---|---|---
|  |  |  |
FID w.r.t. to Summer | 20.423 | 90.128 | 114.602 | 58.436
FID w.r.t. to Winter | 127.738 | 116.896 | 135.517 | 119.847
FID w.r.t. to Night | 173.572 | 169.287 | 161.076 | 171.635
LPIPS w.r.t. to Summer | 0.022 | 0.155 | 0.242 | 0.0764
LPIPS w.r.t. to Winter | 0.626 | 0.599 | 0.621 | 0.606
LPIPS w.r.t. to Night | 0.623 | 0.613 | 0.567 | 0.614
|  0% Winter 50% Night |  100% Winter 100% Night |  75% Winter 0% Night |  0% Winter 75% Night
|  |  |  |
FID w.r.t. to Summer | 52.557 | 136.331 | 82.661 | 90.585
FID w.r.t. to Winter | 155.389 | 146.648 | 120.264 | 133.728
FID w.r.t. to Night | 163.074 | 186.578 | 171.976 | 160.286
LPIPS w.r.t. to Summer | 0.090 | 0.199 | 0.111 | 0.178
LPIPS w.r.t. to Winter | 0.620 | 0.612 | 0.602 | 0.621
LPIPS w.r.t. to Night | 0.606 | 0.600 | 0.613 | 0.582
Table 4: Average FID and L-PIPS distance for summer$\rightarrow$winter+night
transformations on arbitrary configurations. The images at the top are
representatives of the transformation. In bold the set with the smallest
distance row-wise is highlighted.

###  1.5 Transformation quality

We are interested in determining whether the addition of the conditioning
vector $p$ has a detrimental effect on the quality of the reconstruction. To
this end, we measure the FID and LPIPS distances on images produced by
CycleGAN [1] and our method parametrized with $p=1$, with respect to the
target domain. We train both models using the same data and hyperparameters.
In Tab. 5 we show that the quality of our transformations is similar to that
of existing formulations.

| CycleGAN [1] | Ours
---|---|---
Task | FID |  LPIPS $\pm\sigma$ | FID |  LPIPS $\pm\sigma$
SYNTHIA summer$\rightarrow$winter | 118.621 | 0.597 $\pm$ 0.091 | 132.208 | 0.590 $\pm$ 0.097
SYNTHIA summer$\rightarrow$softrain | 201.941 | 0.693 $\pm$ 0.060 | 210.790 | 0.701 $\pm$ 0.055
SYNTHIA summer$\rightarrow$fog | 159.074 | 0.567 $\pm$ 0.082 | 151.934 | 0.566 $\pm$ 0.084
SYNTHIA summer$\rightarrow$dawn | 171.494 | 0.628 $\pm$ 0.059 | 155.643 | 0.631 $\pm$ 0.076
SYNTHIA summer$\rightarrow$sunset | 154.975 |  0.601 $\pm$ 0.073 | 146.279 | 0.612 $\pm$ 0.074
INIT day$\rightarrow$night | 120.990 |  0.621 $\pm$ 0.075 | 149.868 | 0.679 $\pm$ 0.076
Table 5: Comparison between the original CycleGAN formulation and our approach
for full domain transformation ($p=1$). Both models have similar FID and LPIPS
scores. The deviation from the mean measured on LPIPS is also similar for both
models.

###  1.6 Mixing Transformations

In Fig. 6 we also show generated images when all values of $p$ are nonzero,
_i.e_., when asking to mix disjoint transformations. The generator learns to
_mix_ the two transformations, to some extent, generating images with a
mixture of Night and Winter that are not part of the training sets (e.g.,
$p=[0.75,0.75]$). In general, mixing styles can be considered a “tug of war”
between the transformations, where each style “pulls” to bring the image
closer to its target domain. Since multiple styles are doing this
simultaneously, the result image shows several independent transformations
that do not affect each other unless they alter the same area of the image
(_e.g_. adding snow on the side of the road and a night sky at the same time
causes little conflict). The values of the different entries of $p$ determines
the level of compromise to reach. When two styles collide (_e.g_. two sky
transformations) some artifacts appear in the affected region (see Fig. 6 for
$p=[0.5,0.75]$). As expected, the higher the value of an individual element of
$p$ is with respect to others, the more significant its presence. For example,
for the $p=[0.25,0.75]$ case in a summer$\rightarrow$winter+night task, the
amount of snow is fairly small and the image is significantly darkened. When
$p_{1}$ increases, the night style loses its dominance and the images becomes
slightly lighter and snowier, but nevertheless it still shows some darkness.

Night Winter | $p_{1}=0$ | $p_{1}=0.25$ | $p_{1}=0.5$ | $p_{1}=0.75$ | $p_{1}=1$
---|---|---|---|---|---
$p_{2}=0$ |  |  |  |  |
$p_{2}=0.25$ |  |  |  |  |
$p_{2}=0.5$ |  |  |  |  |
$p_{2}=0.75$ |  |  |  |  |
$p_{2}=1$ |  |  |  |  |
Figure 6: Mixture of Winter and Night styles for
summer$\rightarrow$winter+night. The disentangled transformations (only one
style applied) are  highlighted.

###  1.7 Feature-wise transformation

An interesting effect of the interpolation is the fact that, contrary to a
simple blending between two images, the network transforms the global style
and different regions of the image at different steps (sky, terrain, snow,
rain…). In Fig. 7 we show this phenomenon for the summer$\rightarrow$softrain
and summer$\rightarrow$winter cases.

$p=0$ | $p=0.25$ | $p=0.5$ | $p=0.75$ | $p=1$
---|---|---|---|---
|  |  |  |
No changes | Luminosity | Clouds | Light rain | Heavy rain
|  |  |  |
No changes | Luminosity | Clouds | Light snow | Clouds, heavy snow
Figure 7: Different changes at various stages of $p$. The network does not
apply all aspects of the transformation at once, but rather it modifies
progressively different characteristics of the input, see the caption under
each image. Top: summer$\rightarrow$softrain, Bottom:
summer$\rightarrow$winter.

###  1.8 Latent space analysis

As already mentioned, our method implicitly learns an interpolation between
domains and a way of mixing them. We are interested in analyzing how this
behavior reflects on the inner latent space learned by the network. To this
end, we collect the activations of the network right after the image and the
conditioning vector information are mixed for different values of $p$ and plot
them on a 3 dimensional space using PCA decomposition. In the left side of
Fig. 8 we show the results for summer$\rightarrow$winter+night+softrain when
sampling different images from the same SYNTHIA sequence and applying to each
one disentangled values of $p$ ranging from $0$ to $1$. Smooth changes in one
of the parametrization dimensions make the learned image representations move
smoothly in the latent space along well specified and pretty consistent
trajectories. These results agree with the recent findings of [25] that show
how basic transformations are implicitly learned by generative models as
trajectories on a latent space. We show that our model has similar properties
while addressing an image-to-image task and providing an explicit control on
how to move inside the learned trajectories, _i.e_., the parametrization
vector.

On the right part of Fig. 8 we add to the plot the latent space vectors
obtained by mixing multiple transformations (e.g. $[0.25,0.5,1]$) to the same
set of images. The newly added points falls within the three trajectories
described by the disjoint transformations. Therefore our model is able to
remap the content of an image and the transformation to be applied in a shared
latent space where individual domain to domain transformation remap to
trajectories while mixing domains transformation remaps to points within these
trajectories.

Figure 8: Visualizations of the latent space activations for the
summer$\rightarrow$winter+night+softrain task. Left: Latent space activactions
for disentangled transformations. Right: Same disentangled transformations
plus entangled ones. The color code for each dimension of $p$ ranges from blue
for $p=0$ to red ($p=1$). The transformations are laid out following three
different trajectories (one per dimension of $p$, therefore one per style) and
follow a clear trajectories away from $p=0$. The mixed vectors are all
contained in the volume spanned by the disentangled trajectories.

###  1.9 Domain generalization

We also briefly explore the domain generalization capabilities of this method
using real-world datasets for evaluation. In Fig. 9 we evaluate a pre-trained
network for the summer$\rightarrow{}$night+rain task trained with images from
the SYNTHIA dataset on real images acquired by a phone. The network correctly
segments the different components of the image (sky, buildings) and adds the
appropriate effects (clouds, lights).

Rainy Night | $p_{1}=0$ | $p_{1}=0.25$ | $p_{1}=0.5$ | $p_{1}=0.75$ | $p_{1}=1$
---|---|---|---|---|---
$p_{2}=0$ |  |  |  |  |
$p_{2}=0.25$ |  |  |  |  |
$p_{2}=0.5$ |  |  |  |  |
$p_{2}=0.75$ |  |  |  |  |
$p_{2}=1$ |  |  |  |  |
Figure 9: Evaluation of the network trained on SYNTHIA on real-world images.

###  1.10 Implementation details

Our model for soft-parameterization extends CycleGAN with a series of fully
connected layers that transform the conditioning vector by upsampling it to a
64-dimensional vector. This is replicated and concatenated to the feature
dimension of the output of the last ResNet block. In the Book-Covers case, we
additionally use skip connections between the encoder and decoder parts of the
generator, to which we also concatenate the upsampled conditioning vector, as
well as multi-scale discriminators. We train for all experiments using a batch
size of 1 and an image size of $512\times 512$. We implement our framework on
TensorFlow 1 [34] following the architecture of CycleGAN [1]444
https://github.com/tensorflow/gan/tree/master/tensorflow_gan/examples/cyclegan.
with the additions already discussed in Sec. 3.2. We use the Adam [35]
optimizer in all our experiments with a learning rate of 0.0001 and
$\beta_{1}=0.5$. Following the guidelines of the original CycleGAN
architecture, we set $\lambda=10$ and use a batch size of 1. The number of
training iterations varies between different tests but roughly corresponds to
$200$ epochs.

###  1.11 Alternatives for the concatenation of $p$

The conditioning vector $p$ is transformed through a series of fully connected
layers and finally concatenated at a certain step of the generators and
discriminators. As additional ablation studies we analyze the effect of
injecting it at different levels of the architectures and report the results
in Tab. 6. We train multiple variants of our architecture for the
summer$\rightarrow$winter tasks and compare the different alternatives by
measuring FID with respect to a random source images (when conditioning with
$p=0$) and with respect to a random target image (when conditioning with
$p=1$). The performances of all the alternatives are fairly close, but
injecting $p$ just once for both the generator and discriminator has the
advantage of being the simpler strategy and providing the best performance.
For this reason we selected this option and presented it in the main paper.

|  $p$ concatenation | FID
---|---|---
| Generator | Discriminator | vs source $p=0$ | vs target $p=1$
Ablation 1 | once |  $\forall$ conv. | 29.754 | 111.602
Ablation 2 |  $\forall$ encoder conv. | once | 13.285 | 113.442
Ablation 3 |  $\forall$ decoder conv. | once | 13.849 | 109.607
Ablation 4 |  $\forall$ encoder conv. |  $\forall$ conv. | 16.217 | 116.461
Ablation 5 |  $\forall$ decoder conv |  $\forall$ conv. | 13.666 | 112.598
ParGAN | once | once | 13.183 | 109.25
Table 6: Different configurations on how $p$ is included in the generator and
discriminator and its effects on quality. We use the same measurement strategy
as in Tab. 3.


# Parallax estimation for push-frame satellite imagery: application to super-
resolution and 3D surface modeling from Skysat products

Jérémy Anger1,2 Thibaud Ehret1 Gabriele Facciolo1

(1Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France
2Kayrros SAS)

###### Abstract

Recent constellations of satellites, including the Skysat constellation, are
able to acquire burst of images. This new acquisition mode allows for modern
image restoration techniques, including multi-frame super-resolution. As the
satellite moves during the acquisition of the burst, elevation changes in the
scene translate into noticeable parallax. This parallax hinders the results of
the restoration. To cope with this issue, we propose a novel parallax
estimation method. The method is composed of a linear Plane+Parallax
decomposition of the apparent motion and a multi-frame optical flow algorithm
that exploits all frames simultaneously. Using SkySat L1A images, we show that
the estimated per-pixel displacements are important for applying multi-frame
super-resolution on scenes containing elevation changes and that can also be
used to estimate a coarse 3D surface model.

##  1 Introduction

Satellites play a big role in the observation of the Earth: from environmental
monitoring to industry monitoring. Many Earth monitoring applications require
a good ground resolution. For example, fine detection and analysis of human
activity requires a resolution in the range of 30 cm to 1 m / pixel [1].

Earth observation missions were historically owned by national organisations,
constructing high-cost long-term satellites. Starting in the late 90’s, a
similar model was also adopted by actors from the private sector (e.g. IKONOS,
EROS, QuickBird, WorldView). But in recent years, some companies have started
to offer low-cost imagery thanks to new satellite designs. The current trend
is to launch many smaller satellites to lower orbits and with a shorter
lifespan, providing a wider coverage at lower cost. For example, this strategy
allows Planet to provide a daily revisit time on some products. However, low-
cost satellites means that the quality of each individual image is lower, with
higher noise or worse GSD for example. This means that instead of trying to
obtain a 50 cm GSD from the physical design of the satellite, such resolution
has to be reached using computational photography techniques such as multi-
frame super-resolution.

Reference frame W/o parallax est. With parallax est.

std. dev.

Figure 1: Accurate parallax estimation for super-resolution allows to restore
finer details. Top row: reference low-resolution frame, SR without parallax
estimation, SR with parallax estimation using the proposed method. Bottom row:
standard deviation of the frames aligned with the estimated displacements.

The SkySat constellation [1] from Planet follows this trend towards small but
high-resolution satellites. SkySats contain a full-frame sensor able to
capture 40 frames per second and is operated in a _push-frame_ mode with
significant overlap between the frames. This means that the same point on the
ground is seen in several consecutive images, which are combined with
computational photography techniques. Furthermore, thanks to the design of its
optical system, the images are aliased, providing the opportunity to increase
the resolution by means of multi-frame super-resolution.

A simple strategy to multi-frame super-resolution consists in aligning the
frames using a rigid model (assuming that the observed scene is flat) and then
fusing them [2]. However, because the frames are acquired from slightly
different viewpoints, when the scene contains large elevation changes the
parallax effect due to surface relief becomes noticeable and can hinder the
reconstruction. Indeed, as noted in [3], the parallax $\delta\varepsilon$ is
directly proportional to the elevation changes $\delta z$, inversely
proportional to the altitude $h$ of the satellite, and proportional to the
distance between the viewpoints $b$ (baseline)

$\delta\varepsilon\propto\frac{b}{h}\delta z.$ (1)

Since the overlapping frames are acquired with increasingly larger baselines
the parallax will introduce non rigid displacements of the samples
proportional to their elevation.

In order to avoid fusion artefacts, this parallax has to be estimated and
taken into account in the super-resolution. In a calibrated system (with
perfectly known camera models) plus knowledge of the scene topography this
displacement could be easily deduced. However, since camera models are subject
to pointing errors a solution for non-calibrated systems is needed.

The problem of estimating the 3D geometry of a scene from a set of views from
different angles is classic in computer vision. It usually involves first the
estimation of the camera parameters via bundle adjustment [4], then matching
across the views and triangulating the matches [5]. However, Equation (1)
implies that the projective camera model can be assumed affine [6]. In
addition, because of the push-frame acquisition the correspondences between
frames can be reliably estimated using optical flow methods.

Contributions. In this paper we propose to estimate the parallax from a burst
of non-calibrated frames using a linear _Plane+Parallax_ formulation [7],
where the correspondences between frames are provided by an optical flow
algorithm. The frames are first aligned with respect to a common virtual
plane, from which the parallax between all frames can then be estimated using
a multi-frame optical flow algorithm. We show that the estimated parallax
allows to improve super-resolution results on scenes with large elevation
changes and moving objects as shown in Figure 1. In addition, we show that the
estimated parallax can provide coarse 3D information, even if the frames are
not acquired in a stereo setting.

##  2 Related works

The video product of the SkySat constellation was used in [8] to estimate a
depth map. The authors computed several low baseline disparity maps using
phase correlation, which were then warped and stacked to improve the SNR of
the final disparity map. The warp was computed by matching the images to a
reference one. A similar study was also conducted in [9] with the objective of
producing an elevation model also using a SkySat video product. SkySat video
products require a satellite maneuver to keep the object of interest in the
frame during the video, resulting in thousands of frames available. In
contrast, our method exploits the redundancy in the L1A SkySat product, which
consists of a few dozen frames for a given scene. This implies that the
maximum available baseline is much smaller, resulting in a more ill-posed
estimation.

In [7] it is proposed a method for estimating Plane+Parallax motion from
multiple views of a 3D scene. The method alternates between computation of the
parallax motion common to all the frames and the camera epipoles. For that
they use a photometric constraint between the reference and all the other
images. Our method relies on a similar joint multi-frame formulation, but
since we assume affine cameras there is no need to alternate with the epipole
estimation and the parallax can be directly recovered.

##  3 Method

The inputs of the proposed method are a set of satellite images acquired in a
push-frame burst (as possible by the SkySat constellation [1]) and pre-
registered using a translation. The objective is to estimate the parallax in
the scene.

In [2] an affine transform is estimated between each frame and the reference
one. This registration is not sufficient when the scene contains large
elevation changes. To account for these, we propose a refined model based on
one affine transform per frame noted $A_{i}$ and a common dense optical flow
$d$ to model the geometry of the scene.

###  3.1 Plane stabilization

In this section we study how to register the frames in presence of parallax.
The frames are assumed to be taken at uniform intervals and along a
rectilinear trajectory and registered up to a translation. Since the satellite
is far from the scene we can assume that parallax induces a disparity
translation $d$ proportional to the height of the ground [3] and proportional
to the baseline between the frames (i.e. proportional to the frame number)

$\displaystyle v_{0}(x)=v_{i}(x+i\cdot d(x)).$ (2)

Equation 2 corresponds to the case of a perfectly rectified scene, where the
same zero-parallax plane is always registered in the sequence $v_{i}$.
However, due to pointing errors and the limitations of the registration up to
a translation, the actual model might include an additional homography
$H_{i}$, which we can safely approximate by an affine transform $A_{i}$
because of the distance from the satellite to the scene [6]:

$\displaystyle v_{0}(x)=v_{i}(A_{i}(x+i\cdot d(x))).$ (3)

The transforms $A_{i}$ and the vector field $d$ are unknown, and must be
estimated. Note that in (3) the disparity $d$ is composed with the affinities
$A_{i}$. However, since the $A_{i}$ are close to translations (with Jacobian
$J_{i}$ close to the identity) we can safely use the following linear
approximation of the previous model

$\displaystyle v_{0}(x)=v_{i}(A_{i}(x)+J_{i}(i\cdot d(x))),$ (4)

where we assume that $J_{i}=\mathds{1}$. The advantage of this formulation is
that it leads to a linear system of equations on the parameters of $A_{i}$ and
$d$. When $A_{i}\neq\mathds{1}$ the linearization could still be applied but
an rough estimate of $A_{i}$ should be provided (for instance by aligning the
frames [10]) from which the Jacobians can be extracted. We now detail how our
linear system is built and solved.

First, the pairwise optical flows $f_{i}$ are computed between the frames
$u_{i}$ and frame $u_{0}$ and thus verify the relations

$\displaystyle v_{0}(x)=v_{i}(f_{i}(x)).$ (5)

The flows $f_{i}$ provide correspondences between the frames, hence we want to
factorize them as

$f_{i}(x)=A_{i}(x)+i\cdot d(x).$ (6)

The set of equations associated to all the flows form an over-determined
linear system that can be solved by least squares minimization. We solve for
all the $A_{i}$ and a single $d$ using the Conjugate Gradient method. In order
to speed-up this optimization, the linear system can be sub-sampled (e.g. by a
factor 4, thus considering only $\nicefrac{{1}}{{16}}$ equations).

At this point the flow $d$ shows some part of the parallax, but is not very
precise due to the noisy nature of each individual flows $f_{i}$. However, the
affine transformations $A_{i}$ allow to align the frames to a common plane,
which is necessary for the next step: parallax estimation.

###  3.2 Parallax estimation

As a result of the previous step, the affine transforms $A_{i}$ can be used to
align the frames $v_{i}$ on a common plane. For the parallax estimation step,
we propose a multi-frame optical flow method, allowing to fully exploit all
the frames to produce a single disparity map.

The proposed method is an extension of the robust optical flow method [11] to
multiple frames assuming a constant motion model. The flow is computed using
all the images at the same time, contrary to a regular optical flow that is
computed between pairs of frames. As before, we consider a set of $N$ frames
$\\{v_{-N/2},\dots,v_{0},\dots,v_{N/2}\\}$ such that $v_{0}$ corresponds to
the reference frame and that the frames are now aligned on a common plane
using the transforms $A_{i}$. Given that the movement computed corresponds
only to the parallax effect, we assume that the optical flow between the frame
$0$ and the $i$th frame, $w_{0,i}$ is linearly proportional to the optical
flow from $0$ to $1$ such that $w_{0,i}=i\cdot w_{0,1}$. This indicates that a
single optical flow capturing the parallax effect is able to align each frame
to the reference.

Let $d$ be the optical flow to be estimated. The energy of the robust optical
flow method [11] for a single pair $(v_{0},v_{i})$ is

$\displaystyle\begin{aligned} E_{i}(d)&=\sum_{x}\Psi(v_{i}(A_{i}x+i\cdot
d)-v_{0}(A_{0}x))\\\ &+\gamma\sum_{x}\Psi(\nabla v_{i}(A_{i}x+i\cdot d)-\nabla
v_{0}(A_{0}x))\\\ &+\alpha\sum_{x}\Psi(\sqrt{(}|\nabla d_{x}|^{2}+|\nabla
d_{y}|^{2}))\end{aligned}$ (7)

with $\Psi(x)=\sqrt{x^{2}+\epsilon^{2}}$ and $d_{x}$ (respectively $d_{y}$)
corresponds to the $x$ (respectively $y$) component of the optical flow.
Finding $d$ that fits the best all frames corresponds then to minimizing the
following energy

$E(d)=\frac{1}{N}\sum_{i=-N/2}^{N/2}E_{i}(d),$ (8)

which is optimized using successive over-relaxation [11].

##  4 Experiments

In this section we validate the proposed method on two applications: multi-
frame super-resolution and 3D surface modeling. We show that our parallax
estimation method allows for precise displacement estimation between the input
full-frame images.

###  4.1 Application to super-resolution

We first demonstrate the proposed method on a multi-frame super-resolution
from SkySat burst. Multi-frame super-resolution usually relies on a
registration step, which has to be as precise as possible. For this reason, it
is convenient to estimate global parametrized transform (e.g. affine transform
as in [2]). However, in the presence of abrupt elevation changes in the scene,
the change of viewpoint of the satellite between frames induces a parallax
motion. As we have seen, such parallax motion must be estimated locally.

We compare the multi-frame super-resolution method of [2] without and with our
parallax estimation method. Figure 1 shows the result of the two methods on
images of an open pit mine near Sahuarita AZ. We observe that the rigid
registration exhibits artefacts due to the elevation change in the scene,
whereas the proposed registration is able to retrieve fine details.
Furthermore, this figure also shows the temporal standard deviation of the
aligned stack of images (interpolated with splines of order 5). As the rigid
registration is not able to register local structures, the temporal standard
deviation is high around such structures. Instead, the proposed method
successfully minimizes this measure. For a quantitative comparison, the
average of the temporal standard deviation are reported in Table 4.1. We
observe that the standard deviation is lower when using a low optical flow
regularization $\alpha$, which is important of the context of super-
resolution.

Table 1: Comparison of the average standard deviation of the aligned stacks on
three locations.

|  | Proposed
---|---|---
| Rigid transform | $\alpha=10$ | $\alpha=60$
Sahuarita |  |  |

& 0.34511 0.34638 Morenci 0.45282 0.42532 0.43164 Chicago 0.52539 0.45405
0.47601

Moving objects. Objects that have a constant speed are also captured by the
multi-frame optical flow. In this case, the flow can be used to compute the
speed of the objects [9], as well as to provide accurate displacements for the
super-resolution algorithm. Figure 2 shows two super-resolution results, with
and without parallax handling ($\alpha=10$). When fusing images only using
global transforms, the moving objects are accumulated at incorrect positions,
resulting in ghosting artifacts.

Reference frame W/o parallax est. With parallax est.

Figure 2: Parallax estimation allows to register moving objects and remove
ghosting artifacts.

###  4.2 Application to 3D surface modeling

Per-pixel registration using plane stabilization and multi-frame optical flow
provides accurate displacement for each pixel. This displacement is
proportional to the elevation of the scene (up-to the choice of the plane by
the stabilization), and thus gives a coarse estimation of the elevation model.
In order to obtain a smoother result, we set a higher regularization weight of
$\alpha=60$ when estimating the multi-frame optical flow for the application
of coarse 3D modeling.

Figure 3 shows the resulting parallax estimation on a large open-pit mine near
Morenci AZ. This result is obtained from the multi-frame optical flow
estimated using 35 consecutive L1A frames. We compare our result with a DSM
estimated using S2P [5] from the L1B frames acquired in tri-stereo
configuration. While the proposed reconstruction is very coarse, we can
identify the main 3D features.

Figure 3: Qualitative comparison between the result of our coarse 3D modeling
and the DSM result of S2P [5].

##  5 Conclusion

We proposed a novel parallax estimation method from SkySat images. The method
is able to estimate precise per-pixel displacements. When used along a multi-
frame super-resolution algorithm, the proposed method allows to recover more
details and contain less artefacts. Moreover, we showed that parallax
estimation from a single burst can be used as a coarse alternative to 3D
reconstruction from stereo pairs.

Using images with a small baseline to reconstruct the 3D as we propose is a
very ill-posed problem, and our method could probably benefit from more
advanced optical flow priors. Future works will focus on reducing the
computational of the method, for instance using a faster optical flow
implementation.

## Acknowledgements

Work partly financed by IDEX Paris-Saclay IDI 2016, ANR-11-IDEX-0003-02,
Office of Naval research grant N00014-17-1-2552, DGA Astrid project « filmer
la Terre » noANR-17-ASTR-0013-01, MENRT. This work was performed using HPC
resources from GENCI–IDRIS (grant 2020-AD011011795). We thank Planet for
providing the L1A SkySat images.

## References

  * [1] K. Murthy, M. Shearn, B. D. Smiley, A. H. Chau, J. Levine, and M. D. Robinson,  “Skysat-1: very high-resolution imagery from a small satellite,”  in Sensors, Systems, and Next-Generation Satellites, 2014.
  * [2] J. Anger, T. Ehret, C. de Franchis, and G. Facciolo,  “Fast and accurate multi-frame super-resolution of satellite images.,”  ISPRS, 2020.
  * [3] J. Delon and B. Rougé,  “Small baseline stereovision,”  Journal of Mathematical Imaging and Vision, 2007.
  * [4] B. Triggs,  “Plane+ parallax, tensors and factorization,”  in ECCV, 2000.
  * [5] C. De Franchis, E. Meinhardt-Llopis, J. Michel, J.-M. Morel, and G. Facciolo,  “An automatic and modular stereo pipeline for pushbroom images,”  in ISPRS, 2014.
  * [6] C. de Franchis, E. Meinhardt-Llopis, J. Michel, J.-M. Morel, and G. Facciolo,  “On stereo-rectification of pushbroom images,”  in ICIP, 2014.
  * [7] M. Irani, P. Anandan, and M. Cohen,  “Direct recovery of planar-parallax from multiple frames,”  Pattern Analysis and Machine Intelligence, 2002.
  * [8] X. Wan, J. Liu, H. Yan, G. L. Morgan, and T. Sun,  “3D super resolution scene depth reconstruction based on skysat video image sequences,”  in IGARSS, 2016.
  * [9] P. d’Angelo, G. Kuschk, and P. Reinartz,  “Evaluation of skybox video and still image products,”  ISPRS Archives, 2014.
  * [10] T. Briand, G. Facciolo, and J. Sánchez,  “Improvements of the Inverse Compositional Algorithm for Parametric Motion Estimation,”  IPOL, 2018.
  * [11] J. Sánchez Pérez, N. Monzón López, and A. Salgado de la Nuez,  “Robust Optical Flow Estimation,”  IPOL, 2013.