# Disentangling Domain and Content

Dan Andrei Iliescu
University of Cambridge
<EMAIL_ADDRESS>
&Aliaksei Mikhailiuk
University of Cambridge
<EMAIL_ADDRESS>
&Damon Wischik
University of Cambridge
<EMAIL_ADDRESS>
&Rafal Mantiuk
University of Cambridge
<EMAIL_ADDRESS>

###### Abstract

Many real-world datasets can be divided into groups according to certain
salient features (e.g. grouping images by subject, grouping text by font,
etc.). Often, machine learning tasks require that these features be
represented separately from those manifesting independently of the grouping.
For example, image translation entails changing the style of an image while
preserving its content. We formalize these two kinds of attributes as two
complementary generative factors called “domain” and “content”, and address
the problem of disentangling them in a fully unsupervised way. To achieve
this, we propose a principled, generalizable probabilistic model inspired by
the Variational Autoencoder. Our model exhibits state-of-the-art performance
on the composite task of generating images by combining the domain of one
input with the content of another. Distinctively, it can perform this task in
a few-shot, unsupervised manner, without being provided with explicit
labelling for either domain or content. The disentangled representations are
learned through the combination of a group-wise encoder and a novel domain-
confusion loss.

##  1 Introduction

Learning rich, interpretable representations with deep neural networks is one
of the main challenges of current artificial intelligence research. Achieving
such representations would enable us to perform complex and highly useful
operations on high-dimensional data (Bengio et al., 2013). Perhaps the first
milestone that has yet to be reached in this research is learning
representations which easily factorize along the lines of recognizable human
concepts. This property is called “the disentanglement of generative factors”,
and is an accelerating field of inquiry (Tschannen et al., 2018), with many
major contributions coming from models based on the Variational Autoencoder
(Kingma & Welling, 2014; Rezende et al., 2014).

Recent work (Locatello et al., 2019; van Steenkiste et al., 2019) has revealed
limitations in the current methods caused by the inherent ambiguity of the
disentanglement objective. They have pointed out the need for equipping models
with inductive biases appropriate to their respective application. An example
disentanglement task with such increased specificity is the learning to
generate multi-object scenes whereby the representation is trained to
factorize along object lines (Engelcke et al., 2020; Burgess et al., 2019;
Greff et al., 2019).

As a further step towards this goal, we identify another promising
disentanglement objective, namely the separation of domain and content
representations, widely applicable to a variety of tasks, ranging from
unsupervised translation to missing data imputation. In a general sense,
whenever there exists some form of grouping imposed on a dataset, the notion
of domain arises naturally to characterize the attributes of the data which
are common within groups but differ across groups. Such attributes could be
the style of a painting in the context of style transfer, or the language of a
text in the context of neural machine translation. The notion of content then
appears as a counterpart to the domain to encompass the features which occur
independently of the domain features. For instance, in the context of style
transfer, the actual subject of the painting represents the content.

###  1.1 Related Work

There are many research directions which lead into the domain-content
paradigm. Early work on domain adaptation (Ben-David et al., 2010; Ganin et
al., 2016), for example, has highlighted the desirability of learning domain-
invariant (content) representations of the data, in order to perform
classification and regression in a common space. The model of Gonzalez-Garcia
et al. (2018) can successfully separate domain-specific from domain-invariant
features for two domains.

Problems such as image-to-image translation, which entails changing the domain
of an image while preserving its content, have been extensively studied. Major
deep learning innovations have come from this area (Isola et al., 2017; Zhu et
al., 2017), producing results of excellent quality. However, unsupervised
models have been limited by the rigidity of their domain representations. Many
methods can only be trained to map between two domains (Zhu et al., 2017;
Taigman et al., 2017), or a fixed set of domains (Choi et al., 2018, 2020; Lee
et al., 2020). Even models designed to accommodate new domains at test-time
either rely on restricting the domain to stylistic features (Liu et al.,
2019), or requiring re-training for every new example (Benaim & Wolf, 2018).
Moreover, to the best of our knowledge, no model has the capacity to process
sets of examples specifying both the source and target domain at test-time,
but rely either on explicit conditioning or on one single example. All these
constraints limit the model’s ability to understand unseen domains and
transfer knowledge between them.

Conversely, state-of-the-art methods to perform novel view synthesis rely
either on structural assumptions about the geometry of the scene (Sitzmann et
al., 2019; Yoon et al., 2020) or on explicit conditioning on camera viewpoint
(content) (Eslami et al., 2018; Mildenhall et al., 2020). This restricts the
model’s usefulness when the viewpoint or scene structure is missing or
difficult to describe explicitly.

Perhaps the closest inspiration for our approach comes from the influential
work on Semi-Supervised learning by Kingma et al. (2014), where they design a
generative model with two latent variables: class (domain) and ${\mathbf{z}}$
(content). They recognize that a limitation of their method is that the number
of generative likelihood evaluations scales linearly with the number of
classes, since the class variable is categorical.

###  1.2 Our contribution

We wish to address a general formulation of the domain-content problem that
does not rely on any explicit conditioning or constraints on the nature of the
disentangled features. We treat domain and content as independent continuous
random variables. The strength of our model is that it allows for the
specification of new domains at test-time by provision of any number of
examples, and permits content queries through examples as well. The continuous
nature of the latent representations is also useful for measuring similarity
between domains, or for classifying inputs by content irrespective of domain.

In this work:

  * •

We propose a probabilistic model of domain-content with an associated neural
architecture built upon the paradigm of the Variational Autoencoder. Our model
has the capacity to separate domain and content features in an unsupervised,
few-shot manner. The group-wise domain encoder enables it to process unseen
domains at test-time, while its novel domain-confusion loss prevents domain
and content information from mixing in the latent representations during
training.

  * •

We demonstrate qualitatively the ability of our model to perform a generalized
task called domain-content fusion, bringing together image-to-image
translation and novel view synthesis, that requires the model to generate
images by combining the domain of one input with the content of another.

  * •

We measure quantitatively the robustness of our disentanglement by
ascertaining how well our model’s latent representations can predict domain
and content features in the data. We record improvements over other
disentanglement methods.

##  2 Probabilistic Domain-Content

Figure 1: Bayesian networks comparing VAE (left) and our Domain-Content model
(right).

We consider a very general formulation of the domain-content problem: Let
${\mathbf{X}}$ be a dataset of images divided into $N$ packs
$\underline{{\mathbf{x}}}_{i}$ (we use the underline notation to denote a pack
of elements), where $i\in\\{1:N\\}$. Each pack $i$ consists of $K_{i}$ images
$\underline{{\mathbf{x}}}_{i}=\\{{\mathbf{x}}_{ik}\leavevmode\nobreak\
|\leavevmode\nobreak\ k\in\\{1:K_{i}\\}\\}$. By definition, all elements
within a pack belong to the same domain.

Our goal is to create a probabilistic model that exploits the pack structure
of the dataset in order to extract a domain random variable ${\mathbf{m}}_{i}$
and a pack of content random variables $\underline{{\mathbf{o}}}_{i}$ from a
given pack of images $\underline{{\mathbf{x}}}_{i}$. In order to constitute
useful disentangled representations, the inferred latent variables should
satisfy certain intuitive principles:

  1. 1.

The representation should be rich enough that one could recover an accurate
estimate of the input $\underline{{\mathbf{x}}}_{i}$ given the distribution of
the latent variables ${\mathbf{m}}_{i}$ and $\underline{{\mathbf{o}}}_{i}$.

  2. 2.

Each image in the pack $\underline{{\mathbf{x}}}_{i}$ should have the same
associated domain variable ${\mathbf{m}}_{i}$ (we have enforced this by
construction).

  3. 3.

The distribution of individual content variables ${\mathbf{o}}_{ik}$ should be
independent of the domain of the input.

In this work, we propose a probabilistic model with an associated neural
architecture that, by following these principles, achieves effective and
robust disentanglement of the domain and content factors. Our model follows
the Variational Autoencoder paradigm (Kingma & Welling, 2014; Rezende et al.,
2014), wherein the latent inference density is used as a sampling distribution
for training a generative model. We have been particularly inspired by the
semi-supervised approach of Kingma et al. (2014), who have also built a
Variational Autoencoder with two latent variables.

###  2.1 Parametric Generative Model

Our generative model comprises a family of parametric densities $p$ over the
variables $\underline{{\mathbf{x}}}_{i}$, ${\mathbf{m}}_{i}$ and
$\underline{{\mathbf{o}}}_{i}$. The joint distribution of a pack factorizes
according to the Bayesian network in Figure 1:

$p(\underline{{\mathbf{x}}}_{i},{\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i})=p({\mathbf{m}}_{i})\prod_{k=1}^{K_{i}}p({\mathbf{o}}_{ik})p({\mathbf{x}}_{ik}|{\mathbf{m}}_{i},{\mathbf{o}}_{ik})$
(1)

Notice that, when conditioned on ${\mathbf{m}}_{i}$, an individual image
variable ${\mathbf{x}}_{ik}$ is independent of all the other images in the
pack $\underline{{\mathbf{x}}}_{i}\setminus\\{{\mathbf{x}}_{ik}\\}$ and their
corresponding contents
$\underline{{\mathbf{o}}}_{i}\setminus\\{{\mathbf{o}}_{ik}\\}$. We assign
parameters $\alpha$ to the prior over the domain
$p_{\alpha}({\mathbf{m}}_{i})$, $\beta$ to the prior over the content
$p_{\beta}({\mathbf{o}}_{ik})$ and $\theta$ to the generator density
$p_{\theta}({\mathbf{x}}_{ik}|{\mathbf{m}}_{i},{\mathbf{o}}_{ik})$. Only
$\theta$ is a trainable parameter, since it corresponds to the parameters of
our decoder network. Its maximum likelihood estimator is:

$\displaystyle\theta^{\prime}$
$\displaystyle=\underset{\theta}{\operatorname*{arg\,max}}\frac{1}{N}\sum_{i=1}^{N}\log
p_{\theta,\alpha,\beta}(\underline{{\mathbf{x}}}_{i})\textrm{, where }$ (2)
$\displaystyle p_{\theta,\alpha,\beta}(\underline{{\mathbf{x}}}_{i})$
$\displaystyle=\mathbb{E}_{p_{\alpha}({\mathbf{m}}_{i})}\prod_{k=1}^{K_{i}}\mathbb{E}_{p_{\beta}({\mathbf{o}}_{ik})}\log
p_{\theta}({\mathbf{x}}_{ik}|{\mathbf{m}}_{i},{\mathbf{o}}_{ik})$ (3)

###  2.2 Variational Inference in the Domain-Content Model

Figure 2: Neural architecture of the MO model.

Optimizing the likelihood under this formulation would require sampling over
the priors $p_{\alpha}({\mathbf{m}}_{i})$ and $p_{\beta}({\mathbf{o}}_{ik})$,
a procedure that would converge extremely slowly and leave us with no
tractable posterior over the latents conditioned on the images. We, therefore,
introduce a parametric inference density
$q({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})$
over which to perform importance sampling during training for faster
convergence and a tractable inference posterior. This inference density
comprises the domain-content extractor that we set out to create.

According to Kahn & Marshall (1953), the generative latent posterior
$p_{\theta,\alpha,\beta}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})$
is, itself, the optimal inference density with respect to reducing the
variance of the maximum likelihood estimator. We seek, therefore, to design an
inference density
$q({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})$
that preserves the conditional relationships between the variables in the
generative model, in order to be theoretically capable of recovering the
generative posterior. A range of choices are available on how to factorize the
inference posterior while still retaining the aforementioned conditional
relationships. We choose to condition the content on the domain in order to
exploit the conditional independence of contents in a pack given its domain
$q(\underline{{\mathbf{o}}}_{i}|{\mathbf{m}}_{i},\underline{{\mathbf{x}}}_{i})=\prod_{k=1}^{K_{i}}q({\mathbf{o}}_{ik}|{\mathbf{m}}_{i},{\mathbf{x}}_{ik})$.
The inference model, depicted in Figure 1, becomes:

$q({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})=q({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})\prod_{k=1}^{K_{i}}q({\mathbf{o}}_{ik}|{\mathbf{x}}_{ik},{\mathbf{m}}_{i})$
(4)

We assign parameters $\zeta$ to the inference posterior of the domain
$q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})$, and $\xi$ to the
posterior inference over the content
$q_{\xi}({\mathbf{o}}_{ik}|{\mathbf{x}}_{ik},{\mathbf{m}}_{i})$. Both $\zeta$
and $\xi$ are trainable parameters, since the goal is to learn an inference
density accurate enough to recover the image input (Principle 1). They
correspond to the parameters of our domain and content encoder.

By applying Importance Sampling, followed by Jensen’s Inequality, to the
maximum likelihood objective, we obtain the Evidence Lower Bound (ELBO) for
our model (the full derivation is available in Appendix A):

$\displaystyle\log
p_{\theta,\alpha,\beta}(\underline{{\mathbf{x}}}_{i})\geq\leavevmode\nobreak\
$
$\displaystyle\mathbb{E}_{q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})}\sum_{k=1}^{K_{i}}\mathbb{E}_{q_{\xi}({\mathbf{o}}_{ik}|{\mathbf{x}}_{ik},{\mathbf{m}}_{i})}\log
p_{\theta}({\mathbf{x}}_{ik}|{\mathbf{m}}_{i},{\mathbf{o}}_{ik})\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \textrm{(reconstruction)}$ (5)
$\displaystyle-
D_{\mathrm{KL}}[q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})\leavevmode\nobreak\
||\leavevmode\nobreak\ p_{\alpha}({\mathbf{m}}_{i})]\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \textrm{(domain)}$ (6)
$\displaystyle-\mathbb{E}_{q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})}\sum_{k=1}^{K_{i}}D_{\mathrm{KL}}[q_{\xi}({\mathbf{o}}_{ik}|{\mathbf{x}}_{ik},{\mathbf{m}}_{i})\leavevmode\nobreak\
||\leavevmode\nobreak\ p_{\beta}({\mathbf{o}}_{ik})]\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \textrm{(content)}$ (7)

This separates neatly into a reconstruction loss and two regularization
penalties, for the domain and content variables. The expression of the
reconstruction loss optimizes Principle 1 directly, as it encourages precise
estimates of the output image.

###  2.3 Neural Architecture

Algorithm 1 MorphOus

Input: Dataset ${\mathbf{X}}$ formed of $N$ packs each of $K_{i}$ images,
hyperparameter $\lambda=100$.

for $i=1$ to $N$ do

Randomly select two packs $\underline{{\mathbf{x}}}_{i}$ and
$\underline{{\mathbf{x}}}_{j}$

${\mathbf{m}}_{i}\sim
q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})$

$L^{m}_{i}:=\log q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})-\log
p_{\alpha}({\mathbf{m}}_{i})$

for $k=1$ to $K_{i}$ do

${\mathbf{o}}_{ik}\sim
q_{\xi}({\mathbf{o}}_{ik}|{\mathbf{x}}_{ik},{\mathbf{m}}_{i})$

$L^{o}_{ik}:=\log
q_{\xi}({\mathbf{o}}_{ik}|{\mathbf{x}}_{ik},{\mathbf{m}}_{i})-\log
p_{\beta}({\mathbf{o}}_{ik})$

$L^{r}_{ik}:=({\mathbf{x}}_{ik}-G({\mathbf{m}}_{i},{\mathbf{o}}_{ik}))^{2}$

end for

$L_{i}=L^{m}_{i}+\sum_{k=1}^{K_{i}}[L^{r}_{ik}+L^{o}_{ik}]$

/ / repeat for pack $\underline{{\mathbf{x}}}_{j}$

$L_{j}=L^{m}_{j}+\sum_{k=1}^{K_{j}}[L^{r}_{jk}+L^{o}_{jk}]$

/ / domain-confusion loss

$L^{c}:=\textrm{DomConfLoss}(\underline{{\mathbf{o}}}_{i},\underline{{\mathbf{o}}}_{j})$

$L:=L_{i}+L_{j}+\lambda L^{c}$

/ / update parameters

$\theta,\zeta,\xi:=\textrm{Adam}(L)$

end for

Algorithm 2 DomConfLoss

Input: Two packs of content variables
$\underline{{\mathbf{o}}}_{i},\underline{{\mathbf{o}}}_{j}$ of size $K_{i}$
and $K_{j}$.

Output: Loss value verifying whether the packs have the same distribution.

/ / choose random sizes for pack splits

$A_{i}\sim\mathcal{U}(1,K_{i}),\leavevmode\nobreak\ B_{i}:=K_{i}-A_{i}$

$A_{j}\sim\mathcal{U}(1,K_{j}),\leavevmode\nobreak\ B_{j}:=K_{j}-A_{j}$

$\underline{{\mathbf{a}}}_{i},\underline{{\mathbf{b}}}_{i}:=\textrm{split}(\underline{{\mathbf{o}}}_{i},A_{i},B_{i})$

$\underline{{\mathbf{a}}}_{j},\underline{{\mathbf{b}}}_{j}:=\textrm{split}(\underline{{\mathbf{o}}}_{j},A_{j},B_{j})$

/ / get “real” discriminator predictions

${\mathbf{r}}_{i}:=D(\sum_{k=1}^{A_{i}}H({\mathbf{a}}_{ik}),\sum_{l=1}^{B_{i}}H({\mathbf{b}}_{il}))$

${\mathbf{r}}_{j}:=D(\sum_{k=1}^{A_{j}}H({\mathbf{a}}_{jk}),\sum_{l=1}^{B_{j}}H({\mathbf{b}}_{jl}))$

/ / get “fake” discriminator predictions

${\mathbf{f}}_{a}:=D(\sum_{k=1}^{A_{i}}H({\mathbf{a}}_{ik}),\sum_{l=1}^{A_{j}}H({\mathbf{a}}_{jl}))$

${\mathbf{f}}_{b}:=D(\sum_{k=1}^{B_{i}}H({\mathbf{b}}_{ik}),\sum_{l=1}^{B_{j}}H({\mathbf{b}}_{jl}))$

$L^{c}:=\log{\mathbf{r}}_{i}{\mathbf{r}}_{j}+\log(1-{\mathbf{f}}_{a})(1-{\mathbf{f}}_{b})$

/ / update discriminator parameters

$D,H:=\textrm{Adam}(-L^{c})$

return $L^{c}$

Following the VAE paradigm, we implement the three trainable parametric
densities of our model as three normal distributions with diagonal covariance,
whose mean and variance are computed by feed-forward neural architectures. The
generator density takes the form of a normal distribution with fixed variance
and mean computed by the generator network $G$, taking as input the
concatenated domain and content codes. In practice, the output image will be
the mean of the distribution, rather than samples from it.

$p_{\theta}({\mathbf{x}}_{ik}|{\mathbf{m}}_{i},{\mathbf{o}}_{ik})=\mathcal{N}(\mu_{x},1)\textrm{,
where }\leavevmode\nobreak\ \mu_{x}=G({\mathbf{m}}_{i},{\mathbf{o}}_{ik})$ (8)

The domain inference density is a normal that requires its parameters to be
computed by a neural architecture processing a variable number of un-ordered,
exchangeable inputs. For this, we use a Deep Set network architecture (Zaheer
et al., 2017), whereby each input is individually encoded by a the same
network $E_{m}$, then the outputs are averaged together, and the result is
passed through a second network $F_{m}$. We average the outputs instead of
summing them (as used in Zaheer et al. (2017)) because we want the inference
density of the domain to be agnostic to the number of inputs in the pack.

$q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})=\mathcal{N}(\mu_{m},\sigma_{m}^{2})\textrm{,
where }\leavevmode\nobreak\
\mu_{m},\sigma_{m}=F_{m}\left(\frac{1}{K_{i}}\sum_{k=1}^{K_{i}}E_{m}({\mathbf{x}}_{ik})\right)$
(9)

The content inference density is a normal with parameters computed by encoding
an image with a network $E_{o}$, then concatenating the output with the domain
code of the pack, then passing it through another network $F_{o}$.

$q_{\xi}({\mathbf{o}}_{ik}|{\mathbf{x}}_{ik},{\mathbf{m}}_{i})=\mathcal{N}(\mu_{o},\sigma_{o}^{2})\textrm{,
where }\leavevmode\nobreak\
\mu_{o},\sigma_{o}=F_{o}({\mathbf{m}}_{i},E_{o}({\mathbf{x}}_{ik}))$ (10)

Diagrams depicting each of these architectures are displayed in Figure 2. We
employ the reparametrization trick (Kingma & Welling, 2014) to sample from the
inference posterior over the latents. As for the domain and content priors,
they are fixed arbitrary normal distributions with mean 0 and variance 1. A
more complete specifications of the neural implementation is available in
Appendix B.

##  3 Domain-Confusion Loss

Figure 3: Neural architecture of the Domain-Confusion discriminator $\eta$.

So far, Principle 1 is encouraged by optimizing the Evidence Lower Bound,
while Princicple 2 is enforced by construction. We inspect whether the model
also satisfies Principle 3, which requires that the distribution of inferred
content variables in a pack be orthogonal to the domain of the pack. We can
reformulate this principle more precisely to claim that, in the limit of
infinitely large packs, the distribution of a random variable
$\tilde{{\mathbf{o}}}$, denoting the random choice of one content variable
${\mathbf{o}}_{ik}$ from a pack of inferred content variables
$\underline{{\mathbf{o}}}_{i}$, should be the same same regardless of the pack
of origin.

$q_{\zeta,\xi}(\tilde{{\mathbf{o}}}|\underline{{\mathbf{x}}}_{i})=q_{\zeta,\xi}(\tilde{{\mathbf{o}}}|\underline{{\mathbf{x}}}_{j})\textrm{,
where }K_{i},K_{j}\to\infty$ (11)

This statement relies on the fact that, when the size of the pack tends to
infinity, the random picking of one inferred content variable from the pack
defines a distribution over content values conditioned on the “true” domain of
the pack. Since the probability of content features should be independent of
domain features, the empirical density
$q(\tilde{{\mathbf{o}}}|\underline{{\mathbf{x}}}_{i})$ should also stay fixed
as $i$ changes.

#### Claim

When the Domain-Content Evidence Lower Bound (equation 5) is maximized to its
theoretical potential, then equation 11 is satisfied. In other words, when the
inference latent posterior
$q_{\zeta,\xi}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})$
approaches the generative latent posterior
$p_{\theta,\alpha,\beta}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})$,
and the generative data likelihood
$p_{\theta,\alpha,\beta}(\underline{{\mathbf{x}}}_{i})$ produces samples
indistinguishable from the real data, then the distribution of content
variables will become independent of the “true” domain of the pack, in the
limit of large packs. A discussion and proof of this is included in Appendix
A.

This result shows that our probabilistic model is, in theory, sufficient to
satisfy Principle 3. However, this state cannot be achieved in practice,
because of architectural limitations on both the generative and inference
density families. Therefore, in order to encourage the realization of
Principle 3, we can constrain the space of inference densities to those for
which equation 11 holds, at least approximately. This need is reinforced by
empirical observations of the unconstrained model, which reveal that the
distribution of inferred content variables within a pack is highly sensitive
to the task, network architecture and choice of hyperparameters.

In this work, we tackle this problem practically by proposing an adversarial
loss which encourages the homogeneous distribution of content variables across
packs by penalizing differences between pairs of packs in their set of content
values. We call this the Domain-Confusion loss, and we show empirically in
Table 6 that it increases the quality and robustness of the disentanglement.

The loss is built around an adversarial discriminator $\eta$ which receives as
input a pair of packs of content values and is trained to output 1 if the two
packs come from the same distribution, or 0 if they are differently
distributed. This is called a verification task, inspired by Sohn et al.
(2019), and we apply it to contrast pairs of sub-packs coming from the same
pack with pairs of sub-packs coming from different packs. Concretely, every
iteration of training takes as input two packs of content values
$\underline{{\mathbf{o}}}_{i},\underline{{\mathbf{o}}}_{j}$, which we split
randomly into $(\underline{{\mathbf{a}}}_{i},\underline{{\mathbf{b}}}_{i})$
and $(\underline{{\mathbf{a}}}_{j},\underline{{\mathbf{b}}}_{j})$,
respectively. The loss takes the value:

$L^{c}(\underline{{\mathbf{o}}}_{i},\underline{{\mathbf{o}}}_{j})=\log\eta(\underline{{\mathbf{a}}}_{i},\underline{{\mathbf{b}}}_{i})+\log\eta(\underline{{\mathbf{a}}}_{j},\underline{{\mathbf{b}}}_{j})+\log(1-\eta(\underline{{\mathbf{a}}}_{i},\underline{{\mathbf{a}}}_{j}))+\log(1-\eta(\underline{{\mathbf{b}}}_{i},\underline{{\mathbf{b}}}_{j}))$
(12)

The greater the loss, the more the discriminator can distinguish between the
two packs. Because the architecture of $\eta$ needs to accommodate two packs
of varying sizes, we implement it as a Deep Set (Zaheer et al., 2017), just
like in the case of the domain encoder. Unlike to the domain encoder, here we
use summation instead of averaging, since $\eta$ does not need to be agnostic
to the number of inputs. The discriminator takes the form:

$\eta(\underline{{\mathbf{o}}}_{i},\underline{{\mathbf{o}}}_{j})=D\left(\sum_{k=1}^{K_{i}}H({\mathbf{o}}_{ik}),\leavevmode\nobreak\
\sum_{l=1}^{K_{j}}H({\mathbf{o}}_{jl})\right)$ (13)

where $D$ and $H$ are neural networks. A diagram of this architecture is
displayed in Figure 3. The full implementation is available in Algorithm 2.

##  4 Experiments

(a) Disentangling font and character.

(b) Disentangling object and imaging conditions.

(c) Disentangling object shape and object rotation.

Figure 4: Qualitative results of our model on the fusion task are displayed
with a black border. Each row represents a domain and each column represents a
content. The topmost row is the content input and each row of the leftmost 5
columns is a domain input.

We evaluate our model on the generalized domain-content fusion task mentioned
in the introduction. Given a trained model and two unseen image packs
$\underline{{\mathbf{x}}}_{i}$, $\underline{{\mathbf{x}}}_{j}$, we extract the
domain of $\underline{{\mathbf{x}}}_{i}$ and the contents of every image in
$\underline{{\mathbf{x}}}_{j}$, and then use them to generate $K_{j}$ output
images.

This task enables us to visually inspect the quality of the disentanglement by
judging how well the model follows each of the domain-content principles: Are
the images of high-quality? Do they have the same domain features? Do the
content features of each correspond to those of the associated input image?

We apply our model to three datasets: a dataset of font images collected from
Google Fonts, the Small Norb Dataset (LeCun et al., 2004) and an original
dataset, called Silhouettes, comprising 3-dimensional block shapes imaged at
various rotation angles. Detailed descriptions of the datasets are available
in Appendix C. For the Silhouettes and Google Font datasets, we test only on
domains which have been withheld during training, in order to show how well
the model generalizes to new domains. We display results for 5 testing packs
of each dataset in Figure 4. Further results on more datasets are available in
the Appendix D. We also provide results in Appendix D for testing the model on
both unseen domains and unseen contents. The results appear to separate domain
and content features very well without sacrificing the image quality as
compared to the VAE.

###  4.1 Predicting ground-truth factors from the latent representation

In order to obtain more quantitative evidence of disentanglement, we adapt the
factor regression method also used by Greff et al. (2019) to measure factor
information and interpretability in the latent space. The method involves
learning a simple linear mapping between the latent space of the trained model
and the value space of the ground-truth factor. A high predictive accuracy
implies that the latent representation contains the necessary factor
information and also organizes it in an easily interpretable way.

We complete this experiment on our Silhouettes dataset, and learn separate
predictors for the domain features (object shape) and content features
(rotation angle). Predicting the shape is a 27-way binary classification and
predicting the rotation angle is a two-way regression. We provide more details
on the encoding of these features in Appendix C. We compare each latent
representation of our model (domain and content) with a VAE, a FactorVAE (Kim
& Mnih, 2018), considered to be state-of-the-art in disentanglement, and
random guessing. In the case of our model, the goal is that each of the two
representations should predict its own factor as much as possible, and not to
predict the other’s factor. We include a comparison of our model with and
without the Domain-Confusion loss as an ablation study on the impact of this
loss. Details on the experimental setup and measurements, as well as
comparisons with other models on other datasets, are available in Appendix D.

Table 1: How well can the latent representation predict the ground-truth factors | MO (w/ DC) | MO (w/o DC) |  |  |
---|---|---|---|---|---
Factor (metric) | domain | content | domain | content | FactorVAE | VAE | Guessing
Object shape (CE) | -0.023 | -0.449 | -0.051 | -0.402 | -0.211 | -0.236 | -0.451
Rotation (MSE) | 673.4 | 456.2 | 672.8 | 533.2 | 563.1 | 597.8 | 671.3

The results, displayed in Table 6, reveal not only that the latent variables
of our model predict their corresponding factors far better than the FactorVAE
or VAE, but also that the cross-over predictions are no better than random.
This result shows that domain and content features are concentrated
successfully in their designated representations. Moreover, we can see a
marked improvement in the model with the Domain-Confusion loss over the one
without it, in the case of both domain and content factors.

##  5 Conclusion

In this work, we have described a general problem of disentangling the domain
and content generative factors, and proposed a probabilistic model with an
associated neural network to solve this task. We have built the model
according to the VAE paradigm and introduced the Domain-Confusion loss to
compensate for limitations brought upon by the neural architecture. We have
proven the effectiveness of our disentanglement solution by providing both
qualitative results on the domain-content fusion tasks, and quantitative
measures of the predictive power of the latent representations.

One crucial direction to explore in future work is the relationship between
domain-content disentanglement and the notion of invariant risk Arjovsky et
al. (2019) in the context of causal inference. Designing methods that can
separate confounding environmental factors (domain) from the factors of
interest (content) would lead to significant innovations in many research
fields, especially in the study of medical counterfactuals. The method
presented here is not yet able to perform such separation in the case where
content distributions vary across domains, since it uses the homogeneity
across domains as a proxy for identifying content factors.

##  6 Broader Impact

One of the main ethical faults associated with the application of statistical
learning to real-world problems is the acquisition of any biases that might be
present in the training dataset. As Arjovsky et al. (2019) discuss in their
work, classical deep learning methods minimizing the expected risk of their
hypothesis cannot distinguish between spurious correlations and true
mechanisms. Therefore, naive statistical correspondences are drawn between
phenomena that are not causally connected, creating unintended consequences
with potential scientific or social impact. Although our model is still an
expected-risk minimizing algorithm, we note that it could also be used as a
paradigm for diagnosing dataset biases. For example, if two distinct
populations of elements, sharing the same set of classes in a classification
task, vary in their representation of different classes across different
datasets, then the same individual placed in various datasets will produce, in
turn, different content encodings, revealing the biases in the dataset. Our
model is still very theoretical, but we believe it is a step towards a deeper
study of the relationship between element and environment in the context of
deep learning.

## References

  * Arjovsky et al. (2019) Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Ben-David et al. (2010) Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W.  A theory of learning from different domains.  _Machine learning_ , 79(1-2):151–175, 2010.
  * Benaim & Wolf (2018) Benaim, S. and Wolf, L.  One-shot unsupervised cross domain translation.  In _Advances in Neural Information Processing Systems_ , pp. 2104–2114, 2018.
  * Bengio et al. (2013) Bengio, Y., Courville, A., and Vincent, P.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1798–1828, 2013.
  * Burgess et al. (2019) Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A.  Monet: Unsupervised scene decomposition and representation.  _arXiv preprint arXiv:1901.11390_ , 2019.
  * Chen et al. (2018) Chen, T. Q., Li, X., Grosse, R. B., and Duvenaud, D. K.  Isolating sources of disentanglement in variational autoencoders.  In _Advances in Neural Information Processing Systems_ , pp. 2610–2620, 2018.
  * Choi et al. (2018) Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., and Choo, J.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 8789–8797, 2018.
  * Choi et al. (2020) Choi, Y., Ha, Y., Yoo, J., Uh, J.-W., Kim, S., and Choo, J.  Stargan v2: Diverse image synthesis for multiple domains.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Engelcke et al. (2020) Engelcke, M., Kosiorek, A. R., Jones, O. P., and Posner, I.  Genesis: Generative scene inference and sampling with object-centric latent representations.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=BkxfaTVFwH.
  * Eslami et al. (2018) Eslami, S. A., Rezende, D. J., Besse, F., Viola, F., Morcos, A. S., Garnelo, M., Ruderman, A., Rusu, A. A., Danihelka, I., Gregor, K., et al.  Neural scene representation and rendering.  _Science_ , 360(6394):1204–1210, 2018.
  * Ganin et al. (2016) Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V.  Domain-adversarial training of neural networks.  _The Journal of Machine Learning Research_ , 17(1):2096–2030, 2016.
  * Gonzalez-Garcia et al. (2018) Gonzalez-Garcia, A., van de Weijer, J., and Bengio, Y.  Image-to-image translation for cross-domain disentanglement.  In _Advances in Neural Information Processing Systems_ , pp. 1287–1298, 2018.
  * Greff et al. (2019) Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L., Botvinick, M., and Lerchner, A.  Multi-object representation learning with iterative variational inference.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97, pp. 2424–2433. PMLR, 2019.
  * Isola et al. (2017) Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1125–1134, 2017.
  * Kahn & Marshall (1953) Kahn, H. and Marshall, A. W.  Methods of reducing sample size in monte carlo computations.  _Journal of the Operations Research Society of America_ , 1(5):263–278, 1953.
  * Kim & Mnih (2018) Kim, H. and Mnih, A.  Disentangling by factorising.  In _Proceedings of the 35rd International Conference on Machine Learning_ , volume 80, pp. 2649–2658, 2018.
  * Kingma & Welling (2014) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  _2nd International Conference on Learning Representations, ICLR 2019_ , 2014.
  * Kingma et al. (2014) Kingma, D. P., Mohamed, S., Rezende, D. J., and Welling, M.  Semi-supervised learning with deep generative models.  In _Advances in neural information processing systems_ , pp. 3581–3589, 2014.
  * LeCun et al. (2004) LeCun, Y., Huang, F. J., and Bottou, L.  Learning methods for generic object recognition with invariance to pose and lighting.  In _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , volume 2, pp. II–104. IEEE, 2004.
  * Lee et al. (2020) Lee, H.-Y., Tseng, H.-Y., Mao, Q., Huang, J.-B., Lu, Y.-D., Singh, M., and Yang, M.-H.  Drit++: Diverse image-to-image translation via disentangled representations.  _International Journal of Computer Vision_ , 2020.
  * Liu et al. (2019) Liu, M.-Y., Huang, X., Mallya, A., Karras, T., Aila, T., Lehtinen, J., and Kautz, J.  Few-shot unsupervised image-to-image translation.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2019\.
  * Locatello et al. (2019) Locatello, F., Bauer, S., Lucic, M., Rätsch, G., Gelly, S., Schölkopf, B., and Bachem, O.  Challenging common assumptions in the unsupervised learning of disentangled representations.  _International Conference on Machine Learning_ , 2019.
  * Mildenhall et al. (2020) Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R.  Nerf: Representing scenes as neural radiance fields for view synthesis, 2020.
  * Rezende et al. (2014) Rezende, D. J., Mohamed, S., and Wierstra, D.  Stochastic backpropagation and approximate inference in deep generative models.  In _Proceedings of the 31st International Conference on Machine Learning_ , 2014.
  * Sitzmann et al. (2019) Sitzmann, V., Zollhöfer, M., and Wetzstein, G.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In _Advances in Neural Information Processing Systems_ , pp. 1119–1130, 2019.
  * Sohn et al. (2019) Sohn, K., Shang, W., Yu, X., and Chandraker, M.  Unsupervised domain adaptation for distance metric learning.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=BklhAj09K7.
  * Taigman et al. (2017) Taigman, Y., Polyak, A., and Wolf, L.  Unsupervised cross-domain image generation.  In _5th International Conference on Learning Representations, ICLR_ , 2017.
  * Tschannen et al. (2018) Tschannen, M., Bachem, O., and Lucic, M.  Recent advances in autoencoder-based representation learning.  In _Third workshop on Bayesian Deep Learning (NeurIPS 2018)_ , 2018\.  URL http://www.nari.ee.ethz.ch/pubs/p/autoenc2018.
  * van Steenkiste et al. (2019) van Steenkiste, S., Locatello, F., Schmidhuber, J., and Bachem, O.  Are disentangled representations helpful for abstract visual reasoning?  In _Advances in Neural Information Processing Systems_ , pp. 14222–14235, 2019.
  * Watters et al. (2019) Watters, N., Matthey, L., Burgess, C. P., and Lerchner, A.  Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes.  _arXiv preprint arXiv:1901.07017_ , 2019.
  * Yoon et al. (2020) Yoon, J. S., Kim, K., Gallo, O., Park, H. S., and Kautz, J.  Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Zaheer et al. (2017) Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J.  Deep sets.  In _Advances in neural information processing systems_ , pp. 3391–3401, 2017.
  * Zhu et al. (2017) Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A.  Upaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2223–2232, 2017.

## Appendix A - Proofs

### Evidence Lower Bound

$\displaystyle\log p_{\theta,\alpha,\beta}(\underline{{\mathbf{x}}}_{i})$
$\displaystyle=\log\mathbb{E}_{p_{\alpha,\beta}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i})}p_{\theta}(\underline{{\mathbf{x}}}_{i}|{\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i})\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \textrm{(total probability)}$ (14)
$\displaystyle=\log\mathbb{E}_{q_{\zeta,\xi}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})}\left[p_{\theta}(\underline{{\mathbf{x}}}_{i}|{\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i})\frac{p_{\alpha,\beta}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i})}{q_{\zeta,\xi}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})}\right]\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \textrm{(importance sampling)}$ (15)
$\displaystyle\geq\underbrace{\mathbb{E}_{q_{\zeta,\xi}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})}\log
p_{\theta}(\underline{{\mathbf{x}}}_{i}|{\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i})}_{\textrm{reconstruction}}+\underbrace{\mathbb{E}_{q_{\zeta,\xi}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})}\log\frac{p_{\alpha,\beta}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i})}{q_{\zeta,\xi}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})}}_{\textrm{latent
penalties}\leavevmode\nobreak\ ({\mathbf{A}})}\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \textrm{(Jensen's inequality)}$ (16)

Latent penalties:

$\displaystyle-{\mathbf{A}}=$ $\displaystyle\leavevmode\nobreak\
\mathbb{E}_{q_{\zeta,\xi}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})}\log\frac{q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})\prod_{k=1}^{K_{i}}q_{\xi}({\mathbf{o}}_{ik}|{\mathbf{x}}_{ik},{\mathbf{m}}_{i})}{p_{\alpha}({\mathbf{m}}_{i})\prod_{k=1}^{K_{i}}p_{\beta}({\mathbf{o}}_{ik})}\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \textrm{(factorization)}$ (17)
$\displaystyle=$ $\displaystyle\leavevmode\nobreak\
\underbrace{D_{\mathrm{KL}}[q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})||p_{\alpha}({\mathbf{m}}_{i})]}_{\textrm{domain}}+\underbrace{\mathbb{E}_{q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})}\sum_{k=1}^{K_{i}}D_{\mathrm{KL}}[q_{\xi}({\mathbf{o}}_{ik}|{\mathbf{x}}_{ik},{\mathbf{m}}_{i})\leavevmode\nobreak\
||\leavevmode\nobreak\ p_{\beta}({\mathbf{o}}_{ik})]}_{\textrm{content}}$ (18)

We can also rewrite the ELBO in terms of the marginal data likelihood by
rearranging the terms:

$ELBO(\underline{{\mathbf{x}}}_{i})=\underbrace{\log
p_{\theta,\alpha,\beta}(\underline{{\mathbf{x}}}_{i})}_{\textrm{data
likelihood}}-\underbrace{D_{\mathrm{KL}}[q_{\zeta,\xi}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i}|\underline{{\mathbf{x}}}_{i})||p_{\alpha,\beta}({\mathbf{m}}_{i},\underline{{\mathbf{o}}}_{i})]}_{\textrm{variational
gap}}$ (19)

### Content Independence

We first introduce $\tilde{{\mathbf{x}}}$ which represents the image variable
from input pack $\underline{{\mathbf{x}}}_{i}$ which corresponds to the
selected content $\tilde{{\mathbf{o}}}$. $\tilde{{\mathbf{x}}}$ is distributed
according to the empirical density
$h(\tilde{{\mathbf{x}}}|\underline{{\mathbf{x}}}_{i})$. We apply the law of
total probability in order to obtain an expectation of the content inference
density $q_{\xi}$ in terms of $\tilde{{\mathbf{x}}}$ and ${\mathbf{m}}_{i}$.

$\displaystyle
q_{\zeta,\xi}(\tilde{{\mathbf{o}}}|\underline{{\mathbf{x}}}_{i})$
$\displaystyle=\mathbb{E}_{h(\tilde{{\mathbf{x}}}|\underline{{\mathbf{x}}}_{i})}q_{\zeta,\xi}(\tilde{{\mathbf{o}}}|\tilde{{\mathbf{x}}},\underline{{\mathbf{x}}}_{i})\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \textrm{(total probability)}$ (20)
$\displaystyle=\mathbb{E}_{h(\tilde{{\mathbf{x}}}|\underline{{\mathbf{x}}}_{i})}\mathbb{E}_{q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})}q_{\zeta,\xi}(\tilde{{\mathbf{o}}}|\tilde{{\mathbf{x}}},\underline{{\mathbf{x}}}_{i},{\mathbf{m}}_{i})\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \textrm{(total probability)}$ (21)
$\displaystyle=\mathbb{E}_{h(\tilde{{\mathbf{x}}}|\underline{{\mathbf{x}}}_{i})}\mathbb{E}_{q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})}q_{\xi}(\tilde{{\mathbf{o}}}|\tilde{{\mathbf{x}}},{\mathbf{m}}_{i})\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \textrm{(conditional independence)}$
(22)

As $K_{i}\to\infty$, selecting an image variable from the pack
$h(\tilde{{\mathbf{x}}}|\underline{{\mathbf{x}}}_{i})$ becomes equivalent to
sampling an image variable directly from the generative density of the
$p_{\theta,\beta}(\tilde{{\mathbf{x}}}|{\mathbf{m}}_{i}^{\prime})$, where
${\mathbf{m}}_{i}^{\prime}$ is the value of the “true domain” of pack
$\underline{{\mathbf{x}}}_{i}$. For this we have also assumed that the input
images come from the generative distribution
$p_{\theta,\alpha,\beta}(\underline{{\mathbf{x}}}_{i})$, one of the necessary
conditions for which the ELBO is maximised.

$h(\tilde{{\mathbf{x}}}|\underline{{\mathbf{x}}}_{i})\overset{K_{i}\to\infty}{\approx}\mathbb{E}_{p_{\theta,\beta}(\underline{{\mathbf{x}}}_{i}|{\mathbf{m}}_{i}^{\prime})}h(\tilde{{\mathbf{x}}}|\underline{{\mathbf{x}}}_{i})=p_{\theta,\beta}(\tilde{{\mathbf{x}}}|{\mathbf{m}}_{i}^{\prime})$
(23)

We make the same observation for the domain inference density. In order to
express ${\mathbf{m}}_{i}$ in terms of ${\mathbf{m}}_{i}^{\prime}$ we use the
fact that the inference posterior density approaches the generative posterior
density, which is the other necessary condition for which the ELBO is
maximized.

$q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})\overset{K_{i}\to\infty}{\approx}\mathbb{E}_{p_{\theta,\beta}(\underline{{\mathbf{x}}}_{i}|{\mathbf{m}}_{i}^{\prime})}q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})\overset{q_{\zeta,\xi}\to
p_{\theta,\alpha,\beta}}{\approx}p({\mathbf{m}}_{i}|{\mathbf{m}}_{i}^{\prime})$
(24)

$p({\mathbf{m}}_{i}|{\mathbf{m}}_{i}^{\prime})$ is an artificial density which
simply indicates that all the probability mass rests on the value
${\mathbf{m}}_{i}^{\prime}$. We can now re-write the latent posterior as being
the content posterior conditioned on the “true domain”.

$\mathbb{E}_{q_{\zeta}({\mathbf{m}}_{i}|\underline{{\mathbf{x}}}_{i})}q_{\xi}(\tilde{{\mathbf{o}}}|\tilde{{\mathbf{x}}},{\mathbf{m}}_{i})\overset{\textrm{Eq.
\ref{eq:dom_infty}}}{\approx}\mathbb{E}_{p({\mathbf{m}}_{i},{\mathbf{m}}_{i}^{\prime})}q_{\xi}(\tilde{{\mathbf{o}}}|\tilde{{\mathbf{x}}},{\mathbf{m}}_{i})\overset{q_{\zeta,\xi}\to
p_{\theta,\alpha,\beta}}{\approx}p_{\theta,\beta}(\tilde{{\mathbf{o}}}|\tilde{{\mathbf{x}}},{\mathbf{m}}_{i}^{\prime})$
(25)

We now introduce the results we have obtained into the initial expression.

$q_{\zeta,\xi}(\tilde{{\mathbf{o}}}|\underline{{\mathbf{x}}}_{i})\overset{Eq.\ref{eq:img_infty},\leavevmode\nobreak\
\ref{eq:dom_rep}}{\approx}\mathbb{E}_{p_{\theta,\beta}(\tilde{{\mathbf{x}}}|{\mathbf{m}}_{i}^{\prime})}p_{\theta,\beta}(\tilde{{\mathbf{o}}}|\tilde{{\mathbf{x}}},{\mathbf{m}}_{i}^{\prime})=p_{\beta}(\tilde{{\mathbf{o}}}|{\mathbf{m}}_{i}^{\prime})=p_{\beta}(\tilde{{\mathbf{o}}})\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \textrm{(cond. independence)}$ (26)

We have shown that, if $q_{\zeta,\xi}\to p_{\theta,\alpha,\beta}$,
$K_{i}\to\infty$, and the input images are generated by
$p_{\theta,\alpha,\beta}$, then
$q_{\zeta,\xi}(\tilde{{\mathbf{o}}}|\underline{{\mathbf{x}}}_{i})\approx
p_{\beta}(\tilde{{\mathbf{o}}})$ which is independent of the pack or its
domain.

## Appendix B - Architectural Details

The model consists of four networks: domain encoder, content encoder, decoder
and discriminator. In this section we provide their implementation. For all
networks we use Adam optimizer with learning rate of 0.0001 and $\epsilon$ of
1e-8. We set the latent code size for both domain and content to 16, we refer
to them as $S_{D}$ and $S_{C}$ respectively. The encoder and decoder networks
use spatial broadcasting to aid disentanglement, as introduced by (Watters et
al., 2019).

#### Domain encoder:

Before being fed into the encoder images are transformed into the coordinate
matrix - recording the row index and column index for every pixel for every
image in the pack: ($i^{th}$ pack size $K_{i}$)$\times$(image height
$H$)$\times$(image width $W$) $\times$(image channels $C+2$ (one for rows, one
for cols). The summary is provided in Table 2.

The coordinate matrix is concatenated along the input pack dimension and
passed through the convolutional layers. After the inputs are passed through
the convolutional layers the output is reshaped into:
$K_{i}\times(H*W/(16*16/128))$. The latent codes are then averaged along the
pack dimension. The output of the operation has $1\times(H*W/2)$ dimensions.
It is then passed through a fully connected layer which maps the latent space
to the mean and variance of domain distribution with shape $1\times S_{D}$.

Table 2: Architecture of the domain encoder Type | Stride | Size/Ch | Act. Fun | Comment
---|---|---|---|---
Conv 4x4 | 1 | 64 | ReLU |
Conv 4x4 | 1 | 64 | ReLU |
Conv 4x4 | 4 | 128 | ReLU |
Conv 4x4 | 4 | 128 | ReLU |
Reshape | - | - | - | Output dims: $K_{i}\times(H*W/(16*16/128))$
Average | - | - | - | Output dims: $1\times(H*W/2)$
Dense | - | 512 | ReLU |
Dense | - | $K_{i}\times S_{D}$ | - | Outputs the mean of the distribution
Dense | - | $K_{i}\times S_{D}$ | - | Outputs the variance of the distribution

#### Content encoder:

The architecture of the content encoder follows the same general principle as
domain encoder up to the averaging layer. However, now, instead of averaging
along the pack we concatenate a domain code on top of every current code to
get tensor of size $K_{i}\times(H*W/2+S_{D})$.

The next step is the same as with domain encoder for encoding the means and
variances $K_{i}\times S_{C}$. Note that now there is only one domain code and
as many content codes as there are elements in the pack. Detailed architecture
is given in Table 3.

Table 3: Architecture of the content encoder Type | Stride | Size/Ch | Act. Fun | Comment
---|---|---|---|---
Conv 4x4 | 1 | 64 | ReLU |
Conv 4x4 | 1 | 64 | ReLU |
Conv 4x4 | 4 | 128 | ReLU |
Conv 4x4 | 4 | 128 | ReLU |
Reshape |  |  |  | Output dims: $K_{i}\times(H*W/(16*16/128))$
Concatenate |  |  |  | Output dims: $K_{i}\times(H*W/2+S_{D})$
Dense | - | 512 | ReLU |
Dense | - | $K_{i}\times S_{C}$ | - | Outputs the mean of the distribution
Dense | - | $K_{i}\times S_{D}$ | - | Outputs the variance of the distribution

Spatial broadcasting decoder: We follow the same principles as Watters et al.
(2019). The full architecture is given in Table 4. The decoder takes as input
is domain and and content codes. These are concatenated to obtain a array of
$K_{i}\times(S_{D}+S_{C})$ dimensions. The array is then broadcast to the
image shape $K_{i}\times H\times W\times(S_{D}+S_{C})$.

We then concatenate the resulting array with a coordinate matrix of size
$K_{i}\times H\times W\times 2$ (first channel counts the rows and second
channel counts the cols). This way we end up with a array of $K_{i}\times
H\times W\times(S_{D}+S_{C}+2)$. Thus, for every pixel in every image in pack,
we have the whole domain code, the whole content code, row number and column
number.

Table 4: Architecture of the decoder Type | Stride | Size/Ch | Act. Fun | Comment
---|---|---|---|---
Conv 4x4 | 1 | 128 | ReLU |
Conv 4x4 | 1 | 128 | ReLU |
Conv 4x4 | 1 | 128 | ReLU |
Conv 4x4 | 1 | 64 | ReLU |
Conv 4x4 | 1 | 64 | ReLU |
Conv 4x4 | 1 | C | - | Output has image shape $H\times W\times C$

Discriminator: The discriminator takes as input two packs A and B of content
codes with sizes $K_{A}$ and $K_{B}$ respectively. The overall input size is
thus $K_{A}\times S_{C}$, $K_{B}\times S_{C}$. The output of the discriminator
is the probability of the content codes coming from the same domain. The
detailed architecture is given in Table 5.

Table 5: Architecture of the discriminator Type | Stride | Size/Ch | Act. Fun | Comment
---|---|---|---|---
Dense | - | 64 | ReLU | For each pack separately
Dense | - | 128 | ReLU | For each pack separately
Sum | - | 128 | - | Sum along pack dimension
Concat | - | 256 | - | Concatenate packs along the feature axis
Dense | - | 128 | ReLU |
Dense | - | 64 | ReLU |
Dense | - | 32 | ReLU |
Dense | - | 1 | - |

#### Domain Confusion Loss:

When adding the domain-confusion loss to the ELBO, the $\lambda$ coefficient
of the domain-confusion loss is 100.

## Appendix C - Datasets

For training we generate 16000 packs. To generate a pack we first generate a
set of images belonging to the same domain and then randomly sample from this
set. Each pack has a variable number of examples. The size of the pack is
given by $4+Pois(8)$, where $Pois(8)$ is a Poisson distribution with
$\lambda=8$ (i.e. on average each pack contained 12 examples).

#### Silhouettes Dataset:

The dataset consists of $96\times 96\times 3$ procedurally images of 3D
arrangements of cubes imaged at different rotation angles. We treat the
arrangement of the cubes as the domain. We generate images by first defining a
$3\times 3\times 3$ grid. Each grid cell can be empty or occupied by a cube.
We set the probability of a cube being present in a cell by Bernoulli
distribution with $p=\frac{1}{6}$. For each domain we treat rotation (view
angle) of the grid as the content. The angle is controlled by two parameters:
pitch and yaw. Both are uniformly distributed between 0 and 90. We withhold
$1000$ shapes during training (of the possible $2^{2}7$ shapes).

#### Google Fonts:

The dataset consists of $64\times 64\times 1$ images of 64 characters in 2767
font typefaces, downloaded from (https://github.com/google/fonts). We treat
font as the domain and character as a content. During training we withhold 100
fonts chosen randomly. In the unseen-content experiments we also withhold 4
characters chosen randomly.

#### Small Norbs:

The dataset (LeCun et al., 2004) consists of $96\times 96\times 1$ images of
toys 50 photographed under 972 conditions (camera azimuth, rotation and
lighting). The toy represents the domain and the condition represents the
content. We withhold 5 toys during training.

## Appendix D - Experiments

Throughout our experiments, the models are trained for 100 epochs and tested
only on withheld domains. We provide additional qualitative results on the
Google Fonts dataset in Figure 5 and on the Silhouettes dataset in Figure 6.

(a) First row is domain input, second row is content input, third row is
fusion output.

(b) First row is domain input, second row is content input, third row is
fusion output.

(c) First row is domain input, second row is content input, third row is
fusion output.

(d) First row is domain input, second row is content input, third row is
fusion output.

(e) First row is domain input, second row is content input, third row is
fusion output.

(f) First row is domain input, second row is content input, third row is
fusion output.

(g) First row is domain input, second row is content input, third row is
fusion output.

Figure 5: Qualitative results of the fusion task on the Google Fonts dataset.

Figure 6: Further qualitative results of the fusion task on the Silhouettes
dataset. Rows are domains, columns are contents.

### Unseen domain and unseen content

We withhold 4 characters from the Google Font dataset during training, and
then attempt the fusion task where both inputs come from unseen domains, and
the content input has an unseen content. Results are shown in Figure 7.

Figure 7: Qualitative results of the fusion task with unseen domains and
unseen contents. Row is domain, column is content. Leftmost 5 columns are
domain inputs. Top rows are content inputs. Center blocks are results with
unseen domains - seen contents. The rightmost blocks are results with unseen
domains - unseen contents.

### Quantitative results

Table 6 presents more quantitative results on both the Silhouettes and Google
Fonts datasets. We compare both versions of our model (with and without the
Domain-Confusion loss) with the VAE (Kingma & Welling, 2014; Rezende et al.,
2014), FactorVAE (Kim & Mnih, 2018), $\beta$-TCVAE (Chen et al., 2018), and
random guessing. For these models we use the same exact architecture and
hyperparameters as with our Domain-Content model. For the FactorVAE we use the
same discriminator network as in our model, without the summing and
concatenation. We use a $\beta=10$ for the $\beta$-TCVAE and a $\gamma=100$
for the FactorVAE, reflecting the relative weighting that the discriminator
loss has in our model. All the latent representations have dimensionality 16.

We obtain the scores by training a 1-layer affine mapping between the latent
representation and each data factor for each pre-trained model. The models are
trained for 50 epochs, after which the affine mappings are trained for 10
epochs. Each data factor has its own encoding: For the Silhouettes dataset,
the domain factor is a 27-way sigmoid classification (one for each cube in the
grid), while the content factor is a 2-way regression with values between 0
and 90. The scores are given in average cross-entropy (CE) and mean square
error (MSE).

For the Google Fonts dataset, the domain factor is predicted through a
verification task whereby a 2-layer MLP with 64 hidden units takes as input
two latent representations and has to output 1 if they have the same font or 0
if not. When training, half of the pairs have the same font and half do not.
The content factor is a 64-way softmax classification, where each unit
represents one character. The scores are given in average cross-entropy.

Table 6: How well can the latent representation predict the ground-truth
factors. Model (I) is the FactorVAE (Kim & Mnih, 2018) and model (II) is the
$\beta$-TCVAE (Chen et al., 2018).

Silhouettes Dataset
MO (w/ DC) MO (w/o DC) Factor (metric) domain content domain content (I) (II)
VAE Guess domain (CE) -0.023 -0.449 -0.051 -0.402 -0.211 -0.205 -0.236 -0.451
content (MSE) 673.4 456.2 672.8 533.2 563.1 581.6 597.8 671.3
Google Fonts Dataset
MO (w/ DC) MO (w/o DC) Factor (metric) domain content domain content (I) (II)
VAE Guess domain (CE) -0.103 -0.572 -0.162 -0.564 -0.411 -0.369 -0.430 -0.581
content (CE) -0.462 -0.015 -0.443 -0.035 -0.287 -0.321 -0.397 -0.473