# On the Generative Utility of Cyclic Conditionals

Chang Liu1, Haoyue Tang2, Tao Qin1, Jintao Wang2, Tie-Yan Liu1
1 Microsoft Research Asia, Beijing, 100080. 2 Tsinghua University, Beijing,
100084 Correspondence to: Chang Liu<EMAIL_ADDRESS>done during
an internship at Microsoft Research Asia.

###### Abstract

We study whether and how can we model a joint distribution $p(x,z)$ using two
conditional models $p(x|z)$ and $q(z|x)$ that form a cycle. This is motivated
by the observation that deep generative models, in addition to a likelihood
model $p(x|z)$, often also use an inference model $q(z|x)$ for extracting
representation, but they rely on a usually uninformative prior distribution
$p(z)$ to define a joint distribution, which may render problems like
posterior collapse and manifold mismatch. To explore the possibility to model
a joint distribution using only $p(x|z)$ and $q(z|x)$, we study their
_compatibility_ and _determinacy_ , corresponding to the existence and
uniqueness of a joint distribution whose conditional distributions coincide
with them. We develop a general theory for operable equivalence criteria for
compatibility, and sufficient conditions for determinacy. Based on the theory,
we propose a novel generative modeling framework CyGen that only uses the two
cyclic conditional models. We develop methods to achieve compatibility and
determinacy, and to use the conditional models to fit and generate data. With
the prior constraint removed, CyGen better fits data and captures more
representative features, supported by both synthetic and real-world
experiments.

##  1 Introduction

Deep generative models have achieved a remarkable success in the past decade
for generating realistic complex data $x$ and extracting useful
representations through their latent variable $z$. Variational auto-encoders
(VAEs) [45, 67, 14, 15, 46, 80] follow the Bayesian framework and specify a
prior distribution $p(z)$ and a likelihood model $p(x|z)$, so that a joint
distribution $p(x,z)=p(z)p(x|z)$ is defined for generative modeling (the joint
induces a distribution $p(x)$ on data). An inference model $q(z|x)$ is also
used to approximate the posterior distribution $p(z|x)$ (derived from the
joint $p(x,z)$), which serves for extracting representations. Other frameworks
like generative adversarial nets [30, 25, 27], flow-based models [24, 60, 44,
31] and diffusion-based models [74, 38, 76, 49] follow the same structure,
with different choices of the conditional models $p(x|z)$ and $q(z|x)$ and
training objectives. While for the prior $p(z)$, there is often not much
knowledge for complex data (like images, text, audio), and these models widely
adopt an uninformative prior such as a standard Gaussian. This however,
introduces some side effects:

  * •

Posterior collapse [15, 34, 64]: The standard Gaussian prior tends to squeeze
$q(z|x)$ towards the origin for all $x$, which degrades the representativeness
of the inferred $z$ for $x$ and hurts downstream tasks in the latent space
like classification and clustering.

  * •

Manifold mismatch [22, 28, 41]: Typically the likelihood model is continuous
(keeps topology), so the standard Gaussian prior would restrict the modeled
data distribution to a simply-connected support, which limits the capacity for
fitting data from a non-(simply) connected support.

While there are works trying to mitigate the two problems, they require either
a strong domain knowledge [48, 41], or additional cost to learn a complicated
prior model [52, 21, 79] sometimes even at the cost of inconvenient inference
[59, 87].

One question then naturally emerges: _Can we model a joint distribution
$p(x,z)$ only using the likelihood $p(x|z)$ and inference $q(z|x)$ models?_ If
we can, the limitations from specifying or learning a prior are then removed
from the root. Also, the inference model $q(z|x)$ is then no longer a
struggling approximation to a predefined posterior but participates in
defining the joint distribution (avoid “inner approximation”). Modeling
conditionals is also argued to be much easier than modeling marginal or joint
distributions directly [2, 9, 10]. In many cases, one may even have better
knowledge on the conditionals than on the prior, _e.g._ shift/rotation
invariance of image representations (CNNs [51] / SphereNet [20]), and rules to
extract frequency/energy features for audio [65]. It is then more natural and
effective to incorporate this knowledge into the conditionals than using an
uninformative prior.

In this paper, we explore such a possibility, and develop both a systematic
theory and a novel generative modeling framework CyGen (Cyclic-conditional
Generative model).

(1) Theoretical analysis on the question amounts to two sub-problems: can two
given cyclic conditionals correspond to a common joint, and if yes, can they
determine the joint uniquely. We term them _compatibility_ and _determinacy_
of two conditionals, corresponding to the existence and uniqueness of a common
joint. For this, we develop novel compatibility criteria and sufficient
conditions for determinacy. Beyond existing results, ours are operable (vs.
existential [11]) and self-contained (vs. need a marginal [9, 10, 50, 32]),
and are general enough to cover both continuous and discrete cases. Our
compatibility criteria are also equivalence (vs. unnecessary [1, 4, 5, 6])
conditions. The seminal book [6] makes extensive analysis for various
parametric families. Besides the equivalence criteria, we also extend their
general analysis beyond the product support case, and also cover the Dirac
case.

(2) In addition to its independent contribution, the theory also enables
generative modeling using only the two cyclic conditional models, _i.e._ the
CyGen framework. We develop methods for achieving compatibility and
determinacy to make an eligible generative model, and for fitting and
generating data to serve as a generative model. Efficient implementation
techniques are designed. Note CyGen also determines a prior implicitly; it
just does not need an explicit model for the prior (vs. [52, 21, 79, 59]). We
show the practical utility of CyGen in both synthetic and real-world tasks.
The improved performance in downstream classification and data generation
demonstrates the advantage to mitigate the posterior collapse and manifold
mismatch problems.

###  1.1 Related work

Dependency networks ([35]; similarly [39]) are perhaps the first to pursue the
idea of modeling a joint by a set of conditionals. They use Gibbs sampling to
determine the joint and are equivalent to undirected graphical models. They do
not allow latent variables, so compatibility is not a key consideration as the
data already specifies a joint as the common target of the conditionals.
Beyond that, we introduce latent variables to better handle sensory data like
images, for which we analyze the conditions for compatibility and determinacy
and design novel methods to solve this different task.

Denoising auto-encoders (DAEs). AEs [71, 7] aim to extract data features by
enforcing reconstruction through its encoder and decoder, which are
deterministic hence insufficient determinacy (see Sec. 2.2.2). DAEs [82, 9,
10] use a probabilistic encoder and decoder for robust reconstruction against
random data corruption. Their utility as a generative model is first noted
through the equivalence to score matching (implies modeling $p(x)$) for a
Gaussian RBM [81] or an infinitesimal Gaussian corruption [2]. In more general
cases, the utility to modeling the joint $p(x,z)$ is studied via the Gibbs
chain, _i.e._ the Markov chain with transition kernel
$p(x^{\prime}|z^{\prime})q(z^{\prime}|x)$. Under a global [9, 50, 32] or local
[10] shared support condition, its stationary distribution $\pi(x,z)$ exists
uniquely. But this is _not really determinacy_ : even incompatible
conditionals can have this unique existence, in which case $\pi(z|x)\neq
q(z|x)$ [35, 9]. Moreover, the Gibbs chain does not give an explicit
expression of $\pi(x,z)$ (thus intractable likelihood evaluation), and
requires many iterations to converge for data generation and even for training
(Walkback [9], GibbsNet [50]), making the algorithms costly and unstable.

As for _compatibility_ , it is not really covered in DAEs. Existing results
only consider the statistical consistency (unbiasedness under infinite data)
of the $p(x|z)$ estimator by fitting $(x,z)$ data from $p^{*}(x)q(z|x)$ [9,
10, 50, 32], where $p^{*}(x)$ denotes the true data distribution.
Particularly, they require a marginal $p^{*}(x)$ in advance, so that the joint
is already defined by $p^{*}(x)q(z|x)$ regardless of $p(x|z)$, while
compatibility (as well as determinacy) is a matter only of the two
conditionals.

More crucially, the DAE loss is not proper for optimizing $q(z|x)$ as it
promotes a mode-collapse behavior. This hinders both compatibility and
determinacy (Sec. 3.2): one may not use $q(z|x)$ for inference, and data
generation may depend on initialization. In contrast, CyGen explicitly
enforces compatibility and guarantees determinacy, and enables likelihood
evaluation and better generation.

Dual learning considers conversion between two modalities in both directions,
_e.g._ , machine translation [33, 85, 84] and image style transfer [42, 90,
88, 53]. Although we also consider both directions, the fundamental
distinction is that in generative modeling there is no data of the latent
variable $z$ (not even unpaired). Technically, they did not consider
determinacy: they require a marginal to determine a joint. We find their
determinacy is actually insufficient (see Sec. 2.2.2). Their cycle-consistency
loss [42, 90, 88] is a version of our compatibility criterion in the Dirac
case (see Sec. 2.2.1), and we extend it to allow probabilistic conversion (see
Sec. 3.1).

##  2 Compatibility and Determinacy Theory

To be a generative model, a system needs to determine a distribution on the
data variable $x$. With latent variable $z$, this amounts to determining a
joint distribution over $(x,z)$, which calls for compatibility and determinacy
analysis for cyclic conditionals. In this section we build a general theory on
the conditions for compatibility and determinacy. We begin with formalizing
the problems.

Setup. Denote the measure spaces of the two random variables $x$ and $z$ as
$(\mathbb{X},\mathscr{X},\xi)$ and $(\mathbb{Z},\mathscr{Z},\zeta)$111 The
symbol $\mathbb{Z}$ overwrites the symbol for the set of integers, which is
not involved in this paper. , where $\mathscr{X}$, $\mathscr{Z}$ are the
respective sigma-fields, and the base measures $\xi$, $\zeta$ (_e.g._ ,
Lebesgue measure on Euclidean spaces, counting measure on finite/discrete
spaces) are sigma-finite. We use $\mathcal{X}\in\mathscr{X}$,
$\mathcal{Z}\in\mathscr{Z}$ to denote measurable sets, and use
“$\overset{\text{\tiny$\xi$}}{=}$”, “$\subseteq^{\text{$\xi$}}$” as the
extensions of “$=$”, “$\subseteq$” up to a set of $\xi$-measure-zero (Def.
A.1). Following the convention in machine learning, we call a “probability
measure” as a “distribution”. We do not require any further structures such as
topology, metric, or linearity, for the interest of the most general
conclusions that unify Euclidean/manifold and finite/discrete spaces and allow
$\mathbb{X}$, $\mathbb{Z}$ to have different dimensions or types.

Joint and conditional distributions are defined on the product measure space
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z},\xi\\!\otimes\\!\zeta)$,
where “$\\!\times\\!$” is the usual Cartesian product,
$\mathscr{X}\\!\otimes\\!\mathscr{Z}:=\sigma(\mathscr{X}\\!\times\\!\mathscr{Z})$
is the sigma-field generated by measurable rectangles from
$\mathscr{X}\\!\times\\!\mathscr{Z}$, and $\xi\\!\otimes\\!\zeta$ is the
product measure [13, Thm. 18.2]. Define the _slice_ of
$\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$ at $z$ as
$\mathcal{W}_{z}:=\\{x\mid(x,z)\in\mathcal{W}\\}\in\mathscr{X}$ [13, Thm.
18.1(i)], and its _projection_ onto $\mathbb{Z}$ as
$\mathcal{W}^{\mathbb{Z}}:=\\{z\mid\exists
x\in\mathbb{X}\text{~{}s.t.~{}}(x,z)\in\mathcal{W}\\}\in\mathscr{Z}$ (Appx.
A.3). In a similar style, denote the _marginal_ of a joint $\pi$ on
$\mathbb{Z}$ as
$\pi^{\mathbb{Z}}(\mathcal{Z}):=\pi(\mathbb{X}\\!\times\\!\mathcal{Z})$. To
keep the same level of generality, we follow the general definition of
conditionals ([13, p.457]; see also Appx. A.4): the conditional
$\pi(\mathcal{X}|z)$ of a joint $\pi$ is the density function (R-N derivative)
of $\pi(\mathcal{X}\\!\times\\!\cdot)$ w.r.t $\pi^{\mathbb{Z}}$. We highlight
the key characteristic under this generality that $\pi(\cdot|z)$ can be
arbitrary on a set of $\pi^{\mathbb{Z}}$-measure-zero, particularly, outside
the support of $\pi^{\mathbb{Z}}$. Appx. A and B provide more background
details and our technical preparations that are also of independent interest.
The goal of analysis can then be formalized below.

######  Definition 2.1 (compatibility and determinacy).

We say two conditionals $\mu(\mathcal{X}|z)$, $\nu(\mathcal{Z}|x)$ are
_compatible_ , if there exists a joint distribution $\pi$ on
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z})$ such
that $\mu(\mathcal{X}|z)$ and $\nu(\mathcal{Z}|x)$ are its conditional
distributions. We say two compatible conditionals have _determinacy_ on a set
$\mathcal{S}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$, if there is only one
joint distribution concentrated on $\mathcal{S}$ that makes them compatible.

To put the concept into practical use, the analysis aims at operable
conditions for compatibility and determinacy. We consider two cases separately
(still unifying continuous and discrete cases), as they correspond to
different types of generative models, and lead to qualitatively different
conclusions.

###  2.1 Absolutely Continuous Case

We first consider the case where for any $z\in\mathbb{Z}$ and any
$x\in\mathbb{X}$,222There may be problems if absolute continuity holds only
for $\zeta$-a.e. $z$ and $\xi$-a.e. $x$; see Appx. Example C.2. the
conditionals $\mu(\cdot|z)$ and $\nu(\cdot|x)$ are either absolutely
continuous (w.r.t $\xi$ and $\zeta$, resp.) [13, p.448], or zero in the sense
of a measure. Equivalently, they have density functions $p(x|z)$ and $q(z|x)$
(non-negative by definition; may integrate to zero). This case include
“smooth” distributions on Euclidean spaces or manifolds, and _all_
distributions on finite/discrete spaces. Many generative modeling frameworks
use density models thus count for this case, including VAEs [45, 67, 66, 46,
80] and diffusion-based models [74, 38, 76].

####  2.1.1 Compatibility criterion in the absolutely continuous case

One may expect that when absolutely continuous conditionals $p(x|z)$ and
$q(z|x)$ are compatible, their joint is also absolutely continuous (w.r.t
$\xi\\!\otimes\\!\zeta$) with some density $p(x,z)$. This intuition is
verified by our Lem. C.1 in Appx. C.1. One could then safely apply density
function formulae and get
$\frac{p(x|z)}{q(z|x)}=\frac{p(x,z)}{p(z)}/\frac{p(x,z)}{p(x)}=\frac{p(x)}{p(z)}$
factorizes into a function of $x$ and a function of $z$. Conversely, if the
ratio factorizes as such $\frac{p(x|z)}{q(z|x)}=a(x)b(z)$, one could get
$p(x|z)\frac{1}{Ab(z)}=q(z|x)\frac{a(x)}{A}$ where
$A:=\int_{\mathbb{X}}a(x)\xi(\mathrm{d}x)$, which defines a joint density and
compatibility is achieved. This intuition leads to the classical compatibility
criterion [1; 4, Thm. 4.1; 5, Thm. 1; 6, Thm. 2.2.1].

However, the problem is more complicated than imagined. Berti et al. [11,
Example 9] point out that the classical criterion is unfortunately _not
necessary_. The subtlety is about on which region does this factorization have
to hold. The classical criterion requires it to be the positive region of
$p(x|z)$ which also needs to coincide with that of $q(z|x)$. But as mentioned,
conditional $\mu(\cdot|z)$ can be arbitrary outside the support of the
marginal $\pi^{\mathbb{Z}}$ (similarly for $\nu(\cdot|x)$), which may lead to
additional positive regions that violate the requirement.333The flexibility of
$p(x|z)$ on a $\xi$-measure-zero set for a given $z$ (similarly for $q(z|x)$)
is not a vital problem, as one can adjust the conditions to hold only a.e. To
address the problem, Berti et al. [11] give an equivalence criterion (Thm. 8),
but it is _existential_ thus less useful as the definition of compatibility
itself is existential. Moreover, these criteria are restricted to either
Euclidean or discrete spaces.

Next we give our _equivalence_ criterion that is _operable_. In addressing the
subtlety with regions, we first introduce a related concept that helps
identify appropriate regions.

Figure 1: Illustration of a $\xi\\!\otimes\\!\zeta$-complete component
$\mathcal{S}$ of $\mathcal{W}$.

a

######  Definition 2.2 ($\xi\\!\otimes\\!\zeta$-complete component).

For a set $\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$, we say that a
set $\mathcal{S}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$ is a
_$\xi\\!\otimes\\!\zeta$ -complete component_ of $\mathcal{W}$, if
$\mathcal{S}^{\sharp}\cap\mathcal{W}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}$,
where
$\mathcal{S}^{\sharp}:=\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\cup\mathbb{X}\\!\times\\!\mathcal{S}^{\mathbb{Z}}$
is the _stretch_ of $\mathcal{S}$.

Fig. 1 illustrates the concept. Roughly, the stretch $\mathcal{S}^{\sharp}$ of
$\mathcal{S}$ represents the region where the conditionals are a.s. determined
if $\mathcal{S}$ is the _support_ 444 While the typical definition of support
requires a topological structure which is absent under our generality, Def.
B.8 in Appx. B.1 defines such a concept for absolutely continuous
distributions.  of the joint. If $\mathcal{S}$ is a complete component of
$\mathcal{W}$, it is complete under stretching and intersecting with
$\mathcal{W}$. Such a set $\mathcal{S}$ is an a.s. subset of $\mathcal{W}$
(Lem. B.12), while has a.s. the same slice as $\mathcal{W}$ does for almost
all $z\in\mathcal{S}^{\mathbb{Z}}$ and $x\in\mathcal{S}^{\mathbb{X}}$ (Lem.
B.16). This is critical for the normalizedness of distributions in our
criterion. Appx. B.3 shows more facts. With this concept, our compatibility
criterion is presented below.

Figure 2: Illustration of our compatibility criterion in the absolutely
continuous case (Thm. 2.3). The conditionals are uniform on the respective
depicted slices. For condition (i),
$\mathcal{P}_{z}\subseteq^{\text{$\xi$}}\mathcal{Q}_{z}$ is _not_ satisfied on
the left half, _e.g._ $z_{1}$, so $\mathcal{W}_{p,q}$ does not cover the left
half; it is satisfied on the right half, _e.g._ $z_{2}$, so
$\mathcal{W}_{p,q}$ is composed of slices $\mathcal{P}_{z}$ on the right half,
making the top-right quadrant (shaded). Similarly, $\mathcal{W}_{q,p}$ is the
same region, and it is a $\xi\\!\otimes\\!\zeta$-complete component of itself.
It also satisfies other conditions thus is a complete support $\mathcal{S}$.

a

######  Theorem 2.3 (compatibility criterion, absolutely continuous).

Let $p(x|z)$ and $q(z|x)$ be the density functions of two everywhere
absolutely continuous (or zero) conditional distributions, and define:

$\displaystyle\mathcal{P}_{z}:=\\{x\mid p(x|z)>0\\},\mathcal{P}_{x}:=\\{z\mid
p(x|z)>0\\},$ (2) $\displaystyle\mathcal{Q}_{z}:=\\{x\mid
q(z|x)>0\\},\mathcal{Q}_{x}:=\\{z\mid q(z|x)>0\\}.$ (3)

Then they are compatible, if and only if they have a _complete support_
$\mathcal{S}$, defined as a (i) $\xi\\!\otimes\\!\zeta$-complete component of
both

$\displaystyle\mathcal{W}_{p,q}\\!:=\\!\\!\\!\\!\\!\bigcup_{z:\mathcal{P}_{z}\subseteq^{\text{$\xi$}}\mathcal{Q}_{z}}\\!\\!\\!\\!\mathcal{P}_{z}\\!\times\\!\\{z\\},\;\mathcal{W}_{q,p}\\!:=\\!\\!\\!\\!\\!\bigcup_{x:\mathcal{Q}_{x}\subseteq^{\text{$\zeta$}}\mathcal{P}_{x}}\\!\\!\\!\\!\\{x\\}\\!\times\\!\mathcal{Q}_{x},$
(4)

such that: (ii)
$\mathcal{S}^{\mathbb{X}}\subseteq^{\text{$\xi$}}\mathcal{W}_{q,p}^{\mathbb{X}}$,
$\mathcal{S}^{\mathbb{Z}}\subseteq^{\text{$\zeta$}}\mathcal{W}_{p,q}^{\mathbb{Z}}$,
(iii) $(\xi\\!\otimes\\!\zeta)(\mathcal{S})>0$, and (iv)
$\frac{p(x|z)}{q(z|x)}$ factorizes as $a(x)b(z)$, $\xi\\!\otimes\\!\zeta$-a.e.
on $\mathcal{S}$,555 Formally, there exist functions $a$ on
$\mathcal{S}^{\mathbb{X}}$ and $b$ on $\mathcal{S}^{\mathbb{Z}}$ s.t.
$(\xi\\!\otimes\\!\zeta)\\{(x,z)\in\mathcal{S}\mid\frac{p(x|z)}{q(z|x)}\neq
a(x)b(z)\\}=0$. where (v) $a(x)$ is $\xi$-integrable on
$\mathcal{S}^{\mathbb{X}}$. For sufficiency,

$\displaystyle\pi(\mathcal{W}):=\frac{\int_{\mathcal{W}\cap\mathcal{S}}q(z|x)\left\lvert
a(x)\right\rvert(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)}{\int_{\mathcal{S}^{\mathbb{X}}}\left\lvert
a(x)\right\rvert\xi(\mathrm{d}x)},$ (5)

$\forall\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$, is a compatible
joint of them.

Fig. 2 shows an illustration of the conditions. To understand the criterion,
conditions (iv) and (v) stem from the starting inspiration, which also shows a
hint for Eq. (5). Other conditions handle the subtlety to find a region
$\mathcal{S}$ where (iv) and (v) must hold. This is essentially the support of
a compatible joint $\pi$ as there is no need and no way to control
conditionals outside the support.

For necessity, informally, if $z$ is in the support of $\pi^{\mathbb{Z}}$,
then $p(x|z)$ determines the distribution on $\mathbb{X}\\!\times\\!\\{z\\}$;
particularly, the joint $\pi$ should be a.e. positive on $\mathcal{P}_{z}$,
which in turn asks $q(z|x)$ to be so. This means
$\mathcal{P}_{z}\subseteq^{\text{$\xi$}}\mathcal{Q}_{z}$ (unnecessary equal,
since $q(z|x)$ is “out of control” outside the joint support), which leads to
the definitions of $\mathcal{W}_{p,q}$ and $\mathcal{W}_{q,p}$. The joint
support should be contained within the two sets in order to avoid _support
conflict_ (_e.g._ , although the bottom-left quadrant in Fig. 2 is part of the
intersection of positive regions of the conditionals, a joint on it is
required by $p(x|z)$ to also cover the top-left, on which $q(z|x)$ does not
agree). Condition (i) indicates
$\mathcal{S}\subseteq^{\text{$\xi\\!\otimes\\!\zeta$}}\mathcal{W}_{p,q}$ and
$\mathcal{W}_{q,p}$ so $\mathcal{S}$ satisfies this requirement and also makes
the ratio in (iv) a.e. well-defined. The complete-component condition in (i)
also makes the conditionals _normalized_ on $\mathcal{S}$: as mentioned, such
an $\mathcal{S}$ has a.s. the same slice as $\mathcal{W}_{p,q}$ does for a
given $z$ in support $\mathcal{S}^{\mathbb{Z}}$, so the integral of $p(x|z)$
on $\mathcal{S}_{z}$ is the same as that on
$(\mathcal{W}_{p,q})_{z}=\mathcal{P}_{z}$ which is 1 by construction;
similarly for $q(z|x)$. In contrast, Appx. Example C.3 shows
$\mathcal{S}=\mathcal{W}_{p,q}\cap\mathcal{W}_{q,p}$ is inappropriate.
Conditions (ii) and (iii) cannot be guaranteed by condition (i) (Appx. Example
B.13), while are needed to rule out special cases (Appx. Lem. B.14, Example
B.15). Appx. C.2 gives a formal proof. Finally, although the criterion relies
on the _existence_ of such a complete support, candidates are few (if any), so
it is _operable_.

####  2.1.2 Determinacy in the absolutely continuous case

When compatible, absolutely continuous cyclic conditionals are very likely to
have determinacy.

######  Theorem 2.4 (determinacy, absolutely continuous).

Let $p(x|z)$ and $q(z|x)$ be two compatible conditional densities, and
$\mathcal{S}$ be a complete support that makes them compatible (necessarily
exists due to Thm. 2.3). Suppose that
$\mathcal{S}_{z}\overset{\text{\tiny$\xi$}}{=}\mathcal{S}^{\mathbb{X}}$, for
$\zeta$-a.e. $z$ on $\mathcal{S}^{\mathbb{Z}}$, or
$\mathcal{S}_{x}\overset{\text{\tiny$\zeta$}}{=}\mathcal{S}^{\mathbb{Z}}$, for
$\xi$-a.e. $x$ on $\mathcal{S}^{\mathbb{X}}$. Then their compatible joint
supported on $\mathcal{S}$ is unique, which is given by Eq. (5).

Proof is given in Appx. C.4. The condition in the theorem roughly means that
the complete support $\mathcal{S}$ is “rectangular”. From the perspective of
Markov chain, this corresponds to the _irreducibility_ of the Gibbs chain for
the unique existence of a stationary distribution. When the conditionals have
multiple such complete supports, on each of which the compatible joint is
unique, while globally on $\mathbb{X}\\!\times\\!\mathbb{Z}$, they may have
multiple compatible joints. In general, determinacy in the absolutely
continuous case is _sufficient_ , particularly we have the following strong
conclusion in a common case (_e.g._ , for VAEs).

######  Corollary 2.5.

We call two conditional densities _have a.e.-full supports_ , if
$p(x|z)>0,q(z|x)>0$ for $\xi\\!\otimes\\!\zeta$-a.e. $(x,z)$. If they are
compatible, then their compatible joint is unique, since
$\mathbb{X}\\!\times\\!\mathbb{Z}$ is the $\xi\\!\otimes\\!\zeta$-unique
complete support (Prop. C.4 in Appx. C.3), which satisfies the condition in
Thm. 2.4.

###  2.2 Dirac Case

Many other prevailing generative models, including generative adversarial
networks (GANs) [30] and flow-based models [24, 60, 44, 31], use a
deterministic function $x=f(z)$ as the likelihood model. In such cases, the
conditional
$\mu(\mathcal{X}|z)=\delta_{f(z)}(\mathcal{X}):=\mathbb{I}[f(z)\in\mathcal{X}],\forall\mathcal{X}\in\mathscr{X}$
is a Dirac measure. Note it does not have a density function when $\xi$
assigns zero to all single-point sets, _e.g._ the Lebesgue measure on
Euclidean spaces, so we keep the measure notion. This case is not exclusive to
the absolutely continuous case: a Dirac conditional on a discrete space is
also absolutely continuous.

####  2.2.1 Compatibility criterion in the Dirac case

Figure 3: Illustration of our compatibility criterion in the Dirac case (Thm.
2.6).

a

Compatibility criterion is easier to imagine in this case. As illustrated in
Fig. 3, it is roughly that the other-way conditional $\nu(\cdot|x)$ could find
a way to put its mass only on the curve; otherwise support conflict is
rendered.

######  Theorem 2.6 (compatibility criterion, Dirac).

Suppose that $\mathscr{X}$ contains all the single-point sets:
$\\{x\\}\in\mathscr{X},\forall x\in\mathbb{X}$. Conditional distribution
$\nu(\mathcal{Z}|x)$ is compatible with
$\mu(\mathcal{X}|z):=\delta_{f(z)}(\mathcal{X})$ where function
$f:\mathbb{Z}\to\mathbb{X}$ is $\mathscr{X}$/$\mathscr{Z}$-measurable666 For
meaningful discussion, we require $f$ to be
$\mathscr{X}$/$\mathscr{Z}$-measurable, which includes any function between
discrete sets and continuous functions when $\mathscr{X}$ and $\mathscr{Z}$
are the Borel sigma-fields. , if and only if there exists $x_{0}\in\mathbb{X}$
such that $\nu(f^{-1}(\\{x_{0}\\})|x_{0})=1$.

See Appx. C.6 for proof. Note such $x_{0}$ must be in the image set
$f(\mathbb{Z})$, otherwise
$\nu(f^{-1}(\\{x_{0}\\})|x_{0})=\nu(\emptyset|x_{0})=0$. For a typical GAN
generator, the preimage set $f^{-1}(\\{x_{0}\\})$ is discrete, so a compatible
inference model must not be absolutely continuous. What may be counter-
intuitive is that $\nu(\cdot|x)$ is not required to concentrate on the curve
for _any_ $x$; one $x_{0}$ is sufficient as $\delta_{(x_{0},f(x_{0}))}$ is a
compatible joint.

Nevertheless, in practice one often desires the compatibility to hold over a
set $\mathcal{X}$ to make a useful model. When $\nu(\cdot|x)$ is also chosen
in the Dirac form $\delta_{g(x)}$, this can be achieved by minimizing
$\mathbb{E}_{p(x)}\ell\big{(}x,f(g(x))\big{)}$, where $p(x)$ is a distribution
on $\mathcal{X}$ and $\ell$ is a metric on $\mathbb{X}$. This is the _cycle-
consistency loss_ used in dual learning [42, 90, 88, 53]. When $f$ is
invertible, minimizing the loss (_i.e._ , $g=f^{-1}$ a.e. on $\mathcal{X}$) is
also necessary, as $f^{-1}(x)$ only has one element. Particularly, flow-based
models are naturally compatible (so are their injective variants [3, 86, 16]
by Thm. 2.6).

####  2.2.2 Determinacy in the Dirac case

As mentioned, for any $x_{0}$ satisfying the condition, two compatible
conditionals have the determinacy on this point $\\{x_{0}\\}$ with the unique
joint $\delta_{(x_{0},f(x_{0}))}$. But when such $x_{0}$ is not unique, the
distribution over these $x_{0}$ values is not determined, so the two
conditionals do not have the determinacy globally on
$\mathbb{X}\\!\times\\!\mathbb{Z}$. This is similar to the absolutely
continuous case with multiple complete supports; particularly, each
$\\{(x_{0},f(x_{0}))\\}$ is a complete support for discrete $\mathbb{X}$ and
$\mathbb{Z}$. This meets one’s intuition: compatible Dirac conditionals can
only determine a curve in $\mathbb{X}\\!\times\\!\mathbb{Z}$, but cannot
determine a distribution on the curve. One exception is when $f(z)\equiv
x_{0}$ is constant, so this $x_{0}$ is the only candidate. The joint then
degenerates to a distribution on $\mathbb{Z}$, which is fully determined by
$\nu(\cdot|x_{0})$.

In general, determinacy in the Dirac case is _insufficient_ , and this type of
generative models (GANs, flow-based models) have to specify a prior to define
a joint.

##  3 Generative Modeling using Cyclic Conditionals

The theory suggests it is possible that cyclic conditionals achieve
compatibility and a sufficient determinacy, so that they can determine a
useful joint without specifying a prior. Note a certain prior is implicitly
determined by the conditionals; we find we just do not need an explicit model
for it. This inspires CyGen, a novel framework that only uses Cyclic
conditionals for Generative modeling.

For the eligibility as a generative model, compatibility and a sufficient
determinacy are required. For the latter, we just shown a deterministic
likelihood or inference model is not suitable, so we use absolutely continuous
conditionals as the theory suggests. The conditionals can then be modeled by
parameterized densities $p_{\theta}(x|z)$, $q_{\phi}(z|x)$. We consider the
common case where $\mathbb{X}=\mathbb{R}^{d_{\mathbb{X}}}$,
$\mathbb{Z}=\mathbb{R}^{d_{\mathbb{Z}}}$, and $p_{\theta}(x|z)$,
$q_{\phi}(z|x)$ have a.e.-full supports and are differentiable. Determinacy is
then exactly guaranteed by Cor. 2.5. For compatibility, we develop an
effective loss in Sec. 3.1 to enforce it.

For the usage as a generative model, we develop methods to fit the model-
determined data distribution $p_{\theta,\phi}(x)$ to the true data
distribution $p^{*}(x)$ in Sec. 3.2, and to generate data from
$p_{\theta,\phi}(x)$ in Sec. 3.3.

###  3.1 Enforcing Compatibility

In this a.e.-full support case, the entire product space
$\mathbb{X}\\!\times\\!\mathbb{Z}$ is the only possible complete support
(Prop. C.4 in Appx. C.3), so for compatibility, condition (iv) in Thm. 2.3 is
the most critical one. For this, we do not have to find functions $a(x)$,
$b(z)$ in Thm. 2.3, but only need to enforce such a factorization. So we
propose the following loss function to enforce compatibility:

$\displaystyle(\min_{\theta,\phi})\;\;C(\theta,\phi):=\mathbb{E}_{\rho(x,z)}\left\lVert\nabla_{x}\nabla_{z}^{\top}r_{\theta,\phi}(x,z)\right\rVert_{F}^{2},\text{where
}r_{\theta,\phi}(x,z):=\log\big{(}p_{\theta}(x|z)/q_{\phi}(z|x)\big{)}.$ (6)

Here, $\rho$ is some absolutely continuous reference distribution on
$\mathbb{X}\\!\times\\!\mathbb{Z}$, which can be taken as
$p^{*}(x)q_{\phi}(z|x)$ in practice as it gives samples to estimate the
expectation. When $C(\theta,\phi)=0$, we have
$\nabla_{x}\nabla_{z}^{\top}r_{\theta,\phi}(x,z)=0$,
$\xi\\!\otimes\\!\zeta$-a.e. [13, Thm. 15.2(ii)]. By integration, this means
$\nabla_{z}r_{\theta,\phi}(x,z)=V(z)$ hence $r_{\theta,\phi}(x,z)=v(z)+u(x)$,
$\xi\\!\otimes\\!\zeta$-a.e., for some functions $V(z)$, $v(z)$, $u(x)$ s.t.
$V(z)=\nabla v(z)$. So the ratio
$p_{\theta}(x|z)/q_{\phi}(z|x)=\exp\\{r_{\theta,\phi}(x,z)\\}=\exp\\{u(x)\\}\exp\\{v(z)\\}$
factorizes, $\xi\\!\otimes\\!\zeta$-a.s.

In the sense of enforcing compatibility, this loss generalizes the cycle-
consistency loss to probabilistic conditionals. Also, the loss is different
from the Jacobian-norm regularizers in contractive AE [68] and DAE [68, 2],
and explains the “tied weights” trick for AEs [63, 82, 81, 68, 2] (see Appx.
D.1).

Implication on Gaussian VAE which uses additive Gaussian conditional models,
$p_{\theta}(x|z):=\mathcal{N}(x|f_{\theta}(z),\sigma_{d}^{2}I_{d_{\mathbb{X}}})$
and
$q_{\phi}(z|x):=\mathcal{N}(z|g_{\phi}(x),\sigma_{e}^{2}I_{d_{\mathbb{Z}}})$.
It is the vanilla and the most common form of VAE [45]. As its ELBO objective
drives $q_{\phi}(z|x)$ to meet the joint $p(z)p_{\theta}(x|z)$, compatibility
is enforced. Under our view, this amounts to minimizing the compatibility loss
Eq. (6), which then enforces the match of Jacobians:
$(\nabla_{z}f_{\theta}^{\top}(z))^{\top}=(\sigma_{d}^{2}/\sigma_{e}^{2})\nabla_{x}g_{\phi}^{\top}(x)$.
As the two sides indicate the equation is constant of both $x$ and $z$, it
must be a constant, so _$f_{\theta}(z)$ and $g_{\phi}(x)$ must be affine_, and
the joint is also a Gaussian [12; 6, Thm. 3.3.1]. This conclusion coincides
with the theory on additive noise models in causality [89, 62], and explains
the empirical observation that the latent space of such VAEs is quite linear
[73]. It is also the root of recent analyses that the latent space coordinates
the data manifold [21], and the inference model learns an isometric embedding
after a proper rescaling [58].

This finding reveals that the expectation to use deep neural networks for
learning a flexible nonlinear representation will be disappointed in Gaussian
VAE. So we use a non-additive-Gaussian model, _e.g._ a flow-based model [66,
46, 80, 31], for at least one of $p_{\theta}(x|z)$ and $q_{\phi}(z|x)$ (often
the latter).

Efficient implementation. Direct Jacobian evaluation for Eq. (6) is of
complexity $O(d_{\mathbb{X}}d_{\mathbb{Z}})$, which is often prohibitively
large. We thus propose a stochastic but unbiased and much cheaper method based
on Hutchinson’s trace estimator [40]:
$\operatorname{tr}(A)=\mathbb{E}_{p(\eta)}[\eta^{\top}A\eta]$, where $\eta$ is
any random vector with zero mean and identity covariance (_e.g._ , a standard
Gaussian). As the function within expectation is
$\left\lVert\nabla_{x}\nabla_{z}^{\top}r\right\rVert_{F}^{2}=\left\lVert\nabla_{z}\nabla_{x}^{\top}r\right\rVert_{F}^{2}=\operatorname{tr}\big{(}(\nabla_{z}\nabla_{x}^{\top}r)^{\top}\nabla_{z}\nabla_{x}^{\top}r\big{)}$,
applying the estimator yields a formulation that reduces gradient evaluation
complexity to $O(d_{\mathbb{X}}+d_{\mathbb{Z}})$:

$\displaystyle(\min_{\theta,\phi})\;\;C(\theta,\phi)={}$
$\displaystyle\mathbb{E}_{\rho(x,z)}\mathbb{E}_{p(\eta_{x})}\left\lVert\nabla_{z}\big{(}\eta_{x}^{\top}\nabla_{x}r_{\theta,\phi}(x,z)\big{)}\right\rVert_{2}^{2},\text{where
}\mathbb{E}[\eta_{x}]=0,\operatorname{Var}[\eta_{x}]=I_{d_{\mathbb{X}}}.$ (7)

As concluded from the above analysis on Gaussian VAE, we use a flow-based
model for the inference model $q_{\phi}(z|x)$. But in common instances
evaluating the inverse of the flow is intractable [66, 46, 80] or costly [31].
This however, disables the use of automatic differentiation tools for
estimating the gradients in the compatibility loss. Appx. D.2 explains this
problem in detail and shows our solution.

###  3.2 Fitting Data

After achieving compatibility, Cor. 2.5 guarantees the a.e.-fully supported
conditional models uniquely determine a joint, hence a data distribution
$p_{\theta,\phi}(x)$. To fit $p_{\theta,\phi}(x)$ to the true data
distribution $p^{*}(x)$, an explicit expression is required. For this, Eq. (5)
is not helpful as we do not have explicit expressions of $a(x)$, $b(z)$. But
when compatibility is given, we can safely use density function formulae:

$\displaystyle\textstyle
p_{\theta,\phi}(x)=1/\frac{1}{p_{\theta,\phi}(x)}=1/\int_{\mathbb{Z}}\frac{p_{\theta,\phi}(z^{\prime})}{p_{\theta,\phi}(x)}\zeta(\mathrm{d}z^{\prime})=1/\int_{\mathbb{Z}}\frac{q_{\phi}(z^{\prime}|x)}{p_{\theta}(x|z^{\prime})}\zeta(\mathrm{d}z^{\prime})=1/\mathbb{E}_{q_{\phi}(z^{\prime}|x)}[1/p_{\theta}(x|z^{\prime})],$
(8)

which is an explicit expression in terms of the two conditionals. Although
other expressions are possible, this one has a simple form, and the Monte-
Carlo expectation estimation in $\mathbb{Z}$ has a lower variance than in
$\mathbb{X}$ since usually $d_{\mathbb{Z}}\ll d_{\mathbb{X}}$. We can thus fit
data by maximum likelihood estimation:

$\displaystyle(\min_{\theta,\phi})\;\;\mathbb{E}_{p^{*}(x)}[-\log
p_{\theta,\phi}(x)]=\mathbb{E}_{p^{*}(x)}[\log\mathbb{E}_{q_{\phi}(z^{\prime}|x)}[1/p_{\theta}(x|z^{\prime})]].$
(9)

The loss function can be estimated using the reparameterization trick [45] to
reduce variance, and the logsumexp trick is adopted for numerical stability.
This expression can also serve for data likelihood evaluation. The final
training process of CyGen is the joint optimization with the compatibility
loss.

Comparison with DAE. We note that the DAE loss [82, 9]
$\mathbb{E}_{p^{*}(x)q_{\phi}(z^{\prime}|x)}[-\log p_{\theta}(x|z^{\prime})]$
is a _lower bound_ of Eq. (9) due to Jensen’s inequality, so it is not
suitable for maximizing likelihood. In fact, the DAE loss minimizes
$\mathbb{E}_{q_{\phi}(z)}\mathrm{KL}(q_{\phi}(x|z)\|p_{\theta}(x|z))$ for
$p_{\theta}(x|z)$ to match $q_{\phi}(x|z)$, where $q_{\phi}(z)$ and
$q_{\phi}(x|z)$ are induced from the joint $p^{*}(x)q_{\phi}(z|x)$, but it is
not a proper loss for $q_{\phi}(z|x)$ as a _mode-collapse_ behavior is
promoted: the optimal $q_{\phi}(z|x)$ only concentrates on the point(s) of
$\operatorname*{argmin}_{z^{\prime}}p_{\theta}(x|z^{\prime})$, and an
additional entropy term $-\mathbb{E}_{q_{\phi}(z)}\mathbb{H}[q_{\phi}(x|z)]$
is required to optimize the same $\mathrm{KL}$ loss. This behavior _hurts
determinacy_ , as $q_{\phi}(z|x)$ tends to be a (mixture of) Dirac measure
(Sec. 2.2.2). The resulting Gibbs chain may also converge differently
depending on initialization, as ergodicity is broken. This behavior also hurts
compatibility, as $q_{\phi}(x|z)$ deviates from $p_{\theta}(x|z)$ (not Dirac),
and does not match the Gibbs stationary distribution [35, 9]. In contrast,
CyGen follows a more fundamental logic: enforce compatibility explicitly and
follow the maximum likelihood principle faithfully. It leads to a proper loss
for both conditionals that does not hinder determinacy.

###  3.3 Data Generation

Generating samples from the learned data distribution $p_{\theta,\phi}(x)$ is
not as straightforward as typical models that specify a prior, since ancestral
sampling is not available. But it is still tractable via Markov chain Monte
Carlo methods (MCMCs). We propose using _dynamics-based MCMCs_ , which are
often more efficient than Gibbs sampling (used in DAE [9] and GibbsNet [50]).
They only require an _unnormalized_ density function of the target
distribution, which is readily available in CyGen when compatible:
$p_{\theta,\phi}(x)=\frac{p_{\theta,\phi}(x)}{p_{\theta,\phi}(z)}p_{\theta,\phi}(z)=\frac{p_{\theta}(x|z)}{q_{\phi}(z|x)}p_{\theta,\phi}(z)\propto\frac{p_{\theta}(x|z)}{q_{\phi}(z|x)}$
for any $z\in\mathbb{Z}$. In practice, this $z$ can be taken as a sample from
$q_{\phi}(z|x)$ to lie in a high probability region for a confident estimate.

Stochastic gradient Langevin dynamics (SGLD) [83] is a representative
instance, which has been shown to produce complicated realistic samples in
energy-based [26], score-based [75] and diffusion-based [38, 76] models. It
gives the following transition:

$\displaystyle\textstyle\\!x^{(t+1)}=x^{(t)}+\varepsilon\nabla_{x^{(t)}}\log\frac{p_{\theta}(x^{(t)}|z^{(t)})}{q_{\phi}(z^{(t)}|x^{(t)})}+\sqrt{2\varepsilon}\,\eta^{(t)}_{x},\text{where
}z^{(t)}\sim
q_{\phi}(z|x^{(t)}),\eta^{(t)}_{x}\sim\mathcal{N}(0,I_{d_{\mathbb{X}}}),\\!$
(10)

and $\varepsilon$ is a step size parameter. Method to draw $z\sim
p_{\theta,\phi}(z)$ can be developed symmetrically (see Appx. Eq. (91)). Also
applicable are other dynamics-based MCMCs [19, 23, 57], and particle-based
variational inference methods [56, 17, 54, 55, 77] which are more sample-
efficient.

##  4 Experiments

We demonstrate the power of CyGen for data generation and representation
learning. Baselines include DAE, and generative models using Gaussian prior
_e.g._ VAE and BiGAN (Appx. E.1). For a fair comparison, all methods use the
same architecture, which is an additive Gaussian $p_{\theta}(x|z)$ and a
Sylvester flow (Householder version) [80] for $q_{\phi}(z|x)$ (Appx. E.2), as
required by CyGen (Sec. 3.1). It is necessarily probabilistic for determinacy,
so we exclude flow-based generative models and common BiGAN/GibbsNet
architectures, which are deterministic. We also considered GibbsNet [50] which
also aims at the prior issue, but it does not produce reasonable results using
the same architecture, due to its unstable training process (see Appx. E.1).
Codes: https://github.com/changliu00/cygen.

###  4.1 Synthetic Experiments

data | DAE | VAE | BiGAN | CyGen | CyGen(PT)
---|---|---|---|---|---
|  |  |  |  |
class-wise aggregated posterior |  |  |  |  |
Table 4: Generated data (DAE and CyGen use SGLD) and class-wise aggregated posteriors of DAE, VAE, BiGAN and CyGen. Also shows results of CyGen(PT) that is PreTrained as a VAE. (Best view in color.)  iteration | 1100 | 1200 | 1300 | 1400 | 30000
---|---|---|---|---|---
CyGen(PT), using SGLD |  |  |  |  |
CyGen(PT), using Gibbs |  |  |  |  |
compt. loss | $7.0\\!\times\\!10^{3}$ | $5.4\\!\times\\!10^{3}$ | $7.7\\!\times\\!10^{3}$ | $6.2\\!\times\\!10^{3}$ | $4.6\\!\times\\!10^{3}$
CyGen(PT) w/o compt., using SGLD |  |  |  |  |
compt. loss | $1.1\\!\times\\!10^{5}$ | $1.6\\!\times\\!10^{5}$ | $2.6\\!\times\\!10^{5}$ | $8.6\\!\times\\!10^{5}$ | $1.2\\!\times\\!10^{8}$
Table 5: Generated data along the training process of CyGen after VAE
pretraining (iteration 1000), using SGLD (rows 1,3) and Gibbs sampling (row 2)
for generation, and with (rows 1,2) and without the compatibility loss (row 3)
for training. See also Appx. Fig. 13.

For visual verification of the claims, we first consider a 2D toy dataset
(Fig. 4 top-left). Appx. E.3 shows more details and results, including the
investigation on another similar dataset.

Data generation. The learned data distributions (as the histogram of generated
data samples) are shown in Fig. 4 (row 1). We see the five clusters are
blurred to overlap in VAE’s distribution and are still connected in BiGAN’s,
due to the specified prior. In contrast, our CyGen fits this distribution much
better; particularly it clearly separates the five non-connected clusters.
This verifies the advantage to overcome the _manifold mismatch_ problem. As
for DAE, it cannot capture the data distribution due to collapsed inference
model and insufficient determinacy (Sec. 3.2).

Representation. Class-wise aggregated posteriors (as the scatter plot of $z$
samples from $q_{\phi}(z|x)p^{*}(x|y)$ for each class/cluster $y$) in Fig. 4
(row 2) show that CyGen mitigates the _posterior collapse_ problem, as the
learned inference model $q_{\phi}(z|x)$ better separates the classes with a
margin in the latent space. This more informative and representative feature
would benefit downstream tasks like classification or clustering in the latent
space. In contrast, the specified Gaussian prior squeezes the VAE latent
clusters to touch, and the BiGAN latent clusters even to mix. The mode-
collapsed inference model of DAE locates all latent clusters in the same
place.

Incorporating knowledge into conditionals. CyGen alone (without pretraining)
already performs well. When knowledge is available, we can further incorporate
it into the conditional models. Fig. 4 shows pretraining CyGen’s likelihood
model as in a VAE (CyGen(PT)) embodies VAE’s knowledge that the prior is
centered and centrosymmetric, as the (all-class) aggregated posterior
($\approx$ prior) is such. Note its data generation quality is not sacrificed.
Appx. Fig. 14 verifies this directly via the priors.

Comparison of data generation methods. We then make more analysis on CyGen.
Fig. 5 (rows 1,2) shows generated data of CyGen using SGLD and Gibbs sampling.
We see SGLD better recovers the true distribution, and is more robust to
slight incompatibility.

Impact of the compatibility loss. Fig. 5 (rows 1,3) also shows the comparison
with training CyGen without the compatibility loss. We see the compatibility
is then indeed out of control, which invalidates the likelihood estimation Eq.
(9) for fitting data and the gradient estimation in Eq. (10) for data
generation, leading to the failure in row 3. Along the training process of the
normal CyGen, we also find a smaller compatibility loss makes better
generation (esp. using Gibbs sampling).

###  4.2 Real-World Experiments

DAE | VAE | CyGen(PT)
---|---|---
|  |
|  |
Table 6: Generated data on the MNIST and SVHN datasets.

We test the performance of CyGen on real-world image datasets MNIST and SVHN.
We consider the VAE-pretrained version, CyGen(PT), for more stable training.
Appx. E.4 shows more details. On these datasets, even BiGAN cannot produce
reasonable results using the same architecture, similar to GibbsNet.

Data generation. From Fig. 6, We see that CyGen(PT) generates both sharp and
diverse samples, as a sign to mitigate _manifold mismatch_. DAE samples are
mostly imperceptible, due to the mode-collapsed $q_{\phi}(z|x)$ and the
subsequent lack of determinacy (Sec. 3.2).

Table 7: Downstream classification accuracy (%) using learned representation
by various models.
†: Results from [50] using a different, deterministic architecture (not suitable for CyGen).  | DAE | VAE | BiGAN† | GibbsNet† | CyGen(PT)
---|---|---|---|---|---
MNIST | 98.0$\pm\\!$0.1 | 94.5$\pm\\!$0.3 | 91.0 | 97.7 | 98.3$\pm\\!$0.1
SVHN | 74.5$\pm\\!$1.0 | 30.8$\pm\\!$0.2 | 66.7 | 79.6 | 75.8$\pm\\!$0.5

VAE samples are a little blurry as a typical behavior due to the simply-
connected prior. This observation is also quantitatively supported by the FID
score [36, 72] on SVHN: CyGen achieves 102, while DAE 157 and VAE 128 (lower
is better).

Representation. We then show in Table 7 that CyGen(PT)’s latent representation
is more informative for the downstream classification task, as an indicator to
avoid _posterior collapse_. BiGAN and GibbsNet make random guess using the
same probabilistic flow architecture, and their reported results in [50] using
a different, deterministic architecture (not suitable for CyGen due to
insufficient determinacy) are still not always better, due to the prior
constraint. We conclude that CyGen achieves both superior generation and
representation learning performance.

##  5 Conclusions and Discussions

In this work we investigate the possibility of defining a joint distribution
using two conditional distributions, under the motivation for generative
modeling without an explicit prior. We develop a systematic theory with novel
and operable equivalence criteria for compatibility and sufficient conditions
for determinacy, and propose a novel generative modeling framework CyGen that
only uses cyclic conditional models. Methods for achieving compatibility and
determinacy, fitting data and data generation are developed. Experiments show
the benefits of CyGen over DAE and prevailing generative models that specify a
prior in overcoming manifold mismatch and posterior collapse.

The novel CyGen framework broadens the starting point to build a generative
model, and the general theory could also foster a deeper understanding of
other machine learning paradigms, _e.g._ , dual learning and self-supervised
learning, and inspire more efficient algorithms.

## References

  * Abrahams and Thomas [1984] J. Abrahams and J. B. Thomas.  A note on the characterization of bivariate densities by conditional densities.  _Communications in Statistics-Theory and Methods_ , 13(3):395–400, 1984.
  * Alain and Bengio [2014] G. Alain and Y. Bengio.  What regularized auto-encoders learn from the data-generating distribution.  _The Journal of Machine Learning Research_ , 15(1):3563–3593, 2014.
  * Ardizzone et al. [2019] L. Ardizzone, J. Kruse, C. Rother, and U. Köthe.  Analyzing inverse problems with invertible neural networks.  In _Proceedings of the International Conference on Learning Representations (ICLR 2019)_ , 2019.
  * Arnold and Press [1989] B. C. Arnold and S. J. Press.  Compatible conditional distributions.  _Journal of the American Statistical Association_ , 84(405):152–156, 1989.
  * Arnold et al. [2001] B. C. Arnold, E. Castillo, J. M. Sarabia, et al.  Conditionally specified distributions: an introduction.  _Statistical Science_ , 16(3):249–274, 2001.
  * Arnold et al. [2012] B. C. Arnold, E. Castillo, and J.-M. S. Alegria.  _Conditionally specified distributions_ , volume 73.  Springer Science & Business Media, 2012.
  * Baldi and Hornik [1989] P. Baldi and K. Hornik.  Neural networks and principal component analysis: Learning from examples without local minima.  _Neural Networks_ , 2(1):53–58, 1989.
  * Behrmann et al. [2019] J. Behrmann, W. Grathwohl, R. T. Chen, D. Duvenaud, and J.-H. Jacobsen.  Invertible residual networks.  In _International Conference on Machine Learning_ , pages 573–582. PMLR, 2019.
  * Bengio et al. [2013] Y. Bengio, L. Yao, G. Alain, and P. Vincent.  Generalized denoising auto-encoders as generative models.  In _Advances in Neural Information Processing Systems_ , 2013.
  * Bengio et al. [2014] Y. Bengio, E. Laufer, G. Alain, and J. Yosinski.  Deep generative stochastic networks trainable by backprop.  In _International Conference on Machine Learning_ , pages 226–234, 2014.
  * Berti et al. [2014] P. Berti, E. Dreassi, and P. Rigo.  Compatibility results for conditional distributions.  _Journal of Multivariate Analysis_ , 125:190–203, 2014\.
  * Bhattacharyya [1943] A. Bhattacharyya.  On some sets of sufficient conditions leading to the normal bivariate distribution.  _Sankhyā: The Indian Journal of Statistics_ , pages 399–406, 1943\.
  * Billingsley [2012] P. Billingsley.  _Probability and Measure_.  John Wiley & Sons, New Jersey, 2012.  ISBN 978-1-118-12237-2.
  * Bornschein et al. [2016] J. Bornschein, S. Shabanian, A. Fischer, and Y. Bengio.  Bidirectional Helmholtz machines.  In _International Conference on Machine Learning_ , pages 2511–2519. PMLR, 2016.
  * Bowman et al. [2016] S. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio.  Generating sentences from a continuous space.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 10–21, 2016.
  * Brehmer and Cranmer [2020] J. Brehmer and K. Cranmer.  Flows for simultaneous manifold learning and density estimation.  In _Advances in Neural Information Processing Systems_ , volume 33, 2020.
  * Chen et al. [2018] C. Chen, R. Zhang, W. Wang, B. Li, and L. Chen.  A unified particle-optimization framework for scalable Bayesian sampling.  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI 2018)_ , Monterey, California USA, 2018. Association for Uncertainty in Artificial Intelligence.
  * Chen et al. [2019] R. T. Chen, J. Behrmann, D. Duvenaud, and J.-H. Jacobsen.  Residual flows for invertible generative modeling.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Chen et al. [2014] T. Chen, E. Fox, and C. Guestrin.  Stochastic gradient Hamiltonian Monte Carlo.  In _Proceedings of the 31st International Conference on Machine Learning (ICML 2014)_ , pages 1683–1691, Beijing, China, 2014. IMLS.
  * Coors et al. [2018] B. Coors, A. P. Condurache, and A. Geiger.  SphereNet: Learning spherical representations for detection and classification in omnidirectional images.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 518–533, 2018.
  * Dai and Wipf [2019] B. Dai and D. Wipf.  Diagnosing and enhancing VAE models.  In _Proceedings of the International Conference on Learning Representations (ICLR 2019)_ , 2019.
  * Davidson et al. [2018] T. R. Davidson, L. Falorsi, N. De Cao, T. Kipf, and J. M. Tomczak.  Hyperspherical variational auto-encoders.  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI 2018)_ , 2018.
  * Ding et al. [2014] N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven.  Bayesian sampling using stochastic gradient thermostats.  In _Advances in Neural Information Processing Systems_ , pages 3203–3211, Montréal, Canada, 2014. NIPS Foundation.
  * Dinh et al. [2015] L. Dinh, D. Krueger, and Y. Bengio.  NICE: Non-linear independent components estimation.  In _Workshop on the International Conference on Learning Representations_ , 2015.
  * Donahue et al. [2017] J. Donahue, P. Krähenbühl, and T. Darrell.  Adversarial feature learning.  In _Proceedings of the International Conference on Learning Representations (ICLR 2017)_ , 2017.
  * Du and Mordatch [2019] Y. Du and I. Mordatch.  Implicit generation and modeling with energy based models.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Dumoulin et al. [2017] V. Dumoulin, I. Belghazi, B. Poole, O. Mastropietro, A. Lamb, M. Arjovsky, and A. Courville.  Adversarially learned inference.  In _Proceedings of the International Conference on Learning Representations (ICLR 2017)_ , 2017.
  * Falorsi et al. [2018] L. Falorsi, P. de Haan, T. R. Davidson, N. De Cao, M. Weiler, P. Forré, and T. S. Cohen.  Explorations in homeomorphic variational auto-encoding.  _arXiv preprint arXiv:1807.04689_ , 2018.
  * Galambos [1995] J. Galambos.  _Advanced probability theory_ , volume 10.  CRC Press, 1995.
  * Goodfellow et al. [2014] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems_ , pages 2672–2680, Montréal, Canada, 2014. NIPS Foundation.
  * Grathwohl et al. [2019] W. Grathwohl, R. T. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud.  FFJORD: Free-form continuous dynamics for scalable reversible generative models.  In _Proceedings of the International Conference on Learning Representations (ICLR 2019)_ , 2019.
  * Grover and Ermon [2019] A. Grover and S. Ermon.  Uncertainty autoencoders: Learning compressed representations via variational information maximization.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pages 2514–2524. PMLR, 2019.
  * He et al. [2016] D. He, Y. Xia, T. Qin, L. Wang, N. Yu, T.-Y. Liu, and W.-Y. Ma.  Dual learning for machine translation.  In _Advances in Neural Information Processing Systems_ , pages 820–828, 2016.
  * He et al. [2019] J. He, D. Spokoyny, G. Neubig, and T. Berg-Kirkpatrick.  Lagging inference networks and posterior collapse in variational autoencoders.  In _Proceedings of the International Conference on Learning Representations (ICLR 2019)_ , 2019.
  * Heckerman et al. [2000] D. Heckerman, D. M. Chickering, C. Meek, R. Rounthwaite, and C. Kadie.  Dependency networks for inference, collaborative filtering, and data visualization.  _Journal of Machine Learning Research_ , 1(Oct):49–75, 2000.
  * Heusel et al. [2017] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  GANs trained by a two time-scale update rule converge to a local Nash equilibrium.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Higgins et al. [2017] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner.  Beta-VAE: Learning basic visual concepts with a constrained variational framework.  In _Proceedings of the International Conference on Learning Representations (ICLR 2017)_ , 2017.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Hofmann and Tresp [1998] R. Hofmann and V. Tresp.  Nonlinear Markov networks for continuous variables.  _Advances in Neural Information Processing Systems_ , pages 521–527, 1998.
  * Hutchinson [1989] M. F. Hutchinson.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines.  _Communications in Statistics-Simulation and Computation_ , 18(3):1059–1076, 1989.
  * Kalatzis et al. [2020] D. Kalatzis, D. Eklund, G. Arvanitidis, and S. Hauberg.  Variational autoencoders with riemannian brownian motion priors.  In _International Conference on Machine Learning_ , pages 5053–5066. PMLR, 2020.
  * Kim et al. [2017] T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim.  Learning to discover cross-domain relations with generative adversarial networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 1857–1865. JMLR.org, 2017.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma and Dhariwal [2018] D. P. Kingma and P. Dhariwal.  Glow: generative flow with invertible 1$\\!\times\\!$ 1 convolutions.  In _Advances in Neural Information Processing Systems_ , pages 10236–10245, 2018.
  * Kingma and Welling [2014] D. P. Kingma and M. Welling.  Auto-encoding variational Bayes.  In _Proceedings of the International Conference on Learning Representations (ICLR 2014)_ , Banff, Canada, 2014. ICLR Committee.
  * Kingma et al. [2016] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling.  Improved variational inference with inverse autoregressive flow.  In _Advances in Neural Information Processing Systems_ , pages 4743–4751, Barcelona, Spain, 2016. NIPS Foundation.
  * Klenke [2013] A. Klenke.  _Probability theory: a comprehensive course_.  Springer Science & Business Media, 2013.
  * Kocaoglu et al. [2018] M. Kocaoglu, C. Snyder, A. G. Dimakis, and S. Vishwanath.  CausalGAN: Learning causal implicit generative models with adversarial training.  In _Proceedings of the International Conference on Learning Representations_ , 2018.
  * Kong et al. [2020] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro.  DiffWave: A versatile diffusion model for audio synthesis.  _arXiv preprint arXiv:2009.09761_ , 2020.
  * Lamb et al. [2017] A. M. Lamb, D. Hjelm, Y. Ganin, J. P. Cohen, A. C. Courville, and Y. Bengio.  GibbsNet: Iterative adversarial inference for deep graphical models.  In _Advances in Neural Information Processing Systems_ , pages 5089–5098, 2017.
  * LeCun et al. [1989] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  _Neural computation_ , 1(4):541–551, 1989.
  * Li et al. [2018] C. Li, M. Welling, J. Zhu, and B. Zhang.  Graphical generative adversarial networks.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Lin et al. [2019] J. Lin, Z. Chen, Y. Xia, S. Liu, T. Qin, and J. Luo.  Exploring explicit domain supervision for latent space disentanglement in unpaired image-to-image translation.  _IEEE transactions on pattern analysis and machine intelligence_ , 2019.
  * Liu et al. [2019a] C. Liu, J. Zhuo, P. Cheng, R. Zhang, J. Zhu, and L. Carin.  Understanding and accelerating particle-based variational inference.  In K. Chaudhuri and R. Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97, pages 4082–4092, Long Beach, California USA, 2019a. IMLS, PMLR.
  * Liu et al. [2019b] C. Liu, J. Zhuo, and J. Zhu.  Understanding MCMC dynamics as flows on the Wasserstein space.  In K. Chaudhuri and R. Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97, pages 4093–4103, Long Beach, California USA, 2019b. IMLS, PMLR.
  * Liu and Wang [2016] Q. Liu and D. Wang.  Stein variational gradient descent: A general purpose Bayesian inference algorithm.  In _Advances in Neural Information Processing Systems_ , pages 2370–2378, Barcelona, Spain, 2016. NIPS Foundation.
  * Ma et al. [2021] Y.-A. Ma, N. S. Chatterji, X. Cheng, N. Flammarion, P. L. Bartlett, and M. I. Jordan.  Is there an analog of Nesterov acceleration for gradient-based MCMC?  _Bernoulli_ , 27(3):1942–1992, 2021.
  * Nakagawa et al. [2021] A. Nakagawa, K. Kato, and T. Suzuki.  Quantitative understanding of VAE as a non-linearly scaled isometric embedding.  In _Proceedings of the 38th International Conference on Machine Learning_ , 2021.
  * Pang et al. [2020] B. Pang, T. Han, E. Nijkamp, S.-C. Zhu, and Y. N. Wu.  Learning latent space energy-based prior model.  In _Advances in Neural Information Processing Systems_ , volume 33, 2020.
  * Papamakarios et al. [2017] G. Papamakarios, T. Pavlakou, and I. Murray.  Masked autoregressive flow for density estimation.  In _Advances in Neural Information Processing Systems_ , pages 2335–2344, 2017.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al.  PyTorch: An imperative style, high-performance deep learning library.  _Advances in Neural Information Processing Systems_ , 32:8026–8037, 2019.
  * Peters et al. [2014] J. Peters, J. M. Mooij, D. Janzing, and B. Schölkopf.  Causal discovery with continuous additive noise models.  _Journal of Machine Learning Research_ , 15(1):2009–2053, 2014.
  * Ranzato et al. [2007] M. Ranzato, Y.-L. Boureau, and Y. LeCun.  Sparse feature learning for deep belief networks.  _Advances in Neural Information Processing Systems_ , 20:1185–1192, 2007.
  * Razavi et al. [2019] A. Razavi, A. v. d. Oord, B. Poole, and O. Vinyals.  Preventing posterior collapse with delta-VAEs.  In _Proceedings of the International Conference on Learning Representations (ICLR 2019)_ , 2019.
  * Ren et al. [2020] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  _arXiv preprint arXiv:2006.04558_ , 2020.
  * Rezende and Mohamed [2015] D. Rezende and S. Mohamed.  Variational inference with normalizing flows.  In _Proceedings of the 32nd International Conference on Machine Learning (ICML 2015)_ , pages 1530–1538, Lille, France, 2015. IMLS.
  * Rezende et al. [2014] D. J. Rezende, S. Mohamed, and D. Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In _International Conference on Machine Learning_ , pages 1278–1286, 2014.
  * Rifai et al. [2011] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio.  Contractive auto-encoders: Explicit invariance during feature extraction.  In _Proceedings of the International Conference on Machine Learning_ , 2011.
  * Rifai et al. [2012] S. Rifai, Y. Bengio, Y. N. Dauphin, and P. Vincent.  A generative process for sampling contractive auto-encoders.  In _Proceedings of the International Conference on Machine Learning_ , pages 1811–1818, 2012.
  * Rinaldo [2018] A. Rinaldo.  Advanced probability, February 2018.
  * Rumelhart et al. [1986] D. E. Rumelhart, G. E. Hinton, and R. J. Williams.  Learning representations by back-propagating errors.  _Nature_ , 323(6088):533–536, 1986.
  * Seitzer [2020] M. Seitzer.  pytorch-fid: FID Score for PyTorch.  https://github.com/mseitzer/pytorch-fid, August 2020.  Version 0.2.1.
  * Shao et al. [2018] H. Shao, A. Kumar, and P. Thomas Fletcher.  The Riemannian geometry of deep generative models.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 315–323, 2018.
  * Sohl-Dickstein et al. [2015] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pages 2256–2265. PMLR, 2015.
  * Song and Ermon [2019] Y. Song and S. Ermon.  Generative modeling by estimating gradients of the data distribution.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Song et al. [2021] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole.  Score-based generative modeling through stochastic differential equations.  In _Proceedings of the International Conference on Learning Representations (ICLR 2021)_ , 2021.
  * Taghvaei and Mehta [2019] A. Taghvaei and P. Mehta.  Accelerated flow for probability distributions.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 6076–6085. PMLR, 2019.
  * Teshima et al. [2020] T. Teshima, I. Ishikawa, K. Tojo, K. Oono, M. Ikeda, and M. Sugiyama.  Coupling-based invertible neural networks are universal diffeomorphism approximators.  _arXiv preprint arXiv:2006.11469_ , 2020.
  * Vahdat and Kautz [2020] A. Vahdat and J. Kautz.  NVAE: A deep hierarchical variational autoencoder.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Van Den Berg et al. [2018] R. Van Den Berg, L. Hasenclever, J. M. Tomczak, and M. Welling.  Sylvester normalizing flows for variational inference.  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence_ , pages 393–402. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
  * Vincent [2011] P. Vincent.  A connection between score matching and denoising autoencoders.  _Neural Computation_ , 23(7):1661–1674, 2011\.
  * Vincent et al. [2008] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol.  Extracting and composing robust features with denoising autoencoders.  In _Proceedings of the International Conference on Machine Learning_ , pages 1096–1103, 2008.
  * Welling and Teh [2011] M. Welling and Y. W. Teh.  Bayesian learning via stochastic gradient Langevin dynamics.  In _Proceedings of the 28th International Conference on Machine Learning (ICML 2011)_ , pages 681–688, Bellevue, Washington USA, 2011. IMLS.
  * Xia et al. [2017a] Y. Xia, J. Bian, T. Qin, N. Yu, and T.-Y. Liu.  Dual inference for machine learning.  In _Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI-17)_ , pages 3112–3118, 2017a.
  * Xia et al. [2017b] Y. Xia, T. Qin, W. Chen, J. Bian, N. Yu, and T.-Y. Liu.  Dual supervised learning.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 3789–3798. JMLR.org, 2017b.
  * Xiao et al. [2020] M. Xiao, S. Zheng, C. Liu, Y. Wang, D. He, G. Ke, J. Bian, Z. Lin, and T.-Y. Liu.  Invertible image rescaling.  In _European Conference on Computer Vision_ , pages 126–144. Springer, 2020.
  * Xiao et al. [2021] Z. Xiao, K. Kreis, J. Kautz, and A. Vahdat.  VAEBM: A symbiosis between variational autoencoders and energy-based models.  In _International Conference on Learning Representations (ICLR 2021)_ , 2021.
  * Yi et al. [2017] Z. Yi, H. Zhang, P. Tan, and M. Gong.  DualGAN: Unsupervised dual learning for image-to-image translation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2849–2857, 2017.
  * Zhang and Hyvärinen [2009] K. Zhang and A. Hyvärinen.  On the identifiability of the post-nonlinear causal model.  In _Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI 2009)_ , pages 647–655. AUAI Press, 2009.
  * Zhu et al. [2017] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2223–2232, 2017.

## Supplementary Materials

##  Appendix A Background in Measure Theory

###  A.1 The Integral

The _integral_ of a nonnegative measurable function $f$ on a measure space
$(\Omega,\mathscr{F},\mu)$ is defined as:

$\displaystyle\int
f\,\mathrm{d}\mu:=\sup\sum_{i}\mu(\mathcal{W}^{(i)})\inf_{\omega\in\mathcal{W}^{(i)}}f(\omega),$
(11)

where the supremum is taken over all finite decompositions
$\\{\mathcal{W}^{(i)}\\}$ of $\Omega$ into $\mathscr{F}$-sets [13, p.211]. For
a general measurable function, its integral is defined as the subtraction from
the integral of its positive part $f^{+}(\omega):=\max\\{0,f(\omega)\\}$ with
the integral of its negative part $f^{-}(\omega):=\max\\{0,-f(\omega)\\}$. A
measurable function is said to be _$\mu$ -integrable_ [13, p.212] if both
integrals of its positive and negative parts are finite.

(i) This is a general definition of integral. When $\Omega$ is an Euclidean
space and $\mu$ is the Lebesgue measure on it, this integral reduces to the
Lebesgue integral (which in turn coincides with the Riemann integral when the
latter exists). When $\Omega$ is a discrete set (_i.e._ , a finite or
countable set) and $\mu$ is the counting measure, this integral reduces to
summation.

(ii) The integral satisfies common properties like linearity and monotonicity
[13, Thm. 16.1], continuity under boundedness [13, Thm. 16.4, Thm. 16.5],
_etc._ For a nonnegative function $f$, $\int f\,\mathrm{d}\mu=0$ if and only
if $f=0$, $\mu$-a.e. [13, Thm. 15.2].

(iii) The integral over a set $\mathcal{W}\in\mathscr{F}$ is defined as
$\int_{\mathcal{W}}f\,\mathrm{d}\mu:=\int\mathbb{I}_{\mathcal{W}}f\,\mathrm{d}\mu$
[13, p.226], where $\mathbb{I}_{\mathcal{W}}$ is the indicator function.
(1) We thus sometimes also write $\int_{\Omega}f\,\mathrm{d}\mu$ for $\int
f\,\mathrm{d}\mu$ to highlight the integral area. By this definition,
$\int_{\mathcal{W}}f\,\mathrm{d}\mu=0$ if $\mu(\mathcal{W})=0$ [13, p.226].
(2) For two measurable functions $f$ and $g$, if $f=g$, $\mu$-a.e., then
$\int_{\mathcal{W}}f\,\mathrm{d}\mu=\int_{\mathcal{W}}g\,\mathrm{d}\mu$ for
any $\mathcal{W}\in\mathscr{F}$ [13, Thm. 15.2]. The inverse also holds if $f$
and $g$ are nonnegative and $\mu$ is sigma-finite, or $f$ and $g$ are
integrable [13, Thm. 16.10(i,ii)]777 13, Thm. 16.10(iii): $f=g$, $\mu$-a.e.,
if $\int_{\mathcal{W}}f\,\mathrm{d}\mu=\int_{\mathcal{W}}g\,\mathrm{d}\mu$ for
any $\mathcal{W}$ from a pi-system $\Pi$ that generates $\mathscr{F}$, and
$\Omega$ is a finite or countable union of $\Pi$-sets. .
(3) If $f$ is a nonnegative measurable function, then
$\nu(\mathcal{W}):=\int_{\mathcal{W}}f\,\mathrm{d}\mu$,
$\forall\mathcal{W}\in\mathscr{F}$, is a measure on $(\Omega,\mathscr{F})$
[13, p.227]888 Its countable additivity is guaranteed by Billingsley [13, Thm.
16.9]. . Such a measure $\nu$ is finite, if and only if $f$ is
$\mu$-integrable.

###  A.2 Absolute Continuity and Radon-Nikodym Derivative

For two measures $\mu$ and $\nu$ on the same measurable space
$(\Omega,\mathscr{F})$, $\nu$ is said to be _absolutely continuous_ w.r.t
$\mu$, denoted as $\nu\ll\mu$, if $\mu(\mathcal{W})=0$ indicates
$\nu(\mathcal{W})=0$ for $\mathcal{W}\in\mathscr{F}$ [13, p.448]. If $\mu$ and
$\nu$ are sigma-finite and $\nu\ll\mu$, the _Radon-Nikodym theorem_ [13, Thm.
32.2] asserts that there exists a $\mu$-unique nonnegative function $f$ on
$\Omega$, such that
$\nu(\mathcal{W})=\int_{\mathcal{W}}f(\omega)\mu(\mathrm{d}\omega)$ for any
$\mathcal{W}\in\mathscr{F}$. Such a function $f$ is called the _Radon-Nikodym
(R-N) derivative_ of $\nu$ w.r.t $\mu$, and is also denoted as
$\frac{\mathrm{d}\nu}{\mathrm{d}\mu}$. It represents the density function of
$\nu$ w.r.t base measure $\mu$.

(i) Since the general definition of integral includes summation in the
discrete case, this density function also includes the probability mass
function in the discrete case.

(ii) The Dirac measure
$\delta_{\omega_{0}}(\mathcal{W}):=\mathbb{I}_{\mathcal{W}}(\omega_{0})$
($\mathbb{I}$ is the indicator function) at a single point
$\omega_{0}\in\Omega$ is not absolutely continuous on Euclidean spaces w.r.t
the Lebesgue measure, which assigns measure 0 to the set $\\{\omega_{0}\\}$.
To be strict, the Dirac delta function is not a proper density function, since
its integrals covering $\omega_{0}$ involve the indefinite $\infty\cdot 0$ on
the component $\\{\omega_{0}\\}$ of the integral domain. Its characteristic
that such integrals equal to one, is a standalone structure from being a
function. So it is better treated as a measure of functional.

###  A.3 Product Measure Space

Two measure spaces $(\mathbb{X},\mathscr{X},\xi)$ and
$(\mathbb{Z},\mathscr{Z},\zeta)$ induce a _product measure space_
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z},\xi\\!\otimes\\!\zeta)$.

(i) The _product sigma-field_
$\mathscr{X}\\!\otimes\\!\mathscr{Z}:=\sigma(\mathscr{X}\\!\times\\!\mathscr{Z})$
is the smallest sigma-field on $\mathbb{X}\\!\times\\!\mathbb{Z}$ containing
$\mathscr{X}\\!\times\\!\mathscr{Z}$ (29, Thm. 22; 70, Def. 7.1; equivalently,
47, Remark 14.10; 47, Def. 14.4). Note that the Cartesian product
$\mathscr{X}\\!\times\\!\mathscr{Z}$, representing the set of _measurable
rectangles_ , is only a semiring (thus also a pi-system). So we need to extend
for a sigma-field. For any
$\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$, its _slice_ (or section)
at $z\in\mathbb{Z}$, defined by:

$\displaystyle\mathcal{W}_{z}:=\\{x\mid(x,z)\in\mathcal{W}\\},$ (12)

lies in $\mathscr{X}$, and similarly $\mathcal{W}_{x}\in\mathscr{Z}$ [13, Thm.
18.1(i)]. We define the _projection_ (or restriction) of $\mathcal{W}$ onto
$\mathbb{Z}$, as $\mathcal{W}^{\mathbb{Z}}:=\\{z\mid\exists
x\in\mathbb{X}\text{~{}s.t.~{}}(x,z)\in\mathcal{W}\\}$. By definition, for any
$z\in\mathbb{Z}\setminus\mathcal{W}^{\mathbb{Z}}$,
$\mathcal{W}_{z}=\emptyset$.

(ii) The _product measure_ $\xi\\!\otimes\\!\zeta$ is characterized by
$(\xi\\!\otimes\\!\zeta)(\mathcal{X}\\!\times\\!\mathcal{Z})=\xi(\mathcal{X})\zeta(\mathcal{Z})$
for measurable rectangles
$\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}$.
Some common conclusions require $\xi$ and $\zeta$ to be sigma-finite on
$\mathscr{X}$ and $\mathscr{Z}$, respectively.
(1) In the characterization
$(\xi\\!\otimes\\!\zeta)(\mathcal{X}\\!\times\\!\mathcal{Z})=\xi(\mathcal{X})\zeta(\mathcal{Z})$,
if the indefinite $0\cdot\infty$ is met, it is zero. To see this, consider two
sets $\mathcal{X}$ and $\mathcal{Z}$ that satisfy $\xi(\mathcal{X})=0$ and
$\zeta(\mathcal{Z})=0$. Since $\zeta$ is sigma-finite, there are finite or
countable disjoint $\mathscr{Z}$-sets
$\mathcal{Z}^{(1)},\mathcal{Z}^{(2)},\cdots$ such that
$\zeta(\mathcal{Z}^{(i)})<\infty$ for any $i\geqslant 1$ and
$\bigcup_{i=1}^{\infty}\mathcal{Z}^{(i)}=\mathbb{Z}$. Redefining
$\mathcal{Z}^{(i)}$ as $\mathcal{Z}^{(i)}\cap\mathcal{Z}$, we have
$\bigcup_{i=1}^{\infty}\mathcal{Z}^{(i)}=\mathcal{Z}$ while still
$\zeta(\mathcal{Z}^{(i)})<\infty$. So
$(\xi\\!\otimes\\!\zeta)(\mathcal{X}\\!\times\\!\mathbb{Z})=(\xi\\!\otimes\\!\zeta)(\mathcal{X}\\!\times\\!\bigcup_{i=1}^{\infty}\mathcal{Z}^{(i)})=(\xi\\!\otimes\\!\zeta)(\bigcup_{i=1}^{\infty}\mathcal{X}\\!\times\\!\mathcal{Z}^{(i)})$.
Recalling that a measure is countably additive by definition, this is
$=\sum_{i=1}^{\infty}(\xi\\!\otimes\\!\zeta)(\mathcal{X}\\!\times\\!\mathcal{Z}^{(i)})=\sum_{i=1}^{\infty}\xi(\mathcal{X})\zeta(\mathcal{Z}^{(i)})=0$.
(2) In this case, such a $\xi\\!\otimes\\!\zeta$ is sigma-finite on
$\mathscr{X}\\!\times\\!\mathscr{Z}$, and the characterization on the pi-
system $\mathscr{X}\\!\times\\!\mathscr{Z}$ determines a unique sigma-finite
measure on
$\sigma(\mathscr{X}\\!\times\\!\mathscr{Z})=\mathscr{X}\\!\otimes\\!\mathscr{Z}$
[13, Thm. 10.3]. See also Galambos [29, Thm. 22]; Klenke [47, Thm. 14.14];
Rinaldo [70, Thm. 7.9]. Moreover, we have [13, Thm. 18.2]:

$\displaystyle(\xi\\!\otimes\\!\zeta)(\mathcal{W})=\int_{\mathbb{Z}}\xi(\mathcal{W}_{z})\zeta(\mathrm{d}z)=\int_{\mathbb{X}}\zeta(\mathcal{W}_{x})\xi(\mathrm{d}x),\;\forall\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}.$
(13)

Since for any $z\in\mathbb{Z}\setminus\mathcal{W}^{\mathbb{Z}}$,
$\mathcal{W}_{z}=\emptyset$ (see (i)) thus $\xi(\mathcal{W}_{z})=0$, we also
have (by leveraging the additivity of integrals over a countable partition
[13, Thm. 16.9] and that an a.e. zero function gives a zero integral [13, Thm.
15.2(i)]):

$\displaystyle(\xi\\!\otimes\\!\zeta)(\mathcal{W})=\int_{\mathcal{W}^{\mathbb{Z}}}\xi(\mathcal{W}_{z})\zeta(\mathrm{d}z)=\int_{\mathcal{W}^{\mathbb{X}}}\zeta(\mathcal{W}_{x})\xi(\mathrm{d}x),\;\forall\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}.$
(14)

(iii) For a function $f$ on $\mathbb{X}\\!\times\\!\mathbb{Z}$, if it is
$\mathscr{X}\\!\otimes\\!\mathscr{Z}$-measurable, then $f(x,\cdot)$ is
$\mathscr{Z}$-measurable for any $x\in\mathbb{X}$, and $f(\cdot,z)$ is
$\mathscr{X}$-measurable for any $z\in\mathbb{Z}$ [13, Thm. 18.1(ii)]. When
$f$ is $\xi\\!\otimes\\!\zeta$-integrable, _Fubini’s theorem_ [13, Thm. 18.3]
asserts its integral on $\mathbb{X}\\!\times\\!\mathbb{Z}$ can be computed
iteratedly in either order:

$\displaystyle\int_{\mathbb{X}\\!\times\\!\mathbb{Z}}f(x,z)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)=\int_{\mathbb{Z}}\left(\int_{\mathbb{X}}f(x,z)\xi(\mathrm{d}x)\right)\zeta(\mathrm{d}z)=\int_{\mathbb{X}}\left(\int_{\mathbb{Z}}f(x,z)\zeta(\mathrm{d}z)\right)\xi(\mathrm{d}x).$
(15)

For any $\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$, the same
equalities hold for function $\mathbb{I}_{\mathcal{W}}f$. For the first
iterated integral, we have
$\int_{\mathbb{X}}\mathbb{I}_{\mathcal{W}}(x,z)f(x,z)\xi(\mathrm{d}x)=\int_{\mathbb{X}}\mathbb{I}_{\mathcal{W}_{z}}(x)f(x,z)\xi(\mathrm{d}x)=\int_{\mathcal{W}_{z}}f(x,z)\xi(\mathrm{d}x)$,
and on the region $\mathbb{Z}\setminus\mathcal{W}^{\mathbb{Z}}$, the integral
$\int_{\mathcal{W}_{z}}f(x,z)\xi(\mathrm{d}x)=0$ [13, p.226] since
$\mathcal{W}_{z}=\emptyset$ on that region (see (i)). So we have a more
general form of Fubini’s theorem:

$\displaystyle\int_{\mathcal{W}}f(x,z)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)=\int_{\mathcal{W}^{\mathbb{Z}}}\left(\int_{\mathcal{W}_{z}}f(x,z)\xi(\mathrm{d}x)\right)\zeta(\mathrm{d}z)=\int_{\mathcal{W}^{\mathbb{X}}}\left(\int_{\mathcal{W}_{x}}f(x,z)\zeta(\mathrm{d}z)\right)\xi(\mathrm{d}x).$
(16)

(iv) For a measure $\pi$ on the product measurable space
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z})$,
define its marginal distributions:
$\pi^{\mathbb{X}}(\mathcal{X}):=\pi(\mathcal{X}\\!\times\\!\mathbb{Z}),\forall\mathcal{X}\in\mathscr{X}$,
and
$\pi^{\mathbb{Z}}(\mathcal{Z}):=\pi(\mathbb{X}\\!\times\\!\mathcal{Z}),\forall\mathcal{Z}\in\mathscr{Z}$.

###  A.4 Conditional Distributions

In the most general case, a distribution (probability measure) $\pi$ on a
measurable space $(\Omega,\mathscr{F})$ gives a _conditional distribution_
(conditional probability) $\pi(\mathcal{W}|\omega)$ for
$\mathcal{W}\in\mathscr{F}$ w.r.t a sub-sigma-field
$\mathscr{G}\subseteq\mathscr{F}$.

(i) For any $\mathcal{W}\in\mathscr{F}$, the function
$\mathscr{G}\to\mathbb{R}^{\geqslant
0},\mathcal{G}\mapsto\pi(\mathcal{G}\cap\mathcal{W})$ gives a measure on
$\mathscr{G}$. It is absolutely continuous w.r.t
$\pi^{\mathscr{G}}:\mathscr{G}\to\mathbb{R}^{\geqslant
0},\mathcal{G}\mapsto\pi(\mathcal{G})$, the projection of $\pi$ onto
$\mathscr{G}$, due to the monotonicity (or (sub-)additivity) of measures. So
the R-N derivative on $\mathscr{G}$ exists, which defines the conditional
distribution [13, p.457]:

$\displaystyle\pi(\mathcal{W}|\omega):=\frac{\mathrm{d}\pi(\cdot\cap\mathcal{W})}{\mathrm{d}\pi^{\mathscr{G}}(\cdot)}(\omega),$
(17)

where $\omega\in\Omega$. Note that as defined as an R-N derivative, the
conditional distribution is only $\pi^{\mathscr{G}}$-unique.

(ii) As a function of $\omega$, $\pi(\mathcal{W}|\omega)$ is
$\mathscr{G}$-measurable and $\pi$-integrable, and satisfies [13, p.457, Thm.
33.1]:

$\displaystyle\int_{\mathcal{G}}\pi(\mathcal{W}|\omega)\pi^{\mathscr{G}}(\mathrm{d}\omega)=\pi(\mathcal{G}\cap\mathcal{W}),\forall\mathcal{G}\in\mathscr{G}.$
(18)

This could serve as an alternative definition of conditional probability.

(iii) For $\pi^{\mathscr{G}}$-a.e. $\omega$, $\pi(\cdot|\omega)$ is a
distribution (probability measure) on $(\Omega,\mathscr{F})$ [13, Thm. 33.2].

(iv) Conditional distributions on a product measurable space
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z})$.
Consider the sub-sigma-field
$\mathscr{G}:=\\{\mathbb{X}\\}\\!\times\\!\mathscr{Z}$. By construction, any
$\mathcal{G}\in\mathscr{G}$ can be formed by
$\mathcal{G}=\mathbb{X}\\!\times\\!\mathcal{Z}$ for some
$\mathcal{Z}\in\mathscr{Z}$. So
$\pi^{\mathscr{G}}(\mathcal{G}):=\pi(\mathcal{G})=\pi(\mathbb{X}\\!\times\\!\mathcal{Z})=:\pi^{\mathbb{Z}}(\mathcal{Z})$,
and Eq. (18) becomes
$\pi\big{(}(\mathbb{X}\\!\times\\!\mathcal{Z})\cap\mathcal{W}\big{)}=\int_{\mathbb{X}\\!\times\\!\mathcal{Z}}\pi(\mathcal{W}|x,z)\pi^{\mathscr{G}}(\mathrm{d}x\mathrm{d}z)=\int_{\mathbb{X}\\!\times\\!\mathcal{Z}}\pi(\mathcal{W}|x,z)\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathcal{Z}}\pi(\mathcal{W}|x,z)\pi^{\mathbb{Z}}(\mathrm{d}z)$.
This indicates that the conditional probability $\pi(\mathcal{W}|x,z)$ in this
case is constant w.r.t $x$. We hence denote it as $\pi(\mathcal{W}|z)$.

Consider $\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$ in the form
$\mathcal{W}=\mathcal{X}\\!\times\\!\mathbb{Z}$ for some
$\mathcal{X}\in\mathscr{X}$. For any
$\mathcal{G}=\mathbb{X}\\!\times\\!\mathcal{Z}\in\mathscr{G}$, we have from
Eq. (18) that
$\int_{\mathcal{G}}\pi(\mathcal{W}|z)\pi^{\mathscr{G}}(\mathrm{d}x\mathrm{d}z)=\pi(\mathcal{G}\cap\mathcal{W})=\pi(\mathcal{X}\\!\times\\!\mathcal{Z})$.
From the above deduction, the l.h.s is
$\int_{\mathbb{X}\\!\times\\!\mathcal{Z}}\pi(\mathcal{W}|z)\pi^{\mathscr{G}}(\mathrm{d}x\mathrm{d}z)=\int_{\mathcal{Z}}\pi(\mathcal{X}\\!\times\\!\mathbb{Z}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)$.
Defining $\pi(\mathcal{X}|z)$ as $\pi(\mathcal{X}\\!\times\\!\mathbb{Z}|z)$
for any $\mathcal{X}\in\mathscr{X}$, we have:

$\displaystyle\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{Z}}\pi(\mathcal{X}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathcal{X}}\pi(\mathcal{Z}|x)\pi^{\mathbb{X}}(\mathrm{d}x),\;\forall\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}.$
(19)

This is the conditional distribution in the usual sense. Note again that as
defined as R-N derivatives, the conditional distributions $\pi(\mathcal{X}|z)$
and $\pi(\mathcal{Z}|x)$ are only $\pi^{\mathbb{Z}}$-unique and
$\pi^{\mathbb{X}}$-unique, respectively.

For any $\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$, define
${\tilde{\pi}}(\mathcal{W}):=\int_{\mathbb{Z}}\pi(\mathcal{W}_{z}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)$.
It is easy to verify that ${\tilde{\pi}}$ is a distribution (probability
measure; thus finite and sigma-finite) on
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z})$ [13,
p.227], since $\pi(\mathcal{X}|z)$ is a distribution (and thus nonnegative) on
$(\mathbb{X},\mathscr{X})$ for $\pi^{\mathbb{Z}}$-a.e. $z$ [13, Thm. 33.2].
For any
$\mathcal{W}=\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}$,
${\tilde{\pi}}(\mathcal{W})=\int_{\mathbb{Z}}\pi(\mathcal{X}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathcal{Z}}\pi(\mathcal{X}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=\pi(\mathcal{X}\\!\times\\!\mathcal{Z})$
due to Eq. (19). So ${\tilde{\pi}}$ and $\pi$ agree on the pi-system
$\mathscr{X}\\!\times\\!\mathscr{Z}$, which indicates that they agree on
$\sigma(\mathscr{X}\\!\times\\!\mathscr{Z})=\mathscr{X}\\!\otimes\\!\mathscr{Z}$
due to Billingsley [13, Thm. 10.3, Thm. 3.3]. This means that (see the
argument in (ii) (2) in Supplement A.3 for the second line of the equation):

$\displaystyle\pi(\mathcal{W})={}$
$\displaystyle\int_{\mathbb{Z}}\pi(\mathcal{W}_{z}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathbb{X}}\pi(\mathcal{W}_{x}|x)\pi^{\mathbb{X}}(\mathrm{d}x)$
(20) $\displaystyle={}$
$\displaystyle\int_{\mathcal{W}^{\mathbb{Z}}}\pi(\mathcal{W}_{z}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathcal{W}^{\mathbb{X}}}\pi(\mathcal{W}_{x}|x)\pi^{\mathbb{X}}(\mathrm{d}x),\;\forall\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}.$
(21)

Finally, we formalize some definitions in the main text below.

######  Definition A.1.

Consider a general measure space $(\Omega,\mathscr{F},\mu)$. (i) We say that
two measurable sets $\mathcal{S},{\tilde{\mathcal{S}}}\in\mathscr{F}$ are
_$\mu$ -a.s. the same_, denoted as
“$\mathcal{S}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}$”, if
$\mu(\mathcal{S}\triangle{\tilde{\mathcal{S}}})=0$, where “$\triangle$”
denotes the symmetric difference between two sets. (ii) We say that
$\mathcal{S}$ is a _$\mu$ -a.s. subset_ of $\mathcal{W}$, denoted as
“$\mathcal{S}\subseteq^{\text{$\mu$}}\mathcal{W}$”, if
$\mu(\mathcal{S}\setminus\mathcal{W})=0$.

##  Appendix B Lemmas

###  B.1 Lemmas for General Probability

######  Lemma B.1.

Let $\mathcal{O}$ be a measure-zero set, $\mu(\mathcal{O})=0$, on a measure
space $(\Omega,\mathscr{F},\mu)$. Then for any measurable set $\mathcal{W}$,
we have
$\mu(\mathcal{O}\setminus\mathcal{W})=\mu(\mathcal{W}\cap\mathcal{O})=0$, and
$\mu(\mathcal{W}\cup\mathcal{O})=\mu(\mathcal{W}\setminus\mathcal{O})=\mu(\mathcal{W})$.

  * Proof.

Due to the monotonicity of a measure [13, Thm. 16.1], we have
$\mu(\mathcal{O}\setminus\mathcal{W})\leqslant\mu(\mathcal{O})=0$ and
$\mu(\mathcal{W}\cap\mathcal{O})\leqslant\mu(\mathcal{O})=0$, so we get
$\mu(\mathcal{O}\setminus\mathcal{W})=\mu(\mathcal{W}\cap\mathcal{O})=0$.
Since
$\mu(\mathcal{W}\cup\mathcal{O})=\mu(\mathcal{W}\cup(\mathcal{O}\setminus\mathcal{W}))$
and the two sets are disjoint, it equals to
$\mu(\mathcal{W})+\mu(\mathcal{O}\setminus\mathcal{W})$, which is
$\mu(\mathcal{W})$ by the above conclusion. So we get
$\mu(\mathcal{W}\cup\mathcal{O})=\mu(\mathcal{W})$. When applying this
conclusion to $\mathcal{W}\setminus\mathcal{O}$, we have
$\mu((\mathcal{W}\setminus\mathcal{O})\cup\mathcal{O})=\mu(\mathcal{W}\setminus\mathcal{O})$,
while the l.h.s is $\mu(\mathcal{W}\cup\mathcal{O})$ which is
$\mu(\mathcal{W})$ by the same conclusion. So we get
$\mu(\mathcal{W}\setminus\mathcal{O})=\mu(\mathcal{W})$. ∎

######  Lemma B.2.

Let $\pi$ be an absolutely continuous distribution (probability measure) on a
measure space $(\Omega,\mathscr{F},\mu)$ with a density function $f$, and let
$\mathcal{S}\in\mathscr{F}$ be a measurable set. Then $\pi(\mathcal{S})=1$ if
and only if
$\pi(\mathcal{W})=\int_{\mathcal{W}\cap\mathcal{S}}f\,\mathrm{d}\mu,\forall\mathcal{W}\in\mathscr{F}$.

  * Proof.

“Only if”: Since $\mathcal{S}\subseteq\Omega$, we have
$\pi(\Omega\setminus\mathcal{S})=\pi(\Omega)-\pi(\mathcal{S})=0$. For any
$\mathcal{W}\in\mathscr{F}$, we have
$\pi(\mathcal{W})=\pi(\mathcal{W}\cap\mathcal{S})+\pi(\mathcal{W}\cap(\Omega\setminus\mathcal{S}))$,
while
$0\leqslant\pi(\mathcal{W}\cap(\Omega\setminus\mathcal{S}))\leqslant\pi(\Omega\setminus\mathcal{S})=0$.
So we have
$\pi(\mathcal{W})=\pi(\mathcal{W}\cap\mathcal{S})=\int_{\mathcal{W}\cap\mathcal{S}}f\,\mathrm{d}\mu$.

“If”:
$1=\pi(\Omega)=\int_{\Omega\cap\mathcal{S}}f\,\mathrm{d}\mu=\int_{\mathcal{S}}f\,\mathrm{d}\mu=\int_{\mathcal{S}\cap\mathcal{S}}f\,\mathrm{d}\mu=\pi(\mathcal{S})$.
∎

######  Lemma B.3.

Let $\mathcal{S}$ and ${\tilde{\mathcal{S}}}$ be two measurable sets on a
measure space $(\Omega,\mathscr{F},\mu)$ such that
$\mathcal{S}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}$. Then
$\mu(\mathcal{S}\setminus{\tilde{\mathcal{S}}})=\mu({\tilde{\mathcal{S}}}\setminus\mathcal{S})=0$,
and
$\mu(\mathcal{S})=\mu({\tilde{\mathcal{S}}})=\mu(\mathcal{S}\cup{\tilde{\mathcal{S}}})=\mu(\mathcal{S}\cap{\tilde{\mathcal{S}}})$.

  * Proof.

Let $\mathcal{D}^{+}:={\tilde{\mathcal{S}}}\setminus\mathcal{S}$ and
$\mathcal{D}^{-}:=\mathcal{S}\setminus{\tilde{\mathcal{S}}}$. By construction,
we have $\mathcal{D}^{+}\cap\mathcal{S}=\emptyset$ and
$\mathcal{D}^{-}\subseteq\mathcal{S}$, so we also have
$\mathcal{D}^{+}\cap\mathcal{D}^{-}=\emptyset$, and
${\tilde{\mathcal{S}}}=(\mathcal{S}\setminus\mathcal{D}^{-})\cup\mathcal{D}^{+}=(\mathcal{S}\cup\mathcal{D}^{+})\setminus\mathcal{D}^{-}$.
By definition,
$\mathcal{S}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}$ indicates
$0=\mu(\mathcal{S}\triangle{\tilde{\mathcal{S}}})=\mu(\mathcal{D}^{+}\cup\mathcal{D}^{-})=\mu(\mathcal{D}^{+})+\mu(\mathcal{D}^{-})$,
so we have both $\mu(\mathcal{D}^{+})=0$ and $\mu(\mathcal{D}^{-})=0$.
Subsequently,
$\mu({\tilde{\mathcal{S}}})=\mu\big{(}(\mathcal{S}\setminus\mathcal{D}^{-})\cup\mathcal{D}^{+}\big{)}=\mu(\mathcal{S}\setminus\mathcal{D}^{-})+\mu(\mathcal{D}^{+})=\mu(\mathcal{S}\setminus\mathcal{D}^{-})=\mu(\mathcal{S})-\mu(\mathcal{D}^{-}\cap\mathcal{S})=\mu(\mathcal{S})-\mu(\mathcal{D}^{-})=\mu(\mathcal{S})$,
and
$\mu(\mathcal{S}\cup{\tilde{\mathcal{S}}})=\mu(\mathcal{S}\cup\mathcal{D}^{+})=\mu(\mathcal{S})+\mu(\mathcal{D}^{+})=\mu(\mathcal{S})$.
Noting also that
$\mathcal{S}\cup{\tilde{\mathcal{S}}}=(\mathcal{S}\cap{\tilde{\mathcal{S}}})\cup(\mathcal{S}\triangle{\tilde{\mathcal{S}}})$
and that this is a disjoint union, we have
$\mu(\mathcal{S}\cup{\tilde{\mathcal{S}}})=\mu(\mathcal{S}\cap{\tilde{\mathcal{S}}})+\mu(\mathcal{S}\triangle{\tilde{\mathcal{S}}})=\mu(\mathcal{S}\cap{\tilde{\mathcal{S}}})$.
∎

######  Lemma B.4.

On a measure space $(\Omega,\mathscr{F},\mu)$,
“$\cdot\overset{\text{\tiny$\mu$}}{=}\cdot$” is an equivalence relation.

  * Proof.

Symmetry and reflexivity are obvious. For transitivity, let $\mathcal{A}$,
$\mathcal{B}$ and $\mathcal{C}$ be three measurable sets such that
$\mathcal{A}\overset{\text{\tiny$\mu$}}{=}\mathcal{B}$ and
$\mathcal{B}\overset{\text{\tiny$\mu$}}{=}\mathcal{C}$. Since
$\mathcal{A}\setminus\mathcal{C}=\big{(}(\mathcal{A}\setminus\mathcal{C})\cap\mathcal{B}\big{)}\cup\big{(}(\mathcal{A}\setminus\mathcal{C})\setminus\mathcal{B}\big{)}=\big{(}\mathcal{A}\cap(\mathcal{B}\setminus\mathcal{C})\big{)}\cup\big{(}(\mathcal{A}\setminus\mathcal{B})\setminus\mathcal{C}\big{)}\subseteq(\mathcal{B}\setminus\mathcal{C})\cup(\mathcal{A}\setminus\mathcal{B})$,
we have
$\mu(\mathcal{A}\setminus\mathcal{C})\leqslant\mu(\mathcal{B}\setminus\mathcal{C})+\mu(\mathcal{A}\setminus\mathcal{B})=0$
due to Lemma B.3. Similarly, $\mu(\mathcal{C}\setminus\mathcal{A})=0$. So
$\mu(\mathcal{A}\triangle\mathcal{C})=\mu(\mathcal{A}\setminus\mathcal{C})+\mu(\mathcal{C}\setminus\mathcal{A})=0$.
∎

######  Lemma B.5.

Let $\mathcal{S}$ and ${\tilde{\mathcal{S}}}$ be two measurable sets on a
measure space $(\Omega,\mathscr{F},\mu)$ such that
$\mathcal{S}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}$. Then for any
measurable set $\mathcal{W}$, we have
$\mathcal{S}\cup\mathcal{W}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}\cup\mathcal{W}$,
$\mathcal{S}\cap\mathcal{W}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}\cap\mathcal{W}$,
$\mathcal{S}\setminus\mathcal{W}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}\setminus\mathcal{W}$
and
$\mathcal{W}\setminus\mathcal{S}\overset{\text{\tiny$\mu$}}{=}\mathcal{W}\setminus{\tilde{\mathcal{S}}}$.

  * Proof.

Let $\mathcal{D}^{+}:={\tilde{\mathcal{S}}}\setminus\mathcal{S}$ and
$\mathcal{D}^{-}:=\mathcal{S}\setminus{\tilde{\mathcal{S}}}$. By Lemma B.3, we
have $\mu(\mathcal{D}^{+})=0$ and $\mu(\mathcal{D}^{-})=0$.

For any measurable set $\mathcal{W}$, we have
$({\tilde{\mathcal{S}}}\cup\mathcal{W})\setminus(\mathcal{S}\cup\mathcal{W})={\tilde{\mathcal{S}}}\setminus\mathcal{S}\setminus\mathcal{W}=\mathcal{D}^{+}\setminus\mathcal{W}$,
and similarly
$(\mathcal{S}\cup\mathcal{W})\setminus({\tilde{\mathcal{S}}}\cup\mathcal{W})=\mathcal{D}^{-}\setminus\mathcal{W}$.
So
$\mu\big{(}(\mathcal{S}\cup\mathcal{W})\triangle({\tilde{\mathcal{S}}}\cup\mathcal{W})\big{)}=\mu\big{(}\big{(}(\mathcal{S}\cup\mathcal{W})\setminus({\tilde{\mathcal{S}}}\cup\mathcal{W})\big{)}\cup\big{(}({\tilde{\mathcal{S}}}\cup\mathcal{W})\setminus(\mathcal{S}\cup\mathcal{W})\big{)}\big{)}=\mu\big{(}(\mathcal{D}^{-}\setminus\mathcal{W})\cup(\mathcal{D}^{+}\setminus\mathcal{W})\big{)}=\mu(\mathcal{D}^{-}\setminus\mathcal{W})+\mu(\mathcal{D}^{+}\setminus\mathcal{W})\leqslant\mu(\mathcal{D}^{-})+\mu(\mathcal{D}^{+})=0$,
that is
$\mathcal{S}\cup\mathcal{W}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}\cup\mathcal{W}$.

Since
$({\tilde{\mathcal{S}}}\cap\mathcal{W})\setminus(\mathcal{S}\cap\mathcal{W})=({\tilde{\mathcal{S}}}\setminus\mathcal{S})\cap\mathcal{W}=\mathcal{D}^{+}\cap\mathcal{W}$
and similarly
$(\mathcal{S}\cap\mathcal{W})\setminus({\tilde{\mathcal{S}}}\cap\mathcal{W})=\mathcal{D}^{-}\cap\mathcal{W}$,
we have
$\mu\big{(}(\mathcal{S}\cap\mathcal{W})\triangle({\tilde{\mathcal{S}}}\cap\mathcal{W})\big{)}=\mu\big{(}\big{(}(\mathcal{S}\cap\mathcal{W})\setminus({\tilde{\mathcal{S}}}\cap\mathcal{W})\big{)}\cup\big{(}({\tilde{\mathcal{S}}}\cap\mathcal{W})\setminus(\mathcal{S}\cap\mathcal{W})\big{)}\big{)}=\mu\big{(}(\mathcal{D}^{-}\cap\mathcal{W})\cup(\mathcal{D}^{+}\cap\mathcal{W})\big{)}=\mu(\mathcal{D}^{-}\cap\mathcal{W})+\mu(\mathcal{D}^{+}\cap\mathcal{W})\leqslant\mu(\mathcal{D}^{-})+\mu(\mathcal{D}^{+})=0$,
so
$\mathcal{S}\cap\mathcal{W}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}\cap\mathcal{W}$.

Since
$({\tilde{\mathcal{S}}}\setminus\mathcal{W})\setminus(\mathcal{S}\setminus\mathcal{W})={\tilde{\mathcal{S}}}\setminus\mathcal{W}\setminus\mathcal{S}={\tilde{\mathcal{S}}}\setminus\mathcal{S}\setminus\mathcal{W}=\mathcal{D}^{+}\setminus\mathcal{W}$
and similarly
$(\mathcal{S}\setminus\mathcal{W})\setminus({\tilde{\mathcal{S}}}\setminus\mathcal{W})=\mathcal{D}^{-}\setminus\mathcal{W}$,
we have
$\mu\big{(}(\mathcal{S}\setminus\mathcal{W})\triangle({\tilde{\mathcal{S}}}\setminus\mathcal{W})\big{)}=\mu\big{(}\big{(}(\mathcal{S}\setminus\mathcal{W})\setminus({\tilde{\mathcal{S}}}\setminus\mathcal{W})\big{)}\cup\big{(}({\tilde{\mathcal{S}}}\setminus\mathcal{W})\setminus(\mathcal{S}\setminus\mathcal{W})\big{)}\big{)}=\mu\big{(}(\mathcal{D}^{-}\setminus\mathcal{W})\cup(\mathcal{D}^{+}\setminus\mathcal{W})\big{)}=\mu(\mathcal{D}^{-}\setminus\mathcal{W})+\mu(\mathcal{D}^{+}\setminus\mathcal{W})\leqslant\mu(\mathcal{D}^{-})+\mu(\mathcal{D}^{+})=0$,
so
$\mathcal{S}\setminus\mathcal{W}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}\setminus\mathcal{W}$.

Since
$(\mathcal{W}\setminus{\tilde{\mathcal{S}}})\setminus(\mathcal{W}\setminus\mathcal{S})=\mathcal{W}\setminus(\mathcal{W}\setminus\mathcal{S})\setminus{\tilde{\mathcal{S}}}=(\mathcal{W}\cap\mathcal{S})\setminus{\tilde{\mathcal{S}}}=(\mathcal{S}\setminus{\tilde{\mathcal{S}}})\cap\mathcal{W}=\mathcal{D}^{-}\cap\mathcal{W}$
and similarly
$(\mathcal{W}\setminus\mathcal{S})\setminus(\mathcal{W}\setminus{\tilde{\mathcal{S}}})=\mathcal{D}^{+}\cap\mathcal{W}$,
we have
$\mu\big{(}(\mathcal{W}\setminus\mathcal{S})\triangle(\mathcal{W}\setminus{\tilde{\mathcal{S}}})\big{)}=\mu\big{(}\big{(}(\mathcal{W}\setminus\mathcal{S})\setminus(\mathcal{W}\setminus{\tilde{\mathcal{S}}})\big{)}\cup\big{(}(\mathcal{W}\setminus{\tilde{\mathcal{S}}})\setminus(\mathcal{W}\setminus\mathcal{S})\big{)}\big{)}=\mu\big{(}(\mathcal{D}^{+}\cap\mathcal{W})\cup(\mathcal{D}^{-}\cap\mathcal{W})\big{)}=\mu(\mathcal{D}^{+}\cap\mathcal{W})+\mu(\mathcal{D}^{-}\cap\mathcal{W})\leqslant\mu(\mathcal{D}^{+})+\mu(\mathcal{D}^{-})=0$,
so
$\mathcal{W}\setminus\mathcal{S}\overset{\text{\tiny$\mu$}}{=}\mathcal{W}\setminus{\tilde{\mathcal{S}}}$.
∎

######  Definition B.6.

We say that a set satisfying a certain condition is _$\mu$ -unique_, if for
any two such sets $\mathcal{S}$ and ${\tilde{\mathcal{S}}}$, it holds that
$\mathcal{S}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}$.

######  Lemma B.7.

Let $\pi$ be an absolutely continuous distribution (probability measure) on a
measure space $(\Omega,\mathscr{F},\mu)$ with a density function $f$. If a set
$\mathcal{S}\in\mathscr{F}$ satisfies $\pi(\mathcal{S})=1$ and that $f>0$,
$\mu$-a.e. on $\mathcal{S}$, then such an $\mathcal{S}$ is $\mu$-unique.

  * Proof.

Suppose we have two such sets $\mathcal{S}$ and ${\tilde{\mathcal{S}}}$. By
Lemma B.2, we know that for any $\mathcal{W}\in\mathscr{F}$,
$\pi(\mathcal{W})=\int_{\mathcal{W}\cap\mathcal{S}}f\,\mathrm{d}\mu=\int_{\mathcal{W}}\mathbb{I}_{\mathcal{S}}f\,\mathrm{d}\mu=\int_{\mathcal{W}}\mathbb{I}_{\tilde{\mathcal{S}}}f\,\mathrm{d}\mu$.
So by Billingsley [13, Thm. 16.10(ii)], we know that
$\mathbb{I}_{\mathcal{S}}f=\mathbb{I}_{\tilde{\mathcal{S}}}f$, $\mu$-a.e.

Since $f>0$, $\mu$-a.e. on $\mathcal{S}$, we know that
$\mathbb{I}_{\mathcal{S}}=\mathbb{I}_{\tilde{\mathcal{S}}}$, $\mu$-a.e. on
$\mathcal{S}$. This means that
$\mu\\{\omega\in\mathcal{S}\mid\mathbb{I}_{\mathcal{S}}\neq\mathbb{I}_{\tilde{\mathcal{S}}}\\}=\mu\\{\omega\in\mathcal{S}\mid\omega\notin{\tilde{\mathcal{S}}}\\}=\mu(\mathcal{S}\setminus{\tilde{\mathcal{S}}})=0$.
Symmetrically, since $f>0$, $\mu$-a.e. also on ${\tilde{\mathcal{S}}}$, we
know that $\mu({\tilde{\mathcal{S}}}\setminus\mathcal{S})=0$. So we have
$\mu(\mathcal{S}\triangle{\tilde{\mathcal{S}}})=\mu((\mathcal{S}\setminus{\tilde{\mathcal{S}}})\cup({\tilde{\mathcal{S}}}\setminus\mathcal{S}))=\mu(\mathcal{S}\setminus{\tilde{\mathcal{S}}})+\mu({\tilde{\mathcal{S}}}\setminus\mathcal{S})=0$,
which means that
$\mathcal{S}\overset{\text{\tiny$\mu$}}{=}{\tilde{\mathcal{S}}}$. ∎

The $\mu$-unique set $\mathcal{S}$ in the lemma serves as another form of the
_support_ of a distribution. The standard definition of the support requires a
topological structure and $\mathscr{F}$ is the corresponding Borel sigma-
field. If given absolute continuity $\pi\ll\mu$, this lemma enables the
generality that does not require a topological structure. The condition
$\pi(\mathcal{S})=1$ prevents $\mathcal{S}$ to be too small, while the
condition that $f>0$, $\mu$-a.e. on $\mathcal{S}$ prevents $\mathcal{S}$ to be
too large.

######  Definition B.8 (support of an absolutely continuous distribution
(without topology)).

Define the _support_ of an absolutely continuous distribution (probability
measure) $\pi$ on a measure space $(\Omega,\mathscr{F},\mu)$, as the
$\mu$-unique set $\mathcal{S}\in\mathscr{F}$ such that $\pi(\mathcal{S})=1$
and for any density function $f$ of $\pi$, it holds that $f>0$, $\mu$-a.e. on
$\mathcal{S}$.

###  B.2 Lemmas for Product Probability

In this subsection and the following, let
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z},\xi\\!\otimes\\!\zeta)$
be the product measure space by the two individual ones
$(\mathbb{X},\mathscr{X},\xi)$ and $(\mathbb{Z},\mathscr{Z},\zeta)$, where
$\xi$ and $\zeta$ are sigma-finite.

######  Lemma B.9.

For a measure $\pi$ on the product measure space
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z},\xi\\!\otimes\\!\zeta)$,
if $\pi\ll\xi\\!\otimes\\!\zeta$, then $\pi^{\mathbb{X}}\ll\xi$ and
$\pi^{\mathbb{Z}}\ll\zeta$.

  * Proof.

For any $\mathcal{X}\in\mathscr{X}$ such that $\xi(\mathcal{X})=0$, we have
$(\xi\\!\otimes\\!\zeta)(\mathcal{X}\\!\times\\!\mathbb{Z})=\xi(\mathcal{X})\zeta(\mathbb{Z})=0$,
where the last equality is verified in (ii) (1) in Supplement A.3 when
$\zeta(\mathbb{Z})=\infty$. Since $\pi\ll\xi\\!\otimes\\!\zeta$, this means
that $\pi(\mathcal{X}\\!\times\\!\mathbb{Z})=\pi^{\mathbb{X}}(\mathcal{X})=0$.
So $\pi^{\mathbb{X}}\ll\xi$. Similarly, $\pi^{\mathbb{Z}}\ll\zeta$. ∎

######  Lemma B.10.

For an assertion $t(x,z)$ on
$\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$, $t(x,z)$ holds
$\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{W}$, if and only if $t(x,z)$ holds
$\xi$-a.e. on $\mathcal{W}_{z}$, for $\zeta$-a.e. $z$ on
$\mathcal{W}^{\mathbb{Z}}$.

  * Proof.

By the definition of “$t(x,z)$ holds $\xi\\!\otimes\\!\zeta$-a.e. on
$\mathcal{W}$”, we have:

$\displaystyle(\xi\\!\otimes\\!\zeta)\\{(x,z)\in\mathcal{W}\mid\neg
t(x,z)\\}=0$ (Since $\xi$ and $\zeta$ are sigma-finite, from Eq. (14),)
$\displaystyle\Longleftrightarrow{}$
$\displaystyle\int_{\mathcal{W}^{\mathbb{Z}}}\xi\\{x\in\mathcal{W}_{z}\mid\neg
t(x,z)\\}\zeta(\mathrm{d}z)=0$ (22) (Since $\xi(\cdot)$ is nonnegative, from
Billingsley [13, Thm. 15.2],) $\displaystyle\Longleftrightarrow{}$
$\displaystyle\xi\\{x\in\mathcal{W}_{z}\mid\neg t(x,z)\\}=0,\text{for
$\zeta$-a.e. $z$ on $\mathcal{W}^{\mathbb{Z}}$},$

which is “$t(x,z)$ holds $\xi$-a.e. on $\mathcal{W}_{z}$, for $\zeta$-a.e. $z$
on $\mathcal{W}^{\mathbb{Z}}$”. ∎

######  Lemma B.11.

Let $\mathcal{X},{\tilde{\mathcal{X}}}\in\mathscr{X}$ such that
$\mathcal{X}\overset{\text{\tiny$\xi$}}{=}{\tilde{\mathcal{X}}}$. Then
$\mathcal{X}\\!\times\\!\mathbb{Z}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}{\tilde{\mathcal{X}}}\\!\times\\!\mathbb{Z}$.

  * Proof.

Since
$(\mathcal{X}\\!\times\\!\mathbb{Z})\triangle({\tilde{\mathcal{X}}}\\!\times\\!\mathbb{Z})=\big{(}(\mathcal{X}\\!\times\\!\mathbb{Z})\setminus({\tilde{\mathcal{X}}}\\!\times\\!\mathbb{Z})\big{)}\cup\big{(}({\tilde{\mathcal{X}}}\\!\times\\!\mathbb{Z})\setminus(\mathcal{X}\\!\times\\!\mathbb{Z})\big{)}=\big{(}(\mathcal{X}\setminus{\tilde{\mathcal{X}}})\cup({\tilde{\mathcal{X}}}\setminus\mathcal{X})\big{)}\\!\times\\!\mathbb{Z}$,
we can verify that
$(\xi\\!\otimes\\!\zeta)\big{(}(\mathcal{X}\\!\times\\!\mathbb{Z})\triangle({\tilde{\mathcal{X}}}\\!\times\\!\mathbb{Z})\big{)}=(\xi\\!\otimes\\!\zeta)\big{(}\big{(}(\mathcal{X}\setminus{\tilde{\mathcal{X}}})\cup({\tilde{\mathcal{X}}}\setminus\mathcal{X})\big{)}\\!\times\\!\mathbb{Z}\big{)}=\xi\big{(}(\mathcal{X}\setminus{\tilde{\mathcal{X}}})\cup({\tilde{\mathcal{X}}}\setminus\mathcal{X})\big{)}\zeta(\mathbb{Z})=\xi(\mathcal{X}\triangle{\tilde{\mathcal{X}}})\zeta(\mathbb{Z})=0$,
where the last equality is verified in (ii) (1) in Supplement A.3 when
$\zeta(\mathbb{Z})=\infty$. ∎

###  B.3 Lemmas for $\xi\\!\otimes\\!\zeta$-Complete Component

Echoing Def. 2.2, a set $\mathcal{S}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$ is
called a _$\xi\\!\otimes\\!\zeta$ -complete component_ of
$\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$, if

$\displaystyle\mathcal{S}^{\sharp}\cap\mathcal{W}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S},\text{where
}\mathcal{S}^{\sharp}:=\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\cup\mathbb{X}\\!\times\\!\mathcal{S}^{\mathbb{Z}}.$
(23)

This means that $\mathcal{S}$ is complete under _stretching_ and intersecting
with $\mathcal{W}$.

######  Lemma B.12.

Let $\mathcal{S}$ be a $\xi\\!\otimes\\!\zeta$-complete component of
$\mathcal{W}$. Then
$\mathcal{S}\subseteq^{\text{$\xi\\!\otimes\\!\zeta$}}\mathcal{W}$.

  * Proof.

By construction, we have $\mathcal{S}\subseteq\mathcal{S}^{\sharp}$ so
$\mathcal{S}\setminus\mathcal{W}=\mathcal{S}\setminus(\mathcal{S}\cap\mathcal{W})=\mathcal{S}\setminus(\mathcal{S}^{\sharp}\cap\mathcal{W})$.
Hence,
$(\xi\\!\otimes\\!\zeta)(\mathcal{S}\setminus\mathcal{W})=(\xi\\!\otimes\\!\zeta)\big{(}\mathcal{S}\setminus(\mathcal{S}^{\sharp}\cap\mathcal{W})\big{)}=0$
by definition Eq. (23) and Lemma B.3. ∎

Figure 8: Example B.13 showing that a $\xi\\!\otimes\\!\zeta$-complete
component of $\mathcal{W}$ may not have its projection be an a.s. subset of
that of $\mathcal{W}$.

a

######  Example B.13.

Note that when $\mathcal{S}$ is a $\xi\\!\otimes\\!\zeta$-complete component
of $\mathcal{W}$, it may not hold that
$\mathcal{S}^{\mathbb{X}}\subseteq^{\text{$\xi$}}\mathcal{W}^{\mathbb{X}}$ and
$\mathcal{S}^{\mathbb{Z}}\subseteq^{\text{$\zeta$}}\mathcal{W}^{\mathbb{Z}}$.
Fig. 8 shows an example, where $(\mathbb{X},\mathscr{X},\xi)$ and
$(\mathbb{Z},\mathscr{Z},\zeta)$ are the one dimensional Euclidean spaces with
line Borel sigma-field and line Lebesgue measure,
$(\mathbb{R},\mathscr{R},\lambda)$, and $\mathcal{W}:=[0,1]^{2}$ and
$\mathcal{S}:=[0,1]^{2}\cup([1,2]\\!\times\\!\\{\frac{1}{2}\\})$. We have
$\mathcal{S}^{\mathbb{X}}=[0,2]$ so
$\mathcal{S}^{\sharp}=([0,2]\\!\times\\!\mathbb{R})\cup(\mathbb{R}\\!\times\\![0,1])$
and $\mathcal{S}^{\sharp}\cap\mathcal{W}=\mathcal{W}$. Since
$\mathcal{S}\triangle\mathcal{W}=[1,2]\\!\times\\!\\{\frac{1}{2}\\}$ is a line
segment that has measure zero under the plane Lebesgue measure
$\xi\\!\otimes\\!\zeta=\lambda^{2}$, we have
$\mathcal{S}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{W}$ so
$\mathcal{S}$ is a $\xi\\!\otimes\\!\zeta$-complete component of
$\mathcal{W}$. But
$\xi(\mathcal{S}^{\mathbb{X}}\setminus\mathcal{W}^{\mathbb{X}})=\lambda([0,2]\setminus[0,1])=\lambda(1,2]=1$
is not zero, so
$\mathcal{S}^{\mathbb{X}}\subseteq^{\text{$\xi$}}\mathcal{W}^{\mathbb{X}}$
does not hold.

######  Lemma B.14.

Let $\mathcal{S}$ be a $\xi\\!\otimes\\!\zeta$-complete component of
$\mathcal{W}$, and ${\tilde{\mathcal{S}}}$ be a measurable set such that
${\tilde{\mathcal{S}}}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}$,
${\tilde{\mathcal{S}}}^{\mathbb{X}}\overset{\text{\tiny$\xi$}}{=}\mathcal{S}^{\mathbb{X}}$
and
${\tilde{\mathcal{S}}}^{\mathbb{Z}}\overset{\text{\tiny$\zeta$}}{=}\mathcal{S}^{\mathbb{Z}}$.
Then this ${\tilde{\mathcal{S}}}$ is also a $\xi\\!\otimes\\!\zeta$-complete
component of $\mathcal{W}$.

  * Proof.

By Lemma B.11, we know that
${\tilde{\mathcal{S}}}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}$,
$\mathbb{X}\\!\times\\!{\tilde{\mathcal{S}}}^{\mathbb{Z}}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathbb{X}\\!\times\\!\mathcal{S}^{\mathbb{Z}}$.
Repeatedly applying Lemma B.5, we have
${\tilde{\mathcal{S}}}^{\sharp}:={\tilde{\mathcal{S}}}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\cup\mathbb{X}\\!\times\\!{\tilde{\mathcal{S}}}^{\mathbb{Z}}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\cup\mathbb{X}\\!\times\\!{\tilde{\mathcal{S}}}^{\mathbb{Z}}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\cup\mathbb{X}\\!\times\\!\mathcal{S}^{\mathbb{Z}}=:\mathcal{S}^{\sharp}$,
and
${\tilde{\mathcal{S}}}^{\sharp}\cap\mathcal{W}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}^{\sharp}\cap\mathcal{W}$,
which
$\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}{\tilde{\mathcal{S}}}$.
From the transitivity (Lemma B.4), we have
${\tilde{\mathcal{S}}}^{\sharp}\cap\mathcal{W}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}{\tilde{\mathcal{S}}}$.
∎

Figure 9: Example B.15 showing that in Lem. B.14, only being
$\xi\\!\otimes\\!\zeta$-a.s. the same as a $\xi\\!\otimes\\!\zeta$-complete
component ${\tilde{\mathcal{S}}}$ of $\mathcal{W}$ is not sufficient for
${\tilde{\mathcal{S}}}$ to be also a $\xi\\!\otimes\\!\zeta$-complete
component of $\mathcal{W}$.

a

######  Example B.15.

Note that only the
${\tilde{\mathcal{S}}}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}$
condition is not sufficient. Fig. 9 shows such an example, where
$(\mathbb{X},\mathscr{X},\xi)$ and $(\mathbb{Z},\mathscr{Z},\zeta)$ are the
one dimensional Euclidean spaces with line Borel sigma-field and line Lebesgue
measure, $(\mathbb{R},\mathscr{R},\lambda)$, and
$\mathcal{W}:=[0,1]^{2}\cup[1,2]^{2}$, $\mathcal{S}:=[0,1]^{2}$, and
${\tilde{\mathcal{S}}}:=[0,1]^{2}\cup([1,2]\\!\times\\!\\{\frac{1}{2}\\})$. We
have
$\mathcal{S}^{\sharp}=([0,1]\\!\times\\!\mathbb{R})\cup(\mathbb{R}\\!\times\\![0,1])$
so $\mathcal{S}^{\sharp}\cap\mathcal{W}=\mathcal{S}$, justifying that
$\mathcal{S}$ is a $\xi\\!\otimes\\!\zeta$-complete component of
$\mathcal{W}$. On the other hand, since
$\mathcal{S}\triangle{\tilde{\mathcal{S}}}=[1,2]\\!\times\\!\\{\frac{1}{2}\\}$
is a line segment that has measure zero under the plane Lebesgue measure
$\xi\\!\otimes\\!\zeta=\lambda^{2}$, we have
${\tilde{\mathcal{S}}}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}$.
But ${\tilde{\mathcal{S}}}^{\mathbb{X}}=[0,2]$ so
${\tilde{\mathcal{S}}}^{\sharp}=([0,2]\\!\times\\!\mathbb{R})\cup(\mathbb{R}\\!\times\\![0,1])$,
which leads to ${\tilde{\mathcal{S}}}^{\sharp}\cap\mathcal{W}=\mathcal{W}$.
Since
${\tilde{\mathcal{S}}}\triangle\mathcal{W}=([1,2]\\!\times\\!\\{\frac{1}{2}\\})\cup([1,2]\\!\times\\![1,2])$
has a nonzero measure under $\lambda^{2}$ (it equals to 1), we know that
${\tilde{\mathcal{S}}}$ is not a $\xi\\!\otimes\\!\zeta$-complete component of
$\mathcal{W}$.

######  Lemma B.16.

Let $\mathcal{S}$ be a $\xi\\!\otimes\\!\zeta$-complete component of
$\mathcal{W}$, and $f$ be an either nonnegative or
$\xi\\!\otimes\\!\zeta$-integrable function on
$\mathbb{X}\\!\times\\!\mathbb{Z}$. Then for any measurable sets
$\mathcal{Z}\subseteq\mathcal{S}^{\mathbb{Z}}$ and
$\mathcal{X}\subseteq\mathcal{S}^{\mathbb{X}}$, we have:

$\displaystyle\int_{\mathcal{Z}}\int_{\mathcal{W}_{z}}f(x,z)\xi(\mathrm{d}x)\zeta(\mathrm{d}z)=\int_{\mathcal{Z}}\int_{\mathcal{S}_{z}}f(x,z)\xi(\mathrm{d}x)\zeta(\mathrm{d}z),$
(24)
$\displaystyle\int_{\mathcal{X}}\int_{\mathcal{W}_{x}}f(x,z)\zeta(\mathrm{d}z)\xi(\mathrm{d}x)=\int_{\mathcal{X}}\int_{\mathcal{S}_{x}}f(x,z)\zeta(\mathrm{d}z)\xi(\mathrm{d}x).$
(25)

Particularly,
$\int_{\mathcal{S}^{\mathbb{Z}}}\int_{\mathcal{W}_{z}}f(x,z)\xi(\mathrm{d}x)\zeta(\mathrm{d}z)=\int_{\mathcal{S}^{\mathbb{X}}}\int_{\mathcal{W}_{x}}f(x,z)\zeta(\mathrm{d}z)\xi(\mathrm{d}x)=\int_{\mathcal{S}}f(x,z)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)$.

  * Proof.

Since $\mathcal{S}$ is a $\xi\\!\otimes\\!\zeta$-complete component of
$\mathcal{W}$, Eq. (23) holds. By Lemma B.10, we know that for $\zeta$-a.e.
$z$ on $\mathbb{Z}$,
$\xi\big{(}(\mathcal{S}^{\sharp}\cap\mathcal{W})\triangle\mathcal{S}\big{)}_{z}=\xi\big{(}(\mathcal{S}^{\sharp}_{z}\cap\mathcal{W}_{z})\triangle\mathcal{S}_{z}\big{)}=0$.
Noting that $\mathcal{S}^{\sharp}_{z}=\mathbb{X}$ for any
$z\in\mathcal{S}^{\mathbb{Z}}$, this subsequently means that
$\xi(\mathcal{W}_{z}\triangle\mathcal{S}_{z})=0$ for $\zeta$-a.e. $z$ on
$\mathcal{S}^{\mathbb{Z}}$. By the additivity of integrals over a countable
partition [13, Thm. 16.9] and that the integral over a measure-zero set is
zero [13, p.226], we have
$\int_{\mathcal{W}_{z}}f(x,z)\xi(\mathrm{d}x)=\int_{\mathcal{S}_{z}}f(x,z)\xi(\mathrm{d}x)$
for $\zeta$-a.e. $z$ on $\mathcal{S}^{\mathbb{Z}}$. Since a.e.-equal functions
have the same integral [13, Thm. 15.2(v)], we have for any measurable
$\mathcal{Z}\subseteq\mathcal{S}^{\mathbb{Z}}$,
$\int_{\mathcal{Z}}\int_{\mathcal{W}_{z}}f(x,z)\xi(\mathrm{d}x)\zeta(\mathrm{d}z)=\int_{\mathcal{Z}}\int_{\mathcal{S}_{z}}f(x,z)\xi(\mathrm{d}x)\zeta(\mathrm{d}z)$.
Similarly, for any measurable $\mathcal{X}\subseteq\mathcal{S}^{\mathbb{X}}$,
$\int_{\mathcal{X}}\int_{\mathcal{W}_{x}}f(x,z)\zeta(\mathrm{d}z)\xi(\mathrm{d}x)=\int_{\mathcal{X}}\int_{\mathcal{S}_{x}}f(x,z)\zeta(\mathrm{d}z)\xi(\mathrm{d}x)$.

For $\mathcal{Z}=\mathcal{S}^{\mathbb{Z}}$, we have
$\int_{\mathcal{S}^{\mathbb{Z}}}\int_{\mathcal{W}_{z}}f(x,z)\xi(\mathrm{d}x)\zeta(\mathrm{d}z)=\int_{\mathcal{S}^{\mathbb{Z}}}\int_{\mathcal{S}_{z}}f(x,z)\xi(\mathrm{d}x)\zeta(\mathrm{d}z)$,
which is
$\int_{\mathcal{S}}f(x,z)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)$ by
the generalized form Eq. (16) of Fubini’s theorem. Similarly,
$\int_{\mathcal{S}^{\mathbb{X}}}\int_{\mathcal{W}_{x}}f(x,z)\zeta(\mathrm{d}z)\xi(\mathrm{d}x)=\int_{\mathcal{S}}f(x,z)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)$.
∎

##  Appendix C Proofs

Recall that
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z},\xi\\!\otimes\\!\zeta)$
is the product measure space by the two individual ones
$(\mathbb{X},\mathscr{X},\xi)$ and $(\mathbb{Z},\mathscr{Z},\zeta)$, where
$\xi$ and $\zeta$ are sigma-finite.

###  C.1 The Joint-Conditional Absolute Continuity Lemma

Although this lemma is not formally presented in the main text, we highlight
it here since it answers an important question and the answer is not
straightforward.

The lemma reveals the relation between the absolute continuity of a joint
$\pi$ and that of its conditionals $\pi(\cdot|z)$, $\pi(\cdot|x)$. Roughly,
the former guarantees the latter on the supports of the marginals, and the
reverse also holds, allowing one to safely use density function formulae for
deduction. But given two conditionals, one does not have the knowledge on the
marginals _a priori_. For a more useful sufficient condition, one may consider
the absolute continuity of the conditionals for $\zeta$-a.e. $z$ and
$\xi$-a.e. $x$. Unfortunately this is not sufficient, and an example (C.2) is
given after the proof. The lemma shows it is sufficient if the absolute
continuity of one of the conditionals, say $\pi(\cdot|z)$, holds for _any_
$z\in\mathbb{Z}$. The condition in the compatibility criterion Thm. 2.3 is
also inspired from this lemma.

######  Lemma C.1 (joint-conditional absolute continuity).

(i) For a joint distribution $\pi$ on
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z})$, it
is absolutely continuous $\pi\ll\xi\\!\otimes\\!\zeta$ if and only if
$\pi(\cdot|z)\ll\xi$ for $\pi^{\mathbb{Z}}$-a.e. $z$ and
$\pi(\cdot|x)\ll\zeta$ for $\pi^{\mathbb{X}}$-a.e. $x$. (ii) As a sufficient
condition, $\pi\ll\xi\\!\otimes\\!\zeta$ if $\pi(\cdot|z)\ll\xi$ for
$\zeta$-a.e. $z$ and $\pi(\cdot|x)\ll\zeta$ for _any_ $x\in\mathbb{X}$ (or for
_any_ $z\in\mathbb{Z}$ and $\xi$-a.e. $x$).

For conclusion (i):

  * Proof.

“Only if”: Consider any $\mathcal{X}\in\mathscr{X}$ such that
$\xi(\mathcal{X})=0$. From the definition of conditional distribution Eq.
(19), we have
$\pi^{\mathbb{X}}(\mathcal{X})=\pi(\mathcal{X}\\!\times\\!\mathbb{Z})=\int_{\mathbb{Z}}\pi(\mathcal{X}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=0$,
so $\pi(\mathcal{X}|z)=0$ for $\pi^{\mathbb{Z}}$-a.e. $z$ since
$\pi(\mathcal{X}|z)$ is nonnegative [13, Thm. 15.2(ii)]. This means that
$\pi(\cdot|z)\ll\xi$ for $\pi^{\mathbb{Z}}$-a.e. $z$. The same arguments apply
symmetrically to $\pi(\cdot|x)$.

Note that since $\pi(\cdot|z)$ is defined as the R-N derivative, it is allowed
to take any nonnegative value on a $\pi^{\mathbb{Z}}$-measure-zero set. So we
cannot guarantee its behavior for _any_ $z\in\mathbb{Z}$.

“If”: Consider any $\mathcal{Z}\in\mathscr{Z}$ such that
$\zeta(\mathcal{Z})=0$. Since $\pi(\cdot|x)\ll\zeta$ for
$\pi^{\mathbb{X}}$-a.e. $x$, we have $\pi(\mathcal{Z}|x)=0$ for
$\pi^{\mathbb{X}}$-a.e. $x$. So from Eq. (19) we have
$\pi^{\mathbb{Z}}(\mathcal{Z})=\pi(\mathbb{X}\\!\times\\!\mathcal{Z})=\int_{\mathbb{X}}\pi(\mathcal{Z}|x)\pi^{\mathbb{X}}(\mathrm{d}x)=0$
[13, Thm. 15.2(i)]. This indicates that $\pi^{\mathbb{Z}}\ll\zeta$.

Now consider any $\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$ such that
$(\xi\\!\otimes\\!\zeta)(\mathcal{W})=0$. By the definition of product measure
Eq. (13) [13, Thm. 18.2], we have
$(\xi\\!\otimes\\!\zeta)(\mathcal{W})=\int_{\mathbb{Z}}\xi(\mathcal{W}_{z})\zeta(\mathrm{d}z)=0$,
so $\xi(\mathcal{W}_{z})=0$ for $\zeta$-a.e. $z$ since $\xi(\mathcal{W}_{z})$
is nonnegative [13, Thm. 15.2(ii)]. Due to that $\pi^{\mathbb{Z}}\ll\zeta$,
this means that $\xi(\mathcal{W}_{z})=0$ for $\pi^{\mathbb{Z}}$-a.e. $z$.
Since $\pi(\cdot|z)\ll\xi$ for $\pi^{\mathbb{Z}}$-a.e. $z$, this in turn means
that $\pi(\mathcal{W}_{z}|z)=0$ for $\pi^{\mathbb{Z}}$-a.e. $z$. Subsequently,
we have
$\int_{\mathbb{Z}}\pi(\mathcal{W}_{z}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=0$ [13,
Thm. 15.2(i)], which is $\pi(\mathcal{W})=0$ by Eq. (20). So we get
$\pi\ll\xi\\!\otimes\\!\zeta$. ∎

Figure 10: Illustration of the conditionals in Eq. (26) in Example C.2. Both
conditionals are absolutely continuous for $\zeta$-a.e. $z$ or $\xi$-a.e. $x$,
but they allow a compatible joint that is not absolutely continuous w.r.t
$\xi\\!\otimes\\!\zeta$.

For conclusion (ii):

  * Proof.

Consider any $\mathcal{Z}\in\mathscr{Z}$ such that $\zeta(\mathcal{Z})=0$.
Since $\pi(\cdot|x)\ll\zeta$ for any $x\in\mathbb{X}$, we know that
$\pi(\mathcal{Z}|x)=0$ for any $x\in\mathbb{X}$. So from Eq. (19) we have
$\pi^{\mathbb{Z}}(\mathcal{Z})=\pi(\mathbb{X}\\!\times\\!\mathcal{Z})=\int_{\mathbb{X}}\pi(\mathcal{Z}|x)\pi^{\mathbb{X}}(\mathrm{d}x)=0$.
This indicates that $\pi^{\mathbb{Z}}\ll\zeta$.

Now consider any $\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}$ such that
$(\xi\\!\otimes\\!\zeta)(\mathcal{W})=0$. By the definition of product measure
Eq. (13) [13, Thm. 18.2], we have
$(\xi\\!\otimes\\!\zeta)(\mathcal{W})=\int_{\mathbb{Z}}\xi(\mathcal{W}_{z})\zeta(\mathrm{d}z)=0$,
so $\xi(\mathcal{W}_{z})=0$ for $\zeta$-a.e. $z$ since $\xi(\mathcal{W}_{z})$
is nonnegative [13, Thm. 15.2(ii)]. Due to that $\pi(\cdot|z)\ll\xi$ for
$\zeta$-a.e. $z$, this means that $\pi(\mathcal{W}_{z}|z)=0$ for $\zeta$-a.e.
$z$. Since $\pi^{\mathbb{Z}}\ll\zeta$, this in turn means that
$\pi(\mathcal{W}_{z}|z)=0$ for $\pi^{\mathbb{Z}}$-a.e. $z$. Subsequently, we
have $\int_{\mathbb{Z}}\pi(\mathcal{W}_{z}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=0$
[13, Thm. 15.2(i)], which is $\pi(\mathcal{W})=0$ by Eq. (20). So we get
$\pi\ll\xi\\!\otimes\\!\zeta$. The same arguments apply symmetrically when
$\pi(\cdot|z)\ll\xi$ for _any_ $z\in\mathbb{Z}$ and $\pi(\cdot|x)\ll\zeta$ for
$\xi$-a.e. $x$. ∎

######  Example C.2.

To see why it is not sufficient if the two conditionals are absolutely
continuous only for $\zeta$-a.e. $z$ and $\xi$-a.e. $x$, we show an example
below.

Consider the one-dimensional Euclidean space
$\mathbb{X}=\mathbb{Z}=\mathbb{R}$ with line Borel sigma-field
$\mathscr{X}=\mathscr{Z}=\mathscr{R}$ and line Lebesgue measure
$\xi=\zeta=\lambda$. Let

$\displaystyle\mu(\cdot|z):=\begin{cases}\delta_{z+2},&z\in\mathbb{Q}[0,1],\\\
\mathrm{Unif}[0,1],&z\in\bar{\mathbb{Q}}[0,1],\\\
0,&\text{otherwise},\end{cases}\;\nu(\cdot|x):=\begin{cases}\mathrm{Unif}[0,1],&x\in[0,1],\\\
\delta_{x-2},&x\in\mathbb{Q}[2,3],\\\ 0,&\text{otherwise},\end{cases}$ (26)

where $\mathbb{Q}[0,1]:=[0,1]\cap\mathbb{Q}$ and
$\bar{\mathbb{Q}}:=[0,1]\backslash\mathbb{Q}$ are the rational and irrational
numbers on $[0,1]$. The conditionals are illustrated in Fig. 10. Since
$\lambda(\mathbb{Q})=0$, the two conditionals are absolutely continuous for
$\zeta$-a.e. $z$ and $\xi$-a.e. $x$. Consider the joint distribution on
$\mathbb{X}\\!\times\\!\mathbb{Z}=\mathbb{R}^{2}$:

$\displaystyle\pi:=\frac{1}{2}\mathrm{Unif}([0,1]\\!\times\\![0,1])+\frac{1}{2}\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\delta_{(z+2,z)},$
(27)

where $\varrho$ is a distribution on the rationals $\mathbb{Q}[0,1]$ in
$[0,1]$ with the sigma-field of all the subsets of $\mathbb{Q}[0,1]$. Such a
distribution exists, for example, $\varrho(z)=1/2^{n(z)}$ where
$n:\mathbb{Q}[0,1]\to\mathbb{N}^{*}$ bijective is a numbering function of the
countable set $\mathbb{Q}[0,1]$. In this way, each rational number
$z\in\mathbb{Q}[0,1]$ has a positive probability, meanwhile we have
$\varrho(\mathbb{Q}[0,1])=\sum_{n=1}^{\infty}1/2^{n}=1$. Since
$\pi(\\{(z+2,z)\mid z\in\mathbb{Q}[0,1]\\})=\frac{1}{2}$ but
$\lambda^{2}(\\{(z+2,z)\mid z\in\mathbb{Q}[0,1]\\})=0$ under the square
Lebesgue measure $\lambda^{2}$, $\pi$ is not absolutely continuous.

To verify compatibility, note that $\mu$ and $\nu$ here satisfy the
corresponding measurability and integrability. To verify Eq. (19) reduced from
Eq. (18) for defining a conditional, we first derive the marginals:

$\displaystyle\pi^{\mathbb{X}}=\frac{1}{2}\mathrm{Unif}[0,1]+\frac{1}{2}\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\delta_{z+2},\;\;\pi^{\mathbb{Z}}=\frac{1}{2}\mathrm{Unif}[0,1]+\frac{1}{2}\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\delta_{z}.$
(28)

For any $\mathcal{X}\in\mathscr{X}$ and $\mathcal{Z}\in\mathscr{Z}$, we have:

$\displaystyle\pi(\mathcal{X}\\!\times\\!\mathcal{Z})={}$
$\displaystyle\frac{1}{2}\mathrm{Unif}[0,1](\mathcal{X})\mathrm{Unif}[0,1](\mathcal{Z})+\frac{1}{2}\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\mathbb{I}[(z+2,z)\in\mathcal{X}\\!\times\\!\mathcal{Z}]$
(29) $\displaystyle={}$
$\displaystyle\frac{1}{2}\lambda(\mathcal{X}[0,1])\lambda(\mathcal{Z}[0,1])+\frac{1}{2}\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\mathbb{I}[z\in(\mathcal{X}-2)\cap\mathcal{Z}],$
(30)

where $\mathcal{X}[0,1]:=[0,1]\cap\mathcal{X}$ and
$\mathcal{Z}[0,1]:=[0,1]\cap\mathcal{Z}$. To verify the conditional
distribution $\mu(\mathcal{X}|z)$, we have:

$\displaystyle\int_{\mathcal{Z}}\mu(\mathcal{X}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)={}$
$\displaystyle\int_{\mathcal{Z}\cap\mathbb{Q}[0,1]}\delta_{z+2}(\mathcal{X})\pi^{\mathbb{Z}}(\mathrm{d}z)+\int_{\mathcal{Z}\cap\bar{\mathbb{Q}}[0,1]}\mathrm{Unif}[0,1](\mathcal{X})\pi^{\mathbb{Z}}(\mathrm{d}z)$
(31) $\displaystyle={}$
$\displaystyle\int_{\mathcal{Z}\cap\mathbb{Q}[0,1]}\mathbb{I}[z+2\in\mathcal{X}]\pi^{\mathbb{Z}}(\mathrm{d}z)+\int_{\mathcal{Z}\cap\bar{\mathbb{Q}}[0,1]}\lambda(\mathcal{X}[0,1])\pi^{\mathbb{Z}}(\mathrm{d}z)$
(32) $\displaystyle={}$
$\displaystyle\pi^{\mathbb{Z}}((\mathcal{X}-2)\cap\mathcal{Z}\cap\mathbb{Q}[0,1])+\lambda(\mathcal{X}[0,1])\pi^{\mathbb{Z}}(\mathcal{Z}\cap\bar{\mathbb{Q}}[0,1])$
(33) (Since a countable set has measure zero under $\mathrm{Unif}[0,1]$,
_i.e._ $\lambda(\cdot\cap[0,1])$,) $\displaystyle={}$
$\displaystyle\frac{1}{2}\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\mathbb{I}\big{[}z\in(\mathcal{X}-2)\cap\mathcal{Z}\cap\mathbb{Q}[0,1]\big{]}$
(34)
$\displaystyle{}+\frac{1}{2}\lambda(\mathcal{X}[0,1])\bigg{(}\mathrm{Unif}[0,1](\mathcal{Z}[0,1])+\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\mathbb{I}\big{[}z\in\mathcal{Z}\cap\bar{\mathbb{Q}}[0,1]\big{]}\bigg{)}$
(35) $\displaystyle={}$
$\displaystyle\frac{1}{2}\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\mathbb{I}[z\in(\mathcal{X}-2)\cap\mathcal{Z}]+\frac{1}{2}\lambda(\mathcal{X}[0,1])\lambda(\mathcal{Z}[0,1])=\pi(\mathcal{X}\\!\times\\!\mathcal{Z}).$
(36)

For the conditional distribution $\nu(\mathcal{Z}|x)$, we similarly have:

$\displaystyle\int_{\mathcal{X}}\nu(\mathcal{Z}|x)\pi^{\mathbb{X}}(\mathrm{d}x)={}$
$\displaystyle\int_{\mathcal{X}[0,1]}\mathrm{Unif}[0,1](\mathcal{Z})\pi^{\mathbb{X}}(\mathrm{d}x)+\int_{\mathcal{X}\cap\mathbb{Q}[2,3]}\delta_{x-2}(\mathcal{Z})\pi^{\mathbb{X}}(\mathrm{d}x)$
(37) $\displaystyle={}$
$\displaystyle\lambda(\mathcal{Z}[0,1])\pi^{\mathbb{X}}(\mathcal{X}[0,1])+\pi^{\mathbb{X}}\big{(}\mathcal{X}\cap\mathbb{Q}[2,3]\cap(\mathcal{Z}+2)\big{)}$
(38) $\displaystyle={}$
$\displaystyle\frac{1}{2}\lambda(\mathcal{Z}[0,1])\lambda(\mathcal{X}[0,1])+\frac{1}{2}\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\mathbb{I}[z+2\in\mathcal{X}\cap\mathbb{Q}[2,3]\cap(\mathcal{Z}+2)]$
(39) $\displaystyle={}$
$\displaystyle\frac{1}{2}\lambda(\mathcal{Z}[0,1])\lambda(\mathcal{X}[0,1])+\frac{1}{2}\sum_{z\in\mathbb{Q}[0,1]}\varrho(z)\mathbb{I}[z\in(\mathcal{X}-2)\cap\mathcal{Z}]=\pi(\mathcal{X}\\!\times\\!\mathcal{Z}).$
(40)

So the two conditionals $\mu(\cdot|z)$ and $\nu(\cdot|x)$ are compatible and
$\pi$ is their joint distribution. This example illustrates that the absolute
continuity of $\pi(\cdot|z)$ w.r.t $\xi$ for $\zeta$-a.e. $z$ and that of
$\pi(\cdot|x)$ w.r.t $\zeta$ for $\xi$-a.e. $x$, does not indicate the
absolute continuity of $\pi$ w.r.t $\xi\\!\otimes\\!\zeta$.

This example does not contradict result (i) of the Lemma. For any
$z_{0}\in\mathbb{Q}[0,1]$, we have that $\mu(\cdot|z_{0})=\delta_{z_{0}+2}$ is
not absolutely continuous w.r.t $\xi=\lambda$. But
$\pi^{\mathbb{Z}}(\\{z_{0}\\})=\frac{1}{2}\varrho(z_{0})>0$. So it is not that
$\mu(\cdot|z)\ll\xi$ for $\pi^{\mathbb{Z}}$-a.e. $z$, which aligns with that
$\pi$ is not absolutely continuous w.r.t $\xi\\!\otimes\\!\zeta=\lambda^{2}$.

This example also shows that the absolute continuity of the compatible joint
may depend on the joint itself, apart from the two conditionals. Consider
another joint on $(\mathbb{R}^{2},\mathscr{R}^{2},\lambda^{2})$:

$\displaystyle{\tilde{\pi}}:=\mathrm{Unif}([0,1]\\!\times\\![0,1]).$ (41)

It is easy to see that
${\tilde{\pi}}^{\mathbb{X}}(\mathcal{X})=\mathrm{Unif}[0,1](\mathcal{X})=\lambda(\mathcal{X}[0,1])$
and ${\tilde{\pi}}^{\mathbb{Z}}(\mathcal{Z})=\lambda(\mathcal{Z}[0,1])$. For
any $\mathcal{X}\in\mathscr{X}$ and $\mathcal{Z}\in\mathscr{Z}$, we have
${\tilde{\pi}}(\mathcal{X}\\!\times\\!\mathcal{Z})=\lambda(\mathcal{X}[0,1])\lambda(\mathcal{Z}[0,1])$.
To verify Eq. (19) for defining a conditional, we have:

$\displaystyle\int_{\mathcal{Z}}\mu(\mathcal{X}|z){\tilde{\pi}}^{\mathbb{Z}}(\mathrm{d}z)={}$
$\displaystyle\int_{\mathcal{Z}\cap\mathbb{Q}[0,1]}\delta_{z+2}(\mathcal{X}){\tilde{\pi}}^{\mathbb{Z}}(\mathrm{d}z)+\int_{\mathcal{Z}\cap\bar{\mathbb{Q}}[0,1]}\mathrm{Unif}[0,1](\mathcal{X}){\tilde{\pi}}^{\mathbb{Z}}(\mathrm{d}z)$
(42) $\displaystyle={}$
$\displaystyle\int_{\mathcal{Z}\cap\mathbb{Q}[0,1]}\mathbb{I}[z+2\in\mathcal{X}]{\tilde{\pi}}^{\mathbb{Z}}(\mathrm{d}z)+\int_{\mathcal{Z}\cap\bar{\mathbb{Q}}[0,1]}\lambda(\mathcal{X}[0,1]){\tilde{\pi}}^{\mathbb{Z}}(\mathrm{d}z)$
(43) $\displaystyle={}$
$\displaystyle\lambda((\mathcal{X}-2)\cap\mathcal{Z}\cap\mathbb{Q}[0,1])+\lambda(\mathcal{X}[0,1])\lambda(\mathcal{Z}\cap\bar{\mathbb{Q}}[0,1])$
(44) (Since
$\lambda((\mathcal{X}-2)\cap\mathcal{Z}\cap\mathbb{Q}[0,1])\leqslant\lambda(\mathbb{Q})=0$
and
$\lambda(\mathcal{Z}\cap\bar{\mathbb{Q}}[0,1])=\lambda(\mathcal{Z}\cap\bar{\mathbb{Q}}[0,1])+\lambda(\mathbb{Q}[0,1])=\lambda(\mathcal{Z}[0,1])$,)
$\displaystyle={}$
$\displaystyle\lambda(\mathcal{X}[0,1])\lambda(\mathcal{Z}[0,1])={\tilde{\pi}}(\mathcal{X}\\!\times\\!\mathcal{Z}).$
(45)

For the conditional distribution $\nu(\mathcal{Z}|x)$, we similarly have:

$\displaystyle\int_{\mathcal{X}}\nu(\mathcal{Z}|x){\tilde{\pi}}^{\mathbb{X}}(\mathrm{d}x)={}$
$\displaystyle\int_{\mathcal{X}[0,1]}\mathrm{Unif}[0,1](\mathcal{Z}){\tilde{\pi}}^{\mathbb{X}}(\mathrm{d}x)+\int_{\mathcal{X}\cap\mathbb{Q}[2,3]}\delta_{x-2}(\mathcal{Z}){\tilde{\pi}}^{\mathbb{X}}(\mathrm{d}x)$
(46) $\displaystyle={}$
$\displaystyle\lambda(\mathcal{Z}[0,1]){\tilde{\pi}}^{\mathbb{X}}(\mathcal{X}[0,1])+{\tilde{\pi}}^{\mathbb{X}}\big{(}\mathcal{X}\cap\mathbb{Q}[2,3]\cap(\mathcal{Z}+2)\big{)}$
(47) (Since
${\tilde{\pi}}^{\mathbb{X}}\big{(}\mathcal{X}\cap\mathbb{Q}[2,3]\cap(\mathcal{Z}+2)\big{)}=\lambda\big{(}\mathcal{X}\cap\mathbb{Q}[2,3]\cap(\mathcal{Z}+2)\cap[0,1]\big{)}\leqslant\lambda(\mathbb{Q})=0$,)
$\displaystyle={}$
$\displaystyle\lambda(\mathcal{Z}[0,1])\lambda(\mathcal{X}[0,1])={\tilde{\pi}}(\mathcal{X}\\!\times\\!\mathcal{Z}).$
(48)

So ${\tilde{\pi}}$ is also a compatible joint of $\mu(\cdot|z)$ and
$\nu(\cdot|x)$. In this case, the violation set for $\mu(\cdot|z)\ll\xi$,
_i.e._ $\mathbb{Q}[0,1]$, has measure zero under ${\tilde{\pi}}^{\mathbb{Z}}$,
and the violation set for $\nu(\cdot|x)\ll\zeta$, _i.e._ $\mathbb{Q}[2,3]$,
has measure zero under ${\tilde{\pi}}^{\mathbb{X}}$. So result (i) of the
Lemma asserts that ${\tilde{\pi}}\ll\xi\\!\otimes\\!\zeta$, which aligns with
the example. This example shows that although both $\pi$ and ${\tilde{\pi}}$
are the compatible joint of the same conditionals, they have different
absolute continuity. So the condition in result (i) of the Lemma requires the
knowledge of the marginals $\pi^{\mathbb{Z}}$ and $\pi^{\mathbb{X}}$.

###  C.2 Proof of Theorem 2.3

Figure 11: Illustration of Example C.3. The conditionals are uniform on the
respective depicted slices. On $\mathcal{W}_{p,q}\cap\mathcal{W}_{q,p}$, the
conditional $q(z|x)$ is not normalized.

a

######  Example C.3.

Before presenting the proof, we give an example showing that only being the
intersection of $\mathcal{W}_{p,q}$ and $\mathcal{W}_{q,p}$ is not sufficient
to make a valid support of a compatible joint. The example is illustrated in
Fig. 11. The conditionals are uniform on the respective depicted slices, so
conditions (iv) and (v) in the theorem (2.3) are satisfied. The sets
$\mathcal{W}_{p,q}$ and $\mathcal{W}_{q,p}$ are depicted in the figure, and
their intersection $\mathcal{W}_{p,q}\cap\mathcal{W}_{q,p}$ is the right half.
Although on $\mathcal{W}_{p,q}\cap\mathcal{W}_{q,p}$, the conditionals do not
render support conflict, the conditional $q(z|x)$ is unnormalized for a given
$x$ from the bottom half: it integrates to $1/2$ on
$(\mathcal{W}_{p,q}\cap\mathcal{W}_{q,p})_{x}$. This means that any compatible
joint $\pi$ would also have its conditional $\pi(\cdot|x)$ unnormalized for
such $x$, which is impossible.

One may consider trying $\mathcal{W}_{q,p}$ as the joint support. This renders
a support conflict similar to the example in the main text: a joint on
$\mathcal{W}_{q,p}$ is required by $p(x|z)$ to also cover the top-left
quadrant since $z$ values in the left half are covered, but this contradicts