with the absence of mass by $q(z|x)$. In fact, in this example, there is no
$\xi\\!\otimes\\!\zeta$-complete component of both $\mathcal{W}_{p,q}$ and
$\mathcal{W}_{q,p}$, so the two conditionals are not compatible.

  * Proof.

Let $\mu(\mathcal{X}|z)$ and $\nu(\mathcal{Z}|x)$ be the two everywhere
absolutely continuous conditional distributions of whom $p(x|z)$ and $q(z|x)$
are the density functions. We begin with some useful conclusions.

(1) By construction, for any $(x,z)\in\mathcal{W}_{p,q}$, $p(x|z)>0$. For any
$z\in\mathcal{W}_{p,q}^{\mathbb{Z}}$, we have
$\xi\\{x\in(\mathcal{W}_{p,q})_{z}\mid
q(z|x)=0\\}=\xi\\{x\in\mathcal{P}_{z}\mid
x\notin\mathcal{Q}_{z}\\}=\xi(\mathcal{P}_{z}\setminus\mathcal{Q}_{z})=0$,
which means that $q(z|x)>0$, $\xi$-a.e. on $(\mathcal{W}_{p,q})_{z}$. By Lemma
B.10, we have that $q(z|x)>0$, $\xi\\!\otimes\\!\zeta$-a.e. on
$\mathcal{W}_{p,q}$. Symmetrically, $q(z|x)>0$ on $\mathcal{W}_{q,p}$, and
$p(x|z)>0$, $\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{W}_{q,p}$. Particularly,
the ratio $\frac{p(x|z)}{q(z|x)}$ is well-defined and is positive and finite,
both $\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{W}_{p,q}$ and
$\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{W}_{q,p}$. The conclusions also hold
($\xi\\!\otimes\\!\zeta$-a.e.) on any ($\xi\\!\otimes\\!\zeta$-a.s.) subset of
$\mathcal{W}_{p,q}$ or $\mathcal{W}_{q,p}$.

“Only if” (necessity):

Let $\pi$ be a compatible joint distribution of such conditional distributions
$\mu(\cdot|z)$ and $\nu(\cdot|x)$.

(2) Since “for any” indicates “a.e.” under any measure, the conditions in
Lemma C.1 are satisfied, so we have $\pi\ll\xi\\!\otimes\\!\zeta$. By Lemma
B.9, we also have $\pi^{\mathbb{X}}\ll\xi$ and $\pi^{\mathbb{Z}}\ll\zeta$. So
there exist density functions (R-N derivatives; 13, Thm. 32.2) $u(x)$ and
$v(z)$ such that for any $\mathcal{X}\in\mathscr{X}$ and
$\mathcal{Z}\in\mathscr{Z}$,
$\pi^{\mathbb{X}}(\mathcal{X})=\int_{\mathcal{X}}u(x)\xi(\mathrm{d}x)$ and
$\pi^{\mathbb{Z}}(\mathcal{Z})=\int_{\mathcal{Z}}v(z)\zeta(\mathrm{d}z)$. This
$u(x)$ is obviously $\xi$-integrable on any measurable subset of $\mathbb{X}$,
since the integral is no larger (since $u$ is nonnegative) than
$\int_{\mathbb{X}}u(x)\xi(\mathrm{d}x)=1$ which is finite.

(3) By the definition of conditional distribution Eq. (19), for any
$\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}$, we
have
$\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{Z}}\mu(\mathcal{X}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathcal{Z}}\mu(\mathcal{X}|z)v(z)\zeta(\mathrm{d}z)=\int_{\mathcal{Z}}\int_{\mathcal{X}}p(x|z)\xi(\mathrm{d}x)v(z)\zeta(\mathrm{d}z)=\int_{\mathcal{X}\\!\times\\!\mathcal{Z}}p(x|z)v(z)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)$,
where in the last equality, we have applied Fubini’s theorem Eq. (15) [13,
Thm. 18.3]. Similarly,
$\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{X}\\!\times\\!\mathcal{Z}}q(z|x)u(x)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)$.
Noting that $\mathscr{X}\\!\times\\!\mathscr{Z}$ is the pi-system that
generates $\mathscr{X}\\!\otimes\\!\mathscr{Z}$, this indicates that
$p(x|z)v(z)=q(z|x)u(x)$, $\xi\\!\otimes\\!\zeta$-a.e. on
$\mathbb{X}\\!\times\\!\mathbb{Z}$ [13, Thm. 16.10(iii)]999 The requirement
that $\mathbb{X}\\!\times\\!\mathbb{Z}$ is a finite or countable union of
$\mathscr{X}\\!\times\\!\mathscr{Z}$-sets is satisfied, since from the sigma-
finiteness of $\xi$ and $\zeta$, $\mathbb{X}$ and $\mathbb{Z}$ are a finite or
countable union of ($\xi$-finite) $\mathscr{X}$-sets and ($\zeta$-finite)
$\mathscr{Z}$-sets, respectively. . In other words, both $p(x|z)v(z)$ and
$q(z|x)u(x)$ are density functions of $\pi$.
(3.1) Subsequently, by leveraging Lemma B.10, we have for $\zeta$-a.e. $z$ on
$\mathbb{Z}$, $p(x|z)v(z)=q(z|x)u(x)$, $\xi$-a.e. on $\mathbb{X}$.

(4) Let $\mathcal{U}:=\\{x\mid u(x)>0\\}$ and $\mathcal{V}:=\\{z\mid
v(z)>0\\}$, and define:

$\displaystyle\mathcal{S}:=(\mathcal{U}\\!\times\\!\mathcal{V})\cap\mathcal{W}_{p,q},\;\;{\tilde{\mathcal{S}}}:=(\mathcal{U}\\!\times\\!\mathcal{V})\cap\mathcal{W}_{q,p},$
(49)

Since $u$ and $v$ are integrable thus measurable and $\mathbb{R}^{>0}$ is
Lebesgue-measurable, we know that $\mathcal{U}\in\mathscr{X}$ and
$\mathcal{V}\in\mathscr{Z}$ are also measurable. So $\mathcal{S}$ and
${\tilde{\mathcal{S}}}$ are measurable.

(4.1) We can verify that $\mathcal{S}$ is a $\xi\\!\otimes\\!\zeta$-complete
component of $\mathcal{W}_{p,q}$. Since
$\mathcal{S}\subseteq\mathcal{W}_{p,q}$, we only need to verify that:

$\displaystyle(\xi\\!\otimes\\!\zeta)\big{(}\big{[}(\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\cup\mathbb{X}\\!\times\\!\mathcal{S}^{\mathbb{Z}})\cap\mathcal{W}_{p,q}\big{]}\setminus\mathcal{S}\big{)}$
(50) (Since clearly $\mathcal{S}^{\mathbb{X}}\subseteq\mathcal{U}$,
$\mathcal{S}^{\mathbb{Z}}\subseteq\mathcal{V}$, and measures are monotone,)
$\displaystyle\leqslant{}$
$\displaystyle(\xi\\!\otimes\\!\zeta)\big{(}\big{[}(\mathcal{U}\\!\times\\!\mathbb{Z}\cup\mathbb{X}\\!\times\\!\mathcal{V})\cap\mathcal{W}_{p,q}\big{]}\setminus\mathcal{S}\big{)}$
(Since
$(\mathcal{A}\cap\mathcal{C})\setminus(\mathcal{B}\cap\mathcal{C})=(\mathcal{A}\setminus\mathcal{B})\cap\mathcal{C}$,)
$\displaystyle={}$
$\displaystyle(\xi\\!\otimes\\!\zeta)\big{(}\big{[}(\mathcal{U}\\!\times\\!\mathbb{Z}\cup\mathbb{X}\\!\times\\!\mathcal{V})\setminus(\mathcal{U}\\!\times\\!\mathcal{V})\big{]}\cap\mathcal{W}_{p,q}\big{)}$
(Since
$(\mathcal{A}\cup\mathcal{B})\setminus\mathcal{C}=(\mathcal{A}\setminus\mathcal{C})\cup(\mathcal{B}\setminus\mathcal{C})$,)
$\displaystyle={}$
$\displaystyle(\xi\\!\otimes\\!\zeta)\big{(}\big{[}((\mathcal{U}\\!\times\\!\mathbb{Z})\setminus(\mathcal{U}\\!\times\\!\mathcal{V}))\cup((\mathbb{X}\\!\times\\!\mathcal{V})\setminus(\mathcal{U}\\!\times\\!\mathcal{V}))\big{]}\cap\mathcal{W}_{p,q}\big{)}$
$\displaystyle={}$
$\displaystyle(\xi\\!\otimes\\!\zeta)\big{(}\big{[}\\{(x,z)\mid
u(x)>0,v(z)=0\\}\cup\mathcal{W}_{p,q}\big{]}\cup\big{[}\\{(x,z)\mid
u(x)=0,v(z)>0\\}\cup\mathcal{W}_{p,q}\big{]}\big{)}$ (By Conclusion (3) and
adjusting the set by a set of $\xi\\!\otimes\\!\zeta$-measure-zero according
to Lemma B.1,) $\displaystyle={}$
$\displaystyle(\xi\\!\otimes\\!\zeta)\big{(}\big{[}\\{(x,z)\mid
u(x)>0,v(z)=0,q(z|x)=0\\}\cup\mathcal{W}_{p,q}\big{]}$
$\displaystyle{}\cup\big{[}\\{(x,z)\mid
u(x)=0,v(z)>0,p(x|z)=0\\}\cup\mathcal{W}_{p,q}\big{]}\big{)}$
$\displaystyle\leqslant{}$
$\displaystyle(\xi\\!\otimes\\!\zeta)\big{(}\\{(x,z)\mid
u(x)>0,v(z)=0,q(z|x)=0\\}\cup\mathcal{W}_{p,q}\big{)}$
$\displaystyle{}+(\xi\\!\otimes\\!\zeta)\big{(}\\{(x,z)\mid
u(x)=0,v(z)>0,p(x|z)=0\\}\cup\mathcal{W}_{p,q}\big{)}$ (By Conclusion (1),)

$=0$. Symmetrically, we can also verify that ${\tilde{\mathcal{S}}}$ is a
$\xi\\!\otimes\\!\zeta$-complete component of $\mathcal{W}_{q,p}$.

(4.2) We can also show that $\pi(\mathcal{S})=\pi({\tilde{\mathcal{S}}})=1$.
From Conclusion (3), we have:

$\displaystyle 1={}$
$\displaystyle\pi(\mathbb{X}\\!\times\\!\mathbb{Z})=\int_{\mathbb{Z}}\int_{\mathbb{X}}p(x|z)\xi(\mathrm{d}x)v(z)\zeta(\mathrm{d}z)$
(Since the integral on a region with an a.e. zero value is zero [13, Thm.
15.2(i)],) $\displaystyle={}$
$\displaystyle\int_{\mathcal{V}}\int_{\mathcal{P}_{z}}p(x|z)\xi(\mathrm{d}x)v(z)\zeta(\mathrm{d}z)$
(Since $\mathcal{S}^{\mathbb{Z}}\subseteq\mathcal{V}$ and due to Billingsley
[13, Thm. 16.9],) $\displaystyle={}$
$\displaystyle\Big{(}\\!\int_{\mathcal{S}^{\mathbb{Z}}}\\!\int_{\mathcal{P}_{z}}\\!\\!+\\!\int_{\mathcal{V}\setminus\mathcal{S}^{\mathbb{Z}}}\\!\int_{\mathcal{P}_{z}}\\!\Big{)}p(x|z)\xi(\mathrm{d}x)v(z)\zeta(\mathrm{d}z)$
(Since
$\mathcal{V}\setminus\mathcal{S}^{\mathbb{Z}}=\mathcal{V}\setminus(\mathcal{V}\cap\mathcal{W}_{p,q}^{\mathbb{Z}})=\mathcal{V}\setminus\mathcal{W}_{p,q}^{\mathbb{Z}}$,)
$\displaystyle={}$
$\displaystyle\Big{(}\int_{\mathcal{S}^{\mathbb{Z}}}\int_{\mathcal{P}_{z}}+\int_{\mathcal{V}\setminus\mathcal{W}_{p,q}^{\mathbb{Z}}}\int_{\mathcal{P}_{z}}\Big{)}p(x|z)\xi(\mathrm{d}x)v(z)\zeta(\mathrm{d}z).\ignorespaces\ignorespaces$

For the second iterated integral, note that for any $z$ on $\mathcal{V}$,
$v(z)>0$, so from Conclusion (3.1), for $\zeta$-a.e. $z$ on $\mathcal{V}$, we
have $p(x|z)>0\Longrightarrow q(z|x)>0$, $\xi$-a.e. on $\mathbb{X}$. This
means that for $\zeta$-a.e. $z$ on $\mathcal{V}$, $\xi\\{x\mid
p(x|z)>0,q(z|x)=0\\}=\xi(\mathcal{P}_{z}\setminus\mathcal{Q}_{z})=0$. So the
set
$\mathcal{V}\setminus\mathcal{W}_{p,q}^{\mathbb{Z}}=\\{z\in\mathcal{V}\mid\xi(\mathcal{P}_{z}\setminus\mathcal{Q}_{z})>0\text{
or }\mathcal{P}_{z}=\emptyset\\}$ has the same measure under $\zeta$ as the
set $\\{z\in\mathcal{V}\mid\mathcal{P}_{z}=\emptyset\\}$. This means that the
second iterated integral
$\int_{\mathcal{V}\setminus\mathcal{W}_{p,q}^{\mathbb{Z}}}\int_{\mathcal{P}_{z}}p(x|z)\xi(\mathrm{d}x)v(z)\zeta(\mathrm{d}z)=\int_{\\{z\in\mathcal{V}\mid\mathcal{P}_{z}=\emptyset\\}}\int_{\mathcal{P}_{z}}p(x|z)\xi(\mathrm{d}x)v(z)\zeta(\mathrm{d}z)=0$
[13, p.226, Thm. 15.2(i)].

For the first iterated integral, note that by construction,
$\mathcal{P}_{z}=(\mathcal{W}_{p,q})_{z}$ for any $z$ on
$\mathcal{S}^{\mathbb{Z}}$, since
$\mathcal{S}^{\mathbb{Z}}\subseteq\mathcal{W}_{p,q}^{\mathbb{Z}}$. Moreover,
from Conclusion (4.1) that $\mathcal{S}$ is a $\xi\\!\otimes\\!\zeta$-complete
component of $\mathcal{W}_{p,q}$, we can subsequently apply Lemma B.16, so
$\int_{\mathcal{S}^{\mathbb{Z}}}\int_{(\mathcal{W}_{p,q})_{z}}p(x|z)\xi(\mathrm{d}x)v(z)\zeta(\mathrm{d}z)=\int_{\mathcal{S}}p(x|z)v(z)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)$,
which is $\pi(\mathcal{S})$ by Conclusion (3). This means that
$\pi(\mathcal{S})=1$. The same deduction applies symmetrically to
${\tilde{\mathcal{S}}}$, so we also have $\pi({\tilde{\mathcal{S}}})=1$.

Main procedure (“only if”). We will verify that the set $\mathcal{S}$ given in
Eq. (49) satisfies all the necessary conditions.

Conclusion (4.2) shows that $\pi(\mathcal{S})=1>0$, and in Conclusion (2) we
have verified that $\pi\ll\xi\\!\otimes\\!\zeta$. So we have
$(\xi\\!\otimes\\!\zeta)(\mathcal{S})>0$, which verifies Condition (iii).

To verify Conditions (iv) and (v), by Conclusions (3) and (1), we have
$p(x|z)v(z)=q(z|x)u(x)$ and $q(z|x)>0$, $\xi\\!\otimes\\!\zeta$-a.e. on
$\mathcal{S}$. By the construction of $\mathcal{S}$, we also have $v(z)>0$ and
$p(x|z)>0$ everywhere on $\mathcal{S}$. So the ratio $\frac{p(x|z)}{q(z|x)}$
is finite and positive, $\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{S}$, and it
factorizes as $\frac{p(x|z)}{q(z|x)}=u(x)\frac{1}{v(z)}$,
$\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{S}$. By Conclusion (2), $a(x):=u(x)$
is $\xi$-integrable on $\mathcal{S}^{\mathbb{X}}$.

To verify Condition (ii), note that by construction,
$\mathcal{S}^{\mathbb{Z}}\subseteq\mathcal{V}\cap\mathcal{W}_{p,q}^{\mathbb{Z}}\subseteq\mathcal{W}_{p,q}^{\mathbb{Z}}$
so
$\mathcal{S}^{\mathbb{Z}}\subseteq^{\text{$\zeta$}}\mathcal{W}_{p,q}^{\mathbb{Z}}$.
Note that
$\mathcal{S}^{\mathbb{X}}=\\{x\in\mathcal{U}\mid\mathcal{V}\cap(\mathcal{W}_{p,q})_{x}\neq\emptyset\\}=\\{x\in\mathcal{U}\mid\exists
z\in\mathcal{V}\text{~{}s.t.~{}}x\in\mathcal{P}_{z},\mathcal{P}_{z}\subseteq^{\text{$\xi$}}\mathcal{Q}_{z}\\}\subseteq\\{x\in\mathcal{U}\mid\exists
z\in\mathcal{V}\text{~{}s.t.~{}}x\in\mathcal{P}_{z}\\}=\\{x\mid u(x)>0,\exists
z\text{~{}s.t.~{}}v(z)>0,p(x|z)>0\\}$. By Conclusion (3.1), for $\xi$-a.e. $x$
on $\mathcal{S}^{\mathbb{X}}$, we have $\exists
z\text{~{}s.t.~{}}q(z|x)u(x)=p(x|z)v(z)>0$, so $q(z|x)>0$ hence
$\mathcal{Q}_{x}\neq\emptyset$. Moreover, for $\xi$-a.e. $x$ on
$\mathcal{S}^{\mathbb{X}}$ and $\zeta$-a.e. $z$ on $\mathcal{Q}_{x}$, we have
$p(x|z)v(z)=q(z|x)u(x)>0$, so $p(x|z)>0$ hence
$\mathcal{Q}_{x}\subseteq^{\text{$\zeta$}}\mathcal{P}_{x}$. These two
conclusions means that for $\xi$-a.e. $x$ on $\mathcal{S}^{\mathbb{X}}$, we
have
$x\in\\{x\mid\mathcal{Q}_{x}\neq\emptyset,\mathcal{Q}_{x}\subseteq^{\text{$\zeta$}}\mathcal{P}_{x}\\}$
which is exactly $\mathcal{W}_{q,p}^{\mathbb{X}}$. Hence
$\mathcal{S}^{\mathbb{X}}\subseteq^{\text{$\xi$}}\mathcal{W}_{q,p}^{\mathbb{X}}$.

Now, all the left is to verify Condition (i). Conclusion (4.1) has verified
that $\mathcal{S}$ is a $\xi\\!\otimes\\!\zeta$-complete component of
$\mathcal{W}_{p,q}$. To verify that $\mathcal{S}$ is a
$\xi\\!\otimes\\!\zeta$-complete component also of $\mathcal{W}_{q,p}$, note
that Conclusion (4.1) has also verified that ${\tilde{\mathcal{S}}}$ is a
$\xi\\!\otimes\\!\zeta$-complete component of $\mathcal{W}_{q,p}$, so by Lemma
B.14 it suffices to verify that
$\mathcal{S}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}{\tilde{\mathcal{S}}}$,
$\mathcal{S}^{\mathbb{X}}\overset{\text{\tiny$\xi$}}{=}{\tilde{\mathcal{S}}}^{\mathbb{X}}$
and
$\mathcal{S}^{\mathbb{Z}}\overset{\text{\tiny$\zeta$}}{=}{\tilde{\mathcal{S}}}^{\mathbb{Z}}$.

By construction, for any $(x,z)\in\mathcal{S}$, we have $v(z)>0$ and
$p(x|z)>0$, so from Conclusion (3) the density function $p(x|z)v(z)$ of $\pi$
is positive. Similarly, the density function $q(z|x)u(x)$ of $\pi$ is positive
everywhere on ${\tilde{\mathcal{S}}}$. Moreover, Conclusion (4.2) has shown
that both $\pi(\mathcal{S})=1$ and $\pi({\tilde{\mathcal{S}}})=1$. So by Lemma
B.7, we know that
$\mathcal{S}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}{\tilde{\mathcal{S}}}$.
Also by construction, the density function $u(x)$ of $\pi^{\mathbb{X}}$ is
positive everywhere on $\mathcal{S}^{\mathbb{X}}$ and on
${\tilde{\mathcal{S}}}^{\mathbb{X}}$. Moreover, we have
$\pi^{\mathbb{X}}(\mathcal{S}^{\mathbb{X}})=\pi(\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z})\geqslant\pi(\mathcal{S})=1$
so $\pi^{\mathbb{X}}(\mathcal{S}^{\mathbb{X}})=1$ and similarly
$\pi^{\mathbb{X}}({\tilde{\mathcal{S}}}^{\mathbb{X}})=1$. Again by Lemma B.7,
we have
$\mathcal{S}^{\mathbb{X}}\overset{\text{\tiny$\xi$}}{=}{\tilde{\mathcal{S}}}^{\mathbb{X}}$.
It follows similarly that
$\mathcal{S}^{\mathbb{Z}}\overset{\text{\tiny$\zeta$}}{=}{\tilde{\mathcal{S}}}^{\mathbb{Z}}$.

“If” (sufficiency):

(5) For Conditions (iv) and (v), denote ${\tilde{a}}(x):=\left\lvert
a(x)\right\rvert$ and ${\tilde{b}}(z):=\left\lvert b(z)\right\rvert$, and let
${\tilde{A}}:=\int_{\mathcal{S}^{\mathbb{X}}}{\tilde{a}}(x)\xi(\mathrm{d}x)$.
(5.1) From the definition of integrability in Supplement A.1, Condition (v) is
equivalent to that ${\tilde{a}}(x)$ is $\xi$-integrable on
$\mathcal{S}^{\mathbb{X}}$. Particularly, ${\tilde{A}}<\infty$.
Due to Condition (i) and Lemma B.12, we have
$\mathcal{S}\subseteq^{\text{$\xi\\!\otimes\\!\zeta$}}\mathcal{W}_{p,q}$. So
by Condition (iv) and Conclusion (1), we have
$\frac{p(x|z)}{q(z|x)}=a(x)b(z)>0$, $\xi\\!\otimes\\!\zeta$-a.e. on
$\mathcal{S}$. This means both
$(\xi\\!\otimes\\!\zeta)\\{(x,z)\in\mathcal{S}\mid a(x)b(z)=0\\}=0$ and
$(\xi\\!\otimes\\!\zeta)\\{(x,z)\in\mathcal{S}\mid a(x)b(z)<0\\}=0$, since
their summation is zero.
(5.2) Since $\\{(x,z)\mid
x\in\mathcal{S}^{\mathbb{X}},a(x)=0,z\in\mathcal{S}_{x}\\}\subseteq\\{(x,z)\in\mathcal{S}\mid
a(x)b(z)=0\\}$, the second-last equation in Conclusion (5.1) above means that
$(\xi\\!\otimes\\!\zeta)\\{(x,z)\mid
x\in\mathcal{S}^{\mathbb{X}},a(x)=0,z\in\mathcal{S}_{x}\\}=0$. So
$(\xi\\!\otimes\\!\zeta)\\{(x,z)\mid x\in\mathcal{S}^{\mathbb{X}},a(x)\neq
0,z\in\mathcal{S}_{x}\\}=(\xi\\!\otimes\\!\zeta)\\{(x,z)\mid
x\in\mathcal{S}^{\mathbb{X}},z\in\mathcal{S}_{x}\\}-(\xi\\!\otimes\\!\zeta)\\{(x,z)\mid
x\in\mathcal{S}^{\mathbb{X}},a(x)=0,z\in\mathcal{S}_{x}\\}=(\xi\\!\otimes\\!\zeta)(\mathcal{S})>0$
by Condition (iii). Moreover, by Eq. (14), we have
$(\xi\\!\otimes\\!\zeta)\\{(x,z)\mid x\in\mathcal{S}^{\mathbb{X}},a(x)\neq
0,z\in\mathcal{S}_{x}\\}=\int_{\\{x\in\mathcal{S}^{\mathbb{X}}\mid a(x)\neq
0\\}}\zeta(\mathcal{S}_{x})\xi(\mathrm{d}x)>0$, so
$\xi\\{x\in\mathcal{S}^{\mathbb{X}}\mid a(x)\neq 0\\}>0$ [13, p.226].
Particularly, ${\tilde{A}}>0$ [13, Thm. 15.2(ii)].
(5.3) Since $\\{(x,z)\in\mathcal{S}\mid
a(x)b(z)\neq{\tilde{a}}(x){\tilde{b}}(z)\\}=\\{(x,z)\in\mathcal{S}\mid
a(x)b(z)<0\\}$, the last equation in Conclusion (5.1) above means that
$a(x)b(z)={\tilde{a}}(x){\tilde{b}}(z)$, $\xi\\!\otimes\\!\zeta$-a.e. on
$\mathcal{S}$. So we have
$\frac{p(x|z)}{q(z|x)}={\tilde{a}}(x){\tilde{b}}(z)$,
$\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{S}$, following Condition (iv).

(6) Based on Conclusions (5.1) and (5.2), we can define the following finite
and nonnegative functions on $\mathbb{X}$ and $\mathbb{Z}$:

$\displaystyle
u(x):=\begin{cases}\frac{1}{{\tilde{A}}}{\tilde{a}}(x),&\text{if
$x\in\mathcal{S}^{\mathbb{X}}$ and ${\tilde{a}}(x)>0$},\\\
0,&\text{otherwise},\end{cases}\;\;v(z):=\begin{cases}\frac{1}{{\tilde{A}}{\tilde{b}}(z)},&\text{if
$z\in\mathcal{S}^{\mathbb{Z}}$ and ${\tilde{b}}(z)>0$},\\\
0,&\text{otherwise}.\end{cases}$ (51)

(6.1) By construction, ${\tilde{a}}(x){\tilde{b}}(z)=u(x)/v(z)$ on
$\mathcal{S}\cap\\{(x,z)\mid{\tilde{b}}(z)>0\\}$. So from Conclusion (5.3), we
have $p(x|z)v(z)=q(z|x)u(x)$, $\xi\\!\otimes\\!\zeta$-a.e. on
$\mathcal{S}\cap\\{(x,z)\mid{\tilde{b}}(z)>0\\}$. Moreover, following a
similar deduction as in Conclusion (5.2), we know that
$(\xi\\!\otimes\\!\zeta)(\mathcal{S}\setminus\\{(x,z)\mid{\tilde{b}}(z)>0\\})=(\xi\\!\otimes\\!\zeta)\\{(x,z)\mid
z\in\mathcal{S}^{\mathbb{Z}},b(z)=0,x\in\mathcal{S}_{z}\\}=0$. So we have
$p(x|z)v(z)=q(z|x)u(x)$, $\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{S}$.
(6.2) By construction,
$\int_{\mathbb{X}}u(x)\xi(\mathrm{d}x)=\int_{\mathcal{S}^{\mathbb{X}}}u(x)\xi(\mathrm{d}x)=1$.

Main procedure (“if”). We will show that the following function on
$\mathscr{X}\\!\otimes\\!\mathscr{Z}$, which is the same as Eq. (5) in the
theorem, is a distribution on $\mathbb{X}\\!\times\\!\mathbb{Z}$ such that
$\mu(\mathcal{X}|z)$ and $\nu(\mathcal{Z}|x)$ are its conditional
distributions:

$\displaystyle\pi(\mathcal{W}):=\int_{\mathcal{W}\cap\mathcal{S}}q(z|x)u(x)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z),\;\;\forall\mathcal{W}\in\mathscr{X}\\!\otimes\\!\mathscr{Z}.$
(52)

Consider any measurable rectangle
$\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}$. We
have:

$\displaystyle\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{X}\\!\times\\!\mathcal{Z}\cap\mathcal{S}}q(z|x)u(x)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)$
(53) (By the generalized form Eq. (16) of Fubini’s theorem,)
$\displaystyle={}$
$\displaystyle\int_{\mathcal{X}\cap\mathcal{S}^{\mathbb{X}}}\int_{\mathcal{Z}\cap\mathcal{S}_{x}}q(z|x)\zeta(\mathrm{d}z)u(x)\xi(\mathrm{d}x)$
(By Condition (i) and applying Lemma B.16,) $\displaystyle={}$
$\displaystyle\int_{\mathcal{X}\cap\mathcal{S}^{\mathbb{X}}}\int_{\mathcal{Z}\cap(\mathcal{W}_{q,p})_{x}}q(z|x)\zeta(\mathrm{d}z)u(x)\xi(\mathrm{d}x)$
(54) (Since
$\mathcal{S}^{\mathbb{X}}\subseteq^{\text{$\xi$}}\mathcal{W}_{q,p}^{\mathbb{X}}$
by Condition (ii) and $(\mathcal{W}_{q,p})_{x}=\mathcal{Q}_{x}$ on
$\mathcal{W}_{q,p}^{\mathbb{X}}$,) $\displaystyle={}$
$\displaystyle\int_{\mathcal{X}\cap\mathcal{S}^{\mathbb{X}}}\int_{\mathcal{Z}\cap\mathcal{Q}_{x}}q(z|x)\zeta(\mathrm{d}z)u(x)\xi(\mathrm{d}x)$
(55) (Since by construction, $u(x)=0$ outside $\mathcal{S}^{\mathbb{X}}$ and
$q(z|x)=0$ outside $\mathcal{Q}_{x}$,) $\displaystyle={}$
$\displaystyle\int_{\mathcal{X}}\int_{\mathcal{Z}}q(z|x)\zeta(\mathrm{d}z)u(x)\xi(\mathrm{d}x)$
(Recalling that $q(z|x)$ is the density function of $\nu(\cdot|x)$,)
$\displaystyle={}$
$\displaystyle\int_{\mathcal{X}}\nu(\mathcal{Z}|x)u(x)\xi(\mathrm{d}x).\ignorespaces\ignorespaces$

Moreover, due to Conclusion (6.1), we have
$\pi(\mathcal{W})=\int_{\mathcal{W}\cap\mathcal{S}}p(x|z)v(z)(\xi\\!\otimes\\!\zeta)(\mathrm{d}x\mathrm{d}z)$
on $\mathscr{X}\\!\otimes\\!\mathscr{Z}$ [13, Thm. 15.2(v)]. Using this form
of $\pi$ and noting that the symmetrized conditions in the above deduction
also hold, we have:

$\displaystyle\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{Z}}\mu(\mathcal{X}|z)v(z)\zeta(\mathrm{d}z).$
(56)

Since both $q(z|x)$ and $u(x)$ are finite and nonnegative on
$\mathbb{X}\\!\times\\!\mathbb{Z}$, $\pi$ is a measure on
$\mathscr{X}\\!\otimes\\!\mathscr{Z}$ by its definition Eq. (52) [13, p.227].
Moreover, from Eq. (LABEL:eqn:pi-is-int-over-nu), we have
$\pi(\mathbb{X}\\!\times\\!\mathbb{Z})=\int_{\mathbb{X}}u(x)\xi(\mathrm{d}x)=1$
by Conclusion (6.2). So $\pi$ is a distribution (probability measure) on
$\mathbb{X}\\!\times\\!\mathbb{Z}$.

From Eq. (LABEL:eqn:pi-is-int-over-nu), we have
$\pi^{\mathbb{X}}(\mathcal{X}):=\pi(\mathcal{X}\\!\times\\!\mathbb{Z})=\int_{\mathcal{X}}u(x)\xi(\mathrm{d}x)$.
So $u(x)$ is a density function of $\pi^{\mathbb{X}}$, and Eq. (LABEL:eqn:pi-
is-int-over-nu) in turn becomes
$\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{X}}\nu(\mathcal{Z}|x)\pi^{\mathbb{X}}(\mathrm{d}x)$.
This indicates that $\nu(\mathcal{Z}|x)$ is a conditional distribution of
$\pi$ w.r.t sub-sigma-field $\mathscr{X}\\!\times\\!\\{\mathcal{Z}\\}$ for any
$\mathcal{Z}\in\mathscr{Z}$, due to Eq. (19).

Similarly, from Eq. (56), we have
$\pi^{\mathbb{Z}}(\mathcal{Z}):=\pi(\mathbb{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{Z}}v(z)\zeta(\mathrm{d}z)$.
So $v(z)$ is a density function of $\pi^{\mathbb{Z}}$, and Eq. (56) in turn
becomes
$\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{Z}}\mu(\mathcal{X}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)$.
This indicates that $\mu(\mathcal{X}|z)$ is a conditional distribution of
$\pi$ w.r.t sub-sigma-field $\\{\mathcal{X}\\}\\!\times\\!\mathscr{Z}$ for any
$\mathcal{X}\in\mathscr{X}$, due to Eq. (19). The proof is completed. ∎

###  C.3 Complete Support Proposition under the a.e.-Full Support Condition

######  Proposition C.4.

If $p(x|z)$ and $q(z|x)$ have a.e.-full supports, then
$\mathcal{W}_{p,q}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{W}_{q,p}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathbb{X}\\!\times\\!\mathbb{Z}$,
and $\mathbb{X}\\!\times\\!\mathbb{Z}$ is the $\xi\\!\otimes\\!\zeta$-unique
complete support of them when compatible.

  * Proof.

By definition, $p(x|z)>0$ and $q(z|x)>0$, $\xi\\!\otimes\\!\zeta$-a.e. By
Lemma B.10, this means that for $\xi$-a.e. $x$, $p(x|z)>0$ $\zeta$-a.e. thus
$\mathcal{P}_{x}\overset{\text{\tiny$\zeta$}}{=}\mathbb{Z}$, and for
$\zeta$-a.e. $z$, $\mathcal{Q}_{z}\overset{\text{\tiny$\xi$}}{=}\mathbb{X}$.
Similarly, we also have for $\xi$-a.e. $x$,
$\mathcal{Q}_{x}\overset{\text{\tiny$\zeta$}}{=}\mathbb{Z}$ thus
$\mathcal{Q}_{x}\overset{\text{\tiny$\zeta$}}{=}\mathcal{P}_{x}$ by the
transitivity (Lemma B.4) and subsequently
$\mathcal{Q}_{x}\subseteq^{\text{$\zeta$}}\mathcal{P}_{x}$. Similarly, for
$\zeta$-a.e. $z$, $\mathcal{P}_{z}\subseteq^{\text{$\xi$}}\mathcal{Q}_{z}$.
This means that for $\xi$-a.e. $x$,
$(\mathcal{W}_{q,p})_{x}=\mathcal{Q}_{x}\overset{\text{\tiny$\zeta$}}{=}\mathbb{Z}$
thus $z\in(\mathcal{W}_{q,p})_{x}$, $\zeta$-a.e. By Lemma B.10, this means
that for $\xi\\!\otimes\\!\zeta$-a.e. $(x,z)$, we have
$(x,z)\in\mathcal{W}_{q,p}$, so
$\mathcal{W}_{q,p}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathbb{X}\\!\times\\!\mathbb{Z}$.
Similarly, we have
$\mathcal{W}_{p,q}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathbb{X}\\!\times\\!\mathbb{Z}$.

Let $\mathcal{S}$ be a complete support of $p(x|z)$ and $q(z|x)$ when they are
compatible. Since $(\xi\\!\otimes\\!\zeta)(\mathcal{S})=1>0$ from Condition
(iii) in Theorem 2.3, we know that $\xi(\mathcal{S}^{\mathbb{X}})>0$ by Eq.
(14) and Billingsley [13, p.226]. So there is an
$x\in\mathcal{S}^{\mathbb{X}}$ such that
$(\mathcal{W}_{q,p})_{x}\overset{\text{\tiny$\zeta$}}{=}\mathbb{Z}$, otherwise
there would be a non-measure-zero set of $x$ violating
$(\mathcal{W}_{q,p})_{x}\overset{\text{\tiny$\zeta$}}{=}\mathbb{Z}$. As a
$\xi\\!\otimes\\!\zeta$-complete component of $\mathcal{W}_{q,p}$ by Condition
(i) in Theorem 2.3, we have
$\mathcal{S}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}(\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\cap\mathcal{W}_{q,p})\cup(\mathbb{Z}\\!\times\\!\mathcal{S}^{\mathbb{Z}}\cap\mathcal{W}_{q,p})\supseteq\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\cap\mathcal{W}_{q,p}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}$.
This means that
$(\xi\\!\otimes\\!\zeta)(\mathcal{S}^{\mathbb{X}}\\!\times\\!\mathbb{Z}\setminus\mathcal{S})=\int_{\mathcal{S}^{\mathbb{X}}}\zeta(\mathbb{Z}\setminus\mathcal{S}_{x})\xi(\mathrm{d}x)=0$
by Eq. (14), so for $\xi$-a.e. $x$ on $\mathcal{S}^{\mathbb{X}}$,
$\zeta(\mathbb{Z}\setminus\mathcal{S}_{x})=0$ [13, Thm. 15.2(ii)] thus
$\mathcal{S}_{x}\overset{\text{\tiny$\zeta$}}{=}\mathbb{Z}$. So
$\mathcal{S}^{\mathbb{Z}}\overset{\text{\tiny$\zeta$}}{=}\mathbb{Z}$.
Moreover, we also have
$\mathcal{S}\supseteq^{\text{$\xi\\!\otimes\\!\zeta$}}\mathbb{X}\\!\times\\!\mathcal{S}^{\mathbb{Z}}$
which
$\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathbb{X}\\!\times\\!\mathbb{Z}$,
so we have
$\mathcal{S}\overset{\text{\tiny$\xi\\!\otimes\\!\zeta$}}{=}\mathbb{X}\\!\times\\!\mathbb{Z}$.
Similarly, from that $\mathcal{S}$ is a $\xi\\!\otimes\\!\zeta$-complete
component also of $\mathcal{W}_{p,q}$, we have the same conclusion. ∎

###  C.4 Proof of Theorem 2.4

  * Proof.

Let $\pi$ and ${\tilde{\pi}}$ be two compatible joints of $p(x|z)$ and
$q(z|x)$, and they are supported on the same complete support $\mathcal{S}$.
By Conclusions (2) and (3) in the proof (Supplement C.2) of Theorem 2.3, there
exist functions $u(x)$, $v(z)$ and ${\tilde{u}}(x)$, ${\tilde{v}}(z)$ such
that $p(x|z)v(z)$ and $q(z|x)u(x)$ are the densities of $\pi$, and
$p(x|z){\tilde{v}}(z)$ and $q(z|x){\tilde{u}}(x)$ of ${\tilde{\pi}}$, and
$p(x|z)v(z)=q(z|x)u(x)$ and $p(x|z){\tilde{v}}(z)=q(z|x){\tilde{u}}(x)$,
$\xi\\!\otimes\\!\zeta$-a.e. By the definition of a support in Lemma B.7, we
know that the densities of $\pi$ and ${\tilde{\pi}}$ are positive
$\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{S}$, and
$\int_{\mathcal{S}^{\mathbb{X}}}u\,\mathrm{d}\xi=\int_{\mathcal{S}^{\mathbb{X}}}{\tilde{u}}\,\mathrm{d}\xi=\int_{\mathcal{S}^{\mathbb{Z}}}v\,\mathrm{d}\zeta=\int_{\mathcal{S}^{\mathbb{Z}}}{\tilde{v}}\,\mathrm{d}\zeta=1$.

Consequently, we have
$\frac{p(x|z)}{q(z|x)}=\frac{u(x)}{v(z)}=\frac{{\tilde{u}}(x)}{{\tilde{v}}(z)}$,
$\xi\\!\otimes\\!\zeta$-a.e. on $\mathcal{S}$. By Lemma B.10, for $\zeta$-a.e.
$z$ on $\mathcal{S}^{\mathbb{Z}}$, we have
$\frac{u(x)}{v(z)}=\frac{{\tilde{u}}(x)}{{\tilde{v}}(z)}$ for $\xi$-a.e. $x$
on $\mathcal{S}_{z}$, which means that
$\int_{\mathcal{S}_{z}}\frac{u(x)}{v(z)}\xi(\mathrm{d}x)=\int_{\mathcal{S}_{z}}\frac{{\tilde{u}}(x)}{{\tilde{v}}(z)}\xi(\mathrm{d}x)$.
Since for $\zeta$-a.e. $z$ on $\mathcal{S}^{\mathbb{Z}}$,
$\mathcal{S}_{z}\overset{\text{\tiny$\xi$}}{=}\mathcal{S}^{\mathbb{X}}$, we
have by Lemma B.3 that
$\int_{\mathcal{S}^{\mathbb{X}}}\frac{u(x)}{v(z)}\xi(\mathrm{d}x)=\int_{\mathcal{S}^{\mathbb{X}}}\frac{{\tilde{u}}(x)}{{\tilde{v}}(z)}\xi(\mathrm{d}x)$,
which in turn gives
$\frac{1}{v(z)}\int_{\mathcal{S}^{\mathbb{X}}}u\,\mathrm{d}\xi=\frac{1}{v(z)}=\frac{1}{{\tilde{v}}(z)}\int_{\mathcal{S}^{\mathbb{X}}}{\tilde{u}}\,\mathrm{d}\xi=\frac{1}{{\tilde{v}}(z)}$.
So $v(z)={\tilde{v}}(z)$ for $\zeta$-a.e. $z$ on $\mathcal{S}^{\mathbb{Z}}$,
and similarly $u(x)={\tilde{u}}(x)$ for $\xi$-a.e. $x$ on
$\mathcal{S}^{\mathbb{X}}$. Subsequently, the density $p(x|z)v(z)$ or
$q(z|x)u(x)$ of $\pi$ is $\xi\\!\otimes\\!\zeta$-a.e. the same as the density
$p(x|z){\tilde{v}}(z)$ or $q(z|x){\tilde{u}}(x)$ of ${\tilde{\pi}}$. Hence,
$\pi$ and ${\tilde{\pi}}$ are the same distribution. ∎

###  C.5 The Dirac Compatibility Lemma

Before proving the main instructive compatibility theorem (2.6) for the Dirac
case, we first present an existential equivalent criterion for compatibility,
which provides insights to the problem.

######  Lemma C.5 (Dirac compatibility, existential).

Conditional distribution $\nu(\mathcal{Z}|x)$ is compatible with
$\mu(\mathcal{X}|z):=\delta_{f(z)}(\mathcal{X})$ where function
$f:\mathbb{Z}\to\mathbb{X}$ is $\mathscr{X}$/$\mathscr{Z}$-measurable, if and
only if there is a distribution $\beta$ on $(\mathbb{Z},\mathscr{Z})$ such
that $\nu(\mathcal{Z}|x)=\frac{\mathrm{d}\beta(\mathcal{Z}\cap
f^{-1}(\cdot))}{\mathrm{d}\beta(f^{-1}(\cdot))}(x)$, and this $\beta$ is the
marginal $\pi^{\mathbb{Z}}$ of a compatible joint $\pi$ of them.

  * Proof.

We first show the validity of the R-N derivative. Since $\beta$ is a
distribution thus a finite measure, $\beta(\mathcal{Z}\cap f^{-1}(\cdot))$ and
$\beta(f^{-1}(\cdot))$ are also finite thus sigma-finite. For any
$\mathcal{X}\in\mathscr{X}$ such that $\beta(f^{-1}(\mathcal{X}))=0$, we have
$\beta(\mathcal{Z}\cap
f^{-1}(\mathcal{X}))\leqslant\beta(f^{-1}(\mathcal{X}))=0$ since
$\mathcal{Z}\cap f^{-1}(\mathcal{X})\subseteq f^{-1}(\mathcal{X})$ and
measures are monotone. So $\beta(\mathcal{Z}\cap f^{-1}(\mathcal{X}))=0$ and
$\beta(\mathcal{Z}\cap f^{-1}(\cdot))\ll\beta(f^{-1}(\cdot))$. By the R-N
theorem [13, Thm. 32.2], the R-N derivative exists.

“Only if” (necessity): Let $\pi$ be a compatible joint. Since $\mu$ and $\nu$
are its conditional distributions, by Eq. (19), we have:

$\displaystyle\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{Z}}\mu(\mathcal{X}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathcal{X}}\nu(\mathcal{Z}|x)\pi^{\mathbb{X}}(\mathrm{d}x),\;\;\forall\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}.$
(57)

The first integral is
$\int_{\mathcal{Z}}\mathbb{I}[f(z)\in\mathcal{X}]\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathcal{Z}}\mathbb{I}[z\in
f^{-1}(\mathcal{X})]\pi^{\mathbb{Z}}(\mathrm{d}z)=\pi^{\mathbb{Z}}(\mathcal{Z}\cap
f^{-1}(\mathcal{X}))$. Particularly,
$\pi^{\mathbb{X}}(\mathcal{X})=\pi(\mathcal{X}\\!\times\\!\mathbb{Z})=\pi^{\mathbb{Z}}(f^{-1}(\mathcal{X}))$,
_i.e._ $\pi^{\mathbb{X}}$ is the transformed (pushed-forward) distribution
from $\pi^{\mathbb{Z}}$ by measurable function $f$ [13, p.196]. On the other
hand, the equality to the second integral means that
$\pi^{\mathbb{Z}}(\mathcal{Z}\cap
f^{-1}(\mathcal{X}))=\int_{\mathcal{X}}\nu(\mathcal{Z}|x)\pi^{\mathbb{X}}(\mathrm{d}x)=\int_{\mathcal{X}}\nu(\mathcal{Z}|x)\pi^{\mathbb{Z}}(f^{-1}(\mathrm{d}x))$.
This means that $\nu(\mathcal{Z}|x)$ is the R-N derivative of
$\mathcal{X}\mapsto\pi^{\mathbb{Z}}(\mathcal{Z}\cap f^{-1}(\mathcal{X}))$
w.r.t $\mathcal{X}\mapsto\pi^{\mathbb{Z}}(f^{-1}(\mathcal{X}))$. Taking
$\beta$ as $\pi^{\mathbb{Z}}$, which is a distribution on
$(\mathbb{Z},\mathscr{Z})$, yields the necessary condition.

“If” (sufficiency): For any measurable rectangle
$\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}$,
define
$\pi(\mathcal{X}\\!\times\\!\mathcal{Z}):=\int_{\mathcal{Z}}\mu(\mathcal{X}|z)\beta(\mathrm{d}z)$
and ${\tilde{\pi}}(\mathcal{X}\\!\times\\!\mathcal{Z}):=\beta(\mathcal{Z}\cap
f^{-1}(\mathcal{X}))$. Since for any $z\in\mathbb{Z}$, $f(z)\in\mathbb{X}$, so
$\pi(\mathbb{X}\\!\times\\!\mathbb{Z})=\int_{\mathbb{Z}}\mu(\mathbb{X}|z)\beta(\mathrm{d}z)=\int_{\mathbb{Z}}\beta(\mathrm{d}z)=1$.
Since $f^{-1}(\mathbb{X})=\mathbb{Z}$, we have
${\tilde{\pi}}(\mathbb{X}\\!\times\\!\mathbb{Z})=\beta(\mathbb{Z})=1$. So both
$\pi$ and ${\tilde{\pi}}$ are finite thus sigma-finite. Moreover, for any
$\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}$, we
have
$\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{Z}}\mathbb{I}[f(z)\in\mathcal{X}]\beta(\mathrm{d}z)=\int_{\mathcal{Z}}\mathbb{I}[z\in
f^{-1}(\mathcal{X})]\beta(\mathrm{d}z)=\int_{\mathcal{Z}\cap
f^{-1}(\mathcal{X})}\beta(\mathrm{d}z)=\beta(\mathcal{Z}\cap
f^{-1}(\mathcal{X}))={\tilde{\pi}}(\mathcal{X}\\!\times\\!\mathcal{Z})$,
_i.e._ $\pi$ and ${\tilde{\pi}}$ agree on the pi-system
$\mathscr{X}\\!\times\\!\mathscr{Z}$. So by Billingsley [13, Thm. 10.3], $\pi$
and ${\tilde{\pi}}$ extend to the same distribution (probability measure) on
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z})$.

On the other hand, we have
$\pi^{\mathbb{Z}}(\mathcal{Z})=\pi(\mathbb{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{Z}}\mu(\mathbb{X}|z)\beta(\mathrm{d}z)=\int_{\mathcal{Z}}\beta(\mathrm{d}z)=\beta(\mathcal{Z})$,
and furthermore from this, $\mu$ is a conditional distribution of $\pi$ due to
its construction and Eq. (19). Moreover,
${\tilde{\pi}}^{\mathbb{X}}(\mathcal{X})={\tilde{\pi}}(\mathcal{X}\\!\times\\!\mathbb{Z})=\beta(\mathbb{Z}\cap
f^{-1}(\mathcal{X}))=\beta(f^{-1}(\mathcal{X}))$, and by the definition of
$\nu(\mathcal{Z}|x)$ as an R-N derivative, we have $\beta(\mathcal{Z}\cap
f^{-1}(\mathcal{X}))=\int_{\mathcal{X}}\nu(\mathcal{Z}|x)\beta(f^{-1}(\mathrm{d}x))$,
which is
${\tilde{\pi}}(\mathcal{X}\\!\times\\!\mathcal{Z})=\int_{\mathcal{X}}\nu(\mathcal{Z}|x){\tilde{\pi}}^{\mathbb{X}}(\mathrm{d}x)$.
So again due to Eq. (19), $\nu$ is a conditional distribution of
${\tilde{\pi}}$. Since $\pi$ and ${\tilde{\pi}}$ are the same distribution on
$(\mathbb{X}\\!\times\\!\mathbb{Z},\mathscr{X}\\!\otimes\\!\mathscr{Z})$, we
know that $\mu$ and $\nu$ are compatible. ∎

Key insights. Let $\pi$ be a compatible joint of
$\mu(\mathcal{X}|z):=\delta_{f(z)}(\mathcal{X})$ and $\nu(\mathcal{Z}|x)$. For
any $\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}$,
we have:

$\displaystyle\pi(\mathcal{X}\\!\times\\!\mathcal{Z})=\pi^{\mathbb{Z}}(\mathcal{Z}\cap
f^{-1}(\mathcal{X}))=\int_{\mathcal{X}}\nu(\mathcal{Z}|x)\pi^{\mathbb{Z}}(f^{-1}(\mathrm{d}x))=\int_{f^{-1}(\mathcal{X})}\nu(\mathcal{Z}|f(z))\pi^{\mathbb{Z}}(\mathrm{d}z),$
(58)

where the last equality holds due to the rule of change of variables [13, Thm.
16.13]. Let
$f^{-1}(\mathscr{X}):=\sigma(\\{f^{-1}(\mathcal{X})\mid\mathcal{X}\in\mathscr{X}\\})$
be the pulled-back sigma-field from $\mathscr{X}$ by $f$. It is a sub-sigma-
field of $\mathscr{Z}$ as every $f^{-1}(\mathcal{X})\in\mathscr{Z}$ since $f$
is measurable. So the last equality means that:

$\displaystyle\nu(\mathcal{Z}|f(z))=\frac{\mathrm{d}\pi^{\mathbb{Z}}(\mathcal{Z}\cap\cdot)}{\mathrm{d}\pi^{\mathbb{Z}}(\cdot)}\bigg{|}_{f^{-1}(\mathscr{X})}(z).$
(59)

The expression on the left makes sense since for all values of $z$ that yield
the same value of $f(z)$, the R-N derivative is the same. The second equality
also gives:

$\displaystyle\nu(\mathcal{Z}|x)=\frac{\mathrm{d}\pi^{\mathbb{Z}}(\mathcal{Z}\cap
f^{-1}(\cdot))}{\mathrm{d}\pi^{\mathbb{Z}}(f^{-1}(\cdot))}\bigg{|}_{\mathscr{X}}(x).$
(60)

###  C.6 Proof of Theorem 2.6

  * Proof.

“Only if” (necessity): Suppose that $\nu(\mathcal{Z}|x)$ and
$\mu(\mathcal{X}|z):=\delta_{f(z)}(\mathcal{X})$ are compatible but for _any_
$x\in\mathbb{X}$, $\nu(f^{-1}(\\{x\\})|x)<1$. Consider the set
$\mathcal{S}:=\\{(f(z),z)\mid z\in\mathbb{Z}\\}$. Since $f$ is
$\mathscr{X}$/$\mathscr{Z}$-measurable, this set $\mathcal{S}$ is
$\mathscr{X}\\!\otimes\\!\mathscr{Z}$-measurable. It is also easy to verify
that $\mathcal{S}_{z}=\\{f(z)\\}$ and $\mathcal{S}_{x}=f^{-1}(\\{x\\})$. Now
let $\pi$ be _any_ of their compatible joint distribution. From Eq. (20), we
know that
$\pi(\mathcal{S})=\int_{\mathbb{Z}}\mu(\mathcal{S}_{z}|z)\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathbb{Z}}\delta_{f(z)}(\\{f(z)\\})\pi^{\mathbb{Z}}(\mathrm{d}z)=\int_{\mathbb{Z}}\pi^{\mathbb{Z}}(\mathrm{d}z)=\pi^{\mathbb{Z}}(\mathbb{Z})=1$.
On the other hand, also from Eq. (20) and due to the compatibility, we have
$\pi(\mathcal{S})=\int_{\mathbb{X}}\nu(\mathcal{S}_{x}|x)\pi^{\mathbb{X}}(\mathrm{d}x)=\int_{\mathbb{X}}\nu(f^{-1}(\\{x\\})|x)\pi^{\mathbb{X}}(\mathrm{d}x)<\int_{\mathbb{X}}\pi^{\mathbb{X}}(\mathrm{d}x)=\pi^{\mathbb{X}}(\mathbb{X})=1$,
which leads to a contradiction. So if $\nu(\mathcal{Z}|x)$ and
$\mu(\mathcal{X}|z)$ are compatible, then there is $x_{0}\in\mathbb{X}$ such
that $\nu(f^{-1}(\\{x_{0}\\})|x_{0})=1$.

“If” (sufficiency): Let
$\beta(\mathcal{Z}):=\nu(f^{-1}(\\{x_{0}\\})\cap\mathcal{Z}|x_{0})$ be a set
function on $\mathscr{Z}$. We can verify that this $\beta$ is a distribution
(probability measure) on $(\mathbb{Z},\mathscr{Z})$ since $\nu(\cdot|x_{0})$
is. Particularly, since $f$ is $\mathscr{X}$/$\mathscr{Z}$-measurable and
$\\{x_{0}\\}\in\mathscr{X}$ due to the assumption, we know that
$f^{-1}(\\{x_{0}\\})$ thus $f^{-1}(\\{x_{0}\\})\cap\mathcal{Z}$ for any
$\mathcal{Z}\in\mathscr{Z}$ are in $\mathscr{Z}$;
$\beta(\emptyset)=\nu(\emptyset|x_{0})=0$;
$\beta(\mathbb{Z})=\nu(f^{-1}(\\{x_{0}\\})|x_{0})=1$ according to the
assumption; for any countable disjoint $\mathscr{Z}$-sets
$\mathcal{Z}^{(1)},\mathcal{Z}^{(2)},\cdots$, it holds that
$\mathcal{Z}^{(1)}\cap f^{-1}(\\{x_{0}\\}),\mathcal{Z}^{(2)}\cap
f^{-1}(\\{x_{0}\\}),\cdots$ are also disjoint $\mathscr{Z}$-sets, so
$\beta(\bigcup_{i=1}^{\infty}\mathcal{Z}^{(i)})=\nu(f^{-1}(\\{x_{0}\\})\cap\bigcup_{i=1}^{\infty}\mathcal{Z}^{(i)}|x_{0})=\nu(\bigcup_{i=1}^{\infty}f^{-1}(\\{x_{0}\\})\cap\mathcal{Z}^{(i)}|x_{0})=\sum_{i=1}^{\infty}\nu(f^{-1}(\\{x_{0}\\})\cap\mathcal{Z}^{(i)}|x_{0})=\sum_{i=1}^{\infty}\beta(\mathcal{Z}^{(i)})$.

Now we prove that $\beta(\mathcal{Z}\cap
f^{-1}(\mathcal{X}))=\int_{\mathcal{X}}\nu(\mathcal{Z}|x)\beta(f^{-1}(\mathrm{d}x)),\forall\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}$
which is sufficient due to Lemma C.5. For any
$\mathcal{X}\\!\times\\!\mathcal{Z}\in\mathscr{X}\\!\times\\!\mathscr{Z}$, the
l.h.s is $\beta(\mathcal{Z}\cap
f^{-1}(\mathcal{X}))=\nu(f^{-1}(\\{x_{0}\\})\cap
f^{-1}(\mathcal{X})\cap\mathcal{Z}|x_{0})=\nu(f^{-1}(\\{x_{0}\\})\cap\mathcal{Z}|x_{0})\mathbb{I}[x_{0}\in\mathcal{X}]$,
where the last equality holds since $z\in f^{-1}(\\{x_{0}\\})\cap
f^{-1}(\mathcal{X})$ if and only if $f(z)=x_{0}\in\mathcal{X}$. The integral
on the r.h.s is
$\int_{\mathcal{X}}\nu(\mathcal{Z}|x)\nu(f^{-1}(\\{x_{0}\\})\cap
f^{-1}(\mathrm{d}x)|x_{0})$. Since the measure
$\mathcal{X}\mapsto\nu(f^{-1}(\\{x_{0}\\})\cap f^{-1}(\mathcal{X})|x_{0})$ is
zero on the set $\mathcal{X}\setminus\\{x_{0}\\}$ (if there is any $z\in
f^{-1}(\\{x_{0}\\})\cap f^{-1}(\mathcal{X}\setminus\\{x_{0}\\})$, then we have
$f(z)=x_{0}$ and $f(z)\in\mathcal{X}\setminus\\{x_{0}\\}$, which is a
contradiction), the integral can be reduced on $\\{x_{0}\\}\cap\mathcal{X}$
[13, Thm. 16.9]:
$\int_{\\{x_{0}\\}\cap\mathcal{X}}\nu(\mathcal{Z}|x)\nu(f^{-1}(\\{x_{0}\\})\cap
f^{-1}(\mathrm{d}x)|x_{0})=\mathbb{I}[x_{0}\in\mathcal{X}]\nu(\mathcal{Z}|x_{0})\nu(f^{-1}(\\{x_{0}\\})|x_{0})=\nu(\mathcal{Z}|x_{0})\mathbb{I}[x_{0}\in\mathcal{X}]$.
Moreover, $\nu(\mathcal{Z}|x_{0})=\nu(\mathcal{Z}\cap
f^{-1}(\\{x_{0}\\})|x_{0})+\nu(\mathcal{Z}\setminus
f^{-1}(\\{x_{0}\\})|x_{0})$ where $\nu(\mathcal{Z}\setminus
f^{-1}(\\{x_{0}\\})|x_{0})\leqslant\nu(\mathbb{Z}\setminus
f^{-1}(\\{x_{0}\\})|x_{0})=1-\nu(f^{-1}(\\{x_{0}\\})|x_{0})=0$, we have
$\nu(\mathcal{Z}\setminus f^{-1}(\\{x_{0}\\})|x_{0})=0$ and
$\nu(\mathcal{Z}|x_{0})=\nu(\mathcal{Z}\cap f^{-1}(\\{x_{0}\\})|x_{0})$. So
the integral on the r.h.s is $\nu(\mathcal{Z}\cap
f^{-1}(\\{x_{0}\\})|x_{0})\mathbb{I}[x_{0}\in\mathcal{X}]$, which is the same
as the l.h.s. So the equality is verified. ∎

##  Appendix D Topics on the Methods of CyGen

###  D.1 Relation to other auto-encoder regularizations

There are methods that consider regularizing the standard auto-encoder (AE)
[71, 7] with deterministic encoder $g(x)$ and decoder $f(z)$ for certain
robustness. These regularizations are introduced in addition to the standard
AE loss, _i.e._ the reconstruction loss:
$\mathbb{E}_{p^{*}(x)}\ell(x,f(g(x)))$, where $\ell(x,x^{\prime})$ is a
measure of similarity between $x$ and $x^{\prime}$. If $\ell(x,f(z))$ can be
treated as a (scaled) negative log-likelihood $-\log p(x|z)$ on $\mathbb{X}$
(_e.g._ , squared 2-norm $\ell$ for a Gaussian $p(x|z)$, cross entropy $\ell$
for a Bernoulli/categorical $p(x|z)$), then we can adopt a distributional view
of the decoder as $p(x|z)$ and the encoder as $\delta_{g(x)}(z)$ 101010This is
the notation of a Dirac’s delta function, which is not a function in the usual
sense. We adopt this form for the similarity to the DAE loss., and reformulate
the reconstruction loss also under the distributional view:
$\mathbb{E}_{p^{*}(x)}[-\log
p(x|g(x))]=\mathbb{E}_{p^{*}(x)\delta_{g(x)}(z)}[-\log p(x|z)]$.

Comparison with Jacobian norm regularizations. Contractive AE (CAE) [68, 69]
regularizes the Jacobian norm of the encoder,
$\lambda\mathbb{E}_{p^{*}(x)}\left\lVert\nabla_{x}g^{\top}(x)\right\rVert_{F}^{2}$
($\lambda$ controls the scale), in hope to encourage the robustness of the
encoded representation against local changes around training data. When it is
combined with the reconstruction loss which preserves data variation in the
representation for reconstruction, the robustness is confined to the
orthogonal direction to the data manifold, which often does not reflect
semantic meanings of interest. In other words, the variation in this
orthogonal direction is contracted in the representation, hence the name. When
applied to a linear encoder, this becomes the well-known weight-decay
regularizer. Note that CAE does not have a generative modeling utility, as it
uses a deterministic encoder which leads to insufficient determinacy (Sec.
2.2.2).

Denoising AE (DAE) [82, 9, 10] considers the robustness to random
corruption/perturbation on data, so its encoding process is
$z=g(x+\epsilon_{e})$ where
$\epsilon_{e}\sim\mathcal{N}(0,\sigma_{e}^{2}I_{d_{\mathbb{X}}})$ (or any
other distribution with $\mathbb{E}[\epsilon_{e}]=0$ and
$\operatorname{Var}[\epsilon_{e}]=\sigma_{e}^{2}I_{d_{\mathbb{X}}}$), which
defines a probabilistic encoder $q(z|x)$ (note that this is different from an
additive Gaussian encoder). The goal for training a DAE is thus to try to
reconstruct the input under the random corruption, by minimizing the DAE loss:

$\displaystyle\mathbb{E}_{p^{*}(x)q(z|x)}[-\log p(x|z)],$ (61)

which resembles the distributional form of the standard reconstruction loss.
For infinitesimal corruption variance $\sigma_{e}^{2}$ and squared 2-norm
$\ell$, the DAE loss Eq. (61) is roughly equivalent to regularizing the
standard reconstruction loss with
$\sigma_{e}^{2}\mathbb{E}_{p^{*}(x)}\left\lVert\nabla_{x}(f\circ
g)^{\top}(x)\right\rVert_{F}^{2}$, _i.e._ the Jacobian norm of the
reconstruction function [68, 2]. So DAE can be viewed to promote the
robustness of reconstruction while CAE of the representation [68].

In contrast, for additive Gaussian decoder (_i.e._ , squared 2-norm $\ell$)
and encoder, our compatibility regularization Eq. (6) is
$\mathbb{E}_{\rho(x,z)}\left\lVert\frac{1}{\sigma_{d}^{2}}\big{(}\nabla_{z}f^{\top}(z)\big{)}^{\top}-\frac{1}{\sigma_{e}^{2}}\nabla_{x}g^{\top}(x)\right\rVert_{F}^{2}$,
which is different from CAE and DAE regularizations. Ideologically, the
compatibility loss is an intrinsic constraint to make use of the
distributional nature of the encoder and decoder, and is not motivated from
the additional requirement of robustness in some sense.

Comparison with a more accurate DAE reformulation. In fact, the analysis in
[68, 2] for DAE as a regularization of the reconstruction loss is inaccurate.
Key ingredients for the analysis are the Taylor expansions: $\left\lVert
x+\varepsilon\right\rVert_{2}^{2}=\left\lVert
x\right\rVert_{2}^{2}+2x^{\top}\varepsilon+\varepsilon^{\top}\varepsilon$,
$\exp\\{x+\varepsilon\\}=\exp\\{x\\}(1+\varepsilon+\frac{1}{2}\varepsilon^{2})+o(\varepsilon^{2})$,
and
$\log(1+\varepsilon)=\varepsilon-\frac{1}{2}\varepsilon^{2}+o(\varepsilon^{2})$.
In the following, we consider $\ell(x,x^{\prime})=\left\lVert
x-x^{\prime}\right\rVert_{2}^{2}$, corresponding to an additive Gaussian
decoder $p(x|z)$.

First consider the additive Gaussian encoder,
$q(z|x)=\mathcal{N}(z|g(x),\sigma_{e}^{2}I_{d_{\mathbb{Z}}})$, or
$z=g(x)+\epsilon_{e},\epsilon_{e}\sim\mathcal{N}(0,\sigma_{e}^{2}I_{d_{\mathbb{Z}}})$.
Consider the case for infinitesimal $\sigma_{e}$. For the DAE loss Eq. (61),
we have (omitting the expectation over $p^{*}(x)$):

$\displaystyle\mathbb{E}_{q(z|x)}[-\log
p(x|z)]=\mathbb{E}_{q(z|x)}[\ell(x,f(z))]=\mathbb{E}_{q(z|x)}\left\lVert
x-f(z)\right\rVert_{2}^{2}=\mathbb{E}_{p(\epsilon_{e})}\left\lVert
x-f(g(x)+\epsilon_{e})\right\rVert_{2}^{2}$ (62) $\displaystyle={}$
$\displaystyle\mathbb{E}_{p(\epsilon_{e})}\left[\left\lVert x-f(g(x))-(\nabla
f^{\top})^{\top}\epsilon_{e}-\frac{1}{2}\epsilon_{e}^{\top}(\nabla^{2}f)\epsilon_{e}+o(\epsilon_{e}^{2})\right\rVert_{2}^{2}\right]$
(63) $\displaystyle={}$
$\displaystyle\mathbb{E}_{p(\epsilon_{e})}\Big{[}\left\lVert
x-f(g(x))\right\rVert_{2}^{2}-2\big{(}x-f(g(x))\big{)}^{\top}\Big{(}(\nabla
f^{\top})^{\top}\epsilon_{e}+\frac{1}{2}\epsilon_{e}^{\top}(\nabla^{2}f)\epsilon_{e}\Big{)}$
(64) $\displaystyle{}+\epsilon_{e}^{\top}(\nabla f^{\top})(\nabla
f^{\top})^{\top}\epsilon_{e}+o(\epsilon_{e}^{2})\Big{]}$ (65)
$\displaystyle={}$
$\displaystyle\ell\big{(}x,f(g(x))\big{)}-\sigma_{e}^{2}\big{(}x-f(g(x))\big{)}^{\top}\Delta
f+\sigma_{e}^{2}\left\lVert\nabla
f^{\top}\right\rVert_{F}^{2}+o(\sigma_{e}^{2}),$ (66)

where $(\nabla f^{\top})^{\top}$ is the Jacobian of $f$,
$(\epsilon_{e}^{\top}(\nabla^{2}f)\epsilon_{e})_{i}:=\sum_{j,k=1..d_{\mathbb{Z}}}(\epsilon_{e})_{i}(\epsilon_{e})_{j}\partial_{z_{i}}\partial_{z_{j}}f_{i}$,
and $(\Delta
f)_{i}:=\sum_{j=1..d_{\mathbb{Z}}}\partial_{z_{j}}\partial_{z_{j}}f_{i}$, and
they are evaluated at $z=g(x)$. Note that in addition to the Jacobian norm
regularization term discovered in [68, 2], there is a second regularization
term $-\sigma_{e}^{2}\big{(}x-f(g(x))\big{)}^{\top}\Delta f$ that DAE imposes.

For the data-fitting loss of CyGen Eq. (9), a similar approximation can be
derived (again, omitting the expectation over $p^{*}(x)$):

$\displaystyle\log\mathbb{E}_{q(z|x)}[1/p(x|z)]=\log\mathbb{E}_{q(z|x)}\exp\\{-\log
p(x|z)\\}=\log\mathbb{E}_{q(z|x)}\exp\\{\ell(x,f(z))\\}$ (67)
$\displaystyle={}$ $\displaystyle\log\mathbb{E}_{q(z|x)}\exp\\{\left\lVert
x-f(z)\right\rVert_{2}^{2}\\}=\log\mathbb{E}_{p(\epsilon_{e})}\exp\\{\left\lVert
x-f(g(x)+\epsilon_{e})\right\rVert_{2}^{2}\\}$ (68) $\displaystyle={}$
$\displaystyle\log\mathbb{E}_{p(\epsilon_{e})}\exp\left\\{\left\lVert
x-f(g(x))-(\nabla
f^{\top})^{\top}\epsilon_{e}-\frac{1}{2}\epsilon_{e}^{\top}(\nabla^{2}f)\epsilon_{e}+o(\epsilon_{e}^{2})\right\rVert_{2}^{2}\right\\}$
(69) $\displaystyle={}$
$\displaystyle\log\mathbb{E}_{p(\epsilon_{e})}\exp\Big{\\{}\left\lVert
x-f(g(x))\right\rVert_{2}^{2}-2\big{(}x-f(g(x))\big{)}^{\top}\Big{(}(\nabla
f^{\top})^{\top}\epsilon_{e}+\frac{1}{2}\epsilon_{e}^{\top}(\nabla^{2}f)\epsilon_{e}\Big{)}$
(70) $\displaystyle{}+\epsilon_{e}^{\top}(\nabla f^{\top})(\nabla
f^{\top})^{\top}\epsilon_{e}+o(\epsilon_{e}^{2})\Big{\\}}$ (71)
$\displaystyle={}$
$\displaystyle\log\mathbb{E}_{p(\epsilon_{e})}\Big{[}\exp\\{\left\lVert
x-f(g(x))\right\rVert_{2}^{2}\\}\Big{(}1-2\big{(}x-f(g(x))\big{)}^{\top}\Big{(}(\nabla
f^{\top})^{\top}\epsilon_{e}+\frac{1}{2}\epsilon_{e}^{\top}(\nabla^{2}f)\epsilon_{e}\Big{)}$
(72) $\displaystyle{}+\epsilon_{e}^{\top}(\nabla f^{\top})(\nabla
f^{\top})^{\top}\epsilon_{e}+2\Big{(}\big{(}x-f(g(x))\big{)}^{\top}(\nabla
f^{\top})^{\top}\epsilon_{e}\Big{)}^{2}+o(\epsilon_{e}^{2})\Big{)}\Big{]}$
(73) $\displaystyle={}$ $\displaystyle\log\Big{[}\exp\\{\left\lVert
x-f(g(x))\right\rVert_{2}^{2}\\}\Big{(}1-\sigma_{e}^{2}\big{(}x-f(g(x))\big{)}^{\top}\Delta
f$ (74) $\displaystyle{}+\sigma_{e}^{2}\left\lVert\nabla
f^{\top}\right\rVert_{F}^{2}+2\sigma_{e}^{2}\left\lVert(\nabla
f^{\top})\big{(}x-f(g(x))\big{)}\right\rVert_{2}^{2}+o(\sigma_{e}^{2})\Big{)}\Big{]}$
(75) $\displaystyle={}$
$\displaystyle\ell\big{(}x-f(g(x))\big{)}-\sigma_{e}^{2}\big{(}x-f(g(x))\big{)}^{\top}\Delta
f+\sigma_{e}^{2}\left\lVert\nabla
f^{\top}\right\rVert_{F}^{2}+2\sigma_{e}^{2}\left\lVert(\nabla
f^{\top})\big{(}x-f(g(x))\big{)}\right\rVert_{2}^{2}+o(\sigma_{e}^{2}).$ (76)

This is different from the regularization interpretation of DAE Eq. (66) as a
third regularization term $2\sigma_{e}^{2}\left\lVert(\nabla
f^{\top})\big{(}x-f(g(x))\big{)}\right\rVert_{2}^{2}$ is presented.

The compatibility loss Eq. (6) in CyGen becomes:
$\mathbb{E}_{\rho(x,z)}\left\lVert\frac{1}{\sigma_{d}^{2}}\big{(}\nabla_{z}f^{\top}(z)\big{)}^{\top}-\frac{1}{\sigma_{e}^{2}}\nabla_{x}g^{\top}(x)\right\rVert_{F}^{2}$,
where $\sigma_{d}^{2}$ is the Gaussian variance of the decoder $p(x|z)$
(inverse scale for $\ell(\cdot,\cdot)$). When $\rho(x,z)=p^{*}(x)q(z|x)$, this
can be further reduced to (omitting the expectation over $p^{*}(x)$):

$\displaystyle\mathbb{E}_{q(z|x)}\left\lVert\frac{1}{\sigma_{d}^{2}}\big{(}\nabla_{z}f^{\top}(z)\big{)}^{\top}-\frac{1}{\sigma_{e}^{2}}\nabla_{x}g^{\top}(x)\right\rVert_{F}^{2}=\mathbb{E}_{p(\epsilon_{e})}\left\lVert\frac{1}{\sigma_{d}^{2}}\big{(}\nabla
f^{\top}(g(x)+\epsilon_{e})\big{)}^{\top}-\frac{1}{\sigma_{e}^{2}}\nabla
g^{\top}\right\rVert_{F}^{2}$ (77) $\displaystyle={}$
$\displaystyle\mathbb{E}_{p(\epsilon_{e})}\left\lVert\frac{1}{\sigma_{d}^{2}}\Big{(}(\nabla
f^{\top})^{\top}+(\nabla^{2}f^{\top})^{\top}\epsilon_{e}+\frac{1}{2}\epsilon_{e}^{\top}(\nabla^{3}f^{\top})^{\top}\epsilon_{e}+o(\epsilon_{e}^{2})\Big{)}-\frac{1}{\sigma_{e}^{2}}\nabla
g^{\top}\right\rVert_{F}^{2}$ (78) $\displaystyle={}$
$\displaystyle\left\lVert\frac{1}{\sigma_{d}^{2}}(\nabla
f^{\top})^{\top}-\frac{1}{\sigma_{e}^{2}}\nabla
g^{\top}\right\rVert_{F}^{2}+\frac{\sigma_{e}^{2}}{\sigma_{d}^{4}}(\nabla
f^{\top}):(\nabla\Delta
f^{\top})+\frac{\sigma_{e}^{2}}{\sigma_{d}^{4}}\left\lVert\nabla^{2}f^{\top}\right\rVert_{F}^{2}+o(\sigma_{e}^{2}),$
(79)

where
$\big{(}(\nabla^{2}f^{\top})^{\top}\epsilon_{e}\big{)}_{ij}:=\sum_{k=1..d_{\mathbb{Z}}}(\epsilon_{e})_{k}\partial_{z_{k}}\partial_{z_{j}}f_{i}$,
$\big{(}\epsilon_{e}^{\top}(\nabla^{3}f^{\top})^{\top}\epsilon_{e}\big{)}_{ij}:=\sum_{k,k^{\prime}=1..d_{\mathbb{Z}}}(\epsilon_{e})_{k}(\epsilon_{e})_{k^{\prime}}\partial_{z_{k}}\partial_{z_{k^{\prime}}}\partial_{z_{j}}f_{i}$,
and $(\nabla f^{\top}):(\nabla\Delta
f^{\top}):=\sum_{\begin{subarray}{c}i=1..d_{\mathbb{X}},\\\
j,k=1..d_{\mathbb{Z}}\end{subarray}}(\partial_{z_{j}}f_{i})(\partial_{z_{j}}\partial_{z_{k}}\partial_{z_{k}}f_{i})$,
$\left\lVert\nabla^{2}f^{\top}\right\rVert_{F}^{2}:=\sum_{\begin{subarray}{c}i=1..d_{\mathbb{X}},\\\
j,k=1..d_{\mathbb{Z}}\end{subarray}}\big{(}\partial_{z_{k}}\partial_{z_{j}}f_{i}\big{)}^{2}$,
and all terms are evaluated at $z=g(x)$. This is different from the
regularization of CAE and the regularization explanation of DAE.

For the corruption encoder,
$z=g(x+\epsilon_{e}),\epsilon_{e}\sim\mathcal{N}(0,\sigma_{e}^{2}I_{d_{\mathbb{X}}})$,
approximations of the DAE loss Eq. (61) and the data-fitting loss of CyGen Eq.
(9) (_i.e._ , negative data likelihood loss) are similar to the above
expansions except that derivatives of $f$ are replaced with those of $f\circ
g$. Particularly, from Eq. (66), we find that the conclusion in [68, 2] missed
the term $-\sigma_{e}^{2}\big{(}x-f(g(x))\big{)}^{\top}\Delta(f\circ g)$ that
is also of order $\sigma_{e}^{2}$. For the compatibility loss, as there is no
explicit expression of $\log q(z|x)$ (unless $g(x)$ is invertible), the above
expression does not hold. But anyway, it is different from CAE and DAE
regularizations.

Relation to the tied weights trick. The compatibility loss also explains the
“tied weights” trick in AE, which is widely adopted and is vital for the
success of AE [63, 82, 81, 68, 2]. The trick is considered when components of
$x$ and $z$ are binary, and a one-layer, product-of-Bernoulli encoder
$q(z|x)=\prod_{j=1}^{d_{\mathbb{Z}}}\mathrm{Bern}(z_{j}|s((W_{e})_{j,:}x+(b_{e})_{j}))$
and decoder
$p(x|z)=\prod_{i=1}^{d_{\mathbb{X}}}\mathrm{Bern}(x_{i}|s((W_{d})_{i,:}z+(b_{d})_{i}))$
are used, where $s(l):=1/(1+\exp\\{-l\\})$ denotes the sigmoid activation
function. For the encoder, we have
$q(z|x)=\frac{\exp\\{z^{\top}(W_{e}x+b_{e})\\}}{\prod_{j=1}^{d_{\mathbb{Z}}}(1+\exp\\{(W_{e})_{j,:}x+(b_{e})_{j}\\})}$
thus $\log
q(z|x)=z^{\top}(W_{e}x+b_{e})-\sum_{j=1}^{d_{\mathbb{Z}}}\log(1+\exp\\{(W_{e})_{j,:}x+(b_{e})_{j}\\})$,
so $\nabla_{x}\nabla_{z}^{\top}\log q(z|x)=W_{e}^{\top}$. Similarly for the
decoder, we have $\nabla_{x}\nabla_{z}^{\top}\log p(x|z)=W_{d}$. So the
compatibility loss Eq. (6) in this case is $\left\lVert
W_{d}-W_{e}^{\top}\right\rVert_{F}^{2}$, which leads to $W_{d}=W_{e}^{\top}$
when it is zero. This recovers the tied weight trick.

In this Bernoulli case, the CAE regularizer is
$\mathbb{E}_{p^{*}(x)}\sum_{j=1}^{d_{\mathbb{Z}}}\frac{\exp\\{-2((W_{e})_{j,:}x+(b_{e})_{j})\\}\sum_{i=1}^{d_{\mathbb{X}}}(W_{e})_{ji}^{2}}{\big{(}1+\exp\\{-((W_{e})_{j,:}x+(b_{e})_{j})\\}\big{)}^{4}}=\mathbb{E}_{p^{*}(x)}\sum_{j=1}^{d_{\mathbb{Z}}}s((W_{e})_{j,:}x+(b_{e})_{j})^{2}\big{(}1-s((W_{e})_{j,:}x+(b_{e})_{j})\big{)}^{2}\sum_{i=1}^{d_{\mathbb{X}}}(W_{e})_{ji}^{2}$
and DAE does not have the Jacobian-norm regularization explanation, so they
are different from the compatibility loss.

###  D.2 Gradient estimation for flow-based models without tractable inverse

Flow-based density models. As the insight we draw from the analysis on
Gaussian VAE in Sec. 3.1, it is inappropriate to implement both conditionals
$p_{\theta}(x|z)$, $q_{\phi}(z|x)$ using additive Gaussian models. So we need
more flexible and expressive probabilistic models that also allow explicit
density evaluation (so implicit models like GANs are not suitable). Flow-based
models [24, 60, 44, 8, 18] are a good choice. They also allow direct sampling
with reparameterization for efficiently estimating and optimizing the data-
fitting loss Eq. (9) (for which energy-based models are costly), and have been
used as the inference model $q_{\phi}(z|x)$ of VAEs [66, 46, 80, 31]. For a
connection to these prior works, we use a flow-based model also for the
inference model $q_{\phi}(z|x)$. An additive-Gaussian likelihood model
$p_{\theta}(x|z)$ is then allowed for learning nonlinear representations.

To define the distribution $q_{\phi}(z|x)$, a flow-based model uses a
parameterized _invertible_ differentiable transformation $z=T_{\phi}(e|x)$ to
map a random seed $e$ (of the same dimension $d_{\mathbb{Z}}$) following a
simple base distribution $p(e)$ 111111 Although some flow-based models (_e.g._
, the Sylvester flow [80]) also incorporate the dependency on $x$ in the base
distribution ${\tilde{p}}_{\phi}({\tilde{e}}|x)$ (_e.g._ ,
$\mathcal{N}({\tilde{e}}|\mu_{\phi}(x),\Sigma_{\phi}(x))$), we can
reparameterize this distribution as transformed from a “more basic” parameter-
free base distribution $p(e)$ (_e.g._ , $\mathcal{N}(0,I_{d_{\mathbb{Z}}})$)
by an $x$-dependent transformation (_e.g._ ,
${\tilde{e}}=\mu_{\phi}(x)+\Sigma_{\phi}(x)^{1/2}e$) and concatenate this
transformation to the original one as $z=T_{\phi}(e|x)$. (_e.g._ , a standard
Gaussian) to $\mathbb{Z}=\mathbb{R}^{d_{\mathbb{Z}}}$. By deliberately
designed architectures, the transformation $T_{\phi}(\cdot|x)$ is guaranteed
to be invertible, yet still being expressive, with some examples that are even
universal approximators [78]. Benefited from the invertibility, the defined
density can be explicitly given by the rule of change of variables [13, Thm.
17.2]:

$\displaystyle
q_{\phi}(z|x)=p(e=T^{-1}_{\phi}(z|x))\left\lvert\nabla_{z}T^{-\top}_{\phi}(z|x)\right\rvert,$
(80)

where $\left\lvert\nabla_{z}T^{-\top}_{\phi}(z|x)\right\rvert$ is the absolute
value of the determinant of the Jacobian of $T^{-1}_{\phi}(z|x)$ (w.r.t $z$).

Problem for evaluating the compatibility loss. Although $T_{\phi}(z|x)$ is
guaranteed to be invertible, in common instances computing its inverse is
intractable [66, 46, 80] or costly [31, 8, 18] (however, they all guarantee an
easy calculation of the Jacobian determinant
$\left\lvert\nabla_{z}T^{-\top}_{\phi}(z|x)\right\rvert$ for efficient density
evaluation). This means that density estimation of $q_{\phi}(z|x)$ is
intractable for an arbitrary $z$ value, but is only possible for a generated
$z$ value, whose inverse $e$ is known in advance (the generated $z$ is
computed from this $e$). This however, introduces problems when computing the
gradients $\nabla_{x}\log q_{\phi}(z|x)$, $\nabla_{z}\log q_{\phi}(z|x)$ for
the compatibility loss (Eq. (6) or Eq. (7)).

To see this, it is important to distinguish the “formal arguments” and “actual
arguments” of a function. It makes a difference when taking derivatives if the
actual arguments are fed to formal arguments in an involved way. What we need
is the derivatives w.r.t the formal arguments, but automatic differentiation
tools (_e.g._ , the autograd utility in PyTorch [61]) could only compute the
derivatives w.r.t the actual arguments. We use capital subscripts for formal
arguments and lowercase letters for actual arguments. Following this rule, we
denote $\log q^{\phi}_{Z|X}(z|x)$ for $\log q_{\phi}(z|x)$ above, so
$\nabla_{Z}\log q^{\phi}_{Z|X}$ denotes the gradient function that
differentiates the first formal argument $Z$ of $\log q^{\phi}_{Z|X}$, and
similarly for $\nabla_{X}\log q^{\phi}_{Z|X}$. Then at a generated value of
$z=T_{\phi}(e|x)$ from a random seed $e$, the required gradients in the
compatibility loss are w.r.t to the formal arguments $\nabla_{Z}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)$ and $\nabla_{X}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)$, while automatic differentiation tools could
only give the gradients w.r.t the actual arguments $\nabla_{e}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)$ and $\nabla_{x}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)$, which are not the desired gradients. Note
that we do not know the exact calculation rule of $\log q^{\phi}_{Z|X}(z|x)$
for arbitrary $z$ and $x$, but can only evaluate $h^{\phi}(e,x):=\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)$ from a given $e$ and $x$. Automatic
differentiation could only evaluate the gradients of this $h^{\phi}(e,x)$ but
not of $\log q^{\phi}_{Z|X}(z|x)$.

Solution. An explicit deduction is thus required for an expression of the
correct gradients in terms of what automatic differentiation could evaluate.
From the chain rule, we have:

$\displaystyle\nabla_{e}h^{\phi}(e,x)=\nabla_{e}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)={}$
$\displaystyle\big{(}\nabla_{e}T_{\phi}^{\top}(e|x)\big{)}\big{(}\nabla_{Z}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)\big{)},$ (81)
$\displaystyle\nabla_{x}h^{\phi}(e,x)=\nabla_{x}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)={}$
$\displaystyle\big{(}\nabla_{x}T_{\phi}^{\top}(e|x)\big{)}\big{(}\nabla_{Z}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)\big{)}$ (82) $\displaystyle{}+\nabla_{X}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x).$ (83)

The first equation gives one of the desired gradients: $\nabla_{Z}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)=\big{(}\nabla_{e}T_{\phi}^{\top}(e|x)\big{)}^{-1}\big{(}\nabla_{e}h^{\phi}(e,x)\big{)}$.
The term $\nabla_{e}h^{\phi}(e,x)$ can be evaluated using automatic
differentiation, as mentioned. The other term, _i.e._ the Jacobian
$\nabla_{e}T_{\phi}^{\top}(e|x)$, can also use automatic differentiation by
tracking the forward flow computation $z=T_{\phi}(e|x)$, but it is often
available in closed-form for flow-based models, as flow-based models need to
evaluate its determinant anyway so the architecture is designed to give its
closed-form expression.

The second equation gives an expression of the other desired gradient:
$\nabla_{X}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)=\nabla_{x}h^{\phi}(e,x)-\big{(}\nabla_{x}T_{\phi}^{\top}(e|x)\big{)}\big{(}\nabla_{Z}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)\big{)}$. Again, the first term
$\nabla_{x}h^{\phi}(e,x)$ can be evaluated using automatic differentiation.
The term $\big{(}\nabla_{Z}\log q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)\big{)}$ can be
evaluated using the expression we just derived above. For the rest term,
_i.e._ the Jacobian $\nabla_{x}T_{\phi}^{\top}(e|x)$, it can also be evaluated
using automatic differentiation by tracking the forward flow computation
$z=T_{\phi}(e|x)$. For computation efficiency, this can be implemented by
taking the gradient of $z=T_{\phi}(e|x)$ w.r.t $x$ with the grad_outputs
argument of torch.autograd.grad fed by $\nabla_{Z}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)$ (gradients w.r.t $x$ will not be back-
propagated through this $\nabla_{Z}\log q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)$).
This reduces computation complexity from $O(d_{\mathbb{X}}d_{\mathbb{Z}})$
down to $O(d_{\mathbb{X}}+d_{\mathbb{Z}})$. In summary, the desired gradients
can be computed via the following expressions:

$\displaystyle\nabla_{Z}\log q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)={}$
$\displaystyle\big{(}\nabla_{e}T_{\phi}^{\top}(e|x)\big{)}^{-1}\big{(}\nabla_{e}h^{\phi}(e,x)\big{)},$
(84) $\displaystyle\nabla_{X}\log q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)={}$
$\displaystyle\nabla_{x}h^{\phi}(e,x)-\big{(}\nabla_{x}T_{\phi}^{\top}(e|x)\big{)}\big{(}\nabla_{Z}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)\big{)}.$ (85)

Second-order differentiations for the compatibility loss can be done in a
similar way.

A simplified compatibility loss. For the compatibility loss Eq. (7) in the
form of Hutchinson’s trace estimator, a further simplification is possible.
The loss is given by:

$\displaystyle
C(\theta,\phi)=\mathbb{E}_{\rho(x,z)}\mathbb{E}_{p(\eta_{x})}\left\lVert\nabla_{Z}\big{(}\eta_{x}^{\top}\nabla_{X}\log
p^{\theta}_{X|Z}(x|z)-\eta_{x}^{\top}\nabla_{X}\log
q^{\phi}_{Z|X}(z|x)\big{)}\right\rVert_{2}^{2},$ (86)

with any random vector $\eta_{x}$ satisfying
$\mathbb{E}[\eta_{x}]=0,\operatorname{Var}[\eta_{x}]=I_{d_{\mathbb{X}}}$. The
reference distribution $\rho(x,z)$ can be taken as $p^{*}(x)q_{\phi}(z|x)$ for
practical sampling for estimating the expectation. For a flow-based
$q_{\phi}(z|x)$, sampling from $(x,z)\sim p^{*}(x)q_{\phi}(z|x)$ is equivalent
to $(x,T_{\phi}(e|x)),e\sim p(e)$. So the loss can be reformulated as:

$\displaystyle
C(\theta,\phi)=\mathbb{E}_{p^{*}(x)p(e)}\mathbb{E}_{p(\eta_{x})}\left\lVert\nabla_{Z}\big{(}\eta_{x}^{\top}\nabla_{X}\log
p^{\theta}_{X|Z}(x|T_{\phi}(e|x))-\eta_{x}^{\top}\nabla_{X}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)\big{)}\right\rVert_{2}^{2}.$ (87)

Note from Eq. (84), the gradient w.r.t $Z$ is an invertible linear
transformation of the gradient w.r.t $e$, so its norm equals zero if and only
if the gradient w.r.t $e$ has a zero norm. So to avoid this matrix inversion,
we consider a simpler loss that achieves the same optimal solution:

$\displaystyle{\tilde{C}}(\theta,\phi):=\mathbb{E}_{p^{*}(x)p(e)}\mathbb{E}_{p(\eta_{x})}\left\lVert\nabla_{e}\big{(}\eta_{x}^{\top}\nabla_{X}\log
p^{\theta}_{X|Z}(x|T_{\phi}(e|x))-\eta_{x}^{\top}\nabla_{X}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)\big{)}\right\rVert_{2}^{2}.$ (88)

For additive Gaussian $p^{\theta}_{X|Z}$, its gradient $\nabla_{X}\log
p^{\theta}_{X|Z}$ is available in closed-form. For $\nabla_{X}\log
q^{\phi}_{Z|X}(T_{\phi}(e|x)|x)$ in the second term, it can be estimated using
Eq. (85) we just developed. The subsequent gradient w.r.t $e$ can be evaluated
by automatic differentiation. So this loss is tractable to estimate and
optimize.

##  Appendix E Experiment Details

###  E.1 Baseline Methods

We compare our proposed CyGen with bi-directional models (composed of both a
likelihood model and an inference model) including Denoising Auto-Encoder
(DAE), Variational Auto-Encoder (VAE) and BiGAN. Sketches of these models are
introduced below.

DAE [82] first corrupts a real input data $x$ with a local noise and then pass
it through an encoder to define $q_{\phi}(z|x)$. The latent code $z$ is then
decoded to the data space by a decoder $p_{\theta}(x|z)$. The objective is to
minimize the reconstruction error (Eq. (61)):
$\mathbb{E}_{p^{*}(x)}\mathbb{E}_{q_{\phi}(z|x)}[-\log p_{\theta}(x|z)]$.
Compared with VAE, it can be seen as the version with $\beta=0$, _i.e._ it
does not involve a prescribed prior $p(z)$. Nevertheless, optimizing the
objective w.r.t $\phi$ may lead to undesired results. Particularly, for any
given $x$, it may drive $q_{\phi}(z|x)$ to only concentrate on $z$ values that
maximizes $\log p_{\theta}(x|z)$. This renders incompatibility and an
insufficient determinacy (see Sec. 3.2).

VAE [45] defines a joint distribution $p_{\theta}(x,z)=p(z)p_{\theta}(x|z)$
using a specified prior $p(z)$. It learns $p_{\theta}(x|z)$ to match data
distribution $p^{*}(x)$ with the help of an inference model $q_{\phi}(z|x)$,
using the Evidence Lower BOund (ELBO) objective:

$\displaystyle\min_{\theta,\phi}\mathbb{E}_{p^{*}(x)}\mathbb{E}_{q_{\phi}(z|x)}\left[-\log
p_{\theta}(x|z)\right]+\beta\mathbb{E}_{p^{*}(x)}[\mathrm{KL}(q_{\phi}(z|x)\|p(z))].$
(89)

When $\beta=1$, the negative objective is a lower bound of the data likelihood
(evidence) $\mathbb{E}_{p^{*}(x)}[\log p_{\theta}(x)]$ where
$p_{\theta}(x):=\int_{\mathbb{Z}}p(z)p_{\theta}(x|z)\mathrm{d}z$, hence the
name. Optimizing it w.r.t $\phi$ also drives $q_{\phi}(z|x)$ to the true
posterior $p_{\theta}(z|x)$ and also makes the bound tighter. A $\beta$ other
than 1 is considered when there is some desideratum on the latent variable,
_e.g._ , disentanglement [37].

BiGAN [25, 27]. In addition to learning the data distribution $p^{*}(x)$ using
GAN [30], BiGAN also aims to learn a representation extractor, so it
introduces an inference model $q_{\phi}(z|x)$ which is often deterministic
(_i.e._ , a Dirac distribution). The likelihood model (generator)
$p_{\theta}(x|z)$ defines a joint $p(z)p_{\theta}(x|z)$ with the help of a
prescribed prior $p(z)$, while the inference model also defines a joint
$p^{*}(x)q_{\phi}(z|x)$. Samples from both distributions can be easily drawn,
so BiGAN seeks to match them using the GAN loss (Jensen-Shannon divergence)
with the help of a discriminator $D(x,z)$. In each training step, the
discriminator $D(x,z)$ is first updated on a mini-batch of
$p^{*}(x)q_{\phi}(z|x)$ data $x^{+}\sim p^{*}(x),z^{+}\sim
q_{\phi}(\cdot|x^{+})$ with positive labels $y^{+}=1$ and a mini-batch of
$p(z)p_{\theta}(x|z)$ data $z^{-}\sim p(z),x^{-}\sim p_{\theta}(\cdot|z^{-})$
with negative labels $y^{-}=0$. The goal of the training the discriminator is
to minimize the binary cross entropy loss
$\text{BCE}(D(x^{+},z^{+}),y^{+})+\text{BCE}(D(x^{-},z^{-}),y^{-})$. The
conditional models $q_{\phi}(z|x)$ and $p_{\theta}(x|z)$ are then updated to
maximize the same loss
$\text{BCE}(D(x^{+},z^{+}),y^{+})+\text{BCE}(D(x^{-},z^{-}),y^{-})$.

GibbsNet [50] is also considered, which is similar to BiGAN, except that
BiGAN’s prior-driven joint sample generation $z^{-}\sim p(z),x^{-}\sim
p(x|z^{-})$ is modified to run through multiple cycles: $z_{0}\sim
p(z),x_{0}\sim p(x|z_{0}),z_{1}\sim q(z|x_{0}),x_{1}\sim
p(x|z_{1}),\cdots,z^{-}\sim q(z|x_{K-1}),x^{-}\sim p(x|z^{-})$. This resembles
a Gibbs chain, and is considered in GibbsNet for reducing the influence of a
specified prior, as the stationary distribution of the Markov chain does not
rely on the initial distribution but is determined by the two conditional
models (see Sec. 1.1 (paragraph 2) for its limitation). But this iterated
application of the likelihood and inference models makes gradient back-
propagation involved. The gradient accumulates with the cycling iteration,
resulting in a scale much larger than usual. This makes gradient-based
optimization unstable and even leads to numerical instability. We did not find
a reasonable result in any experiment using the same architecture so we omit
the comparison.

###  E.2 Model Architecture

Figure 12: Flow architecture of the inference model $q_{\phi}(z|x)$. See Table
16 for detailed specification.

Our code is developed based on the repositories of the Sylvester
flow121212https://github.com/riannevdberg/sylvester-flows [80] and
FFJORD131313https://github.com/rtqichen/ffjord [31] for the task environment
and flow architectures. VAE, DAE and CyGen share the same architecture of
$p_{\theta}(x|z)$ and of $q_{\phi}(z|x)$, which are detailed in Table 16. The
inference model $q_{\phi}(z|x)$ adopts the architecture of Sylvester flow
[80], illustrated in Fig. 12. It consists of a neural network (denoted as
C-QNN) that outputs $q_{\text{nn}}$, a reparameterization module, and a set of
consecutive $N$ flows. The outputs $q_{\mu}$ and $q_{\sigma}$ are used to
parameterize the diagonal Gaussian distribution for initializing $z_{0}$, the
input to the flows. For implementation simplicity, we choose the Householder
version of the Sylvester flow. For each flow layer, the output
$\mathbf{z}_{t}$ of the flow given input $\mathbf{z}_{t-1}$ is:

$\displaystyle\mathbf{z}_{t}=\mathbf{z}_{t-1}+\mathbf{A}_{t}h(\mathbf{B}_{t}\mathbf{z}_{t-1}+\mathbf{b}_{t}),$
(90)

where $\mathbf{A}_{t}=\mathbf{Q}_{t}\mathbf{R}_{t}$,
$\mathbf{B}_{t}=\tilde{\mathbf{R}}_{t}\mathbf{Q}_{t}$ and $\mathbf{b}_{t}$ are
parameters of the $t$-th flow and $h$ is the hyperbolic-tangent activation
function. Let $\mathbf{A}=\mathbf{Q}\mathbf{R}$,
$\mathbf{B}=\tilde{\mathbf{R}}\mathbf{Q}^{\top}$, where $\mathbf{R}$ and
$\tilde{\mathbf{R}}$ are upper triangular matrices, and
$\mathbf{Q}=\prod_{i=1}^{H}(\mathbf{I}-\frac{2\mathbf{v}_{i}\mathbf{v}_{i}^{\top}}{\mathbf{v}_{i}^{\top}\mathbf{v}_{i}})$
is a sequence of $H=8$ Householder transformations. All the flow parameters,
_i.e._ $\mathbf{v}_{1:H}$, $\mathbf{b}$, $\mathbf{R}$ and
$\tilde{\mathbf{R}}$, depend on $q_{\text{nn}}$ via a flow-parameter block.

###  E.3 Synthetic Experiments

The synthetic datasets (“pinwheel” in the main text and “8gaussians” in this
appendix) are adopted from the above mentioned repositories of the Sylvester
flow and FFJORD. The dimension of the data space is $d_{\mathbb{X}}=2$, and we
take the latent space to be of the same dimension $d_{\mathbb{Z}}=2$.

For the inference model $q_{\phi}(z|x)$, we use a three-layer MLP for the
C-QNN component, with 8 hidden nodes in each layer. After the
reparameterization block, consecutive $N=32$ Householder flow layers are
concatenated. Each flow layer has $H=2$ Householder transformations141414To
make an invertible transformation, the number of Householder transformations
$H$ needs to be no larger than the dimension of $z$, which is 2 in this
synthetic experiment. For the likelihood model $p_{\theta}(x|z)$, it is
implemented as an additive Gaussian model. Its mean function is a three-layer
MLP with 16 hidden nodes in each layer, and its variance is taken isotropic
with fixed scale $0.01$.

For training, we use the Adam optimizer [43] with batch-size 1000 and weight
decay parameter $1\\!\times\\!10^{-5}$ for all methods. All methods use a
learning rate of $1\\!\times\\!10^{-3}$ except for DAE which uses
$1\\!\times\\!10^{-4}$. For BiGAN, the generator is updated once per 128
updates of the discriminator using step size $1\\!\times\\!10^{-4}$. For
CyGen, conditional models $p_{\theta}(x|z)$ and $q_{\phi}(z|x)$ are trained by
minimizing: $1\\!\times\\!10^{-5}\\!\times\\!{}$ compatibility loss Eq. (88)
${}+{}$ data-fitting loss Eq. (9), where the expectation in the data-fitting
loss is estimated using 16 samples from $q_{\phi}(z|x)$ with
reparameterization [45]. For the version CyGen(PT) with PreTraining, the
conditional models are first pretrained as in a VAE by minimizing the ELBO
objective Eq. (89) (with $\beta=1$) using the standard Gaussian prior for 1000
epochs, and are then trained as in CyGen by minimizing the above objective
with a $10$-times smaller learning rate for the likelihood model
$p_{\theta}(x|z)$ (same learning rate for the inference model
$q_{\phi}(z|x)$). DAE is also pretrained in this way.

For data generation, VAE and BiGAN use ancestral sampling: first draw a sample
of $z$ from the standard Gaussian prior $p(z)$, and then draw a data sample
$x$ from the likelihood model $p_{\theta}(x|z)$. For DAE and CyGen, they do
not have a prior model for ancestral sampling. They use MCMC methods,
including Gibbs sampling and SGLD (see Sec. 3.3). One difference for the
synthetic experiment is that the SGLD version is done by passing through the
likelihood model $p_{\theta}(x|z)$ with prior samples drawn via SGLD in the
latent space $\mathbb{Z}$ similar to Eq. (10):

$\displaystyle\textstyle\\!\\!z^{(t+1)}=z^{(t)}+\varepsilon\nabla_{z^{(t)}}\log\frac{q_{\phi}(z^{(t)}|x^{(t)})}{p_{\theta}(x^{(t)}|z^{(t)})}+\sqrt{2\varepsilon}\,\eta^{(t)}_{z},\text{where
}x^{(t)}\\!\sim
p_{\theta}(x|z^{(t)}),\eta^{(t)}_{z}\\!\sim\mathcal{N}(0,I_{d_{\mathbb{Z}}}),$
(91)

and $\varepsilon$ is a step size parameter, taken as $3\\!\times\\!10^{-4}$.
Both Gibbs sampling and SGLD are run for 100 iterations (transition steps).
Their comparison for CyGen is shown in Fig. 5, where SGLD is much better. For
DAE, using Gibbs sampling and using SGLD produce similar data generation
results.

We show more results next.

iteration 1000 VAE pretraining compt. loss $1.6\\!\times\\!10^{4}$ | iteration | 1100 | 1200 | 1300 | 1400 | 30000
---|---|---|---|---|---|---
CyGen(PT) |  |  |  |  |
compt. loss | $7.0\\!\times\\!10^{3}$ | $5.4\\!\times\\!10^{3}$ | $7.7\\!\times\\!10^{3}$ | $6.2\\!\times\\!10^{3}$ | $4.6\\!\times\\!10^{3}$
CyGen(PT) w/o compt. loss |  |  |  |  |
compt. loss | $1.1\\!\times\\!10^{5}$ | $1.6\\!\times\\!10^{5}$ | $2.6\\!\times\\!10^{5}$ | $8.6\\!\times\\!10^{5}$ | $1.2\\!\times\\!10^{8}$
| DAE |  |  |  |  |
| compt. loss | $6.3\\!\times\\!10^{3}$ | $2.2\\!\times\\!10^{3}$ | $1.9\\!\times\\!10^{3}$ | $9.7\\!\times\\!10^{2}$ | $2.2\\!\times\\!10^{2}$
Table 13: Generated data using $\mathbb{Z}$-space SGLD (Eq. (91)) along the
training process after VAE pretraining (iter. 1000) using CyGen(PT), CyGen(PT)
without compatibility loss, and DAE. The last DAE result is at iteration 9200,
after which numerical overflow occurs.

Impact of the compatibility loss. Fig. 13 shows the training process of
CyGen(PT), CyGen(PT) without compatibility loss, and DAE, all after pretrained
as a VAE (iter. 1000; leftmost panel), in terms of generated data
distribution. We see that the normal CyGen(PT) behaves stably to the end and
well approximates the data distribution along the training process. Its
compatibility loss is indeed decreasing. On the other hand, CyGen(PT) without
the compatibility loss diverges eventually, with an exploding compatibility
loss. Although it well optimizes the data-fitting loss Eq. (9), if
compatibility is not enforced, the loss is not the data likelihood that we
want to optimize.

Note that CyGen(PT) without compatibility loss improves generation quality
upon the VAE-pretrained model in the first few training iterations (_e.g._ ,
at iter. 1100). This is because the ELBO objective (Eq. (89)) of VAE also
drives $q_{\phi}(z|x)$ towards the true posterior $p_{\theta}(z|x)\propto
p(z)p_{\theta}(x|z)$ (defined with a specified prior $p(z)$) so compatibility
approximately holds in the first few iterations, which makes the data-fitting
loss (Eq. (9)) effective.

The collapse process of DAE. Fig. 13 (rows 1,3) also shows the comparison with
DAE. We see that after pretraining, DAE quickly shrinks its data distribution,
ending up with a collapsed data distribution, and even finally comes to a
numerical problem. This is due to the mode-collapse behavior of the inference
model $q_{\phi}(z|x)$ from minimizing the DAE loss Eq. (61) and the subsequent
insufficient determinacy, as explained in Sec. 3.2. Although its compatibility
loss is also decreasing, this comes at the cost of the insufficient
determinacy which hinders capturing the data distribution and also makes the
training process unstable.

VAE | CyGen | CyGen(PT)
---|---|---
|  |
Table 14: Prior distributions $p(z)$ of VAE, CyGen (without pretraining), and
CyGen(PT) (with VAE pretraining). Prior samples of CyGen/CyGen(PT) are drawn
using $\mathbb{Z}$-space SGLD (Eq. (91)).

Incorporating knowledge into the conditionals. We plot the prior distributions
in Fig. 14 of VAE, CyGen, and CyGen(PT) with VAE pretraining, in the form of
the histogram of the drawn $z$ samples. For CyGen/CyGen(PT), samples are drawn
by $\mathbb{Z}$-space SGLD (Eq. (91)) using the same step size
$\varepsilon=3\\!\times\\!10^{-4}$ and number of iterations 100. Compared with
VAE, the priors learned by CyGen and CyGen(PT) are more expressive. For CyGen
which is not subjected to any further constraints, there may be multiple
$p_{\theta}(x|z)$ and $q_{\phi}(z|x)$ that are compatible and well match the
given data distribution. Using a standard Gaussian prior for pretraining
successfully incorporates the knowledge of a centered and centrosymmetric
prior into the conditional model $p_{\theta}(x|z)$. The arbitrariness of
possible $p_{\theta}(x|z)$ and $q_{\phi}(z|x)$ is largely mitigated in this
way. This observation meets the discussion in Sec. 4.1 (paragraph 4) on the
aggregated posteriors in Fig. 4.

data | DAE | VAE | BiGAN | CyGen | CyGen(PT)
---|---|---|---|---|---
|  |  |  |  |
class-wise aggregated posterior |  |  |  |  |
Table 15: Generated data (DAE and CyGen use $\mathbb{Z}$-space SGLD Eq. (91))
and class-wise aggregated posteriors of DAE, VAE, BiGAN and CyGen, on the
“8gaussians” dataset. Also shows results of CyGen(PT) that is PreTrained as a
VAE. (Best view in color.)

Results on another similar dataset (8gaussians). We repeat all the settings
above (for the “pinwheel” dataset) on another similar synthetic dataset
“8gaussians” (Fig. 15 top-left). Its data distribution also has a (nearly)
non-connected support, with 8 connected components. The only difference in
settings is that BiGAN uses a learning rate $1\\!\times\\!10^{-4}$ for
updating its generator and $3\\!\times\\!10^{-5}$ for its discriminator. We
see similar observations from the results in Fig. 15.

For data generation, DAE again produces a collapsed data distribution. VAE’s
data distribution is blurred and the 8 clusters are touching, making the
support connected due to using the standard Gaussian prior. BiGAN is unstable
on this dataset and does not produce a reasonable data distribution, despite
some parameter tuning. In contrast, CyGen and CyGen(PT) still recover the data
distribution faithfully, with clear 8 clusters. Again, the advantage to
overcome manifold mismatch is demonstrated.

For representation learning, DAE again collapses the class-wise aggregated
posteriors of all classes and puts them in the same place. VAE identifies the
latent clusters but which are compressed to the origin and squeezed together
to border each other. BiGAN’s aggregated posteriors of different classes
largely overlap each other. In contrast, CyGen and CyGen(PT) separate the
latent clusters clearly. CyGen(PT) additionally embodies the knowledge from
the VAE-pretrained likelihood model that the prior hence the (all-class)
aggregated posterior is centered and centrosymmetric, without suffering
fitting the data distribution.

In all, these observations again verify that CyGen achieves both superior
generation and representation learning performances.

###  E.4 Real-World Experiments

Model architecture. All methods use the same architecture of the inference
model (encoder) and the likelihood model (decoder), illustrated in Fig. 12 and
detailed in Table 16 (except for the reported results from [50] of BiGAN and
GibbsNet listed in Table 7, which are around random guess using the same flow
architecture). The Gaussian variance of the likelihood model $p_{\theta}(x|z)$
is selected to be $0.01$ for all dimensions. All methods use the Adam
optimizer [43] with learning rate $1\\!\times\\!10^{-4}$ and batch-size 100
for 100 epochs.

Data generation. VAE uses ancestral sampling, and DAE uses its standard Gibbs
sampling procedure initialized from
$z_{0}\sim\mathcal{N}(0,I_{d_{\mathbb{Z}}})$. CyGen(PT) generates data by
running $\mathbb{X}$-space SGLD (Eq. (10)) with step size
$\varepsilon=1\\!\times\\!10^{-3}$ initialized from $x_{0}\sim
p_{\theta}(\cdot|z_{0})$ where $z_{0}\sim\mathcal{N}(0,I_{d_{\mathbb{Z}}})$.

Downstream classification. For the downstream classification task on the
latent space $\mathbb{Z}$, we sample one latent representation $z$ for each
data point $x$ directly from the learned inference model $q_{\phi}(z|x)$, and
then train a 2-layer MLP classifier with 10 hidden nodes on top of the latent
representation. Results of DAE, VAE and CyGen(PT) in Table 7 are averaged over
10 random trials. All downstream classifiers are trained for 100 epochs.

Training strategy of our method. CyGen(PT) first pretrains its likelihood and
inference models as in a VAE using the ELBO objective Eq. (89) (with $\beta=1$
on MNIST and $\beta=0.01$ on SVHN) for 100 epochs, and then trains them using
CyGen ($1\\!\times\\!10^{-3}\\!\times\\!$ compatibility loss Eq. (88) ${}+{}$
data-fitting loss Eq. (9)) with a $10$-times smaller learning rate for the
likelihood model (same learning rate for the inference model). On SVHN, we
clamp the standard Gaussian random seed $e$ in the reparameterization step
that initializes $z_{0}$ to be within the interval $[-0.1,0.1]$ (element-
wise).

Remark on the VAE pretraining. For real-world images, the optimization process
of CyGen from a cold start is unstable, possibly because the data distribution
roughly concentrates on a low-dimensional space thus is nearly not absolutely
continuous, while the CyGen-defined distribution is absolutely continuous (see
Lem. C.1 in Appx. C.1). Improved techniques to handle this issue are important
future work. Moreover, the VAE pretraining downweighs the effect of the prior
(_i.e._ , using a small $\beta$ on SVHN), so it is approximately a DAE, which
does not show a reasonable result (Fig. 6). So CyGen is the key to the high-
quality results.

Table 16: Inference and likelihood model architectures for MNIST and SVHN Layers | In-Out Size | Stride
---|---|---
Inference Model $q_{\phi}(z|x)$ Architecture for MNIST-C-QNN
Input $x$ | $1\\!\times\\!28\\!\times\\!28$ |
$5\\!\times\\!5$ GatedConv2d (32), Sigmoid | $32\\!\times\\!28\\!\times\\!28$ | 1
$5\\!\times\\!5$ GatedConv2d (32), Sigmoid | $32\\!\times\\!14\\!\times\\!14$ | 2
$5\\!\times\\!5$ GatedConv2d (64), Sigmoid | $64\\!\times\\!14\\!\times\\!14$ | 1
$5\\!\times\\!5$ GatedConv2d (64), Sigmoid | $64\\!\times\\!7\\!\times\\!7$ | 2
$5\\!\times\\!5$ GatedConv2d (64), Sigmoid | $64\\!\times\\!7\\!\times\\!7$ | 1
$7\\!\times\\!7$ GatedConv2d (256), Sigmoid | $256\\!\times\\!1\\!\times\\!1$ | 1
Output $q_{\text{nn}}$, squeeze | 256 |
Inference Model $q_{\phi}(z|x)$ Architecture for SVHN-C-QNN
Input $x$ | $3\\!\times\\!32\\!\times\\!32$ |
$5\\!\times\\!5$ Conv2d (32), LReLU | $32\\!\times\\!28\\!\times\\!28$ | 1
$4\\!\times\\!4$ Conv2d (64), LReLU | $64\\!\times\\!13\\!\times\\!13$ | 2
$4\\!\times\\!4$ Conv2d (128), LReLU | $128\\!\times\\!10\\!\times\\!10$ | 1
$4\\!\times\\!4$ Conv2d (256), LReLU | $256\\!\times\\!4\\!\times\\!4$ | 2
$4\\!\times\\!4$ Conv2d (512), LReLU | $512\\!\times\\!1\\!\times\\!1$ | 1
$4\\!\times\\!4$ Conv2d (256), Sigmoid | $256\\!\times\\!1\\!\times\\!1$ | 1
Output $q_{\text{nn}}$, squeeze | 256 |
Reparameterization Block for $q_{\phi}(z|x)$ for MNIST and SVHN
Input $q_{\text{nn}}$ | 256 |
Output-1 $q_{\mu}$: Linear $256\\!\times\\!64$ | 64 |
Draw $e\sim\mathcal{N}(0,I_{d_{\mathbb{Z}}})$ and output $z_{0}=q_{\mu}+e\odot q_{\sigma}$ | 64 |
Flow-Parameter Block for $q_{\phi}(z|x)$ for MNIST and SVHN
Input $q_{\text{nn}}$ | 256 |
Output-1 $\mathbf{v}_{1:8}$: Linear $256\\!\times\\!512$ | 512 |
Output-2 $\mathbf{b}$: Linear $256\\!\times\\!8$ | 512 |
Output-3 $\mathbf{R}$: Linear $256\\!\times\\!(64\\!\times\\!64)$ | $(64\\!\times\\!64)$ |
Output-4 $\tilde{\mathbf{R}}$: Linear $256\\!\times\\!(64\\!\times\\!64)$ | $(64\\!\times\\!64)$ |
Likelihood Model $p_{\theta}(z|x)$ Architecture for MNIST
Input $z$ | $64\\!\times\\!1\\!\times\\!1$ |
$7\\!\times\\!7$ GatedConvT2d (64), Sigmoid | $64\\!\times\\!7\\!\times\\!7$ | 1
$5\\!\times\\!5$ GatedConvT2d (64), Sigmoid | $64\\!\times\\!7\\!\times\\!7$ | 1
$5\\!\times\\!5$ GatedConvT2d (64), Sigmoid | $64\\!\times\\!14\\!\times\\!14$ | 2
$5\\!\times\\!5$ GatedConvT2d (32), Sigmoid | $32\\!\times\\!14\\!\times\\!14$ | 1
$5\\!\times\\!5$ GatedConvT2d (32), Sigmoid | $32\\!\times\\!28\\!\times\\!28$ | 2
$5\\!\times\\!5$ GatedConvT2d (32), Sigmoid | $32\\!\times\\!28\\!\times\\!28$ | 1
$1\\!\times\\!1$ GatedConv2d (1), Sigmoid | $1\\!\times\\!28\\!\times\\!28$ | 2
Output $x$ | $1\\!\times\\!28\\!\times\\!28$ |
Likelihood Model $p_{\theta}(z|x)$ Architecture for SVHN
Input $z$ | $64\\!\times\\!1\\!\times\\!1$ |
$4\\!\times\\!4$ ConvT2d (256), LReLU | $256\\!\times\\!4\\!\times\\!4$ | 1
$4\\!\times\\!4$ ConvT2d (128), LReLU | $128\\!\times\\!10\\!\times\\!10$ | 1
$4\\!\times\\!4$ ConvT2d (64), LReLU | $64\\!\times\\!13\\!\times\\!13$ | 1
$4\\!\times\\!4$ ConvT2d (32), LReLU | $32\\!\times\\!28\\!\times\\!28$ | 2
$5\\!\times\\!5$ ConvT2d (32), LReLU | $32\\!\times\\!32\\!\times\\!32$ | 1
$1\\!\times\\!1$ ConvT2d (32), LReLU | $32\\!\times\\!32\\!\times\\!32$ | 1
$1\\!\times\\!1$ Conv2d (32), Sigmoid | $32\\!\times\\!32\\!\times\\!32$ | 1
Output $x$ | $3\\!\times\\!32\\!\times\\!32$ |