# Excheangeability and irreducible rotational invariance

Paolo Baldi, Domenico Marinucci, Stefano Trapani  Department of Mathematics,
University of Rome Tor Vergata

###### Abstract.

In this note we prove that a finite family $\\{X_{1},\dots,X_{d}\\}$ of real
r.v.’s that is exchangeable and such that $(X_{1},\dots,X_{d})$ is invariant
with respect to a subgroup of $SO(d)$ acting irreducibly, is actually
invariant with respect to the action of the full group $SO(d)$. Three
immediate consequences are deduced: a characterization of isotropic spherical
random eigenfunctions whose Fourier coefficients are exchangeable, an
extension of Bernstein’s characterization of the Gaussian and a
characterization of the Lebesgue measure on the sphere.

AMS Classification: 20C30, 20G45, 22E46, 60B15, 60G60, 60G09

Keywords and Phrases: Group Representations, Exchangeability, Invariant Random
Fields, Fourier Coefficients, Random Eigenfunctions, Characterizations of the
Gaussian.

111We acknowledge financial support of the MUR Department of Excellence
Programme MatModTov, and of the Prin 2022 _Grafia_

##  1\. Introduction

In this paper, we investigate characterizations of exchangeable random
variables (r.v.’s) when, in addition, they enjoy some invariance with respect
to a group of rotations.

As is well-known, an infinite sequence $\\{X_{1},X_{2},\dots\\}$ of random
variables on some probability space is exchangeable if and only if for any
permutation $\pi:\mathbb{N}\rightarrow\mathbb{N}$ one has that

$\\{X_{1},\dots,X_{n},\dots\\}\mathop{=}^{Law}\\{X_{\pi(1)},\dots,X_{\pi(n)},\dots\\}.$

Under this condition, the De Finetti-Hewitt-Savage Theorem implies the
existence of a random variable $Y$ on the same probability space such that the
r.v.’s $\\{X_{1},X_{2},\dots\\}$ are conditionally independent given $Y$. The
original statement of De Finetti’s Theorem referred to Bernoulli’s r.v.’s, but
it has been shown to hold for r.v.’s taking values in any Polish space;
research in this area is still active, to the point that it is considered by
some author a foundational result not only in probability/statistics but for
mathematics as a whole (see [1] and the references therein).

While the De Finetti-Hewitt-Savage results hold in great generality for
infinite sequences, it is also well-known that they fail in general for finite
ones. A classical reference by Diaconis and Freedman shows that one can at
most obtain upper bounds in total variation between finite-exchangeable
sequences and mixture of independent random variables; these bounds are sharp,
indeed there exist explicit counterexamples where the bound is saturated, see
[7]. Another celebrated paper by the same authors ([8]) shows that uniform
random vectors on a sphere of growing dimension $n$ any $k$-dimensional set of
coordinates suitably normalized is asymptotically close to Gaussian in total
variation distance, provided that $k=o(n)$ as $n\rightarrow\infty$.

Our purpose here is to investigate some further characterization of
exchangeability for finite-dimensional sequences. In particular, in the next
section we prove an algebraic result which implies that invariance with
respect to a subgroup of $SO(d)$ acting irreducibly in addition to
exchangeability implies invariance with respect to the full group $SO(d)$.

Next, in §3 we deduce properties of the Fourier coefficients of an isotropic
random field on the sphere under the assumption that they are exchangeable. En
passant we obtain a characterization of the Lebesgue measure of the spheres
$\mathbb{S}^{d-1}$ of $\mathbb{R}^{d}$; finally in §4 we give a proof of a
stronger version of Bernstein’s Theorem on the characterization of the
Gaussian.

##  2\. The main decomposition statement

Let as consider the special orthogonal groups $SO(k),SO(N),$ with the
corresponding Lie algebras $so(k),so(N)$; assume $k\geq 3,N\geq 3$. Let

$\phi:SO(k)\to SO(N)$

be a homomorphism, i.e. a representation of $SO(k)$ on $\mathbb{R}^{N}$. Let
$S_{N}$ be the symmetric group of $N$ letters and, for $\sigma\in S_{N}$ let
$E_{\sigma}$ be the permutation matrix corresponding to the permutation
$\sigma$. Then we have the following main result.

######  Theorem 2.1.

Assume that the representation $\phi$ has no one dimensional invariant
subspaces, then the subgroup of $SO(N)$ generated by the elements
$E_{\sigma}^{-1}\phi(g)E_{\sigma}$ with $g\in SO(k)$ and $\sigma\in S_{N}$ is
all of $SO(N)$.

In particular if $\sigma_{0}$ is a fixed permutation of sign $-1$ then the
subgroup of $O(N)$ generated by
$\\{E_{\sigma_{0}},E_{\sigma}^{-1}\phi(g)E_{\sigma},\ \sigma\in S_{N},g\in
SO(k)\\}$ is all of $O(N)$.

###### Proof.

Recall that the Lie algebras $so(k)$ for $k\geq 3,k\neq 4$ are simple, i.e.
they have no non trivial ideals, and that $so(4)$ is the direct sum of two
simple three dimensional ideals (see [9] theorem 12.1, and [12] p. 352).

Let us first prove that, the image of $\phi$ has necessarily dimension $\geq
3$. If $k\geq 3$ and $k\not=4$; since the kernel of the differential
$d\phi_{e}$ ($e$ is the identity element of $SO(k)$) is an ideal of $so(k)$,
then $\ker\phi$ can only be either $so(k)$ or $\\{0\\}$. Since $\phi$ has no
one dimensional invariant subspaces, the first occurrence cannot happen and
$\phi$ is injective. If $k=4$ then $\ker d\phi_{e}=so(4)$ is impossible as
above and $\ker d\phi_{e}$ can only be $\\{0\\}$ or one of the two simple
three dimensional ideals of $so(4)$, as $so(4)$ has no other proper ideals
(see Serre’s book [16], chapter 6). In any case $\dim\ker d\phi_{e}\leq 3$ so
that, as $so(4)$ has dimension $6$, the image of $d\phi_{e}$ is $\geq 3$ in
any case. In particular if $N=3,$ the map $\phi$ is onto, and the proposition
is proved, so we may assume $N\geq 4$.

Next, let $\sigma_{1},\sigma_{2},\ldots,\sigma_{N!}$ be an enumeration of the
permutations in $S_{N},$ and let us consider the map $F:SO(k)^{N!}\to SO(N),$

$F(g_{1},g_{2},\ldots,g_{N!})=(E_{\sigma_{1}}\phi(g_{1})E_{\sigma_{1}}^{-1})(E_{\sigma_{2}}\phi(g_{2})E_{\sigma_{2}}^{-1})\cdots(E_{\sigma_{N!}}\phi(g_{N!})E_{\sigma_{N!}}^{-1})\
.$

Note that, if $e_{2}$ denotes the identity element in $SO(N)$, then
$F(e,\ldots,e)=e_{2}$. Let $W$ be the image of the differential of $F$ at
$(e,\ldots,e)$. Then $W$ is a vector subspace of the tangent space to $SO(N)$
at $e_{2}$ identified with the Lie algebra $so(N)$ which is the space of real
$N\times N$ antisymmetric matrices. By the Leibnitz rule we have:

$W=E_{\sigma_{1}}d\phi_{e}(so(k))E_{\sigma_{1}}^{-1}+(E_{\sigma_{2}})d\phi_{e}(so(k))E_{\sigma_{2}}^{-1}+\cdots+E_{\sigma_{N!}}d\phi_{e}(so(k))E_{\sigma_{N!}}^{-1}.$

Therefore $W$ is a vector subspace of the Lie algebra $so(N)$ which is
invariant under conjugation with respect to any permutation matrix. The
remainder of the proof is concerned with showing that $W=so(N)$. This entails
that that the differential at $(e,\ldots,e)$ of the map $F$ is onto and in
particular the image of $F$ contains a neighborhood of $e$ in $SO(N)$. Since
$SO(N)$ is connected, the group generated by a neighborhood of the identity is
all of $SO(N)$ and this will conclude the proof.

To this goal we shall take advantage of the decomposition of the complexified
Lie algebra $so(N)^{\mathbb{C}}$ under the action of the permutation group
$S_{N}$ that is provided in [5].

We already know that $W$ is invariant under the representation of $S_{N}$
$\Lambda_{\sigma}:A\mapsto E_{\sigma}AE_{\sigma}^{-1},$, $\sigma\in S_{N}$,
$A\in so(N)$.

Let us consider the complexified representation $\Lambda^{\mathbb{C}}$
obtained considering the action $\Lambda_{\sigma}$ on the space
$so(N)^{\mathbb{C}}$ of complex antisymmetric matrices. In [13] it is proved
that, for $N\geq 4$, $\Lambda^{\mathbb{C}}$ decomposes into the direct sum of
two irreducible non isomorphic subrepresentations $V^{\mathbb{C}}_{1},$
$V^{\mathbb{C}}_{2}$ and the authors determine the isomorphism class of each
irreducible component.

In particular $V^{\mathbb{C}}_{1}$ is isomorphic to the so called standard
representation of $S_{N}$ i.e. to the representation of $S_{N}$ by permutation
of the coordinates on the subspace of $\mathbb{C}^{N}$
$\\{z\in\mathbb{C}^{N};z_{1}+\ldots+z_{N}=0\\}$, so that
$\dim(V^{\mathbb{C}}_{1})=N-1$, and, as $\dim(so(N))=\frac{1}{2}\,N(N-1)$,
$\dim(V^{\mathbb{\mathbb{C}}}_{2})=\frac{1}{2}\,(N-1)(N-2)$.

As for $V^{\mathbb{\mathbb{C}}}_{2}$, let
$v_{0}=\,^{t}\\!(1,,\dots,1)\in\mathbb{C}^{N}$, so that $v_{0}$ is fixed under
the action of $S_{N}$. Note that if $v\in\mathbb{C}^{N}$ is in the kernel of
the complex antisymmetric matrix $A$ and $\sigma\in S_{N},$ then $E_{\sigma}v$
is in the kernel of
$\Lambda_{\sigma}^{\mathbb{C}}A=E_{\sigma}AE_{\sigma}^{-1}$. Therefore the
subspace $U=\\{A\in so(N)^{\mathbb{\mathbb{C}}}:Av_{0}=0\\}$ is a
$\Lambda^{\mathbb{C}}$-invariant subspace. Moreover it is a Lie algebra and
more precisely the Lie algebra of the subgroup of
$SO(N)^{\mathbb{\mathbb{C}}}$ fixing the vector $v_{0},$ whose dimension is
$\frac{1}{2}\,(N-1)(N-2)$. If $N>4$ clearly $V^{\mathbb{C}}_{2}=U$. If $N=4$
the subspaces $V^{\mathbb{C}}_{1}$ and $V^{\mathbb{C}}_{2}$ both have
dimension $3$, however looking at the character on a two cycle of the action
of $S_{4}$ on $U$ and on $V^{\mathbb{C}}_{1}$ we see that again
$U=V^{\mathbb{C}}_{2}$. Details are postponed in the subsequent Lemma 2.2.

Let $h(A,B)=tr(A^{t}\overline{B})$ be the $so(N)$ conjugation-invariant
positive Hermitian product. As the orthogonal to $V^{\mathbb{\mathbb{C}}}_{2}$
with respect to $h$ is also $\Lambda^{\mathbb{C}}$-invariant,
$V^{\mathbb{C}}_{1}$ is the $h$-orthogonal subspace to
$V^{\mathbb{\mathbb{C}}}_{2}$. Observe that both subspaces
$V^{\mathbb{C}}_{1}$ and $V^{\mathbb{C}}_{2}$ are invariant with respect to
complex conjugation so that the spaces $V_{1},V_{2}$ of their respective real
points give rise to two irreducible real subrepresentations of $\Lambda$ with
$so(N)=V_{1}+V_{2}$. As $W$ is itself a $\Lambda$-invariant subspace of
$so(N)$, we therefore need to show that $W\nsubseteq V_{1}$ and $W\nsubseteq
V_{2},$ i.e. that $d\phi_{e}(so(k))\nsubseteq V_{1}$ and
$d\phi_{e}(so(k))\nsubseteq V_{2}$.

As $V_{2}$ is the subspace of the matrices that annihilate the vector $v_{0}=$
${}^{t}\\!~{}(1,\dots,1)$, if $d\phi_{e}(so(k))\subseteq V_{2}$ then the
subspace $\mathbb{R}v_{0}$ of $\mathbb{R}^{N}$ would be $\phi$ invariant,
against the assumptions.

Let us finally prove that also $d\phi_{e}(so(k))\subseteq V_{1}$ cannot be.
Let $f_{0}=\,^{t}\\!(1,0,\dots,0)$ and let $B\in SO(N)$ be a matrix such that
$Bf_{0}=\frac{1}{\sqrt{N}}\,{v_{0}}$. Since the inner product $\Re(h)$ on
$so(N)$ induced by $h$ is also $SO(N)$ invariant with respect to conjugation
by $B$, we see that $B^{-1}V_{1}B$ and $B^{-1}V_{2}B$ are $\Re(h)$-orthogonal.
Now, as the matrices of $V_{2}$ annihilate $v_{0}$, $B^{-1}V_{2}B$ is the set
of matrices that annihilate $f_{0}$, i.e. of the form

$\begin{bmatrix}0&0\\\ 0&A\end{bmatrix}$

with $A\in so(N-1)$ and $B^{-1}V_{1}B$, being the $h$-orthogonal to
$B^{-1}V_{2}B$, is the set of matrices of the form

$\begin{bmatrix}0&-^{t}\\!X\\\ X&{\bf{0}}\end{bmatrix}$

with $X\in\mathbb{R}^{N-1}$ and ${\bf{0}}$ is the zero matrix in $so(N-1)$.

Assume by contradiction that $B^{-1}d\phi_{e}(so(k))B\subseteq B^{-1}V_{1}B$.

Choose two matrices $C_{1},C_{2}$ in $B^{-1}d\phi_{e}(so(k))B\subseteq
B^{-1}V_{1}B$ with $C_{1}\neq 0$. Since $B^{-1}d\phi_{e}(so(k))B$ is a Lie
algebra, the matrices $C_{1},C_{2},$ and $[C_{1},C_{2}]=C_{1}C_{2}-C_{2}C_{1}$
also must belong to $B^{-1}V_{1}B$. This means that the entries
$(C_{1})_{i,j}$, $(C_{2})_{i,j}$ and $(C_{1}C_{2}-C_{2}C_{1})_{i,j}$ vanish
for $i>1,j>1$. Choose $i_{0}>0$ such that $(C_{1})_{1,i_{0}}\neq 0$ for some
$i_{0}>1$ and fix an index $j$ with $2\leq j\leq N$. We have

$\halign{\hbox to\displaywidth{$\hfil\displaystyle#\hfil$}\cr
0.0pt{\hfil$\displaystyle
0=[C_{1},C_{2}]_{i_{0},j}=(C_{1}C_{2})_{i_{0},j}-(C_{2}C_{1})_{i_{0},j}\cr
0.0pt{\hfil$\displaystyle=\sum_{k}\bigl{(}(C_{1})_{i_{0},k}(C_{2})_{k,j}-(C_{2})_{i_{0},k}(C_{1})_{k,j}\bigr{)}\cr
0.0pt{\hfil$\displaystyle=(C_{1})_{i_{0},1}(C_{2})_{1,j}-(C_{2})_{i_{0},1}(C_{1})_{1,j}\cr}}}}$

which gives

$(C_{2})_{1,j}=\frac{(C_{2})_{1,i_{0}}(C_{1})_{1,j}}{(C_{1})_{1,i_{0}}}\
\cdotp$

Therefore each value $(C_{2})_{1,j}$ is determined by $C_{1}$ and by
$(C_{2})_{1,i_{0}}$. This means that the space $B^{-1}d\phi_{e}(so(k))B$ is
one dimensional, but we know that it is at least three dimensional. This gives
the desired contradiction. ∎

######  Lemma 2.2.

Let $T_{1}=\\{Z\in\mathbb{C}^{4}:\sum_{j}z_{j}=0\\},$ and consider the
representation $\alpha_{1}$ of the symmetric group $S_{4}$ on $T_{1}$ by
permutation of the coordinates (the so called standard representation).

Let $so(4,\mathbb{C})$ be the space of complex antisymmetric $4\times 4$
matrices, and let $\alpha_{2}$ be the representation of $S_{4}$ on $so(4)$
given by $\alpha_{2}(\sigma)(A)=E_{\sigma}AE_{\sigma}^{-1}$, where
$E_{\sigma}$ is the permutation matrix corresponding to $\sigma\in S_{4}.$ The
subspace $U$ of matrices in $so(4)$ having the vector ${}^{t}(1,\ldots,1)$ in
their kernel is invariant under $\alpha_{2}.$

Let $\sigma_{0}$ be the permutation exchanging $1$ and $2$ and keeping $3$ and
$4$ fixed. Then the character of $\alpha_{1}$ on $\sigma_{0}$ is $1$ and the
character of ${\alpha_{2}}_{|U}$ on $\sigma_{0}$ is $-1$.

###### Proof.

The value of the character of $\alpha_{1}$ on $\sigma_{0}$ is $1$ as shown for
example in Fulton-Harris book [10], §2.3 p. 19).

Now if

$A=\begin{pmatrix}0&a&b&c\\\ -a&0&d&e\\\ -b&-d&0&f\\\ -c&-e&-f&0\end{pmatrix}$

then the subspace $U$ of matrices in $so(4)$ which have the vector
${}^{t}(1,\ldots,1)$ in their kernel corresponds to the vector subspace
$U\subseteq\mathbb{C}^{6}$

$\halign{\hbox to\displaywidth{$\hfil\displaystyle#\hfil$}\cr
0.0pt{\hfil$\displaystyle\\{^{t}(a,b,c,d,e,f)\in\mathbb{C}^{6}:a+b+c=0,-a+d+e=0,\qquad\qquad\qquad\cr
0.0pt{\hfil$\displaystyle\qquad\qquad\qquad\qquad\qquad-b-d+f=0,-c-e-f=0\\}\cr
0.0pt{\hfil$\displaystyle=\\{^{t}(a,b,-a-b,d,a-d,b+d)\in\mathbb{C}^{6},^{t}(a,b,d)\in\mathbb{C}^{3}\\}\
.\cr}}}}$

So a basis of $U$ is
$\beta:=\\{^{t}(1,0,-1,0,1,0),^{t}\\!(0,1,-1,0,0,1),^{t}\\!(0,0,0,1,-1,1)\\}.$
On the other hand for $A$ as above

$\alpha_{2}(\sigma_{0})(A)=\begin{pmatrix}0&-a&d&e\\\ a&0&b&c\\\ -d&-b&0&f\\\
-e&-c&-f&0\end{pmatrix}$

Then the $3\times 3$ matrix $B$ associated to the linear map
$\alpha_{2}(\sigma_{0})_{|U}$ with respect to the base $\beta$ is:

$B=\left(\begin{array}[]{lcc}-1&0&0\\\ 0&0&1\\\ 0&1&0\end{array}\right)$

and $tr(B)=-1.$ ∎

######  Remark 2.3.

Note that any irreducible representation $SO(3)\to GL(N,\mathbb{C})$ is a real
representation (see [5]) i.e. there exists a basis of $\mathbb{C}^{N}$ such
that in this basis the representation takes values in $SO(N)$.

######  Remark 2.4.

Since any semisimple Lie algebra is a direct sum of simple ideals and any
simple Lie algebra has dimension at least $3$, in the above theorem $SO(k)$
can be replaced by any connected semisimple Lie group. G.

##  3\. Random eigenfunctions with exchangeable coefficients

Let $(\Omega,\mathscr{F},{\rm P})$ be a probability space and let us consider
a finite-variance, isotropic random fields on the sphere of $\mathbb{R}^{3}$,
i.e. an application $T:\Omega\times\mathbb{S}^{2}\rightarrow\mathbb{R}$ such
that $\mathbb{E}[T^{2}]<\infty$ and

$T(g\,\cdot)\enspace\mathop{=}^{\mathscr{L}}\enspace T(\cdot)\text{ for all
}g\in SO(3)\ ,$

where the identity in law holds in the sense of processes; from now on,
without loss of generality, we will take all these fields to be zero-mean,
i.e. $\mathbb{E}[T(x)]=0$ for every $x\in\mathbb{S}^{2}$.

Recall that the Laplace-Beltrami operator on $\mathbb{S}^{2}$ can be written
in coordinates as

$\Delta_{\mathbb{S}^{2}}=\frac{1}{\sin\theta}\frac{\partial}{\partial\theta}\sin\theta\frac{\partial}{\partial\theta}+\frac{1}{\sin^{2}\theta}\frac{\partial}{\partial\varphi}$

with eigenvalues $\lambda_{\ell}=-\ell(\ell+1)$; the corresponding eigenspaces
have dimension $2\ell+1$ respectively and the group $SO(3)$ acts irreducibly
on them. Note that the “small” group $SO(3)$ is able to act irreducibly on
euclidean spaces of arbitrarily large dimension. Their elements are known as
spherical harmonics. It is possible to fix for each of them a (real-valued)
orthonormal basis, whose elements we will write $Y_{\ell
m}:\mathbb{S}^{2}\rightarrow\mathbb{R},\ell=0,1,2,\dots,m=-\ell,\dots,\ell$.
There are some standard choices for these elements, most notably the so-called
fully normalized spherical harmonics, (see e.g. [15], p. 64) but their exact
analytic expressions are not relevant for our results, which invariant
invariant with respect to the choice of basis.

A standard result in the theory of isotropic spherical random fields is the
well-known spectral representation theorem, stating that the following
representation holds, in the mean-square sense:

(3.1) $T(x)=\sum_{\ell=0}^{\infty}\sum_{m=-\ell}^{\ell}a_{\ell m}Y_{\ell
m}(x)\ ,$

for $(a_{\ell m})_{\ell,m}$ a triangular array of real-valued random
coefficients with zero mean and uncorrelated, i.e.

$\mathbb{E}[a_{\ell
m}a_{\ell^{\prime}m^{\prime}}]=C_{\ell}\delta_{\ell}^{\ell^{\prime}}\delta_{m}^{m^{\prime}}\
,$

where the nonnegative sequence $(C_{\ell})_{\ell=0,1,\dots}$ is the angular
power spectrum of the field and satisfies

$\mathbb{E}[T^{2}]=\sum_{\ell=1}^{\infty}\frac{2\ell+1}{4\pi}\,C_{\ell}<\infty.$

As the spherical random harmonic coefficients are always uncorrelated for
isotropic fields, then they are independent in the Gaussian case. More
surprisingly, the converse is also true: i.e., for isotropic random fields, if
the spherical random harmonic coefficients are independent, then necessarily
they are also Gaussian as well as the field. This fact was shown first in [2]
and then extended to more general random fields on compact spaces in [3], [4].

We now provide a characterization of the random spherical harmonic
coefficients of isotropic random fields under the assumptions that these
coefficients are exchangeable. Indeed, for $\ell=1,2,\dots$ let $\\{u_{\ell
m}\\}_{m=-\ell,\dots,\ell}$ denote the components of a sequence of
$(2\ell+1)$-dimensional random vectors $u_{\ell.}$, having uniform
distribution on the sphere $\mathbb{S}^{2\ell+1}$. The main result of this
section is the following:

######  Theorem 3.1.

Let $a_{\ell.}=(a_{\ell m})_{m=-\ell,\dots,\ell}$ be the
$(2\ell+1)$-dimensional random vector of the random coefficients of the
$\ell$-th component of an isotropic spherical random field. Then, if the
$a_{\ell m}$ are exchangeable, their law is rotationally invariant and there
exist a nonnegative random variable $\eta_{\ell}$, independent of $a_{\ell.}$,
such that $\mathbb{E}[\eta_{\ell}^{2}]=(2\ell+1)C_{\ell}$ and the following
identity in distribution hold, for all $\ell=1,2,\dots$

(3.2) $a_{\ell.}=u_{\ell.}\times\eta_{\ell}\ .$

Theorem 3.1 is an immediate consequence of Proposition 3.4 below.

######  Remark 3.2.

In [2] (see also [3], [4]) the authors consider the set $(Y_{\ell m})_{\ell
m}$ of the fully normalized spherical harmonics. These are complex valued
functions, so that, in order to obtain a real random field, the coefficients
$a_{\ell m}$ must satisfy the relation

$\overline{a_{\ell,-m}}=a_{\ell,m}\ .$

The authors prove that, for an isotropic random field, if the coefficients
$a_{\ell,0},\dots,a_{\ell,m}$ are independent, then they are necessarily
Gaussian. Note that no assumption of independence is made concerning the real
and the imaginary parts of the $a_{\ell m}$. This assumption is therefore in
some sense not comparable with the exchangeability of this vector.

Note however that when the $(Y_{\ell m})_{\ell m}$ are the fully normalized
spherical harmonics, then the complex random r.v. $a_{\ell m}$ is rotationally
invariant, so that, for any isotropic random field, the real and imaginary
parts of $a_{\ell m}$ are exchangeable r.v.’s.

######  Remark 3.3.

Due to the results [8] mentioned above, it is clear that in the ”high energy”
limit $\ell\rightarrow\infty$, for all $k\in\mathbb{N}$ every subset
$(a_{\ell,\pi(1)},\dots,a_{\ell,\pi(k)})$ of cardinality $k=o(\ell)$ of
exchangeable spherical harmonic coefficients, suitably normalized, converges
in total variation to the mixture of a vector of i.i.d. Gaussian variables
with a random standard deviation. In view of these characterizations, one can
consider Theorem 3.1 as an “approximate” De Finetti style result; note that
for finite $\ell$ the components of $u_{\ell.}$ are identically distributed,
but not independent.

######  Proposition 3.4.

Let $X=^{t}\\!\\!(X_{1},\dots,X_{d})$ be a random vector. Then if $X$ is
invariant with respect to a the action of a subgroup of rotations acting
irreducibly on $\mathbb{R}^{d}$ and the family $\\{X_{1},\dots,X_{d}\\}$ is
exchangeable, then $X$ is rotationally invariant and the r.v.’s $|X|$ and
$\frac{X}{|X|}$ are independent.

###### Proof.

By Theorem 2.1 the law of $X$ is invariant by the action of $SO(d)$ hence the
rotational invariance. It is well known that if an $d$-dimensional r.v. $X$
has a law that is rotationally invariant, then the angular component
$\frac{X}{|X|}$ and the radial one $|X|$ are independent. Let us however
detail a proof of this fact for thoroughness sake.

Let $A\in\mathscr{B}(\mathbb{S}^{d-1})$ and $B\in\mathscr{B}(\mathbb{R}^{+})$.
Then, for every orthogonal matrix $O\in SO(d)$, we have

${\rm P}\Bigl{(}\frac{X}{|X|}\in A,|X|\in B\Bigr{)}={\rm
P}\Bigl{(}\frac{OX}{|X|}\in A,|X|\in B\Bigr{)}$

which means that, for every fixed $B\in\mathscr{B}(\mathbb{R}^{+})$, the
finite measure

$\mathscr{B}(\mathbb{S}^{d-1})\ni A\mapsto{\rm P}\Bigl{(}\frac{X}{|X|}\in
A,|X|\in B\Bigr{)}$

is rotationally invariant, hence a multiple of the normalized Lebesgue
measure, $\lambda$ say, of $\mathbb{S}^{d-1}$. The total mass of such a
measure is deduced by choosing $A=\mathbb{S}^{d-1}$, i.e. ${\rm P}(|X|\in B)$.
Therefore

${\rm P}\Bigl{(}\frac{X}{|X|}\in A,|X|\in B\Bigr{)}=\lambda(A){\rm
P}\bigl{(}|X|\in B\bigr{)}={\rm P}\Bigl{(}\frac{X}{|X|}\in A\Bigr{)}{\rm
P}\bigl{(}|X|\in B\bigr{)}\ .$

$\Box$ mm

######  Remark 3.5.

In general the property of exchangeabilty of the components of a vector
$X=^{t}\\!\\!(X_{1},\dots,X_{d})$ is not invariant with respect to rotations,
so that a priori the condition of exchangeability of $X$ depends on the choice
of the orthogonal basis that is chosen. However, as Theorem 3.1, states
rotational invariance, it also states that if the coefficients are
exchangeable with respect to one basis, then they are also exchangeable with
respect to any other basis.

######  Remark 3.6.

It is well-known that the Lebesgue measure on the sphere $\mathbb{S}^{d-1}$
and its multiples are the unique measures on $\mathbb{S}^{d-1}$ that are
invariant under the action of the rotation group $SO(d)$.

In the same spirit as in the previous section, Proposition 3.4 allows to give
a weaker characterization.

In terms of probabilities, if $X=(X_{1},\dots,X_{d})$ is an
$\mathbb{S}^{d-1}$-valued r.v. such that $\\{X_{1},\dots,X_{d}\\}$ is
exchangeable and whose law, $\mu$ say, is invariant under the action of a
subgroup of $SO(d)$ acting irreducibly on $\mathbb{R}^{d}$, then $\mu$ is the
normalized Lebesgue measure on $\mathbb{S}^{d-1}$.

##  4\. An extension of Bernstein’s theorem

Bernstein’s theorem (see [14], [6] p.74 e.g.) states that a probability on
$\mathbb{R}^{d}$ which is invariant with respect to the action of $SO(d)$ and
whose components are independent is necessarily Gaussian. What if we assumed
invariance only with respect to the action of a subgroup of $SO(d)$ acting
irreducibly on $\mathbb{R}^{d}$?

######  Theorem 4.1.

(Bernstein revisited) Let $X=(X_{1},\dots,X_{d})$ be a $d$-dimensional r.v.
such that

a) a) its law is invariant with respect to a subgroup of $SO(d)$ acting
irreducibly on $\mathbb{R}^{d}$;

a) b) the r.v.’s $X_{1},\dots,X_{d}$ have the same law and are independent.

Then $X$ is Gaussian.

Theorem 4.1 follows from Theorem 2.1 and the true Bernstein Theorem: condition
b) actually states that $(X_{1},\dots,X_{d})$ is exchangeable, hence invariant
with respect to the action of the permutation group so that by Theorem 2.1 $X$
is invariant with respect to the action of the full group $SO(d)$ and
Bernstein’s Theorem allows to conclude.

With respect to the original Bernstein Theorem, Theorem 4.1 requires a
stronger distributional assumption (equality of the laws of the components
$X_{i}$) but a much weaker invariance hypothesis, as a subgroup of of $SO(d)$
acting irreducibly on $\mathbb{R}^{d}$ can be “very small”: as noted in §3 the
group $SO(3)\subset SO(2d+1)$ can act irreducibly on $\mathbb{R}^{2d+1}$ for
every $d$.

######  Remark 4.2.

It should be noted that a result analogous to Theorem 4.1 follows also from
the so-called Skitovich-Darmois Theorem (see e.g. [11]), which is indeed
slightly more general, as it does not require the single components to be
equi-distributed. However, our purpose here is not to claim new results, as to
show what we consider elegant and simple consequences of our main Theorem 2.1.

## References

  * [1] I. Alam, _Generalizing the de Finetti-Hewitt-Savage Theorem_ , arXiv: 2008.08754, 2023.
  * [2] P. Baldi and D. Marinucci, _Some characterizations of the spherical harmonics coefficients for isotropic random fields_ , Statist. Probab. Lett. 77 (2007), no. 5, 490–496.
  * [3] P. Baldi, D. Marinucci, and V. S. Varadarajan, _On the characterization of isotropic Gaussian fields on homogeneous spaces of compact groups_ , Electron. Comm. Probab. 12 (2007), 291–302 (electronic).
  * [4] P. Baldi and S. Trapani, _Fourier coefficients of invariant random fields on homogeneous spaces of compact Lie groups_ , Ann. Inst. Henri Poincaré Probab. Stat. 51 (2015), no. 2, 648–671.
  * [5] M. B. Can and M. Jones, _Adjoint representations of symmetric groups_ , Current trends in analysis, its applications and computation, Trends Math. Res. Perspect., Birkhäuser/Springer, Cham, 2022, pp. 465–473.
  * [6] L. Chaumont and M. Yor, _Exercises in probability_ , second ed., Cambridge Series in Statistical and Probabilistic Mathematics, vol. 35, Cambridge University Press, Cambridge, 2012, A guided tour from measure theory to random processes, via conditioning.
  * [7] P. Diaconis and D. Freedman, _Finite exchangeable sequences_ , Ann. Probab. 8 (1980), no. 4, 745–764.
  * [8] by same author, _A dozen de Finetti-style results in search of a theory_ , Ann. Inst. H. Poincaré Probab. Statist. 23 (1987), no. 2, 397–423.
  * [9] K. Erdmann and M. J. Wildon, _Introduction to Lie algebras_ , Springer Undergraduate Mathematics Series, Springer-Verlag London, Ltd., London, 2006.
  * [10] W. Fulton and J. Harris, _Representation theory_ , Graduate Texts in Mathematics, vol. 129, Springer-Verlag, New York, 1991, A first course, Readings in Mathematics.
  * [11] S. G. Ghurye and I. Olkin, _A characterization of the multivariate normal distribution_ , Ann. Math. Statist. 33 (1962), 533–541.
  * [12] S. Helgason, _Differential geometry, Lie groups, and symmetric spaces_ , Pure and Applied Mathematics, vol. 80, Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York-London, 1978.
  * [13] G. Itzkowitz, S. Rothman, and H. Strassberg, _A note on the real representations of ${\rm SU}(2,{\bf C})$_, J. Pure Appl. Algebra 69 (1991), no. 3, 285–294.
  * [14] M. Kac, _On a characterization of the normal distribution_ , Amer. J. Math. 61 (1939), 726–728.
  * [15] D. Marinucci and G. Peccati, _Random Fields on the Sphere_ , London Mathematical Society Lecture Note Series, vol. 389, Cambridge University Press, Cambridge, 2011.
  * [16] J.-P. Serre, _Lie algebras and Lie groups_ , W. A. Benjamin, Inc., New York-Amsterdam, 1965, Lectures given at Harvard University, 1964.


# Covering up bias in CelebA-like datasets with Markov blankets:
A post-hoc cure for attribute prior avoidance

Vinay Prabhu  Dian Ang Yap  Alexander Wang  John Whaley

###### Abstract

Attribute prior avoidance entails subconscious or willful non-modeling of
(meta)attributes that datasets are oft born with, such as the 40 semantic
facial attributes associated with the CelebA and CelebA-HQ datasets. The
consequences of this infirmity, we discover, are especially stark in state-of-
the-art deep generative models learned on these datasets that just model the
pixel-space measurements, resulting in an inter-attribute bias-laden latent
space. This viscerally manifests itself when we perform face manipulation
experiments based on latent vector interpolations. In this paper, we address
this and propose a post-hoc solution that utilizes an Ising attribute prior
learned in the attribute space and showcase its efficacy via qualitative
experiments.

Machine Learning, Flow Models, CelebA, Bias Corrections, ICML

##  1 Motivation: Attractive Barack Obama $\neq$ White Blonde Barack Obama

GIF animations showcasing nifty manipulation of celebrity faces111
https://youtu.be/G06dEcZ-QTg by vector operations in the discovered latent
space have emerged as a modality of choice in order to advertise the
effectiveness of image generative models of all hues, be it GANs (Goodfellow
et al., 2014), VAEs (Kingma & Welling, 2013) or Flow-based models (Dinh et
al., 2014). In fact, the idea that deep generative models, especially flow-
based generative models, can be trained without attribute labels by attempting
to directly model the distribution of the input images, $\bm{x}\sim
p(\bm{x})$, and yet yield latent-space representations worthy of downstream
tasks like manipulating the semantic attributes of images is seen as a marquee
feature222The exact terms used are: Useful latent space in Section 1 and
meaningful latent space in Section 6 of (Kingma & Dhariwal, 2018) of the model
(Kingma & Dhariwal, 2018). This is however tantamount to not just ignoring the
rich co-occurrence structure between the image attributes in the dataset that
these models are trained on but also making a faux pixel space - attribute
space marginal independence assumption. That is, for a dataset
$\mathcal{D}=\left\\{{\left({{{\mathbf{x}}_{i}},{{\mathbf{a}}_{i}}}\right)}\right\\}_{i=1}^{N_{\mathit{dataset}}}$
where $\mathbf{x}_{i}$ is the $i^{th}$ image and $\mathbf{a}_{i}$ is the
attribute vector associated with that $i^{th}$ image, modeling just
$p(\mathbf{x})$ is akin to assuming
$p\left({{\mathbf{x}},{\mathbf{a}}}\right)=p({\mathbf{x}})p({\mathbf{a}})$
given that marginalization yields:
$\sum\limits_{\mathbf{a}}{p\left({{\mathbf{x}},{\mathbf{a}}}\right)=}\sum\limits_{\mathbf{a}}{p({\mathbf{x}})p({\mathbf{a}})}=p({\mathbf{x}})$.

Specifically, with regards to CelebA (Liu et al., 2015), we have
$N_{\mathit{dataset}}=30000$ images, each with 40 binary semantic attributes.
If we ignore the co-occurrence of these attributes and not account for it, we
will end up with inter-attribute spill-over effects, an example of which is
showcased in Figure 1. Using the model in (Kingma & Dhariwal, 2018) as an
example, we see the end result of trying to enhance the attractive attribute
in Barack Obama’s and Leonardo DiCaprio’s
333https://www.imdb.com/name/nm0000138/
pictures.444https://openai.com/blog/glow/ Figure 2 shows additional examples
of varying the attractive attribute across a variety of images. Moreover,
since CelebA-HQ was derived and cherry-picked from CelebA (Karras et al.,
2017), the biases are also inherited and are apparent in models trained on
CelebA-HQ.

Figure 1: Interpolating along the Attractive latent attribute in different
scales converges to pictures of white blonde women.

Figure 2: More visualizations of how images of different gender and races
converge to young, white-skinned women with long blonde or red hair as we only
naively interpolate along the ’attractive’ attribute. From top to bottom:
original, attractive: ($\alpha=+1$, attractive: $\alpha=+3$,
attractive:$\alpha=+5$.)

As we increase the weight of influence of the attractive-attribute
manipulation latent vector, we end up rendering Obama and DiCaprio as almost
Caucasian-blonde women. One can attribute this to the possible idiosyncratic
co-prevalence of attributes such as blond_Hair, attractive, and pale_Skin in
the images of the original dataset.

We acknowledge this troublesome nexus555This nexus has also been deeply
explored in works such as (Barocas et al., ) and (Tom Yeh et al., 2017)
between these three attributes and would like to reiterate that this work does
not propose a ‘here’s how you make a face more attractive’ framework but
rather focuses on the issue on ignored statistical priors of the
(meta)attributes that an image dataset is oft shipped with. Specifically, we
address this issue by using graph structured Ising priors to model the inter-
attribute statistical dependence and later harnessing this model to propose a
corrective procedure that helps allay some of the concerns laid thus far.

The rest of the paper is organized as follows. In Section 2, we cover the
estimation procedure for learning the Inter-Attribute Ising Prior (IAIP). In
Section 3, we propose our post hoc bias- corrective framework. In Section 4,
we showcase some qualitative results that highlights the efficacy of our
approach and conclude the paper.

This is a work-in-progress and we have duly open-sourced the implementation
along with an interactive portal provides interfaces to both the IAIP as well
as the corrective procedure here666https://yapdianang.github.io/celeba/. This
work aims to showcase and correct the bias in the image domain, which draw
parallels to the learnt biases in Word2Vec embeddings (Bolukbasi et al., 2016)
and other existing study of biases (Torralba et al., )(Tommasi et al.,
2017)(Binns, 2017). We’d like to re-emphasize we position this work as a
quick-to-implement post-hoc solution and would like to encourage the deep
learning community to not just pay attention to the dubious selection-biases
in the datasets they use to train their generative models, but to also
consider the problem of jointly modeling the attribute space and the pixel-
space of images.

##  2 Learning the Inter-Attribute Ising Prior

Probabilistic graphical models are the unifying framework of choice in areas
such as bioinformatics, statistical physics and communication theory for
modeling complex dependencies among random variables using ideas from both
graph theory and probability theory. This framework not just facilitates
building large-scale multivariate statistical models but also allows for
elegant visualization and knowledge discovery through the learned statistical
dependency graph (Wainwright et al., 2008). In this paper, we choose a
specific type of binary Markov Random Fields (MRFs) termed as Ising Models
(see (Stanley, 1971)) to model the inter-attribute statistical dependencies.
The model specifies a distribution $p(\mathbf{a})$ defined over a substrate
graph, $G^{\textrm{(ising)}}(V,E)$. Specifically the conditional distribution
of an attribute $a_{j}\in\\{0,1\\}$ given the rest of the attributes
$a_{\backslash j}$ and the graph $G^{\textrm{(ising)}}(V,E)$ takes the form:

${p_{\Omega}}\left({{a_{j}}|{a_{\backslash
j}}}\right)=\frac{{\exp\left[{{\tau_{j}}{a_{j}}+{a_{j}}\sum\limits_{k\in{V_{\backslash
j}}}{\left({{\beta_{jk}}{a_{k}}}\right)}}\right]}}{{1+\exp\left[{{\tau_{j}}+\sum\limits_{k\in{V_{\backslash
j}}}{\left({{\beta_{jk}}{a_{k}}}\right)}}\right]}}$ (1)

Here, $\tau_{j}$ and $\beta_{jk}$ are the node parameters and Ising edge
parameters respectively.

In order to learn the underlying graph and the associated node and edge
weights, we use a network estimation procedure termed eLasso that combines
$\ell_{1}$-regularized logistic regression with model selection criterion
based on the Extended Bayesian Information Criterion (EBIC).777eLASSO is
implemented in the R package IsingFit:
http://cran.r-project.org/web/packages/IsingFit/IsingFit.pdf

The EBIC cost function is defined as:

$\mathit{BIC}_{\gamma}\left({{{\hat{\Omega}}_{j}}}\right)=-2\ell\left({{{\hat{\Omega}}_{j}}}\right)+n_{nei}(\rho)\log\left(N_{\mathit{dataset}}\right)\\\
+2\gamma n_{nei}(\rho)\log\left({N_{\mathit{attributes}}-1}\right)$ (2)

Here, $N_{\mathit{dataset}}$ represents the number of observations (which is
30,000 in the case of Celeb-A-HQ dataset), $N_{\mathit{attributes}}-1$
represents the number of covariates (predictors), $n_{nei}(\rho)$ represents
the cardinality of the neighbors set chosen by the logistic regression at a
certain penalty weight $\rho$, and $\gamma$ is the prior strength goodness-of-
fit hyperparameter (Van Borkulo et al., 2014).

So, given the binary attribute matrix
$A\in\\{0,1\\}^{N_{\mathit{dataset}}\times N_{\mathit{attributes}}}$, the log-
likelihood cost function for the $j^{th}$ attribute,
$\ell\left({{{\hat{\Omega}}_{j}}}\right)$ is defined as:

$\ell\left({{{\hat{\Omega}}_{j}}}\right)=\sum\limits_{i=1}^{N_{\mathit{dataset}}}{\left({{\tau_{j}}{A_{ij}}+\sum\limits_{k\in{V_{\backslash
j}}}{\left({{\beta_{jk}}{A_{ij}}{A_{ik}}}\right)-\log\left({1+\exp\left\\{{{\tau_{j}}+\sum\limits_{k\in{V_{\backslash
j}}}{{\beta_{jk}}{A_{ik}}}}\right\\}}\right)}}\right)}$ (3)

After learning the node weights and the edge weights using the IsingFit
procedure, we symmetrize the weights as:

$w_{jk}=\begin{cases}\frac{(\beta_{jk}+\beta_{kj})}{2},&\text{if
}\beta_{jk}\neq 0\textit{ and }\beta_{kj}\neq 0\\\
0,&\text{otherwise}\end{cases}$ (4)

###  2.1 The learned IAIP for the CelebA-HQ dataset

Figure 3: Visualization of Ising model of binary attributes, with green
denoting positive edges and red denoting negative edges.

As we learn in the following section, we’d ideally want the learned graph
$G^{\textrm{(ising)}}(V,E)$ to be connected (no disconnected nodes) as well as
sparse. Therefore, we set the the goodness-of-fit parameter ($\gamma$) of the
eLasso algorithm to be the largest value above which the resultant graph was
disconnected. This variation of the sparsity level of the graph obtained with
regards to the number of disconnected nodes as well as the number of edges (a
sparsity measure) is seen in Figure 9. As seen, that critical value of the
prior strength parameter is $\gamma=6.4$. In Figure 10 and Figure 3, we
visualize the weighted graph representing the Ising model learned for the $40$
binary attributes of the CelebA-HQ dataset (with $\gamma=6.4$).

This graph will not just pave the way for the bias corrective procedure to be
proposed in the upcoming section, but will also serve as a succinct
representation of uncovering anthropocentric bias ingrained during the
labeling process itself by the human annotators, by means of Graph exploration
using popular GUI Graph visualization software such as
Gephi888https://gephi.org/ and NodeXL999https://www.smrfoundation.org/nodexl/.

With regards to this eLasso fitting procedure, we have the following figures.
In Figure 11, we see the attribute-wise thresholds (or node parameters
$\\{\tau_{i}\,i\in V\\})$. This threshold captures the propensity of a
specific attribute to be present in an image in the dataset.

As seen, most of the thresholds (with the exception of Male, No-Beard and
Young) are all negative. This implies most of the attributes in the CelebA-HQ
dataset in-fact exude a normative disposition to be absent. The attributes
Rosy-cheeks, Blurry and Wearing-necktie had the strongest negative thresholds.
This implies that these symptoms had the strongest probability of being
present in a CelebA-HQ image compared to the other attributes. In Figure 12,
we see that variation of the tuning parameter per attribute that was used to
ensure the best-fit set of neighbors (see (Van Borkulo et al., 2014)). As
seen, the features Mouth-slightly-open and Pointy-nose had the largest values
of the tuning parameters associated with them.

##  3 Methodology

In this section, we denote the notation
$\psi\left({}\right):{\mathbf{x}}\to{\mathbf{z}}$ denotes the pixel-space to
latent-space encoder and $\phi\left({}\right):{\mathbf{z}}\to{\mathbf{x}}$
denotes the latent-space to pixel-space decoder. We begin by reviewing the
current procedure101010https://github.com/openai/glow/tree/master/demo for
latent space manipulation and then propose our correction.

###  3.1 Current procedure for latent space manipulation

The current procedure for latent space interpolation firsts encode inputs and
compute the average latent vector of inputs with and without the attribute.
The latent space manipulation vectors are

${{\mathbf{Z}}^{(manip)}}=\left\\{{{\mathbf{z}}_{i}^{(manip)}}\right\\}_{i=1}^{40}$
(5)

where for the $i^{th}$ attribute,

${\mathbf{z}}_{i}^{(manip)}=\frac{{\sum\limits_{t=1}^{{N_{\mathit{dataset}}}}{\psi\left({{{\mathbf{x}}_{t}}\cdot\mathbbm{1}\left[\kern-1.49994pt\left[{{A_{ti}}==+1}\right]\kern-1.49994pt\right]}\right)}}}{{\sum\limits_{t=1}^{{N_{\mathit{dataset}}}}{\mathbbm{1}\left[\kern-1.49994pt\left[{{A_{ti}}==+1}\right]\kern-1.49994pt\right]}}}-\frac{{\sum\limits_{t=1}^{{N_{\mathit{dataset}}}}{\psi\left({{{\mathbf{x}}_{t}}\cdot\mathbbm{1}\left[\kern-1.49994pt\left[{{A_{ti}}==0}\right]\kern-1.49994pt\right]}\right)}}}{{\sum\limits_{t=1}^{{N_{\mathit{dataset}}}}{\mathbbm{1}\left[\kern-1.49994pt\left[{{A_{ti}}==0}\right]\kern-1.49994pt\right]}}}$
(6)

Here, $A\in\\{0,1\\}^{N_{\mathit{dataset}}\times N_{\mathit{attributes}}}$
represents the binary attribute matrix for the dataset and $A_{ti}$ represents
the binary value for the $i^{th}$ attribute in the $t^{th}$ image out of
$N_{\mathit{dataset}}$ images in the dataset.
$\mathbbm{1}\left[\kern-1.49994pt\left[{e}\right]\kern-1.49994pt\right]$
represents the identity function

$\mathbbm{1}\left[\kern-1.49994pt\left[e\right]\kern-1.49994pt\right]=\left\\{{\begin{array}[]{*{20}{c}}1,&\text{if
e = TRUE}\\\ 0,&\text{otherwise.}\end{array}}\right.$ (7)

After obtaining the latent space manipulation vectors, we can manipulate over
the latent space given an input tuple of image, attribute-index and weight of
manipulation $(\mathbf{x},i,\alpha)$. By first encoding a raw image
$\mathbf{x}$ in latent space as $\psi(\mathbf{x})$, One can interpolate over
the latent space by some degree $\alpha$:

${{\mathbf{z}}^{\left({manip\\_orig,i,\alpha}\right)}}=\psi\left({\mathbf{x}}\right)+\alpha{\mathbf{z}}_{i}^{(manip)}$
(8)

To obtain the manipulated image, the interpolated latent vector is decoded to
obtain ${{{\mathbf{z}}^{\left({manip\\_orig,i,\alpha}\right)}}}$:

${{\mathbf{x}}^{\left({manip\\_orig,i,\alpha}\right)}}=\phi\left({{{\mathbf{z}}^{\left({manip\\_orig,i,\alpha}\right)}}}\right)$
(9)

###  3.2 Proposed modified procedure for latent space manipulation

We suggest that we first realign the latent space attribute-wise manipulation
vectors by adding a Markov-blanket correction factor to the vectors computed
in Eq 6 that takes into account the neighboring attributes (nodes) unearthed
in the Ising prior graph learned from the dataset in Section 2. That is, we
generate different latent manipulation vectors derived from Equation 5 as

${{\mathbf{Z}}^{(manip\\_mb)}}=\left\\{{{\mathbf{z}}_{i}^{(manip\\_mb)}}\right\\}_{i=1}^{40}$
(10)

where for the $i^{th}$ attribute,

${\mathbf{z}}_{i}^{(manip\\_mb)}={\mathbf{z}}_{i}^{(manip)}\overbrace{-\mu\sum\limits_{k\in{\mathcal{N}_{i}}\left({G^{\textrm{(ising)}}(V,E)}\right)}{\left({{w_{ik}}{\mathbf{z}}_{k}^{(manip)}}\right)}}^{markov-
blanket-correction}.$ (11)

Here $\mu$ is the Markov-blanket correction strength parameter111111In our
experiments, we empirically found that varying $\mu$ between 0.1 to 0.2
yielded the most aesthetically pleasing results., ${\mathcal{N}_{i}}$ is the
Markov-blanket neighborhood of the $i^{th}$ attribute (node) in the Ising-
graph ${G^{\textrm{(ising)}}(V,E)}$ learned in Section 2.

Now, the output images with Markov-blanket corrections,
${{\mathbf{x}}^{\left({manip\\_mb,i,\alpha}\right)}}$, are generated by:

$\begin{gathered}{{\mathbf{z}}^{\left({manip\\_mb,i,\alpha}\right)}}=\psi\left({\mathbf{x}}\right)+\alpha\left({{\mathbf{z}}_{i}^{(manip)}\overbrace{-\mu\sum\limits_{k\in{N_{i}}\left({{G^{({\text{ising}})}}(V,E)}\right)}{\left({{w_{ik}}{\mathbf{z}}_{k}^{(manip)}}\right)}}^{markov-
blanket-correction={\mathbf{z}}_{i}^{(manip\\_mb)}}}\right)\hfill\\\
{{\mathbf{x}}^{\left({manip\\_mb,i,\alpha}\right)}}=\phi\left({{{\mathbf{z}}^{\left({manip\\_mb,i,\alpha}\right)}}}\right)\hfill\\\
\end{gathered}$ (12)

Given the proposed modified method of correctly manipulating across the latent
space, we define the full algorithm as per Algorithm 1.

Algorithm 1 Corrective Bias Latent Manipulation Algorithm

Input: data $\bm{x}$, priors $\bm{A}$, attributes to interpolate $A$, degree
of interpolation $\alpha$, hyperparameter $\gamma,\mu\in\mathbb{R}^{+}$

$G(V,E)\leftarrow$ Ising Graph of ($\bm{A},\gamma$)

$\psi\leftarrow$ model mapping $\bm{x}$ to $\bm{z}$

$\phi\leftarrow$ model mapping $\bm{z}$ to $\bm{x}$

$z_{m}\leftarrow$ manipulation vectors

Initialize $\bm{z^{\prime}}$ as $\psi(\bm{x})$

for $\alpha_{i}=\alpha_{1},\alpha_{2},\dots,\alpha_{m}$ in $\bm{\alpha}$ do

$\bm{z^{\prime}}\leftarrow\bm{z^{\prime}}+\alpha z_{m_{i}}$

$\mathcal{N}\leftarrow$ neighborhood/adjacent vertices of $z_{m_{i}}$ in $G$

for $k$ in $\mathcal{N}$ do

$\bm{z^{\prime}}\leftarrow\bm{z^{\prime}}+-\alpha\cdot\mu\cdot
w_{ik}(z_{m_{k}})$

end for

end for

$\bm{\hat{x}}\leftarrow\phi(\bm{z^{\prime}})$, the interpolated image with
corrective bias

##  4 Results and Discussion

In this section, we focus on showcasing the efficacy of our procedure by
qualitative methods. Before we dive into the qualitative experiments, we’d
like explore if the (Ising) weights of $G^{\textrm{(ising)}}(V,E)$, that is,
$\left\\{{{w_{ij}}}\right\\};\forall(i,j)\in E$ that are indeed correlated
with the respective (angular) distance between the latent vectors
$\mathbf{z}_{i},\mathbf{z}_{j}$. This would help broadly understand how much
of the separation between the latent space representations is on account of
mere co-occurrence of the attributes and how much can be attributed to the
deep-encoder being able to organically tease out the semantic separation
between the attributes.

Figure 4: Plot of Ising weights against latent manipulation vectors between
the connected nodes in Ising graph.

Figure 4 presents the scatter-plots between the Ising weights $w_{ij}$ and
$d_{ij}$ as the cosine distance between $\mathbf{z}_{i}$ and $\mathbf{z}_{j}$,
the latent manipulation vectors associated with attribute indices $i$ and $j$.

The strong negative correlation $(R_{\mathit{pearson}}=-0.74;p=8.5e^{-47})$
confirms our hunch that the co-occurrence of attributes has strongly
influenced the latent space. That is, two attributes that always co-occurred
in the image dataset (meaning high edge-weight $w_{ij}$ in the Ising prior)
ended up with latent space representations with a low cosine distance between
them. We posit this might be more of a bug than a feature of the model as this
alludes towards the notion that coincidental occurrences will strongly
influence the semantic distancing in the latent space.

Figure 5: Top: Naively manipulating attribute by +2. Bottom: manipulating
attribute by +2 while applying corrective bias penalty to corresponding
Markovian blankets. Figure 6: Naive manipulation across one attribute (top) vs
manipulation with corrective penalty on neighborhood attributes. Figure 7:
Demonstration on how inherent bias in the ’wearing lipstick’ attribute
commonly correlated with blonde hair, young and women can be corrected using
our proposed corrective bias algorithm. Top: wearing lipstick: +2. Bottom:
wearing lipstick: +2 with neighborhood corrective penalty.

Some attributes are mutually associated with others through stereotypes, which
experimentally justifies the need for bias correction. As shown in Figure 6,
increasing High_Cheekbones also subtly increases the attributes of Smiling,
Female and Wearing_Lipstick which are usually associated with feminine
characteristics. On the other hand, increasing attributes such as Bald also
shows increases likelihood of Goatee, Chubby and Male. Applying such
attributes in images in contrasting domains (increasing Lipstick for male,
Bald for females) cause the transformed images to lose intrinsic properties of
the original image by generalizing towards the implicitly learnt bias. This is
also confirmed in Figure 5, where increasing baldness changes the skin color
to be that of a white forehead.

We also demonstrate how biases can be corrected as shown in Figure 7. Using
Wearing_Lipstick as an example, which is implicitly associated with feminine
characteristics, the image learns to transform with biases such as
Blonde_Hair, Female and Wearing_Makeup. However, by accounting for biases, we
show how Wearing_Lipstick can be applied across different characteristic
domains such as gender and race, while preserving the inherent attributes in
the image transformation process.

##  5 Conclusion

We demonstrate that models that directly learn to model the input data without
considering attribute priors while training reveal significant bias. Using
Glow as an example of flow-based generative models, we reveal the existing
bias by manipulating across the latent space, which mirrors common racial and
gender stereotypes.

We also emphasize that this bias also exists in other generative models such
as GANs and VAEs as shown in the Appendix. We propose a post-hoc corrective
measure for models trained with existing bias by using Ising models fitted on
attribute priors.

## References

  * (1) Barocas, S., Hardt, M., and Narayanan, A.  Fairness and machine learning. fairmlbook. org, 2018.  _URL: http://www. fairmlbook. org_.
  * Binns (2017) Binns, R.  Fairness in machine learning: Lessons from political philosophy.  _arXiv preprint arXiv:1712.03586_ , 2017.
  * Bolukbasi et al. (2016) Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., and Kalai, A. T.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In _Advances in neural information processing systems_ , pp. 4349–4357, 2016.
  * Dinh et al. (2014) Dinh, L., Krueger, D., and Bengio, Y.  Nice: Non-linear independent components estimation.  _arXiv preprint arXiv:1410.8516_ , 2014.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pp. 2672–2680, 2014.
  * He et al. (2017) He, Z., Zuo, W., Kan, M., Shan, S., and Chen, X.  Arbitrary facial attribute editing: Only change what you want.  _CoRR_ , abs/1711.10678, 2017.  URL http://arxiv.org/abs/1711.10678.
  * Karras et al. (2017) Karras, T., Aila, T., Laine, S., and Lehtinen, J.  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ , 2017.
  * Kingma & Dhariwal (2018) Kingma, D. P. and Dhariwal, P.  Glow: Generative flow with invertible 1x1 convolutions.  In _Advances in Neural Information Processing Systems_ , pp. 10215–10224, 2018.
  * Kingma & Welling (2013) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Liu et al. (2019) Liu, M., Ding, Y., Xia, M., Liu, X., Ding, E., Zuo, W., and Wen, S.  Stgan: A unified selective transfer network for arbitrary image attribute editing.  2019\.
  * Liu et al. (2015) Liu, Z., Luo, P., Wang, X., and Tang, X.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2015.
  * Sainburg et al. (2018) Sainburg, T., Thielk, M., Theilman, B., Migliori, B., and Gentner, T.  Generative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourage convex latent distributions.  _arXiv preprint arXiv:1807.06650_ , 2018.
  * Stanley (1971) Stanley, H. E.  _Phase transitions and critical phenomena_.  Clarendon Press, Oxford, 1971.
  * Tom Yeh et al. (2017) Tom Yeh, M. et al.  Designing a moral compass for the future of computer vision using speculative analysis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 64–73, 2017.
  * Tommasi et al. (2017) Tommasi, T., Patricia, N., Caputo, B., and Tuytelaars, T.  A deeper look at dataset bias.  In _Domain adaptation in computer vision applications_ , pp. 37–55. Springer, 2017.
  * (16) Torralba, A., Efros, A. A., et al.  Unbiased look at dataset bias.  Citeseer.
  * Van Borkulo et al. (2014) Van Borkulo, C. D., Borsboom, D., Epskamp, S., Blanken, T. F., Boschloo, L., Schoevers, R. A., and Waldorp, L. J.  A new method for constructing networks from binary data.  _Scientific reports_ , 4:5918, 2014.
  * Wainwright et al. (2008) Wainwright, M. J., Jordan, M. I., et al.  Graphical models, exponential families, and variational inference.  _Foundations and Trends® in Machine Learning_ , 1(1–2):1–305, 2008.
  * Xiao et al. (2018) Xiao, T., Hong, J., and Ma, J.  Dna-gan: Learning disentangled representations from multi-attribute images.  _International Conference on Learning Representations, Workshop_ , 2018.
  * Zhou et al. (2017) Zhou, S., Xiao, T., Yang, Y., Feng, D., He, Q., and He, W.  Genegan: Learning object transfiguration and attribute subspace from unpaired data.  _arXiv preprint arXiv:1705.04932_ , 2017.

##  6 Appendix

###  6.1 Visializing Learnt Biases in Generative Models

Figure 8: Bias of CelebA illustrated: Visualization of biases in latent space
of Generative Adversarial Interpolative Autoencoding (GAIA) (Sainburg et al.,
2018). Visualization of latent ‘attractive’, ‘big nose’, and ‘big lips’
correlate to racial profiles that reveal the inherent bias in CelebA.

###  6.2 Visualization of Ising Graphs

Figure 9: Goodness-of-fit parameter ($\gamma$) selection. Figure 10: The
weighted graph representing the Ising model for the binary attributes of the
CelebA-HQ dataset. Figure 11: Attribute-wise thresholds (or node parameters
$\tau_{i}$) used during the eLASSO fitting procedure Figure 12: Attribute-wise
variation of the tuning parameter ($\lambda$) that was used to achieve the
best fit set of neighbors during the Ising prior learning

###  6.3 Generalizing beyond Normalizing Flows: Revealing Bias in Generative
Adversarial Networks Trained on CelebA/CelebA-HQ

We can further confirm that the CelebA dataset is the root cause of the bias
shown experimentally by revealing similar biases in other models, such as
Generative Adversarial Networks. Below, we experiment with two GAN
architectures trained on the CelebA dataset.

Note that the pretrained models only allow the manipulation of 13 attributes:
Bald, Bangs, Black_Hair, Blond_Hair, Brown_Hair, Bushy_Eyebrows, Eyeglasses,
Male, Mouth_Slightly_Open, Mustache, No_Beard, Pale_Skin, Young.

####  6.3.1 AttGAN: Facial Attribute Editing by Only Changing What You Want

AttGAN, by He et al., was created specifically to modify attributes
realistically and in isolation, so that users can “change what [they] want”
(He et al., 2017). Users can choose to modify any number of attributes
simultaneously, as well as adjust the magnitude of adjustment per attribute.

The authors of AttGAN understand that attributes in CelebA are highly
correlated, claiming that in previous models, “adding blond hair always makes
a male become a female because most blond hair objects are female in the
training set”. He et al. correct the effects of this correlation by placing
constraints on the images generated from latent representations, enforcing
these constraints by employing an attribute classifier, a main contribution of
their paper.

We begin by observing that adjusting the Male attribute negatively causes the
AttGAN model to generate long hair and add makeup. Next, we adjust the
Blond_Hair attribute, and observe that this causes the GAN to generate blue
eyes and long hair.

Figure 13: Top: Example of interpolating Male attribute between -2 and +2.
Bottom: Example of interpolating Blond_Hair attribute between -2 and +2.
Leftmost image is original image.

We also increase the Blond_Hair attribute and the Male attributes together.
One might expect increasing the Male attribute to counteract the long-hair
that the Blond_Hair attribute creates. However, as we can see below, it on
occasion acts constructively. It is clear that, even with the addition of an
attribute classifier to constrain the model, the latent representation
manipulations for separate attributes are still not independent. We note,
though, that this behavior is seen mostly when the original images are of
males. He et al. themselves conclude that model errors occur mostly when large
modifications are made to images, such as adding hair to a bald figure.

Lastly, we investigate the manipulation of facial hair. It seems that
mustaches and beards are always generated black regardless of the original
subject’s hair color. In fact, if we attempt to generate a mustache on top of
a subject who already has facial hair, it simply changes the color of the
existing facial hair to black. Additionally, if we generate a mustache on top
of a young or less masculine subject, the rest of the face is morphed into
that of a more elderly and rugged man.

Figure 14: Top: Examples of manipulating both Male and Blond_Hair attributes
by +1. Middle: Manipulating Mustache attribute by +1 on a subject with
existing facial hair. Bottom: Manipulating Mustache attribute by +1 on a young
subject with less masculine features.

####  6.3.2 STGAN: A Unified Selective Transfer Network for Arbitrary Image
Attribute Editing

We also carry out experiments on STGAN, a model built by Liu et al. (Liu et
al., 2019) extending He et al.’s work. Their main contribution is the
selectivity of information passed into their model’s generators, choosing only
to pass in the difference in attribute vector between the source and target
image, so that the generator supposedly only affects the targeted change in
attribute.

As with AttGAN, we begin by manipulating the Male attribute. We can see that
STGAN is less inclined to generate longer hair when decreasing this attribute,
but it generates black splotches in the shape of hair, simply without the
texture and details of hair. It is obvious that STGAN still suffers from the
safe bias in the priors as other models.

Like AttGAN, STGAN changes the eye color of its subject towards blue when
increasing the Blond_Hair attribute. However, it does not generate long hair
like AttGAN does.

Figure 15: Top: Example of interpolating Male attribute between -2 and +2.
Bottom: Example of interpolating Blond_Hair attribute between -2 and +2.
Leftmost image is original image.

Increasing the Blond_Hair and the Male attributes together reproduces the
constructive interaction seen in AttGAN, where even more blond long hair is
generated than increasing just Blond_Hair on its own. This underlying complex
interaction in the latent space seems to be a consistent problem across
generative models, as far as we can tell.

Like AttGAN, generated facial hair is always black, regardless of the original
subject’s hair color, generating a mustache on top of a subject who already
has facial hair simply darkens the color of the existing facial hair, and
generating a mustache on a young/soft-featured subject makes them appear older
and more masculine.

Figure 16: Top: Examples of manipulating both Male and Blond_Hair attributes
by +1. Middle: Manipulating Mustache attribute by +1 on a subject with
existing facial hair. Bottom: Manipulating Mustache attribute by +1 on a young
subject with less masculine features.

####  6.3.3 DNA-GAN: Learning Disentangled Representations from Multi-
Attribute Images

Figure 17: Interpolating on ‘wearing eyeglasses’ changes the racial profile of
the subject matter for DNA-GAN.

DNA-GAN is a supervised method for disentangling multiple factors of variation
simultaneously by using multi-attribute images (Xiao et al., 2018). Trained on
CelebA, it can manipulate several attributes in the latent representations of
images, which is a generalization of GeneGAN (Zhou et al., 2017). DNA-GAN
replaces the explicit nulling loss with the annihilating operation and employs
a single discriminator for guiding images generation on multiple attributes.

While it seeks to disentangle multiple factors and attributes of images, we
see the inherent bias reflected in the CelebA dataset as per Figure 17. While
we manipulate across the ‘wearing eyeglasses’, the racial profile of the
subject matter being manipulated also changes accordingly.

####  6.3.4 Generative Adversarial Interpolative Autoencoding (GAIA)

The Generative Adversarial Interpolative Autoencoder (GAIA) is novel hybrid
between the Generative Adversarial Network (GAN) and the Autoencoder (AE)
(Sainburg et al., 2018). It addresses the issue of GANs which are non-
bidirectional, while also addressing issues of autoencoders which produces
blurry images, and addressing the non-conves latent spaces of autoencoders.

GAIA promotes a convex latent distribution by training adversarially on latent
space interpolations. Trained on CelebA, GAIA produces non-blurry samples that
match both high- and low-level features of the original images. Upon
visualizing the inherent latent spaces learnt, we see that attributes such as
‘big nose’ and ‘big lips’ correlate to racial profiles that reveal the
inherent bias in CelebA, as per Figure 8.