# Entropy Jump and Entropic Central Limit Theorem for Independent Sum

Liuquan Yao $\cdot$ Shuai Yuan

Abstruct It is a manuscript for results about entropic central limit theorem
for independent sum under finite Poincar$\acute{\mbox{e}}$ constant
conditions.

##  1 Introduction

###### Definition 1.1

Consider a random variable $X$ with density $f$, the differential entropy of
$X$ is

$h(X)=h(f):=-\int_{\mathbb{R}}f(x)\ln f(x)dx.$

###### Definition 1.2

Set $X$ is a random variable with absolutely continuous density $f$ and its
Radon-Nikodym derivative $f^{\prime}$, call

$J(X)=J(f):=\int_{\\{f(x)>0\\}}\frac{f^{\prime}(x)^{2}}{f(x)}dx$

the Fisher information of $X$.

If a density $\hat{f}$ satisfies $\hat{f}(x)=f(x),a.s.$ then we put

$J(\hat{f})=J(f).$

In other cases, we set $J(X)=\infty.$

There is a inequility focused by some studies, the Entropy Power
Inequility(EPI)

$e^{2h(X+Y)}\geq e^{2h(X)}+e^{2h(Y)},$

or its corollary, the Entropy Jump Inequility(EJI)

$h(\frac{X+Y}{\sqrt{2}})\geq\dfrac{h(X)+h(Y)}{2},$ (1.1)

for any independent random variables $X,Y$ with densities. Carlen(1991)[3]
used the Ornstein-Uhlenbeck Semigroup to prove the existance of the non-zero
gap $\delta$ in EJI and the entropic central limit theorem can be a directly
corollary.

Johnson(2000)[5] extended Carlen’s work to the independent case with some
special conditions(not just i.i.d.), and Ball etc.(2003)[2], Johnson
etc.(2004)[7] gave a feasible expression of the gap $\delta$ for i.i.d. case
respectively by different methods. Particularly, the paper [7] showed the
speed of convergence in standard Fisher information sense is $O(\frac{1}{n}).$
Ball ect.(2004)[1] also gave the speed and considered the weighted sum case.
We analogize the porjection method in Johnson(2004) [7] to the indepenednt
case(not i.i.d.), and obtain that for two nonsingular random variables $X,Y$
with absolutely continuous densities and their Poincar$\acute{\mbox{e}}$
constant satisfies $R^{*}_{X},R^{*}_{Y}<\infty$, then $\exists
a=a(\sigma^{2}_{x},\sigma^{2}_{Y}),R=R(R_{X}^{*},R_{Y}^{*})>0$,

$h(\dfrac{X+Y}{\sqrt{2}})-\dfrac{h(X)+h(Y)}{2}\geq\dfrac{\min\\{a,1\\}}{\min\\{a,1\\}+2R}\left[\dfrac{1}{2}\ln(2\pi
e\dfrac{\sigma_{X}^{2}+\sigma_{Y}^{2}}{2})-\dfrac{h(X)+h(Y)}{2}\right].$

Recently, Johnson(2020)[6] relaxed the finite Poincar$\acute{\mbox{e}}$
constant condition for entropy gap theory used some functional analysis tools.

##  2 Fisher Information Inequalities for Convolution

The convolution Fisher information inequality is essential theory for entropy
jump and entropic CLT. Paper [7] gave a convolution Fisher information
inequality for i.i.d. case under finite Poincar$\acute{\mbox{e}}$ constant
condition .

###### Definition 2.1

Give a random variable $X$, define

$R_{X}^{*}=\sup_{g\in
H_{1}^{*}(X)}\dfrac{\mathbb{E}[g(X)^{2}]}{\mathbb{E}[g^{\prime}(X)^{2}]},$

where
$H_{1}^{*}(X):=\\{f|f~{}is~{}absolutely~{}continuous,\text{Var}[f(X)]>0,\mathbb{E}[f(X)]=0,\mathbb{E}[f(X)^{2}]<\infty,\mathbb{E}[f^{\prime}(X)]=0\\}.$
$R^{*}_{X}$ is called the Poincar$\acute{\mbox{e}}$ constant of $X$.

By the definition, $R^{*}_{aX}=a^{2}R^{*}_{X},\;\forall a\neq 0.$ Suppose
finite Poincar$\acute{\mbox{e}}$ constant, then Johnson(2004) [7] obtained:

###### Proposition 2.1

( [7] )

Consider $X_{1},X_{2}$ IID with absolutely continuous densities, variance
$\sigma^{2}$ and restricted Poincar$\acute{\mbox{e}}$ constant $R^{*}$, then

$J(\dfrac{X_{1}+X_{2}}{\sqrt{2}})\leq\dfrac{2R^{*}}{\sigma^{2}+2R^{*}}J(X_{1}).$

This Proposition can be expanded to independent situation.

###### Proposition 2.2

Consider two independent random variables $X,Y$, with absolutely continuous
densities and finite variances $\sigma^{2}_{X},\sigma^{2}_{Y}$, and there
exists $0<R<\infty$ s.t.

$R_{X}^{*}\leq R,\;\;R_{Y}^{*}\leq R.$

Denote
$\dfrac{1}{2}[J(X)+J(Y)]=J_{1},J(\frac{X+Y}{\sqrt{2}})=J_{2},\frac{\sigma_{X}^{2}+\sigma_{Y}^{2}}{2}=\sigma^{2}$,
then

$\dfrac{\sigma^{2}J_{1}-1}{\sigma^{2}+2R}\leq
J_{1}-J_{2}\leq\dfrac{\sigma^{2}J_{1}-1}{\sigma^{2}}.$ (2.1)

Proof: In Section 5. $\blacksquare$

##  3 Entropy Jump

Using entropy to prove CLT can be trace back to 1959 [8] and entropy jump is
the basic tool. Paper [3] found that entropy of convolution always increase
with fixed variance unless there are all Gaussian random variables.

######  Theorem 3.1 ([3] )

Let $X_{1}$ and $X_{2}$ be independent random variables with zero mean and
$\sigma^{2}(X_{1})=\sigma^{2}(X_{2})=1$. Suppose

$J(X_{1}),J(X_{2})\leq J_{0}<\infty.$

Let $L$ be a given function on $[0,\infty)$ which decreases to zero at
infinity s.t. $L_{X_{1}},L_{X_{2}}\leq L$. Also let some number $a$ satisfying
$0<a<\frac{1}{2}$ be given. Suppose for some $\varepsilon>0$,

$h(X_{1})\leq h(g)-\varepsilon,$

where $g$ is the density of standard normal distribution. Then there is a
$\delta_{\varepsilon}>0$ depending only on $J_{0},L,a$ and $\varepsilon$ so
that for any $\lambda$ with

$a\leq\lambda^{2}\leq 1-a,$

we have

$h(\lambda
X_{1}+(1-\lambda^{2})^{1/2}X_{2})-\lambda^{2}h(X_{1})-(1-\lambda^{2})h(X_{2})\geq\delta_{\varepsilon}.$

For finite Poincar$\acute{\mbox{e}}$ conatant condition, such as log-concave
density, specific gap for entropy jump can be deduced as a corollary of
convolution Fisher information inequality.

######  Theorem 3.2 ([2] )

Let $X$ be a random variable with variance 1 and finite entropy, whose density
$f$ satisfies the Poincar$\acute{\mbox{e}}$ inequality

$R\int_{\mathbb{R}}fs^{2}\leq\int_{\mathbb{R}}f(s^{\prime})^{2},$

for any function $s$ satisfying

$\int_{\mathbb{R}}fs=0,$

where $R>0$ is a constant.

Set $Y$ is an independent copy of $X$, then

$h(\dfrac{X+Y}{\sqrt{2}})-h(X)\geq\dfrac{R}{2+2R}[h(g)-h(X)],$

where $g$ is the density of standard normal distribution.

Finally, we give the definition of the Lebesgue adjoint to Ornstein-Uhlenbeck
Semigroup which was used in our entropy jump theorem.

###### Definition 3.1

(Lebesgue adjoint to Ornstein-Uhlenbeck Semigroup) Set $t\geq 0$, define a map
$p_{t}^{*}:\mathcal{T}\rightarrow\mathcal{T}$

$(p_{t}^{*}f)(x)=\int_{\mathbb{R}}f(y)g_{1-e^{-2t}}(x-e^{-t}y)dy.$

And an important theorem, de Bruijn’s identity, which shows the relationship
between FIsher information and entropy.

###### Theorem 3.3

(de Bruijn’s identity[4]) Let $X$ be a random variable with density
$f\in\mathcal{T}$, $G$ is standard Gaussian distribution which is independent
with $X$. Then the function $t\mapsto h(X+\sqrt{t}G)$is continuous
differentiable on $(0,\infty)$, and

$\frac{d}{dt}h(X+\sqrt{t}G)=\frac{1}{2}J(X+\sqrt{t}G).$

By de Bruijn’s identity, we could deduce the following theorem from
Proposition 2.2 for the bounds of entropy jump in indepdent case.

###### Theorem 3.4

Consider two independent nonsingular random variables $X,Y$, with absolutely
continuous densities and finite variances $\sigma^{2}_{X},\sigma^{2}_{Y}$, and
there exists $0<R<\infty$ s.t.

$\max\\{R_{X}^{*},R_{Y}^{*},R^{*}_{G}\\}\leq R.$

where $G$ is standard Gaussian distribution. Then there exists
$c=c(\sigma^{2}_{X},\sigma^{2}_{Y},R)$ s.t.

$c\left(\dfrac{1}{2}\ln(2\pi
e\dfrac{\sigma_{X}^{2}+\sigma_{Y}^{2}}{2})-\dfrac{h(X)+h(Y)}{2}\right)\leq
h(\dfrac{X+Y}{\sqrt{2}})-\dfrac{h(X)+h(Y)}{2}.$

Proof: By de Bruijn’s identity(Theorem 3.3),

$h(X)=h(G)-\int_{0}^{\infty}[J(p_{s}^{*}X)-1]ds,$

where $G$ is standard Gaussian distribution and the define of $p_{s}^{*}$ can
be found in Definition 3.1. So

$h(\dfrac{X+Y}{\sqrt{2}})-\dfrac{h(X)+h(Y)}{2}=\int_{0}^{\infty}\dfrac{1}{2}[J(p_{s}^{*}X)+J(p_{s}^{*}Y)]-J(\dfrac{p_{s}^{*}X+p_{s}^{*}Y}{\sqrt{2}})ds.$

Suppose $R$ is bigger than the Poincar$\acute{\mbox{e}}$ constant of $X,Y,G$,
then by the fact in [2] that: for any random variables $X,Y$ satisfying
$R^{*}_{X},R^{*}_{Y}\leq R$, then $\forall\lambda\in[0,1]$,

$R^{*}_{\lambda X+(1-\lambda^{2})^{1/2}Y}\leq R.$ (3.1)

Thus by (2.1),

$\displaystyle h(\dfrac{X+Y}{\sqrt{2}})-\dfrac{h(X)+h(Y)}{2}$
$\displaystyle\geq\int_{0}^{\infty}\dfrac{1-(1-\sigma^{2})e^{-2s}}{1-(1-\sigma^{2})e^{-2s}+2R}[\dfrac{1}{2}J(p_{s}^{*}X)+\dfrac{1}{2}J(p_{s}^{*}Y)-\dfrac{1}{1-(1-\sigma^{2})e^{-2s}}]ds$
$\displaystyle\geq\dfrac{\min\\{a,1\\}}{\min\\{a,1\\}+2R}\left(\int_{0}^{\infty}[\dfrac{1}{2}J(p_{s}^{*}X)+\dfrac{1}{2}J(p_{s}^{*}Y)-1]ds+\int_{0}^{\infty}\dfrac{(\sigma^{2}-1)e^{-2s}}{1-(1-\sigma^{2})e^{-2s}}ds\right)$
$\displaystyle=\dfrac{\min\\{a,1\\}}{\min\\{a,1\\}+2R}\left(\dfrac{1}{2}\ln(2\pi
e)-\dfrac{h(X)+h(Y)}{2}+\int_{0}^{\infty}\dfrac{(\sigma^{2}-1)e^{-2s}}{1-(1-\sigma^{2})e^{-2s}}ds\right)$
$\displaystyle=\dfrac{\min\\{a,1\\}}{\min\\{a,1\\}+2R}[\dfrac{1}{2}\ln(2\pi
e\sigma^{2})-\dfrac{h(X)+h(Y)}{2}],$

where
$\sigma^{2}=\frac{\sigma^{2}_{X}+\sigma^{2}_{Y}}{2},a=\min\\{\sigma^{2}_{X},\sigma^{2}_{Y}\\},$
then we complete the proof by taking

$c=\dfrac{\min\\{a,1\\}}{\min\\{a,1\\}+2R}.$

$\blacksquare$

###### Remark 3.1

If $\sigma_{X},\sigma_{Y}\geq\sigma_{0}$, then we can take

$c=\dfrac{\min\\{\sigma^{2}_{0},1\\}}{\min\\{\sigma^{2}_{0},1\\}+2R}.$ (3.2)

##  4 Central Limit Theorem for Independent Sum

###### Theorem 4.1

Consider an independent nonsingular random variable sequence
$X_{1},X_{2},\cdots$ with absolutely continuous densities, and denote the sum
under the power of 2 as

$\hat{S}_{n}=\dfrac{X_{1}+X_{2}+\cdots+X_{2^{n}}}{2^{\frac{n}{2}}}.$

If the sequence $\\{X_{n},n\geq 1\\}$ satisfies the following conditions:

  1. 1.

Uniformly finite Poincar$\acute{\mbox{e}}$ condition
$R:=\sup_{n}R^{*}_{X_{i}}<\infty,\;\;\forall i\in\mathbb{Z}^{+}$;

  2. 2.

Stable entropy condition

$h(\dfrac{X_{1}+X_{2}\cdots+X_{2^{m}}}{2^{m/2}})=h(\dfrac{X_{2^{m}+1}+X_{2^{m}+2}\cdots+X_{2^{m+1}}}{2^{m/2}}),\;\;\forall
m\in\mathbb{N};$

  3. 3.

Convergent variance condition
$\text{Var}(\hat{S}_{n})\to\sigma^{2}>0,\;n\to\infty$, for a real number
$\sigma$;

  4. 4.

Zero mean condition $\mathbb{E}(X_{i})=0,\;\;\forall i\in\mathbb{Z}^{+}$.

Then

$\|p_{\hat{S}_{n}}-\phi_{\sigma^{2}}\|_{L^{1}}\to 0,\;\;n\to\infty,$ (4.1)

where $p_{\hat{S}_{n}}$ is the density of $\hat{S}_{n}$, $\phi_{\sigma^{2}}$
is the density of $\mathcal{N}(0,\sigma^{2})$.

Proof: For any given $m$, by condtion 1 and (3.1). we can use induction to
show that

$R^{*}_{\frac{X_{1}+X_{2}\cdots+X_{2^{m}}}{2^{m/2}}}\leq
R,R^{*}_{\frac{X_{2^{m}+1}+X_{2^{m}+2}\cdots+X_{2^{m+1}}}{2^{m/2}}}\leq
R,\;\;\forall m\in\mathbb{N}.$

We denote $h_{m}:=h(\hat{S}_{m})$, then due to Theorem 3.4 and condition 2, we
obtain

$h_{m+1}-h_{m}\geq\dfrac{\min\\{b,1\\}}{\min\\{b,1\\}+2R}\left(\frac{1}{2}\ln(2\pi
e\text{Var}(\hat{S}_{m+1}))-h_{m}\right),\;\;\forall m\in\mathbb{N},$ (4.2)

where $b:=\inf_{n}\text{Var}(\hat{S}_{n})>0$ since the condition 3 and the
nonsingular condition hold.

On the other hand, according to EJI (1.1), sequence $\\{h_{n},n\geq 1\\}$ is
increasing with upper bound $\frac{1}{2}\ln(2\pi
e[\sup_{n}\text{Var}(\hat{S}_{n})])<\infty$ since the condition 3, thus the
sequence $\\{h_{n},n\geq 1\\}$ is convergent. This implies

$h_{m+1}-h_{m}\to 0,\;\;m\to\infty.$

Recall that $\frac{1}{2}\ln(2\pi e\text{Var}(\hat{S}_{m+1}))-h_{m}\geq
h_{m+1}-h_{m}\geq 0$, from (4.2) we obatin

$\frac{1}{2}\ln(2\pi e\text{Var}(\hat{S}_{m+1}))-h_{m}\to 0,\;\;m\to\infty,$

therefore by condition 3,

$\frac{1}{2}\ln(2\pi e\text{Var}(\hat{S}_{m}))-h_{m}\to 0,\;\;m\to\infty,$

i.e.

$\text{KL}(\hat{S}_{m}\|\mathcal{N}(0,\text{Var}(\hat{S}_{m})))\to
0,\;\;m\to\infty,$

since condition 4 holds. By Pinsker inequility,

$\|p_{\hat{S}_{n}}-\phi_{\text{Var}(\hat{S}_{m})}\|_{L^{1}}\to
0,\;\;n\to\infty.$

Then the proof is completed since

$\|p_{\hat{S}_{n}}-\phi_{\sigma^{2}}\|_{L^{1}}\leq\|p_{\hat{S}_{n}}-\phi_{\text{Var}(\hat{S}_{n})}\|_{L^{1}}+\|\phi_{\sigma^{2}}-\phi_{\text{Var}(\hat{S}_{n})}\|_{L^{1}},\;\;\forall
n\in\mathbb{Z}^{+},$

and condition 3. $\blacksquare$

###### Corollary 4.1

If the condition 3 in Theorem 4.1 is replaced by

$\text{Var}(X_{i})=\sigma^{2},\;\;\forall i\in\mathbb{Z}^{+},$

then

$\text{KL}(\hat{S}_{n}\|\mathcal{N}(0,\sigma^{2}))=O\left((\dfrac{2R}{\min\\{\sigma^{2},1\\}+2R})^{n}\right),\;\;n\to\infty.$
(4.3)

Proof: By (4.2) and equal variance condition,

$h-h_{m+1}\leq(1-c)(h-h_{m}),\;\;\forall m\in\mathbb{N},$

where $h=\frac{1}{2}\ln(2\pi
e\sigma^{2}),c=\dfrac{\min\\{\sigma^{2},1\\}}{\min\\{\sigma^{2},1\\}+2R}.$
Thus

$\text{KL}(\hat{S}_{n}\|\mathcal{N}(0,\sigma^{2}))=h-h_{n}\leq(1-c)^{n}(h-h_{0}),$

where $h_{0}=h(X_{1})$. Using Pinsker inequility again we complete the proof.
$\blacksquare$

For normal sum $S_{n}:=1/\sqrt{n}\sum_{i=1}^{n}X_{i}$, we have the following
two corollaries from Theorem 4.1.

###### Corollary 4.2

Consider an independent nonsingular random variable sequence
$X_{1},X_{2},\cdots$ with absolutely continuous densities, and denote the sum

$S_{n}:=\dfrac{X_{1}+X_{2}+\cdots+X_{n}}{\sqrt{n}}.$

If the sequence $\\{X_{n},n\geq 1\\}$ satisfies the following conditions:

  1. 1.

Uniformly finite Poincar$\acute{\mbox{e}}$ condition
$R:=\sup_{n}R^{*}_{X_{i}}<\infty,\;\;\forall i\in\mathbb{Z}^{+}$;

  2. 2.

Increasing adding entropy condition

$h(\dfrac{X_{1}+X_{2}\cdots+X_{2^{m}}}{2^{m/2}})\leq
h(\dfrac{X_{2^{m}+1}+X_{2^{m}+2}\cdots+X_{2^{m+1}}}{2^{m/2}}),\;\;\forall
m\in\mathbb{N};$

  3. 3.

Mean and variance condition
$\mathbb{E}(X_{i})=0,\text{Var}(X_{i})=\sigma^{2},\;\;\forall
i\in\mathbb{Z}^{+}$.

Then the entropy sequence $\\{Ent_{n}:=h(S_{n}),n\geq 1\\}$ is convergent if
and only if

$\text{KL}(S_{n}\|\mathcal{N}(0,\sigma^{2}))\to 0,\;\;n\to\infty.$

Proof: Since $\text{KL}(S_{n}\|\mathcal{N}(0,\sigma^{2}))=\frac{1}{2}\ln(2\pi
e\sigma^{2})-Ent_{n}$, the one part of proof is finished.

Next we suppose $\\{Ent_{n},n\geq 1\\}$ is convergent, thus its subsequence
$\\{h_{n},n\geq 1\\}$ defined in Theorem 4.1 is also convergent. then consider
entropy jump $h_{n+1}-\dfrac{h_{n}+\bar{h}_{n}}{2},$ where

$\bar{h}_{n}=h(\dfrac{X_{2^{n}+1}+X_{2^{n}+2}\cdots+X_{2^{n+1}}}{2^{n/2}}).$

For one hand, by Theorem 3.4,

$h_{n+1}-\dfrac{h_{n}+\bar{h}_{n}}{2}\geq 0,$

thus

$\liminf_{n\to\infty}h_{n+1}-\dfrac{h_{n}+\bar{h}_{n}}{2}=\liminf_{n\to\infty}\dfrac{h_{n}-\bar{h}_{n}}{2}\geq
0.$ (4.4)

One the other hand, by condition 2,

$h_{n}-\bar{h}_{n}\leq 0,\forall n\in\mathbb{N}^{+}.$ (4.5)

Combining (4.4) and (4.5), we have

$\lim_{n\to\infty}h_{n}-\bar{h}_{n}=0,$

which implies

$\lim_{n\to\infty}h_{n+1}-\dfrac{h_{n}+\bar{h}_{n}}{2}=0.$

Use Theorem 3.4 again we obtain

$\lim_{n\to\infty}\dfrac{1}{2}\ln(2\pi
e\sigma^{2})-h_{n}=\lim_{n\to\infty}\dfrac{1}{2}\ln(2\pi
e\sigma^{2})-\dfrac{h_{n}+\bar{h}_{n}}{2}=0,$

i.e. a subsequence of $\\{Ent_{n},n\geq 1\\}$ converge to
$\dfrac{1}{2}\ln(2\pi e\sigma^{2})$, then we complete the proof due to the
convergence of the whole sequence $\\{Ent_{n},n\geq 1\\}$.

###### Corollary 4.3

Under the conditions in Corollary 4.1, we have

$\text{KL}(S_{n}\|\mathcal{N}(0,\sigma^{2}))=\begin{cases}O(n^{\log
c}),&\mbox{if~{}}c>1/2,\\\ O(n/\log n),&\mbox{if~{}}c=1/2,\\\
O(n^{-1}),&\mbox{if~{}}c<1/2,\end{cases}\;\;\mbox{as~{}}n\to\infty,$ (4.6)

where

$c=\dfrac{2R}{\min\\{\sigma^{2},1\\}+2R}.$

Proof: Denote $D_{n}=\text{KL}(S_{n}\|\mathcal{N}(0,\sigma^{2}))$. Since
$D_{n}$ is positive and subadditive, we have

$nD_{n}\leq 2^{k}D_{2^{k}}+2^{k-1}D_{2^{k-1}}+\cdots+D_{1},$ (4.7)

where $k=\lfloor\log n\rfloor.$ By the definition of $c$ and Corollary 4.1,
when $c\neq 1/2$ we have

$nD_{n}\leq|\frac{1-2^{k}c^{k}}{1-2c}|.$

Use $2^{k}\leq n$, we obatin

$D_{n}=O(n^{\alpha}),$

where

$\alpha=\max\\{-1,\log c\\}.$

When $c=1/2$, then by (4.7),

$nD_{n}\leq k+1\leq\log n+1.$

##  5  Proof of Proposition 2.2

We would prove the result for Fisher information jump in independent case,
following the method in [2].

###  5.1 Preparation

This part we show some conclusions given by [2], which still hold in just
independent case. We miss the proofs since they are the same as the i.i.d.
case.

The first is about the projection lemma, which is the core of the proof.

###### Lemma 5.1

([2] ) Given two independent random variables $X,Y$ with absolutely continuous
densities, and a function $f$ meeting

$\mathbb{E}[f^{2}(X+Y)]<\infty,\;\;\mathbb{E}[f(X+Y)]=0.$

Define

$g_{1}(u):=\mathbb{E}[f(u+Y)],\;\;g_{2}(v):=\mathbb{E}[f(X+v)].$

Then for any measurable functions $h_{1},h_{2}$,

$\mathbb{E}[f(X+Y)-h_{1}(X)-h_{2}(Y)]^{2}=\mathbb{E}[f(X+Y)-g_{1}(X)-g_{2}(Y)]^{2}+\mathbb{E}[g_{1}(X)-h_{1}(X)]^{2}+\mathbb{E}[g_{2}(Y)-h_{2}(Y)]^{2}.$

The vertical component of the projection is control by
Poincar$\acute{\mbox{e}}$ constant, provided that the expectation of random
variables are all $0$ [2], thus in Section 5, we suppose all random variables
have 0 expectation without specifically statement.

###### Lemma 5.2

([2] ) Given two independent random variables $X,Y$ with absolutely continuous
densities, and a function $f$ meeting

$\mathbb{E}[f^{2}(X+Y)]<\infty,\;\;\mathbb{E}[f(X+Y)]=0.$

Define

$g_{1}(u):=\mathbb{E}[f(u+Y)],\;\;g_{2}(v):=\mathbb{E}[f(X+v)].$

Then for any $\beta\in[0,1]$,

$E[f(X+Y)-h_{1}(X)-h_{2}(Y)]^{2}\geq\dfrac{1}{\bar{J}}\left(\dfrac{\beta}{R_{X}^{*}}\mathbb{E}[g_{1}(X)-\mu
X]^{2}+\dfrac{1-\beta}{R_{Y}^{*}}\mathbb{E}[g_{2}(Y)-\mu Y]^{2}\right),$

where $\bar{J}=(1-\beta)J(X)+\beta J(Y),\mu=-2J(X+Y).$

The next lemma gives the calculation about the score function of $X+Y$.

###### Lemma 5.3

([2] ) Given two independent random variables $X,Y$ with absolutely continuous
densities, then

$\rho_{X+Y}(x)=\mathbb{E}[\rho_{X}(X)|X+Y=x]=\mathbb{E}[\rho_{Y}(Y)|X+Y=x].$

Further,

$\dfrac{J(X)+J(Y)}{2}-J(\dfrac{X+Y}{\sqrt{2}})=2\mathbb{E}[\rho_{X+Y}(X+Y)-\dfrac{\rho_{X}(X)+\rho_{Y}(Y)}{2}]^{2}.$

Finally, some calculations are given:

###### Lemma 5.4

([2] ) Given two independent random variables $X,Y$ with absolutely continuous
densities, define

$f=2\rho_{X+Y},\;\;g_{1}(u):=\mathbb{E}[f(u+Y)].$

Then,

$\mathbb{E}[X\rho_{X}(X)]=\frac{1}{2}\mathbb{E}[g_{1}(X)X]+\frac{1}{2}\mathbb{E}[g_{2}(Y)Y]=-1.$
$\mathbb{E}[\rho_{X+Y}(X+Y)\rho_{X}(X)]=J(X+Y).$
$\mathbb{E}[X+\rho_{X}(X)]^{2}=J(X)-2+\mathbb{E}[X^{2}].$
$\mathbb{E}[\rho_{X}(X)]=\mathbb{E}[g_{1}(X)]=0.$

###  5.2 Proof of (2.1)

###### Theorem 5.1

Given two independent random variables $X,Y$ with absolutely continuous
densities, and constant $R$ meeting $R\geq\max\\{R_{X}^{*},R_{Y}^{*}\\}$, then

$J(\dfrac{X+Y}{\sqrt{2}})-\dfrac{1}{\sigma^{2}}\leq\dfrac{2R}{\sigma^{2}+2R}\dfrac{1}{2}[J(X)+J(Y)-\dfrac{2}{\sigma^{2}}],$

where
$\sigma^{2}=\frac{\sigma_{X}^{2}+\sigma_{Y}^{2}}{2}=\sigma^{2}_{\frac{X+Y}{\sqrt{2}}}.$

Proof:

We would give the estimations of
$\mathbb{E}[\rho_{\frac{X+Y}{\sqrt{2}}}(\frac{X+Y}{\sqrt{2}})-\frac{g_{1}(x)+g_{2}(y)}{\sqrt{2}}]^{2}$,
$\mathbb{E}[g_{1}(X)+X]^{2}$ and $\mathbb{E}[g_{2}(Y)+Y]^{2}$.

On the one hand, due to Lemma 5.1, $\frac{g_{1}(X)+g_{2}(Y)}{\sqrt{2}}$ is the
projection of $\rho_{\frac{X+Y}{\sqrt{2}}}(\frac{X+Y}{\sqrt{2}})$ on the space
constructed by $X,Y$(see Fig.1 in [2], page 400). On the other hand, denote
$K$ as the pedal of $\rho_{\frac{X+Y}{\sqrt{2}}}(\frac{X+Y}{\sqrt{2}})$ on the
line
$\left(\frac{\rho_{X}(X)+\rho_{Y}(Y)}{\sqrt{2}},-\frac{X+Y}{\sqrt{2}}\right)$,
thus

$\mathbb{E}[\frac{g_{1}(X)+g_{2}(Y)}{\sqrt{2}}+\frac{X+Y}{\sqrt{2}}]^{2}\geq\mathbb{E}[K+\frac{X+Y}{\sqrt{2}}]^{2},$
(5.1)

since the hypotenuse of a triangle is the longest.

Note that $K$ can be expressed as
$\lambda[\frac{\rho_{X}(X)+\rho_{Y}(Y)}{\sqrt{2}}]-(1-\lambda)[\frac{X+Y}{\sqrt{2}}],$
thus

$\lambda=\arg\min_{a\in[0,1]}\mathbb{E}\left[\rho_{\frac{X+Y}{\sqrt{2}}}(\frac{X+Y}{\sqrt{2}})-\left(a[\frac{\rho_{X}(X)+\rho_{Y}(Y)}{\sqrt{2}}]-(1-a)[\frac{X+Y}{\sqrt{2}}]\right)\right]^{2}.$

According to

$\arg\min_{a}\mathbb{E}[U-aV]^{2}=\dfrac{\mathbb{E}(UV)}{\mathbb{E}(V^{2})},$

and Lemma 5.4,

$\displaystyle\lambda$
$\displaystyle=\dfrac{\mathbb{E}\left([\rho_{\frac{X+Y}{\sqrt{2}}}(\frac{X+Y}{\sqrt{2}})+\frac{X+Y}{\sqrt{2}}][\frac{\rho_{X}(X)+\rho_{Y}(Y)}{\sqrt{2}}+\frac{X+Y}{\sqrt{2}}]\right)}{\mathbb{E}\left(\frac{\rho_{X}(X)+\rho_{Y}(Y)}{\sqrt{2}}+\frac{X+Y}{\sqrt{2}}\right)^{2}}$
(5.2)
$\displaystyle=\dfrac{J(\frac{X+Y}{\sqrt{2}})-2+(\sigma_{X}^{2}+\sigma_{Y}^{2})/2}{1/2J(X)+1/2J(Y)-2+(\sigma_{X}^{2}+\sigma_{Y}^{2})/2}$
(5.3) $\displaystyle\overset{\Delta}{=}\dfrac{A^{\prime}}{A}.$ (5.4)

Put (5.2) in (5.1),

$\mathbb{E}[\frac{g_{1}(X)+g_{2}(Y)}{\sqrt{2}}+\frac{X+Y}{\sqrt{2}}]^{2}=\dfrac{1}{2}\mathbb{E}[g_{1}(X)+X]^{2}+\dfrac{1}{2}\mathbb{E}[g_{2}(Y)+Y]^{2}\geq\dfrac{A^{\prime
2}}{A}.$ (5.5)

Next, define

$h_{1}(x)=h_{2}(x)=-x.$

By Lemma 5.1 and (5.5)

$\mathbb{E}\left(\rho_{\frac{X+Y}{\sqrt{2}}}(\frac{X+Y}{\sqrt{2}})-\frac{g_{1}(X)+g_{2}(Y)}{\sqrt{2}}\right)^{2}\leq
A^{\prime}-\dfrac{A^{\prime 2}}{A}.$

So due to Lemma 5.2

$\displaystyle A^{\prime}-\dfrac{A^{\prime
2}}{A}\geq\mathbb{E}[\rho_{\frac{X+Y}{\sqrt{2}}}(\frac{X+Y}{\sqrt{2}})-\frac{g_{1}(X)+g_{2}(Y)}{\sqrt{2}}]^{2}$
$\displaystyle\geq\dfrac{1}{2\bar{J}}\left(\dfrac{\beta}{R_{X}^{*}}\mathbb{E}[g_{1}(X)-\mu
X]^{2}+\dfrac{1-\beta}{R_{Y}^{*}}\mathbb{E}[g_{2}(Y)-\mu Y]^{2}\right)$
$\displaystyle=\dfrac{1}{2\bar{J}}[\dfrac{\beta}{R_{X}^{*}}[\mathbb{E}(g_{1}(X)+X)^{2}-2(\mu+1)(\sigma_{X}^{2}-1)+\sigma_{X}^{2}(\mu+1)^{2}]$
$\displaystyle\;\;\;\;+\dfrac{1-\beta}{R_{Y}^{*}}[\mathbb{E}(g_{2}(Y)+Y)^{2}-2(\mu+1)(\sigma_{Y}^{2}-1)+\sigma_{Y}^{2}(\mu+1)^{2}]].$

Take $\beta=\frac{R_{X}^{*}}{R_{X}^{*}+R_{Y}^{*}}$, and use (5.5) again,

$\displaystyle A^{\prime}-\dfrac{A^{\prime 2}}{A}$ (5.6)
$\displaystyle\geq\dfrac{1}{2\bar{J}}\left(\dfrac{\mathbb{E}[g_{1}(X)+X]^{2}+\mathbb{E}[g_{2}(Y)+Y]^{2}-2(\sigma_{X}^{2}+\sigma_{Y}^{2}-2)(1-J(\frac{X+Y}{\sqrt{2}}))+(\sigma_{X}^{2}+\sigma_{Y}^{2})(1-J(\frac{X+Y}{\sqrt{2}}))^{2}}{R_{X}^{*}+R_{Y}^{*}}\right)$
$\displaystyle\geq\dfrac{1}{R_{Y}^{*}J(X)+R_{X}^{*}J(Y)}[\dfrac{A^{\prime
2}}{A}-(\sigma_{X}^{2}+\sigma_{Y}^{2}-2)(1-J(\frac{X+Y}{\sqrt{2}}))+(J(\frac{X+Y}{\sqrt{2}})-1)^{2}\dfrac{\sigma_{X}^{2}+\sigma_{Y}^{2}}{2}].$

Suppose $\dfrac{\sigma_{X}^{2}+\sigma_{Y}^{2}}{2}=1$, then

$A^{\prime}-\dfrac{A^{\prime
2}}{A}\geq\dfrac{1}{R_{Y}^{*}J(X)+R_{X}^{*}J(Y)}[\dfrac{A^{\prime
2}}{A}+A^{\prime 2}].$

Due to $R\geq\max\\{R_{X}^{*},R_{Y}^{*}\\}$, then

$A^{\prime}-\dfrac{A^{\prime
2}}{A}\geq\dfrac{1}{R[J(X)+J(Y)]}[\dfrac{A^{\prime 2}}{A}+A^{\prime
2}]=\dfrac{A^{\prime 2}}{2RA}.$

Thus

$A^{\prime}\leq\dfrac{2R}{1+2R}A,$

i.e.

$J(\dfrac{X+Y}{\sqrt{2}})\leq\dfrac{2R}{1+2R}[\dfrac{1}{2}(J(X)-1)+\dfrac{1}{2}(J(Y)-1)]+1.$

For general case $1/2\sigma_{X}^{2}+1/2\sigma_{Y}^{2}=\sigma^{2}>0$,

$\displaystyle
J(\dfrac{X+Y}{\sqrt{2}})=\dfrac{1}{\sigma^{2}}J(\dfrac{X+Y}{\sigma\sqrt{2}})\leq\dfrac{1}{\sigma^{2}}\left(\dfrac{2R}{\sigma^{2}+2R}[\dfrac{1}{2}(\sigma^{2}J(X)-1)+\dfrac{1}{2}(\sigma^{2}J(Y)-1)]+1\right),$

i.e.

$J(\dfrac{X+Y}{\sqrt{2}})-\dfrac{1}{\sigma^{2}_{\frac{X+Y}{\sqrt{2}}}}\leq\dfrac{2R}{\sigma^{2}+2R}\dfrac{1}{2}[J(X)+J(Y)-\dfrac{4}{\sigma_{X}^{2}+\sigma_{Y}^{2}}].$

Then the first inequility has been proved. Recall $J_{2}\sigma^{2}\geq 1$, the
second inequility is easy to obtain.

$\blacksquare$

## References

  * [1] Shiri Artstein, Keith M. ball, Fanck Barthe, and Assaf Naor.  On the rate of convergence in the entropic central limit tneorem.  Probab. Theory Relat. Fields, 129:381–390, 2004.
  * [2] Keith Ball, Franck Barthe, and Assaf Naor.  Entropy jumps in the presence of a spectral gap.  Duke Mathematical Journal, 119(1), 2003.
  * [3] Soffer A. Carlen, E. A.  Entropy production by block variable summation and central limit theorems.  Communications in Mathematical Physics, 140:339–371, 1991.
  * [4] Thomas M Cover.  Elements of information theory.  John Wiley & Sons, 2006.
  * [5] Oliver Johnson.  Entropy inequalities and the central limit theorem.  Stochastic Processes and their Applications, 88:291–304, 2000.
  * [6] Oliver Johnson.  Maximal correlation and the rate of fisher information convergence in the central limit theorem.  IEEE Transactions on Information Theory, 66(8):4992–5002, 2020\.
  * [7] Oliver Johnson and Andrew Barron.  Fisher information inequalities and the central limit theorem.  Probab. Theory Relat. Fields, 129:391–409, 2004.
  * [8] Linnik.  An information-theoretic proof of the central limit theorem with the lindeberg condition.  Theory Probab. Appl., 4:288–299, 1959.


# Multi-class Generative Adversarial Networks for Semi-supervised Image
Classification

Saman Motamed  Institute of Medical Science, University of Toronto  Department
of Diagnostic Imaging, Neurosciences and Mental Health, The Hospital for Sick
Children<EMAIL_ADDRESS>Farzad Khalvati  Institute of Medical
Science, University of Toronto  Department of Diagnostic Imaging,
Neurosciences and Mental Health, The Hospital for Sick Children  Department of
Mechanical and Industrial Engineering, University of Toronto

###### Abstract

From generating never-before-seen images to domain adaptation, applications of
Generative Adversarial Networks (GANs) spread wide in the domain of vision and
graphics problems. With the remarkable ability of GANs in learning the
distribution and generating images of a particular class, they can be used for
semi-supervised classification tasks. However, the challenge is that if two
classes of images share similar characteristics, the GAN might learn to
generalize and hinder the classification of the two classes. In this paper, we
use various images from MNIST and Fashion-MNIST datasets to illustrate how
similar images cause the GAN to generalize, leading to the poor classification
of images. We propose a modification to the traditional training of GANs that
allows for improved multi-class classification in similar classes of images in
a semi-supervised learning framework.

_K_ eywords Generative Adversarial Networks $\cdot$ Image Classification
$\cdot$ Semi-supervised Classification

##  1 Introduction

Generative Adversarial Networks [1] is one of the most exciting inventions in
machine learning in the past decade. While applications of GANs spread wide in
the field of computer vision, image classification using GANs is relatively
unexplored. One of the early uses of GANs in image classification was
detecting anomalies in images, first introduced by Schlegl _et al._ [2] to
detect and identify anomalies in the form of retinal fluid or hyper-reflective
foci in optical coherence tomography (OCT) images of the retina. By defining a
variation score $V(x)$ (eq. 2), their proposed Anomaly Detection GAN (AnoGAN)
captured the characteristic and visual differences of two images; one
generated by the GAN and one real image. The idea was to, for instance, train
the GAN on only healthy images. When GAN is trained, the generator can
generate images similar to those in the healthy image class. During the test
phase, the variation score $V(x)$ must be low if the test image is healthy and
GAN’s generator (G) can generate a similar image to that of the healthy image.
If the test image is not healthy and contains anomalies, $V(x)$ would be
larger, and the generated image would look visually different than the real
image containing anomalies.

Recently, we proposed RANDGAN [3], a Generative Adversarial Network for binary
classification of COVID-19 negative (healthy and viral pneumonia) and COVID-19
positive chest X-ray images without the need to use any COVID-19 positive
images for training the model. By training two GANs, one on normal images and
one on pneumonia images, we calculated the final $V(x)$ for test image $x$.
The GAN trained on normal images would result in lower variation score for
normal images and higher scores for pneumonia and COVID-19 images. The GAN
trained on pneumonia images would result in low variation score for pneumonia
images and higher for normal and COVID-19. Adding the variation scores for
image $x$ from each GAN would result in higher $V(x)$ for COVID-19 positive
images compared to COVID-19 negative (normal and pneumonia) images [3]. Using
a similar approach to RANDGAN [3], a semi-supervised multi-class (e.g., 3
classes) image classification can be developed. Having labels for two classes
out of 3 classes of images, we used GANs to classify these two classes along
with a third unknown class for which we do not have labels and hence, do not
use any of the unknown images to train our model. The first step of multi-
class classification is to distinguish images of the unknown class from images
of the known classes (e.g., COVID-19 positive from negative images). The
second step is to classify the images that fall in the known images by using
one of the GANs trained on either of the known class of images (e.g. normal or
pneumonia). It is to be mentioned that we explore multi-class classification
in a setting where we have two known classes (C1 and C2) of images and an
unknown class (C3) of images. This approach can be extended to multiple known
classes and an unknown class of images.

We observed that, in some instances, training a GAN on images of a class C1
generated not only low variation scores for test images of the same class, but
also low scores for test images of class C2, hindering the ability to classify
C1 vs. C2. We hypothesized the reason to be the ability of the GAN’s generator
G, being trained on C1 images, generalizing to learn and generate images that
visually look similar to C2 images. In this work, we carried out multiple
experiments using different datasets to understand how visually similar images
affect GAN-based image classification’s performance. We propose _MCGAN_ , a
GAN-based multi-classs classifier, to overcome the challenge of classifying
visually similar images using GANs. By using all labeled images in training
the GANs, we force G to not generalize in a way that can generate similar
images to images of other classes.

##  2 Generative Adversarial Networks

###  2.1 Generative Adversarial Nets

A GAN is a deep learning model comprised of two main parts; Generator (G) and
Discriminator (D). G can be seen as an art forger that tries to reproduce art-
work and pass it as the original. D, on the other hand, acts as an art
authentication expert that tries to tell apart real from forged art.
Successful training of a GAN is a battle between G and D where if successful,
G generates realistic images and D is not able to tell the difference between
G’s generated images compared to real images. G takes as input a random
Gaussian noise vector and generates images through transposed convolution
operations. D is trained to distinguish the real images $(x)$ from generated
fake images $(G(z))$. Optimization of D and G can be thought of as the
following game of minimax [1] with the value function $V(G,D)$:

$\min_{G}\max_{D}V(D,G)=\mathbb{E}_{x_{{\sim_{P}}_{data{(x)}}}}[\log
D(x)]+\mathbb{E}_{z_{{\sim_{P}}_{z{(z)}}}}[\log(1-D(G(z)))]$ (1)

During training, G is trained to minimize D’s ability to distinguish between
real and generated images, while D is trying to maximize the probability of
assigning "real" label to real training images and "fake" label to the
generated images from G. The Generator improves at generating more realistic
images while Discriminator gets better at correctly identifying between real
and generated images. Today, when the term GAN is used, the Deep Convolution
GAN (DCGAN) [4] is the architecture that it refers to.

###  2.2 Multi-class Discriminator GAN

The goal of the proposed GAN-based multi-class (MCGAN) classifier is to
distinguish three classes of data (C1, C2, C3) from one another, while only
two classes (C1 and C2) have labels and the third class (C3) is unknown and
has no labels. In doing so, first, we classify C1 and C2 vs. C3, and then
classify C1 vs. C2. Since we have labels for only C1 and C2, the first step is
to learn the distribution of these two classes using two GANs. Here, we
describe the architecture for learning the distribution of C1 images and then
classifying the images of C1 vs. C2. This can be repeated to learn the
distribution of C2 images and then classifying the images of C2 vs. C1. To
learn the distribution of C1 images, a traditional GAN’s (DCGAN, AnoGAN, etc.)
discriminator takes as input the generator’s output (labeled Fake) and a real
C1 image (labeled Real). This forces the generator to learn the distribution
of the images from the real class. If the images of C1 and C2 shares similar
characteristics, training the GAN on the images of C1 could cause G to learn
and generalize well enough, leading to generating similar images to C2 and
hence, hindering the classification of the two known classes (C1 vs. C2). To
overcome this challenge, we feed a third input to the discriminator; images of
C2.

While these are real images from C2, we label them as Fake. This forces the
generator not to learn to generalize to this similar class (C2) while learning
the characteristics of C1. When G generates an image that could pass as
belonging to C2, the discriminator flags it as a fake image, and G re-
evaluates its learning at those stances. Figure 1 shows the architecture of
Multi-class GAN (MCGAN). In imbalanced datasets where the number of images is
not the same for both C1 and C2, we use batches of randomly selected images of
C2 at each training iteration.

Figure 1: Multi-class Generative Adversarial Network

###  2.3 Variation Score

The Variation score $V(x)$ for the query image $x$, proposed by Schlegl _et
al._ [2], is defined as;

$V(x)=(1-\lambda)\times\mathcal{L}_{R}({z})+\lambda\times\mathcal{L}_{D}({z})$
(2)

where $\mathcal{L}_{R}({z})$ (eq. 3) and $\mathcal{L}_{D}({z})$ (eq. 4) are
the residual and discriminator loss respectively that enforce visual and image
characteristic similarity between real image x and generated image $G(z)$. The
discriminator loss captures image characteristics using the output of an
intermediate layer of the discriminator, $f(.)$, making the discriminator act
as an image encoder. Residual loss is the pixel-wise difference between image
x and $G(z$).

$\mathcal{L}_{R}({z})=\sum|x-G(z)|$ (3)
$\mathcal{L}_{D}({z})=\sum|f(x)-f(G(z)|$ (4)

Before calculating V(x) in test, a point $z_{i}$ has to be found through back-
propagation that tries to generate an image as similar as possible to image x.
The loss function used to find $z_{i}$ is based on residual and discriminator
loss defined below.

$\mathcal{L}({z_{i}})=(1-\lambda)\times\mathcal{L}_{R}({z_{i}})+\lambda\times\mathcal{L}_{D}({z_{i}})$
(5)

$\lambda$ adjusts the weighted sum of the overall loss and variation score. We
used $\lambda=0.2$ to train our proposed MCGAN and AnoGAN [2]. Both
architectures were trained with the same initial conditions for performance
comparison.

##  3 Datasets

We used images from two different datasets. MNIST [5] dataset that contains
60,000 training images of handwritten digits and 10,000 test images. Fashion-
MNIST [6] is a dataset of Zalando’s article images—consisting of a training
set of 60,000 examples and a test set of 10,000 examples. In experiments where
test sets are not balanced, we randomly select the same number of images as
the smaller test set for class _a_ from the bigger test set of class _b_. All
gray-scale images were resized to $64\times 64$ pixels, with pixel intensities
scaled to -1 to 1.

Figure 2: MNIST and Fashion MNIST sample images

##  4 Experiments

To pick a subset of similar classes from MNIST and Fashion-MNIST (F-MNIST)
datasets that could cause generalization in GANs, we used metric learning [7].
The goal of metric learning is to train models that can embed inputs into a
high-dimensional space such that "similar" inputs are located close to each
other. To bring images from the same class closer to each other via the
embedding, the training data was constructed as randomly selected pairs of
images from each class matched to the label of that class, instead of
traditional (X,y) pairs where y is the label for corresponding X as singular
images of each class. By embedding the images using a shallow three-layer CNN,
we computed the similarity between the image pairs by calculating the cosine
similarity of the embeddings. We used these similarities as logits for a
softmax. This moves the pairs of images from the same class closer together.
After the training was complete, we sampled 10 examples from each of the 10
classes, and considered their near neighbours as a form of prediction; that
is, does the example and its near neighbours share the same class. This is
visualized as a confusion matrix shown in figure 3. The numbers that lie on
the diagonal represent the correct classifications and the numbers off the
diagonal represent the wrong labels that were misclassified as the true label.
We intentionally used a shallow three-layer CNN to enforce some
misclassification, as achieving near-perfect results in classifying datasets
such as MNIST using CNNs is easy. Using the information from figure 3, we
picked the class pairs (9, 4) and (8, 3) from the MNIST dataset and (Coat,
Shirt), (Coat, Pullover), and (Boot, Sandal) from F-MNIST dataset.

For semi-supervised multi-class classification of the pair of known images
that we have labels for and an unknown class which we will introduce when
testing our models, we trained two GANs. One GAN was trained to generate
images similar to each class of known images. For instance, for the pair (9,
4), one DCGAN (AnoGAN and DCGAN have the same architecture) was trained on 9s
and one was trained on 4s. Similarly, one MCGAN was trained on 9s, labeled
Real while 4s were labeled Fake and one MCGAN was trained on 9s, labeled Fake
while 4s were labeled Real. The models were trained using an NVIDIA GeForce
RTX 2080 Ti with 11 GB of memory.

Figure 3: Confusion matrix of MNIST and F-MNIST embeddings

##  5 Results

For classification of images from known C1 and C2 and unknown C3 classes,
first we classify the unknown images from known images by looking at the final
variation score which is the sum of variation scores from each trained GAN
(the one GAN trained on C1 and the one trained on C2). After classifying C3
images, we need to classify C1 and C2 images. For this, we have two options;
1) use the GAN trained on C1 images to classify the remaining images as C1 or
C2 or 2) use the GAN trained on C2 images and perform classification. In order
to pick the best instance for step 2, we look at the performance of supervised
binary classification of the known classes C1 and C2. The GAN that performs
better in classifying C1 and C2 is used for the second step of multi-class
classification in the semi-supervised setting when we introduce images of
class C3. Figure 4 shows this process.

###  5.1 Supervised binary classification

We calculated variation scores for both DCGANs and MCGANs. Lower variation
scores would translate to the test image having more probability of belonging
to the class of images the GAN was trained to generate images of, while a
larger variation score decreased this probability. We calculated the area
under the ROC curve (AUC) of each model. Table 1 shows the AUC for binary
classification for each data pair and model. For each pair (C1, C2) for DCGAN,
the first AUC is the result of training GAN on C1 images and the second AUC is
the result of GAN training on C2 images. For MCGAN, the first AUC is the
result of using C1 with Real labels and C2 with Fake, while the second AUC is
for vise versa. In both scenarios of using MCGAN to generate images of class
C1 or C2, MCGAN outperformed DCGAN in binary classification.

MNIST MNIST F-MNIST F-MNIST F-MNIST (8 / 3) (9 / 4) (Boot / Sandal) (Coat /
Shirt) (Coat / Pullover) DCGAN 0.87 / 0.9 0.88 / 0.78 0.65 / 0.72 0.68 / 0.54
0.67 / 0.32 MCGAN 0.92 / 0.95 0.91 / 0.84 0.87 / 0.82 0.79 / 0.74 0.77 / 0.71

Table 1: Classification AUCs

###  5.2 Semi-supervised multi-class classification

For semi-supervised multi-class classification, we added images from class 9
to (3, 8), 8 to (4, 9), Dress to (Coat, Pullover), Bag to (Boot, Sandal) and
Pullover to (Coat / Shirt). First, we calculate the accuracy of classifying
the unknown class from known classes, then, by using results from table 1, we
picked the GAN that performed better in classifying the known pairs, and used
that GAN to label the remaining test images that were not categorized as the
unknown class - C3. Figure 4 shows the two step process of multi-class
classification using the two trained GANs. In step 1, the combined variation
scores from the two trained GANs on known classes scores the unknown class
images higher than known classes. This allows for separation of known classes
from the unknown class. In the second step, we can use either GAN (we picked
the one with better performance according to table 1) to classify the known
classes from one another.

Figure 4: Two step process of multi-class classification using two GANs
trained on known classes

Table 2, with the format (C1, C2 - C3), shows the accuracy of classifying C3
(unknown class) from C1 and C2 (known classes used in training) and C1 from
C2. Number a from the pair (a / b) in the table shows the accuracy of
classifying the unknown class C3 and b shows the accuracy of classifying C1
from C2. MCGAN outperformed multi-class classification compared to DCGAN in
all but one case where DCGAN’s classification accuracy of Bag from Sandal and
Boot was 80% compared to MCGAN’s 79%. For the classification of known classes
however, MCGAN outperformed DCGAN in all instances.

MNIST MNIST F-MNIST F-MNIST F-MNIST (3, 8 - 9) (9, 4 - 8) (Sandal, Boot - Bag)
(Coat, Shirt - Pullover) (Coat, Pullover - Dress) DCGAN 0.67 / 0.57 0.87 /
0.79 0.8 / 0.77 0.66 / 0.42 0.73 / 0.52 MCGAN 0.69 / 0.64 0.9 / 0.84 0.79 /
0.81 0.67 / 0.65 0.79 / 0.65

Table 2: Semi-supervised classification AUCs

Figure 5 shows how a DCGAN, trained on images from C1, can generate images
similar to C2 if both classes have similar characteristics in a way that while
learning to generate C1 images, G learns to produce similar images to C2.
MCGAN, on the other hand, forces the generator to avoid this generalization,
which helps the classification task (table 1). The first row of figure 5 shows
the output of DCGAN and MCGAN’s generators trying to generate a similar image
to the test image from class 4 and 3 while having been trained to generate
images of class 9 and 8 respectively. The DCGAN’s output looks closer to the
test image compared to MCGAN. The second row result from the two GANs trained
to generate images of class Boot and generate images similar to test images
from class Sneaker and Sandal respectively. While DCGAN learns to generate
images similar to Sneaker and Sandal while learning from images from class
Boot, our MCGAN succeeds in not making this generalization. For the pair Coat
and Shirt, we picked the Pullover class as the unknown class on purpose as all
three classes are suggested to be similar according to figure 3.

Figure 5: DCGAN and MCGAN generated images

The semi-supervised classification using MCGAN outperformed DCGAN in
classifying the unknown class in all but one experiment (Boot, Sandal - Bag)
where the accuracy of detecting the unknown class (bag) reduced from 80% to
79% using MCGAN while the classification of the known classes improved from
77% to 81%. MCGAN enabled a better classification of the known classes (Tables
1, and2) while improving the classification of the unknown class in all but
one experiment (Table 2).

##  6 Discussion

In classification settings where we do not have enough labeled images for a
class, semi-supervised modes of training that do not require images of that
class to train are of value. While GANs can be used to classify images, we
showed that in some settings where labeled images share similar
characteristics, the generalization ability of GANs can hinder the performance
of classification. Using images from MNIST and Fashion MNIST datasets, we
showed how, for instance, a GAN trained to generate images of handwritten
digit 8 can also generate images that are similar to digit 3. To use GANs in
classifying 8s from 3s, this generalization would result in not only low
variation scores for images of digit 8, but also for images of digit 3. We
proposed MCGAN, which used both classes in training the GAN’s discriminator.
By labeling the digits 3 as fake, we guided the generator to not generate
images that can identify as 3 while learning to generate images of class 8.
This improved the multi-class classification of both the unknown class from
known classes and the known classes from one another. MCGAN however, as shown
in table 2 with classes (Coat, Shirt - Pullover) and (3, 8 - 9), although
works better than DCGAN, still does not perform well in the task of multi-
class classification. When the unknown class (Pullover, 9) shares similar
characteristics with some / all of the known classes, while MCGAN shows
improved results in classifying the classes compared to DCGAN, the
classification performance suffers as a result.

The goal for this study was not to achieve state of the art classification
results on the two datasets, rather using a simple GAN architecture and
showing how the proposed modification in training the discriminator can
improve classification in settings where over-generalization is possible. With
development of more complicated GAN architectures, such as RANDGAN [3] for
detection of COVID-19 X-rays, this modification can further improve the
accuracy of the models.

##  7 Conclusion

In this work, we demonstrated how GANs could learn to generalize to different
classes of images if they share similar characteristics with the class of
training images. This generalization can hinder the ability of GANs for the
task of image classification. We proposed using all labeled images in training
the discriminator to penalize the generalization. The multi-class
discriminator training showed improved accuracy of semi-supervised image
classification.

##  8 Acknowledgements

This research was funded by Chair in Medical Imaging and Artificial
Intelligence funding, a joint Hospital-University Chair between the University
of Toronto, The Hospital for Sick Children, and the SickKids Foundation.

## References

  * [1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [2] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs.  Unsupervised anomaly detection with generative adversarial networks to guide marker discovery.  In International conference on information processing in medical imaging, pages 146–157. Springer, 2017.
  * [3] Saman Motamed, Patrik Rogalla, and Farzad Khalvati.  Randgan: Randomized generative adversarial network for detection of covid-19 in chest x-ray.  arXiv preprint arXiv:2010.06418, 2020.
  * [4] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [5] Yann LeCun and Corinna Cortes.  MNIST handwritten digit database.  2010\.
  * [6] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:1708.07747, 2017.
  * [7] Brian Kulis et al.  Metric learning: A survey.  Foundations and trends in machine learning, 5(4):287–364, 2012\.


We finish this step.

######  Step 5.

With this step we will finish the proof of Theorem 4.6. In other words, for
any positive integer $d$, we find a structure of generalized klt pair
$(X,B,\boldsymbol{\rm M})$ on $X$, ample $\mathbb{Q}$-divisor $A$, and a
Cartier divisor $D$ satisfying all the conditions of Theorem 4.6.

We fix a positive integer $d$. Pick any positive integer $l$ such that
$lK_{X}$ is Cartier, $K_{X}+(l+1)\pi_{*}L$ is big, and ${\rm
dim}H^{0}(W,\mathcal{O}_{W}(2lH_{W}))\geq d$. We can find such $l$ because
$H_{W}$ is very ample and $\pi_{*}L$ is big ((4) in Step 1). By Step 2, the
pair $(X,0)$ is a $\mathbb{Q}$-factorial klt pair. Since $\pi_{*}L$ is nef and
big ((4) in Step 1), we can find an effective $\mathbb{Q}$-divisor $B$ and an
ample $\mathbb{Q}$-divisor $A$ on $X$ such that $(X,B+A)$ is klt and
$(l+1)\pi_{*}L\sim_{\mathbb{Q}}A+B.$ We define a b-nef $\mathbb{Q}$-b-Cartier
$\mathbb{Q}$-b-divisor $\boldsymbol{\rm N}$ by the closure of
$2f^{*}p_{V}^{*}H_{V}$. Since $(X,B+A)$ is $\mathbb{Q}$-factorial klt, we can
find a rational number $u\in(0,1]$ such that the generalized pair
$(X,B,u\boldsymbol{\rm N})$ is generalized klt. We put $\boldsymbol{\rm
M}=u\boldsymbol{\rm N}$ and $D=-lK_{X}$.

We show that $(X,B,\boldsymbol{\rm M})$, $A$ and $D$ satisfy all the
conditions of Theorem 4.6. Clearly ${\rm dim}X=3$, and $X$ is
$\mathbb{Q}$-factorial by Step 2. Since
$K_{X}+A+B\sim_{\mathbb{Q}}K_{X}+(l+1)\pi_{*}L$, the divisor $K_{X}+B+A$ is
big. Therefore, the first condition of Theorem 4.6 holds true. By (3) in Step
1, we have $K_{X}+\pi_{*}L+\boldsymbol{\rm N}_{X}\sim 0$. Since
$\boldsymbol{\rm M}=u\boldsymbol{\rm N}$ and $D=-lK_{X}$, we obtain

$\begin{split}D=K_{X}-(l+1)K_{X}\sim_{\mathbb{Q}}K_{X}+B+A+\frac{l+1}{u}\boldsymbol{\rm
M}_{X}\end{split}$

Since $u\in(0,1]$, we have $\frac{l+1}{u}>1$. Thus, the second condition of
Theorem 4.6 holds. Finally, Since $D=-lK_{X}$, by Step 4 and our choice of
$l$, we have

${\rm dim}H^{1}(X,\mathcal{O}_{X}(D))\geq{\rm
dim}H^{0}(W,\mathcal{O}_{W}(2lH_{W}))\geq d.$

Therefore, the third condition of Theorem 4.6 holds.

We complete the proof of Theorem 4.6. ∎

###### Proof of Theorem 1.4.

It is clear from Theorem 4.6. ∎

##  5\. Appendix. On non-$\mathbb{R}$-Cartier analogue of non-vanishing
theorem

In this appendix, we give a small remark on the following
non-$\mathbb{R}$-Cartier analogue of Theorem 1.1 (2).

######  Question 5.1 (cf. Theorem 1.1).

Let $(X,B,\boldsymbol{\rm M})$ be a projective generalized lc pair such that
$B$ is a $\mathbb{Q}$-divisor and $\boldsymbol{\rm M}$ is a
$\mathbb{Q}$-b-Cartier $\mathbb{Q}$-b-divisor. Let $A$ be an ample
$\mathbb{Q}$-divisor on $X$. Suppose that $K_{X}+B+A+\boldsymbol{\rm M}_{X}$
is pseudo-effective.

Then, is there an effective $\mathbb{Q}$-divisor $D_{\alpha}$ on $X$ such that
$K_{X}+B+A+\alpha\boldsymbol{\rm M}_{X}\sim_{\mathbb{Q}}D_{\alpha}$ for every
rational number $\alpha>1$?

Theorem 4.4 shows that the statement holds in the case where
$K_{X}+B+A+(1-t)\boldsymbol{\rm M}_{X}$ is pseudo-effective in the sense of
Definition 4.2 for a real number $t>0$. In particular, the statement holds
when $\boldsymbol{\rm M}$ descends to a numerically trivial
$\mathbb{Q}$-Cartier divisor. Indeed, if $\boldsymbol{\rm M}$ descends to a
numerically trivial $\mathbb{Q}$-Cartier divisor, then there is a resolution
$f\colon\tilde{X}\to X$ such that the divisor $f^{*}(K_{X}+B+A+\boldsymbol{\rm
M}_{X})-\boldsymbol{\rm M}_{\tilde{X}}$ is pseudo-effective, so $K_{X}+B+A$ is
pseudo-effective in the sense of Definition 4.2.

If we can solve Question 5.1 affirmatively, there is an application to anti-
nef canonical divisors of klt varieties. The author learned the following
question in a discussion with Gongyo.

######  Question 5.2 (Non-vanishing for nef anti-canonical divisors).

Let $(X,0)$ be a projective klt pair. Suppose that $-K_{X}$ is nef. In this
situation, is there an effective $\mathbb{Q}$-divisor $D$ on $X$ such that
$-K_{X}\sim_{\mathbb{Q}}D$?

######  Theorem 5.3.

Assume that the statement of Question 5.1 holds for all generalized lc pairs
and ample $\mathbb{Q}$-divisors. Then, Question 5.2 can be solved
affirmatively.

###### Proof.

Assume that the statement of Question 5.1 holds for all generalized lc pairs
and ample $\mathbb{Q}$-divisors. Let $(X,0)$ be a projective klt pair such
that $-K_{X}$ is nef. We may assume that $-K_{X}$ is not big because otherwise
the non-vanishing for $-K_{X}$ is obvious.

Fix a very ample Cartier divisor $H$ on $X$, and consider the
$\mathbb{P}^{1}$-bundle

$Y:=\mathbb{P}_{X}(\mathcal{O}_{X}\otimes\mathcal{O}_{X}(-H))\overset{f}{\longrightarrow}X.$

We put $T$ as the unique section of $\mathcal{O}_{Y}(1)$. By construction,
$(Y,T)$ is plt, $T+f^{*}H$ is basepoint free, and we have
$K_{Y}+T+(T+f^{*}H)\sim K_{X}$. Let $\pi\colon Y\to Z$ be the contraction
induced by $T+f^{*}H$. Then, $\pi$ is birational and $T$ is the unique
$\pi$-exceptional divisor. We may find an ample Cartier divisor $A_{Z}$ on $Z$
such that $T+f^{*}H\sim\pi^{*}A_{Z}$. We define a $\mathbb{Q}$-b-divisor
$\boldsymbol{\rm M}$ on $X$ by the closure of $-f^{*}K_{X}$. Since
$K_{Y}+T+\pi^{*}A_{Z}+\boldsymbol{\rm M}_{Y}\sim_{\mathbb{Q}}0$ and $-K_{X}$
is nef, we see that $(Z,0,\boldsymbol{\rm M})$ is a generalized lc pair such
that $K_{Z}+\boldsymbol{\rm M}_{Z}+A_{Z}\sim_{\mathbb{Q}}0$.

By applying the statement of Question 5.1 to $(Z,0,\boldsymbol{\rm M})$ and
$A_{Z}$, we can find an effective $\mathbb{Q}$-divisor $D_{Z}$ on $Z$ such
that

$-\pi_{*}f^{*}K_{X}=\boldsymbol{\rm
M}_{Z}\sim_{\mathbb{Q}}K_{Z}+A_{Z}+2\boldsymbol{\rm
M}_{Z}\sim_{\mathbb{Q}}D_{Z}.$

Pick an integer $m>0$ so that $-m\pi_{*}f^{*}K_{X}\sim mD_{Z}$. Since $T$ is
the unique $\pi$-exceptional divisor, we can find a positive integer $n$ such
that $H^{0}(Y,\mathcal{O}_{Y}(-mf^{*}K_{X}+nT))\neq 0$. For each integer
$n^{\prime}\leq n$, we consider the exact sequence

$\begin{split}0\longrightarrow
H^{0}(Y,\mathcal{O}_{Y}(-mf^{*}K_{X}+(n^{\prime}-1)T))&\longrightarrow
H^{0}(Y,\mathcal{O}_{Y}(-mf^{*}K_{X}+n^{\prime}T))\\\ &\longrightarrow
H^{0}(T,\mathcal{O}_{T}((-mf^{*}K_{X}+n^{\prime}T)|_{T}))\end{split}$

which is induced by the exact sequence

$\begin{split}0\longrightarrow\mathcal{O}_{Y}(-mf^{*}K_{X}+(n^{\prime}-1)T)&\longrightarrow\mathcal{O}_{Y}(-mf^{*}K_{X}+n^{\prime}T)\\\
&\longrightarrow\mathcal{O}_{T}((-mf^{*}K_{X}+n^{\prime}T)|_{T})\longrightarrow
0.\end{split}$

By using $(T+f^{*}H)|_{T}\sim 0$ and $T\simeq X$, we have

$H^{0}(T,\mathcal{O}_{T}((-mf^{*}K_{X}+n^{\prime}T)|_{T}))\simeq
H^{0}(X,\mathcal{O}_{X}(-mK_{X}-n^{\prime}H))$

If $n^{\prime}>0$, then $H^{0}(X,\mathcal{O}_{X}(-mK_{X}-n^{\prime}H))=0$
because otherwise $-K_{X}$ is big which contradicts our assumption. Therefore,
for every $0<n^{\prime}\leq n$ we have

$H^{0}(Y,\mathcal{O}_{Y}(-mf^{*}K_{X}+(n^{\prime}-1)T))\simeq
H^{0}(Y,\mathcal{O}_{Y}(-mf^{*}K_{X}+n^{\prime}T)).$

Since $H^{0}(Y,\mathcal{O}_{Y}(-mf^{*}K_{X}+nT))\neq 0$, by an descending
induction on $n^{\prime}$, we obtain

$H^{0}(Y,-mf^{*}K_{X})\neq 0.$

From this, we can find an effective $\mathbb{Q}$-divisor $D$ on $X$ such that
$D\sim_{\mathbb{Q}}-K_{X}$. We finish the proof of Theorem 5.3. ∎

## References

  * [AK] D. Abramovich, K. Karu, Weak semistable reduction in characteristic $0$, Invent. math. 139 (2000), no. 2, 241–273.
  * [B1] C. Birkar, Anti-pluricanonical systems on Fano varieties, Ann. of Math., 19 (2019), no. 2, 345–463.
  * [B2] C. Birkar, Singularities of linear systems and boundedness of Fano varieties, preprint (2016), arXiv:1609.05543.
  * [B3] C. Birkar, Generalised pairs in birational geometry, expository article (2020),
  * [B4] C. Birkar, On connectedness of non-klt loci of singularities of pairs, preperint (2020), arxiv:2010.08226.
  * [BCHM] C. Birkar, P. Cascini, C. D. Hacon, J. McKernan, Existence of minimal models for varieties of log general type, J. Amer. Math. Soc. 23(2010), no. 2, 405–468.
  * [BH] C. Birkar, Z. Hu, Polarized pairs, log minimal models, and Zariski decompositions, Nagoya Math. J. 215 (2014), 203–224.
  * [BZ] C. Birkar, D. Q. Zhang, Effectivity of Iitaka fibrations and pluricanonical systems of polarized pairs, Publ. Math. Inst. Hautes Études Sci. 123 (2016), no. 1, 283–331.
  * [C] S. R. Choi, The geography of log models and its applications, PhD Thesis, Johns Hopkins University, 2008.
  * [Fi] S. Filipazzi, On a generalized canonical bundle formula and generalized adjunction, preprint (2018), arXiv:1807.04847.
  * [FS] S. Filipazzi, R. Svaldi, On the connectedness principle and dual complexes for generalized pairs, preprint (2020), arXiv:2010.08018.
  * [Fu1] O. Fujino, Special termination and reduction to pl flips. In Flips for $3$-folds and $4$-folds, Oxford University Press (2007).
  * [Fu2] O. Fujino, Non-vanishing theorem for log canonical pairs, J. Algebraic Geom. 20 (2011), no. 4, 771–783.
  * [Fu3] O. Fujino, Fundamental theorems for the log minimal model program, Publ. Res. Inst. Math. Sci. 47 (2011), no. 3, 727–789.
  * [Fu4] O. Fujino, Foundations of the minimal model program, MSJ Mem. 35, Mathematical Society in Japan, Tokyo, 2017.
  * [FG] O. Fujino, Y. Gongyo, Log pluricanonical representations and abundance conjecture, Compos. Math. 150 (2014) no. 4, 593–620.
  * [HM] C. D. Hacon, J. Moraga, On weak Zariski decompositions and termination of flips, to appear in Math. Res. Lett.
  * [HX] C. D. Hacon, C. Xu, On finiteness of $B$-representation and semi-log canonical abundance, Adv, Stud. Pure. Math. 70 (2016), Minimal Models and extremal rays–Kyoto, 2011, 361–378.
  * [HanLiu1] J. Han, W. Liu, On numerical nonvanishing for generalized log canonical pairs, to appear in Doc. Math.
  * [HanLiu2] J. Han, W. Liu, On generalized canonical bundle formula for generically finite morphisms, to appear in Ann. Inst. Fourier (Grenoble).
  * [HanLi] J. Han, Z. Li, Weak Zariski decompositions and log terminal models for generalized polarized pairs, preprint (2018), arXiv:1806.01234.
  * [H1] K. Hashizume, Remarks on special kinds of the relative log minimal model program, Manuscripta Math. 160 (2019), no. 3, 285–314.
  * [H2] K. Hashizume, A class of singularity of arbitrary pairs and log canonicalizations, Asian J. Math. 24 (2020), no. 2, 207–238.
  * [H3] K. Hashizume, Finiteness of log abundant log canonical pairs in log minimal model program with scaling, preprint (2020), arXiv:2005.12253.
  * [HH] K. Hashizume, Z. Hu, On minimal model theory for log abundant lc pairs, J. Reine Angew. Math., 767 (2020), 109–159.
  * [KM] J. Kollár, S. Mori, Birational geometry of algebraic varieties. With the collaboration of C. H. Clemens and A. Corti. Translated from the 1998 Japanese original. Cambridge Tracts in Mathematics, 134. Cambridge University Press, Cambridge, 1998.
  * [LP1] V. Lazić, T. Peternell. On generalised abundance I, Publ. Res. Inst. Math. Sci. 56 (2020), no. 2, 353–389.
  * [LP2] V. Lazić, T. Peternell. On generalised abundance II, Peking Math. J. 3 (2020), no. 1, 1–46.
  * [LT] V. Lazić, N. Tsakanikas, On the existence of minimal models for log canonical pairs, to appear in Publ. Res. Inst. Math. Sci.
  * [L] J. Liu, Sarkisov program for generalized pairs, preprint (2018), arXiv:1802.03926.
  * [M] J. Moraga, Termination of pseudo-effective 4-fold flips, preprint (2018), arXiv:1802.10202.
  * [N] N. Nakayama, Zariski-decomposition and abundance, MSJ Mem., 14, Mathematical Society of Japan, Tokyo, 2004.


††thanks<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Topologically protected quantized changes of the distance between atoms

Ali Emami Kopaei1,2 Krzysztof Giergiel2,3,4 Krzysztof Sacha2,5

###### Abstract

Thouless pumping enables the transport of particles in a one-dimensional
periodic potential if the potential is slowly and periodically modulated in
time. The change in the position of particles after each modulation period is
quantized and depends solely on the topology of the pump cycle, making it
robust against perturbations. Here, we demonstrate that Thouless pumping also
allows for the realization of topologically protected quantized changes of the
distance between atoms if the atomic s-wave scattering length is properly
modulated in time.

##  I Introduction

The study of time-dependent Hamiltonians has emerged as a vibrant and
consequential field within the realm of quantum mechanics. These Hamiltonians
play a crucial role in understanding the behavior of quantum systems governed
by time-varying potentials, which can give rise to a diverse array of
captivating and unforeseen phenomena. One particularly intriguing example that
has garnered significant attention is the phenomenon known as Thouless
pumping, first introduced by David J. Thouless in 1983 Thouless1983. Thouless
pumping is a remarkable manifestation of time-dependent Hamiltonians, where
the periodic modulation of the potential induces the transport of particles
across a sample in a well-controlled and quantized manner. The transport is
topologically protected and thus resistant to perturbations and has been
realized experimentally Atala2013; Folling:2007p55101.

On the other hand, when a periodically moving particle is resonantly driven by
a periodically changing external force, an intriguing phenomenon arises. Its
motion, when observed in a frame moving along an undisturbed periodic orbit,
can be effectively described by a Hamiltonian reminiscent of solid-state
physics. In this analogy, the effective Hamiltonian bears resemblance to that
of an electron moving through a crystalline potential formed by ions Guo2013;
Sacha15a. What is interesting is that when this particle is observed in the
laboratory frame, it exhibits solid-state behavior in the time domain
Sacha15a; sacha16. That is, changes in time of the probability for detecting
the particle at a fixed position close to the periodic orbit reproduces the
solid-state behavior described by the effective Hamiltonian in the moving
frame. The potential in the effective Hamiltonian can be engineered by
skillfully selecting the Fourier components of the periodically changing
external force. By doing so, it becomes possible to explore a wide range of
condensed matter phenomena, which include Anderson localization, topological
molecules, topological crystals, quasi-crystals, Mott-insulator phase, and
many-body localization, all within the realm of time-domain observations
Guo2013; Sacha15a; sacha16; Guo2016; Guo2016a; Giergiel2017; delande17;
Mierzejewski2017; Liang2017; Giergiel2018; Giergiel2018a; Lustig2018;
Giergiel2018b; Matus_2021; Giergiel2021; Giergiel2022; kopaei2022; kopaei2023.
This captivating domain of research is known as condensed matter physics in
time crystals, with comprehensive reviews available Sacha2017rev; Guo2020;
SachaTC2020; GuoBook2021. It is worth emphasizing that time crystal
investigations also encompass systems capable of spontaneously breaking time
translation symmetry Wilczek2012; Bruno2013b; Watanabe2015; Syrwid2017;
Kozin2019. In the case of periodically driven systems, this implies the
spontaneous breaking of discrete time translation symmetry dictated by the
external drive, leading to the formation of discrete time crystals Sacha2015;
Khemani16; ElseFTC; Giergiel2020; Wang2020; Wang2021; yarloo_2020. Excitingly,
experimental demonstrations of discrete time crystals have already been
achieved Zhang2017; Choi2017; Pal2018; Rovny2018; Smits2018.

In this current research, we demonstrate a platform to observe the topological
pumping of the particles: The modulation of the atomic s-wave scattering
length in time. By adiabatically changing periodic modulation of the
scattering length, we can realize quantized changes of the relative distance
between atoms and this process is topologically protected. These findings
contribute to the scientific understanding of controllable interactions and
pave the way for further exploration of topological phenomena in experimental
systems Walter2023; Arguello-Luengo2023.

The paper is organized as follows. In Sec. II we introduce our system by
considering two atoms moving on a one-dimensional (1D) ring and derive an
effective Hamiltonian that describe resonante motion of the atoms. In Sec. III
we describe the Thouless pumping phenomenon that the effective Hamiltonian can
reveal and explain what it means for the two-atom system. Section IV contains
results of numerical simulations of the topologically protected control of the
relative distance between the atoms. The summary is given in Sec V.

##  II Model

Let us delve into the system of two atoms moving on a ring (see Fig.1). We
shall assume that these atoms belong to the same species yet exhibit
differences in their hyperfine states. When the kinetic energies are low, the
interaction between these atoms can be effectively captured by means of a
contact Dirac-delta potential with a strength, $4\pi g(t)$, proportional to
the s-wave scattering length of the atoms. This interaction is periodically
modulated in time which can be achieved through the utilization of either the
Feshbach resonance Chin2010 or the confinement-induced resonance Olshanii1998.
Consequently, the Hamiltonian of the system takes the following form:

$H=\frac{p_{1}^{2}+p_{2}^{2}}{2}+2\pi g(t)\delta(x_{1}-x_{2}),$ (1)

where $x_{i}$ and $p_{i}$ represent the positions and conjugate momenta of the
atoms on a ring, respectively. The periodic modulation $g(t)$ in time can read
as follow,

$g(t)=V_{1}\sin^{2}(\omega t-\phi/2)+V_{2}\sin^{2}(2\omega t+\pi/2).$ (2)

The energy and length of the system are conveniently expressed in units of
$\hbar^{2}/mR^{2}$ and $R$, respectively, where $R$ symbolizes the radius of
the ring and $m$ represents the mass of the atoms.

The interaction potential in Eq.(1) is dependent on the relative distance
between the atoms, resulting in the decoupling of the center-of-mass degree of
freedom from the relative coordinate. In this way, the Hamiltonian can be
expressed in terms of the center-of-mass coordinate, $X=(x_{1}+x_{2})/2$, and
the relative distance coordinate, $x=x_{1}-x_{2}$, as well as their
canonically conjugate momenta, $P=p_{1}+p_{2}$ and $p=(p_{1}-p_{2})/2$,
respectively. The Hamiltonian of the system can be rewritten in terms of this
coordinate as follows:

$H=H_{c.m.}+H_{rel},$ (3)

where

$H_{c.m.}=\frac{P^{2}}{4},$ (4) $H_{rel}=p^{2}+2\pi g(t)\delta(x).$ (5)

According to Eq.(4), the center of mass degree of freedom exhibits behavior
similar to the free particle. However, the relative coordinate exhibits more
interesting behaviors especially when periodic modulation in $g(t)$ is
resonant with the motion of the atoms along the ring. We assume the atoms are
moving in the opposite velocities close to $\pm\omega/2s$ where $s$ is an
integer number. To describe the system it is advantageous to consider the
plane wave basis $\ket{n}$ as our chosen basis and switch to a frame of
reference that moves with the atoms using a unitary transformation,
$H\rightarrow UHU^{\dagger}+i(\frac{\partial}{\partial t}U)U^{\dagger}$, where

$U\ket{n}=\begin{cases}e^{-in\omega t/s}\ket{n}&\text{for }n\geq 0,\\\
e^{+in\omega t/s}\ket{n}&\text{for }n<0.\end{cases}$ (6)

Figure 1: Presents schematic plot of the system considered in the paper. Two
atoms are moving with the opposite velocity along the ring and are resonantly
driven by periodic modulation of the s-wave scattering length.

For the system in close proximity to the resonance, where the wave number $n$
is around $\pm\omega/2s$, we can obtain an effective time-independent
Hamiltonian by averaging the exact Hamiltonian over time (secular
approximation) Berman1977; Lichtenberg1992; Buchleitner2002. However, we have
to consider two resonance subspaces. In the first resonance subspace, where
$n,n^{\prime}>0$, the time averaging leads to the following expression:

$\displaystyle\begin{aligned}
&H_{eff}^{(1,1)}=[(n^{2}-n\omega/s)\delta_{n^{\prime},n}+\Delta(n^{\prime},n)]\ket{n^{\prime}}\bra{n}\end{aligned}$
(7)

in which,

$\displaystyle\Delta(n^{\prime},n)$ $\displaystyle=$
$\displaystyle\int_{0}^{T}\frac{dt}{T}\int_{0}^{2\pi}\frac{dx}{2\pi}e^{ix(n-n^{\prime})}e^{i\omega
t(n^{\prime}-n)/s}4\pi g(t)\delta(x)$ (8) $\displaystyle=$ $\displaystyle
V_{1}(\delta_{n^{\prime},n}/2-\delta_{n^{\prime},n+2s}e^{-i\phi}/4-\delta_{n^{\prime},n-2s}e^{i\phi}/4)$
(9) $\displaystyle+$ $\displaystyle
V_{2}(\delta_{n^{\prime},n}/2+\delta_{n^{\prime},n+4s}/4+\delta_{n^{\prime},n-4s}/4),$
(10)

where $T=2s\pi/\omega$. The same result will hold for considering the second
subspace ($n,n^{\prime}<0$),
$H_{eff}^{(2,2)}=[(n^{2}+n\omega/s)\delta_{n^{\prime},n}+\Delta(n^{\prime},n)]\ket{n^{\prime}}\bra{n}$.
However, there will be a coupling between these two subspaces due to the zero-
range Dirac-delta potential 111The coupling is not present if a sufficiently
smooth version of the Dirac-delta potential is used Giergiel2018; SachaTC2020;
Matus_2021.. In the case of $n>0$ and $n^{\prime}<0$,

$\displaystyle\begin{aligned}
&H_{eff}^{(2,1)}=\Delta(-n^{\prime},n)\ket{n^{\prime}}\bra{n},\end{aligned}$
(11)

with $\Delta$ given in Eq.(10), and similarly for $n<0$ and $n^{\prime}>0$,
${H_{eff}^{(1,2)}=\Delta(n^{\prime},-n)\ket{n^{\prime}}\bra{n}}$.

Thus, the complete effective Hamiltonian can be expressed as,

$\displaystyle H_{eff}=\sum_{i,j=1}^{2}H_{eff}^{(i,j)}.$ (12)

Apart from the omitted constant term, we can block diagonalize the Hamiltonian
of Eq.(12) resulting in,

$\displaystyle{H}_{eff}$ $\displaystyle=$
$\displaystyle(m_{1}-\omega/2s)^{2}\;\delta_{m_{1}^{\prime},m_{1}}\ket{m_{1}^{\prime}}\bra{m_{1}}$
$\displaystyle+$
$\displaystyle\left[(m_{2}-\omega/2s)^{2}\;\delta_{m_{2}^{\prime},m_{2}}+\Delta(m_{2}^{\prime},m_{2})\right]\ket{m_{2}^{\prime}}\bra{m_{2}},$

where $|m_{1,2}\rangle=(|n\rangle\mp|-n\rangle)/\sqrt{2}$. The constant terms
$\omega/2s$ can be eliminated if we redefine $m_{1,2}\rightarrow
m_{1,2}+\omega/2s$. The first term of Eq.(II), describes the free particle
moving on the ring and the dynamics in the corresponding Hilbert subspace is
trivial. However, the second part is non-trivial and the corresponding Hilbert
subspace is a subspace where the Thouless pumping we are interested in can be
realized. Looking at the form of the matrix elements of this part of the
Hamiltonian it is not obvious that in the relevant Hilbert space one can
realize the Thouless pumping. However, this part of the Hamiltonian is
identical to the matrix elements of the following Hamiltonian calculated in
the plane wave basis,

${H}_{eff}=p^{2}+2V_{1}\sin^{2}\left(sx-\frac{\phi}{2}\right)+2V_{2}\sin^{2}\left(2sx+\frac{\pi}{2}\right),$
(14)

and systems described by such a Hamiltonian can reveal the Thouless pumping
(see the next section).

Figure 2: (a) Comparison between the quasi-energies of the exact Hamiltonian,
Eq. (5), (blue dots) and the energies of the effective Hamiltonian, Eq. (14),
(red lines) for $\omega=10^{5}$, $V_{1}=32$, $V_{2}=16$, and $s=4$. The
parameters were selected in order to make the energy levels structure clearly
visible. (b) Differences, $\Delta E$, between the lowest eigenenergy levels
for each band of the effective Hamiltonian and the corresponding exact quasi-
energies of the system as a function of $\omega$ in a semi-log plot for
$\phi=0$. Inset shows the same but in a log-log plot. If $\omega$ is
sufficiently large, the agreement between the effective and exact results is
perfect. The parameters used in this simulation are $V_{1}=64$, $V_{2}=32$,
and $s=4$.

The spectrum of the system (14) as a function of the pumping parameter $\phi$
is demonstrated in Fig. (2)a. The spectrum contains two energy bands, and
these bands are well separated from each other with a gap. Each band consists
of $4s$ energy levels. In Fig.(2)b comparison of the spectra obtained by
diagonalization of the exact Floquet Hamiltonian (5) and the effective
Hamiltonian (14) is presented. If $\omega$ is sufficiently large, we obtain
perfect agreement. This means that the phenomena that we can predict with the
help of $H_{eff}$ are reproduced in the exact description of the system.

##  III Thouless Pumping

The effective Hamiltonian, Eq. (14), describes a system which can exhibit the
Thouless pumping phenomenon which has been shown theoretically and
demonstrated experimentally Atala2013; Peil:2003p61264; Folling:2007p55101;
Trotzky:2008p464; viebahn2023interactioninduced; Greschner; Sergi2024;
Ashirbad2023. Even though for some range of parameters, the system is well
described by the tight-binding approximation, in the following we use the
continuum modelQian2011; Wang2013.

The phenomenon of Thouless pumping is of significant interest as it offers a
means of controlling the flow of particles in a system. This control is
achieved by adiabatically changing the control parameter, $\phi$ in Eq. (14),
which results in the transfer of quantized units of charge through the system.
Here, it corresponds to changes of the average value $\langle x\rangle$ by the
same amount after each adiabatic change of $\phi$ by $2\pi$. The connection
between a Chern topological invariant and charge pumping in the 1D system (14)
is closely linked to the integer quantum Hall effect Thouless1982. In the
quantum Hall effect, charged particles move in a 2D space when a magnetic
field is applied perpendicular to their motion. When the $\phi$ changes
adiabatically, it is like threading magnetic flux through a cylinder. This
creates an electric field in a perpendicular direction and results in
quantized Hall conductance Thouless1983; Lohse2016.

Figure 3: Illustration of the Thouless pumping described by the Hamiltonian
(14) with the pumping control parameter $\phi$ changing adiabaticaly in time,
i.e. $\phi$ changes by $2\pi$ over the pumping period $\chi=4\times 10^{3}T$.
The figure shows quantized changes of the displacement, Eq. (16). Different
colors represent different Wannier states as the initial state. The parameters
used in our simulation are $V_{1}=64$, $V_{2}=32$, and $s=4$.

As discussed in Ref. Resta, in the case of the ring geometry with the periodic
boundary conditions, it is more convenient to analyze not the quantity
$\braket{x}$ but

$D=\mathrm{Im}\ln\bra{\psi}e^{ix}\ket{\psi},$ (15)

which involves an operator explicitly respecting the periodic boundary
conditions. In this way, the position operator $D$ is represented in terms of
a phase. The introduction of a time-varying adiabatic parameter $\phi(t)$
yields the observable $\partial_{t}D(t)/2\pi=J(t)$ representing the particle
current. Notably, the topological pumped charge Resta; Bardyn $\Delta n$ after
a cyclic protocol $t\in[0,\chi]$ with ${\phi(m\chi)-\phi(0)=m2\pi}$ is
quantized as an integer and reads

$\Delta n=\frac{s}{\pi}\left[D(\phi=2m\pi)-D(\phi=0)\right].$ (16)

Figure 3 illustrates the Thouless pumping. It shows how $D$ evolves in time
when different Wannier states of the first energy band of the Hamiltonian (14)
are chosen as initial states and $\phi(t)$ changes adiabatically in time,
i.e., $\phi(t)$ changes by $2\pi$ over the period $\chi=4\times 10^{3}T$ which
turns out to be sufficiently longer than the minimal tunneling time $2\times
10^{3}T$ between neighboring lattice sites of the Hamiltonian (14).

Typically a Hamiltonian of the form of Eq. (14) is related to particles in an
external spatially periodic potential which can reveal quantized translation
of the center of mass position of the particles when the control parameter
$\phi$ is adiabatically changed by $2\pi$. Here, the meaning of such a
Thouless pumping is different because $x$ in Eq. (14) describes the relative
distance between two atoms. In the following we show that the Thouless pumping
allows us to control quantized changes of the relative distance between the
atoms which are topologically protected.

Figure 4: Time evolution of the two-atom system. The initial wavefunction has
been chosen as a product of a Gaussian function (centered at $X=\pi$ and with
the standard deviation $\sigma_{c.m.}=0.2$) for the center of mass degree of
freedom and a single Wannier state of the first energy band of the Hamiltonian
(14) (with $s=4$ and $\phi=0$) for the relative degree of freedom. Panel (a)
shows the probability density in the laboratory frame at $t=0$. Panel (b)
shows the same but at $t=\chi=4\times 10^{3}T$, i.e. after one pumping cycle
when $\phi$ changes from 0 to $2\pi$. The maximum of the probability density
along the direction of the relative position of the atoms (diagonal of the
plot) is shifted by $\pi/4$ indicating the quantized Thouless pumping. The
parameters are the same as in Fig.(3).

##  IV adiabatic evolution

In this section we present results of the evolution of two atoms on the ring
with periodic modulation of the contact interaction strength, $g(t)$, and an
additional adiabatic changes of the pumping control parameter $\phi$, see Eq.
(2) and Eq. (14).

The center of mass degree of freedom of two atoms, denoted by $X$, is
described by the free-particle like Hamiltonian (4) while the relative
position degree of freedom of the atoms by the Hamiltonian (5) which in the
moving frame and in the relevant Hilbert subspace can be approximated by the
effective time-independent Hamiltonian (14).  Starting from a single point
$(x_{cm},x)$ it takes $4\pi$ distance moving in the $x$ direction to return to
the stating point. The corresponding first energy band of (14) consists of
$4s$ states. They form $2s$ pairs of states for $x$ in the range $[0,2\pi)$
connected by the change of $x_{cm}$. We assume that the initial state is a
product, $\psi_{c.m.}(X)\psi_{rel}(x)$, of a Gaussian wavefunction for the
center of mass degree of freedom,
$\psi_{c.m.}(X)\propto\exp[-(X-\pi)^{2}/2\sigma_{c.m.}^{2}]$ with
$\sigma_{c.m.}=0.2$, and a single Wannier state, $\psi_{rel}(x)$, of the first
energy band of the Hamiltonian (14) with $\phi=0$ for the relative degree of
freedom. Results of the time evolution in the laboratory frame with the
adiabatically changing pumping parameter $\phi(t)$ [i.e., $\phi(t)$ changes by
$2\pi$ over the time period $\chi=4\times 10^{3}T$ are presented in Fig. 4.
While the probability density spreads along the direction of the center of
mass position (anti-diagonal direction in Fig. 4), along the direction of the
relative position of the atoms we can see that the maximum of the probability
density is shifted by $\pi/4$. The latter means that the average distance
between the atoms has changed from $0$ to $\pi/4$. Each next pumping cycle
increases the average relative distance always by $\pi/4$ and such quantized
manipulation of the relative particles’ distance is topologically protected.
Results of three consecutive pumping cycles are presented in Fig. 5 but this
time the initial wavefunction for the relative degree of freedom,
$\psi_{rel}(x)$, has been chosen as a Gaussian superposition of the
neighboring Wannier states centered at $x=0$ and with the standard deviation
$\sigma_{rel}=0.17$. Despite the fact that the probability distribution is
spreading due to the tunneling between lattice sites of the potential in (14),
the average displacement after each pumping cycle is always $\pi/4$.

Figure 5: Similar results as presented in Fig. 4 but for more pumping cycles
and for the initial state of the relative degree of freedom of the atoms being
a superposition of the Wannier states with a Gaussian envelop centered at
$x=0$ and with the standard deviation $\sigma_{rel}=0.17$. (a) after each
pumping cycle (i.e., at $t=\chi$, $2\chi$ and $3\chi$ in the plots), the
probability density along the direction of the relative position of the atoms
changes by the same value $\pi/4$ indicating quantization of $\Delta n$, Eq.
(16). (b) shows the probability density integrated along the direction of the
center of mass position of the atoms at $t=0$ and after three consecutive
pumping cycles. The parameters are the same as in Fig.(3).

The quantized changes of the distance between atoms can be performed in ultra-
cold atoms. To give a taste of the experimental parameters, let us consider
the following example. Assume that two 39K atoms in the two different
hyperfine states, $\ket{F=1,m_{F}=0}$ and $\ket{F=1,m_{F}=-1}$, are trapped in
the quasi-1D ring with the toroidal trap of the radius $R=40~{}\mu$m and the
transverse harmonic confinement of the frequency $\omega_{\perp}=2\pi\times
10$ kHz, corresponding to the transverse radius 15 nm Wright2000. The atoms
are initially prepared in the Gaussian wavepackets of the width
$\sigma=6.8~{}\mu$m located on the opposite sides of the ring and with the
average velocities $\pm 25.1$ mm/s. The interspecies s-wave scattering length
can be modulated in time by changing the magnetic field close to the Feshbach
resonance at $113.76$ G Tanzi2018; DErrico_2007 or by means of confinement
induced resonances Chin2004. If the scattering length is periodically
modulated with frequency $\omega=3.2\pi$ kHz and amplitude $7.9$ nm (which
corresponds to $V_{2}=4.0$) and the modulation consists of $k_{m}=8$ harmonics
for $s=2$, then the quantized changes of the relative distance between the
atoms, we consider here, can be observed. In order to realize the effective
Hamiltonian, one has to apply time periodic driving with frequencies $\omega$
and $2\omega$. This requirement can be easily fulfilled in the experiment
because having the driving with the frequency $\omega$ its second harmonic
with the frequency $2\omega$ can be generated.

##  V Summary

We consider the behavior of a two-atom system moving on a ring where the
interaction between the atoms is described using the contact Dirac-delta
potential and its strength is modulated in time by means of either Feshbach
resonance or the confinement induced resonance. When the atoms are moving at
opposite velocities which fulfill the resonance condition, i.e., when the
driving frequency is an integer multiple of the frequency of the motion of the
atoms, we can apply the secular approximation. Then, in the frame moving with
the atoms, the system’s behavior can be accurately reproduced by a time-
independent effective Hamiltonian. Notably, the potential in the effective
Hamiltonian can be engineered to create the Thouless pumping phenomenon by
properly selecting the Fourier components of the periodically changing s-wave
scattering length of the atoms. This effective potential emerges from resonant
connections between various harmonics of atoms translational motion and the
time-periodic modulation of the original atom-atom interactions. We
demonstrated that if we initialize the relative position degree of freedom of
the atoms in a localized Wannier state or superposition of the Wannier states
with a Gaussian envelope, the system exibits the topologically protected
quantized changes of the distance between atoms. It is important to emphasize
that, in our system, the Thouless pumping is distinct from the usual
situation. Here, the quantized changes are intricately connected to the
relative distance between the two atoms, instead of the center of mass
position of particles, as the pumping control parameter undergoes an adiabatic
change.

##  VI ACKNOWLEDGMENTS

We acknowledge the support of the National Science Centre, Poland, via Project
No. 2018/31/B/ST2/00349 (A.E.K. and K.G.) is acknowledged. This research was
also funded in part by the National Science Centre, Poland, Project No.
2021/42/A/ST2/00017 (K.S.). The research has been supported by a grant from
the Priority Research Area (DigiWorld) under the Strategic Programme
Excellence Initiative at Jagiellonian University. The numerical computations
in this work were supported in part by PL-Grid Infrastructure, Project No.
PLG/2023/016644. Some parts of our numerical schemes are based on the
PETScpetsc-user-ref; petsc-efficient and SLEPcHernandez:2005 libraries.