# Bound states of two-dimensional Schrödinger-Newton equations

Joachim Stubbe  EPFL, IMB-FSB, Station 8, CH-1015 Lausanne, Switzerland
<EMAIL_ADDRESS>
(Date: 06th February 2008)

###### Abstract.

We prove an existence and uniqueness result for ground states and for purely
angular excitations of two-dimensional Schrödinger-Newton equations. From the
minimization problem for ground states we obtain a sharp version of a
logarithmic Hardy-Littlewood-Sobolev type inequality.

###### Key words and phrases:

Schrödinger-Newton equations, nonlinear Schrödinger equation, rearrangement
inequality, logarithmic Hardy-Littelwood-Sobolev inequality

###### 1991 Mathematics Subject Classification:

35Q55, 35Q40,47J10

I am are indebted to Marc Vuffray for providing us his numerical studies of
the model. I also wish to thank Philippe Choquard for many helpful discussins
during the preparation of this paper.

##  1\. Introduction

We consider the Schrödinger-Newton system

(1.1) $iu_{t}+\Delta u-\gamma Vu=0,\quad\Delta V=|u|^{2}$

in two space dimensions which is equivalent to the nonlinear Schrödinger
equation

(1.2) $iu_{t}+\Delta u-\frac{\gamma}{2\pi}(\ln(|x|)*|u|^{2})u=0$

with nonlocal nonlinear potential

$V(x)=\frac{1}{2\pi}(\ln|x|*|u|^{2})(x,t)=\frac{1}{2\pi}\int_{\mathbb{R}^{2}}\ln(|x-y|)\,|u(t,y)|^{2}\;dy.$

We are interested in the existence of nonlinear bound states of the form

(1.3) $u(t,x)=\phi_{\omega}(x)e^{-i\omega t}.$

The Schrödinger-Newton system (1.1) in three space dimensions has a long
standing history. With $\gamma$ designating appropriate positive coupling
constants it appeared first in 1954, then in 1976 and lastly in 1996 for
describing the quantum mechanics of a Polaron at rest by S. J. Pekar [1], of
an electron trapped in its own hole by Ph. Choquard [2] and of selfgravitating
matter by R. Penrose [3]. The two-dimensional model is studied numerically in
[4]. For the bound state problem there are rigorous results only for the three
dimensional model. In [2] the existence of a unique ground state of the form
(1.3) is shown by solving an appropriate minimization problem. This ground
state solution $\phi_{\omega}(x),\omega<0$ is a positive spherically symmetric
strictly decreasing function. In [5] the existence of infinitely many distinct
spherically symmetric solutions is proven and, in [6], a proof for the
existence of anisotropic bound states is claimed. So far, there are no results
for the two-dimensional model using variational methods. One mathematical
difficulty of the two-dimensional problem is that the Coulomb potential in two
space dimensions is neither bounded from above nor from below and hence does
not define a positive definite quadratic form. However, recently in [7] the
existence of a unique positive spherically symmetric stationary solution
$(u,V)$ of (1.1) such that $V(0)=0$ has been proven by applying a shooting
method to the corresponding system of ordinary differential equations.

In the present paper, we are mainly interested in the ground states of the
model

(1.4) $u(t,x)=\phi_{\omega}(x)e^{-i\omega t},\quad\phi_{\omega}(x)>0.$

We prove the existence of ground states by solving an appropriate minimization
problem for the energy functional $E(u)$. By use of strict rearrangement
inequalities we shall prove that there is a unique minimizer (up to
translations and a phase factor) which is a positive spherically symmetric
decreasing function (theorems 3.2 and theorem 3.3). The existence and
uniqueness of solutions of the form (1.4) for given $\omega$ ,however, depends
on the frequency $\omega$. It will turn out that for any $\omega\leq 0$ there
is a unique ground state of the form (1.4). In addition, we shall prove there
is a positive number $\omega^{*}>0$ independent of the coupling $\gamma$ such
that for any positive $\omega<\omega^{*}$ there are two ground states of the
form (1.4) with different $L^{2}$-norm. In the limit case $\omega=\omega^{*}$
there is again a unique ground state (theorem 3.4). We then apply our
existence result to obtain a sharp Hardy-Littlewood-Sobolev inequality for the
logarithmic kernel (theorem 4.1).

Finally, we prove the existence of non radial solutions of the Schrödinger-
Newton system (1.1) which in polar coordinates $(r,\theta)$ are of the form

(1.5) $\psi(t,r,\theta)=\phi_{m,\omega}(r)e^{im\theta-i\omega
t},\quad\phi_{m,\omega}(r)\geq 0$

for positive integers $m$. These are eigenfunctions of the angular momentum
operator $L=-i\partial_{\theta}$. Again, we prove the existence of such
$\phi_{m,\omega}(r)$ by solving an appropriate minimization problem for the
corresponding energy functional reduced to functions of the form (1.5) for any
given $m$ (theorem 5.1). The minimizers can be interpreted as purely angular
excitations and we also prove their uniqueness in this class of functions
(theorem 5.2).

##  2\. Mathematical Framework

###  2.1. Functional Setting

The natural function space $X$ for the quasi-stationary problem is given by

(2.1) $X=\\{u:\mathbb{R}^{2}\to\mathbb{C}:\;\int_{\mathbb{R}^{2}}|\nabla
u|^{2}+|u|^{2}+\ln(1+|x|)\,|u|^{2}\;dx<\infty\\}.$

The space $X$ is a Hilbert space and by Rellich’s criterion (see, e.g. theorem
XIII.65 of [8]) the embedding $X\hookrightarrow L^{2}$ is compact. We note
$X_{r}$ the space of radial functions in $X$. Formally, the energy $E$
associated to (1.2) is given by

(2.2) $\begin{split}E(u)&=\int_{\mathbb{R}^{2}}|\nabla
u(x)|^{2}\;dx+\frac{\gamma}{4\pi}\int_{\mathbb{R}^{2}}\int_{\mathbb{R}^{2}}\ln(|x-y|)\,|u(x)|^{2}|u(y)|^{2}\;dxdy\\\
&=T(u)+\frac{\gamma}{2}V(u)\\\ \end{split}$

In order to prove that the energy is indeed well defined on $X$ we decompose
the potential energy $V(u)$ into two parts applying the identity

(2.3) $\ln(r)=-\ln(1+\frac{1}{r})+\ln(1+r)$

for all $r>0$. We then define the corresponding functionals

(2.4)
$\begin{split}&V_{1}(u)=-\frac{1}{2\pi}\int_{\mathbb{R}^{2}}\int_{\mathbb{R}^{2}}\ln(1+\frac{1}{|x-y|})\,|u(x)|^{2}|u(y)|^{2}\;dxdy\\\
&V_{2}(u)=\frac{1}{2\pi}\int_{\mathbb{R}^{2}}\int_{\mathbb{R}^{2}}\ln(1+{|x-y|})\,|u(x)|^{2}|u(y)|^{2}\;dxdy\\\
\end{split}$

######  Lemma 2.1.

The energy functional $E:X\longrightarrow\mathbb{R}_{0}^{+}$ is well defined
on $X$ and of class $C^{1}$.

###### Proof.

Since $0\leq\ln(1+\frac{1}{r})\leq\frac{1}{r}$ for any $r>0$ we have by the
Hardy-Littlewood-Sobolev inequality (see e.g. [9]) and Sobolev interpolation
estimates (see e.g. [10])that

$|V_{1}(u)|\leq C_{1}||u||^{4}_{8/3}\leq C_{2}||\nabla u||_{2}||u||_{2}^{3}$

for some constants $C_{1},C_{2}>0$. To bound the second term of the potential
energy we note that

$\ln(1+|x-y|)\leq\ln(1+|x|+|y|)\leq\ln(1+|x|)+\ln(1+|y|)$

and therefore

$|V_{2}(u)|\leq\frac{1}{\pi}\int_{\mathbb{R}^{2}}\ln(1+|x|)\,u(x)|^{2}\int_{\mathbb{R}^{2}}|u(y)|^{2}\;dxdy.\\\
$

The regularity properties of $E(u)$ are obvious. ∎

Finally, we consider the particle number (or charge) defined by

(2.5) $N(u)=\int_{\mathbb{R}^{2}}|u(x)|^{2}\;dx,$

which is also a well-defined quantity on $X$.

###  2.2. Scaling properties

If $\phi_{\omega}(x)$ is a solution of the stationary equation

(2.6)
$-\Delta\phi_{\omega}(x)+\frac{\gamma}{2\pi}\bigg{(}\int_{\mathbb{R}^{2}}\ln(|x-y|)\;|\phi_{\omega}(y)|^{2}\;dy\bigg{)}\;\phi_{\omega}(x)=\omega\phi_{\omega}(x),$

with finite particle number $N_{\omega}=N(\phi_{\omega})$ then by the virial
theorem,

(2.7) $T(\phi_{\omega})=\frac{\gamma}{8\pi}N^{2}(\phi_{\omega}).$

For any $\sigma>0$, the scaled function
$\phi_{\omega,\sigma}(x)=\sigma^{-2}\gamma^{1/2}\phi_{\omega}({x}/{\sigma})$
solves

(2.8)
$\begin{split}&-\Delta\phi_{\omega,\sigma}(x)+\frac{1}{2\pi}\bigg{(}\int_{\mathbb{R}^{2}}\ln(|x-y|)\;|\phi_{\omega,\sigma}(y)|^{2}\;dy\bigg{)}\;\phi_{\omega,\sigma}(x)\\\
&=\\\ &\sigma^{-2}\big{(}\omega+\frac{\gamma
N_{\omega}\ln\sigma}{2\pi}\big{)}\phi_{\omega,\sigma}(x).\\\ \end{split}$

If we choose $\sigma=\sigma_{\omega}$ such that $\omega+\frac{\gamma
N_{\omega}\ln\sigma_{\omega}}{2\pi}=0$, then
$\phi:=\phi_{\omega,\sigma_{\omega}}$ is independent of $\omega,\gamma$ and
satisfies the Schrödinger equation

(2.9)
$-\Delta\phi(x)+\frac{1}{2\pi}\bigg{(}\int_{\mathbb{R}^{2}}\ln(|x-y|)\;|\phi(y)|^{2}\;dy\bigg{)}\;\phi(x)=0.$

Define

(2.10) $\Lambda_{0}:=N(\phi).$

Then the virial theorem (2.7) reads

(2.11) $T(\phi)=\frac{\Lambda_{0}^{2}}{8\pi}.$

From (2.9) we get $V(\phi)=-T(\phi)$, hence

(2.12) $E(\phi)=T(\phi)+\frac{1}{2}V(\phi)=\frac{\Lambda_{0}^{2}}{16\pi}.$

The scaling of the particle number

$\Lambda_{0}=\sigma_{\omega}^{-2}\gamma N_{\omega}$

yields a relation between frequency $\omega$ and the particle number
$N_{\omega}$ which we shall discuss in detail in the following section for
ground state solutions.

##  3\. Ground states

###  3.1. Existence of ground states

We consider the following minimization problem:

(3.1) $e_{0}(\lambda)=\inf\\{E(u),u\in X,N(u)=\lambda\\}.$

We note that the functional $u\to E(u)$ is not convex since the quadratic form
$f\to\int_{\mathbb{R}}\int_{\mathbb{R}}|x-y|f(x)\bar{f}(y)\;dxdy$ is not
positive so that standard convex minimization does not apply. We shall prove
the following theorem:

######  Theorem 3.1.

For any $\lambda>0$ there is a spherically symmetric decreasing
$u_{\lambda}\in X$ such that $e_{0}(\lambda)=E(u_{\lambda})$ and
$N(u_{\lambda})=\lambda$.

###### Proof.

Let $(u_{n})_{n}$ be a minimizing sequence for $e_{0}(\lambda)$, that is
$N(u_{n})=\lambda$ and
$\underset{n\longrightarrow\infty}{\lim}E(u_{n})=e_{0}(\lambda)$. We also may
assume that $E(u_{n})$ is uniformly bounded above. Denoting $u^{*}$ the
spherically symmetric-decreasing rearrangement of $u$ we have (see e.g. lemma
7.17 in [11])

$T(u)\geq T(u^{*}),\quad N(u^{*})=N(u).$

Applying the decomposition $V(u)=V_{1}(u)+V_{2}(u)$ defined in (2.4) we may
apply the strict version of Riesz’s rearrangement inequality (see e.g. theorem
3.9 in [11]) to $V_{1}(u)$ since $\ln(1+1/|x|)$ is positive and strictly
symmetric-decreasing. Therefore

$V_{1}(u)\geq V_{1}(u^{*})$

with equality only if $u(x)=u^{*}(x-x_{0})$ for some $x_{0}\in\mathbb{R}^{2}$.
For the second term $V_{2}(u)$ we apply the following rearrangement
inequality:

######  Lemma 3.2.

Let $f,g$ be two nonnegative functions on $\mathbb{R}$, vanishing at infinity
with spherically symmetric-decreasing rearrangement $f^{*},g^{*}$ ,
respectively. Let $v$ be a nonnegative spherically symmetric increasing
function. Then

(3.2)
$\int_{\mathbb{R}^{2}}\int_{\mathbb{R}^{2}}f(x)v(x-y)g(y)\;dxdy\geq\int_{\mathbb{R}^{2}}\int_{\mathbb{R}^{2}}f^{*}(x)v(x-y)g^{*}(y)\;dxdy$

###### Proof.

The proof follows the same lines as in [12], lemma 3.2, where we proved the
corresponding lemma in one space dimension. We give it here for the sake of
completeness. If $v$ is bounded, $v\leq C$, then $(C-v)^{*}=C-v$ and by
Riesz’s rearrangement inequality (lemma 3.6 in [11]) we have

$\int_{\mathbb{R}^{2}}\int_{\mathbb{R}^{2}}f(x)(C-v(x-y))g(y)\;dxdy\leq\int_{\mathbb{R}^{2}}\int_{\mathbb{R}^{2}}f^{*}(x)(C-v(x-y))g^{*}(y)\;dxdy.$

Since

$\int_{\mathbb{R}^{2}}f(x)\;dx\int_{\mathbb{R}^{2}}g(y)\;dy=\int_{\mathbb{R}^{2}}f^{*}(x)\;dx\int_{\mathbb{R}^{2}}g^{*}(y)\;dy$

the claim follows. If $v$ is unbounded we define a truncation by
$v_{n}(x)=\sup{(v(x),n)}$ and apply the monotone convergence theorem. ∎

By the preceding lemma we have

$V_{2}(u)\geq V_{2}(u^{*})$

and consequently

$V(u)\geq V(u^{*})$

with equality only if $u(x)=e^{i\theta}u^{*}(x-x_{0})$ for some
$\theta\in\mathbb{R}$ and $x_{0}\in\mathbb{R}^{2}$. Therefore we may suppose
that $u_{n}=u^{*}_{n}$. We claim that $u^{*}_{n}\in X$. Indeed, by Newton’s
theorem (see e.g. theorem 9.7 in [11]) we have

(3.3)
$\begin{split}V(u^{*}_{n})&=\frac{1}{2\pi}\int_{\mathbb{R}^{2}}{u^{*}_{n}(x)}^{2}(\ln|x|)\;\bigg{(}\int_{B_{|x|}}{u^{*}_{n}(y)}^{2}\;dy\bigg{)}\;dx\\\
&+\frac{1}{2\pi}\int_{\mathbb{R}^{2}}{u^{*}_{n}(x)}^{2}\bigg{(}\int_{\mathbb{R}^{2}\setminus
B_{|x|}}(\ln|y|)\;{u^{*}_{n}(y)}^{2}\;dy\bigg{)}\;dx\\\ \end{split}$

where $B_{|x|}$ denotes the disc of radius $|x|$ centered at the origin. Since
$\ln|y|\geq\ln|x|$ for all $y\in\mathbb{R}^{2}\setminus B_{|x|}$ we get

$V(u^{*}_{n})\geq\frac{\lambda}{2\pi}\int_{\mathbb{R}^{2}}(\ln|x|)\;{u^{*}_{n}(x)}^{2}\;dx.$

Using

$\ln|x|\geq\ln(1+|x|)-\frac{1}{|x|}$

and the sharp Sobolev inequality between the linear operators $-\Delta$ and
$\frac{1}{|x|}$

(3.4) $\int_{\mathbb{R}^{2}}\frac{1}{|x|}\;|u(x)|^{2}\;dx\leq
2||u||_{2}||\nabla u||_{2}$

we finally get

$V(u^{*}_{n})\geq\frac{\lambda}{2\pi}\int_{\mathbb{R}^{2}}\ln(1+|x|)\;{u^{*}_{n}(x)}^{2}\;dx-\frac{\lambda^{3/2}}{\pi}||\nabla
u^{*}_{n}||_{2}.$

Hence

$E(u^{*}_{n})\geq||\nabla
u^{*}_{n}||_{2}^{2}+\frac{\gamma\lambda}{4\pi}\int_{\mathbb{R}^{2}}\ln(1+|x|)\;{u^{*}_{n}(x)}^{2}\;dx-\frac{\gamma\lambda^{3/2}}{2\pi}||\nabla
u^{*}_{n}||_{2}$

proving our claim. We may extract a subsequence which we denote again by
$(u^{*}_{n})_{n}$ such that $u^{*}_{n}\to u^{*}$ weakly in $X$, strongly in
$L^{2}$ and a.e. where $u^{*}\in X$ is a nonnegative spherically symmetric
decreasing function. Note that $u^{*}\neq 0$ since $N(u^{*})=\lambda$. We want
to show that
$E(u^{*})\leq\underset{n\longrightarrow\infty}{\lim\inf}\;E(u^{*}_{n})$. Since

$T(u^{*})\leq\underset{n\longrightarrow\infty}{\lim\inf}\;T(u^{*}_{n})$

it remains to analyze the functional $V(u)$. Let

$\eta(x)=\int_{B_{|x|}}|u^{*}(y)|^{2}\;dy,\quad\eta_{n}(x)=\int_{B_{|x|}}|u^{*}_{n}(y)|^{2}\;dy.$

Then $\eta_{n}(x)\to\eta(x)$ uniformly since

$||\eta_{n}(x)-\eta(x)||_{\infty}\leq||u^{*}_{n}-u^{*}||_{2}\;||u^{*}_{n}+u^{*}||_{2}\leq
2\sqrt{\lambda}||u^{*}_{n}-u^{*}||_{2}.$

We note that for any spherically symmetric density $|u(x)|^{2}$ with $u\in X$
we may simplify (3.3) to

$V(u^{*}_{n})=\frac{1}{\pi}\int_{\mathbb{R}^{2}}{u^{*}_{n}(x)}^{2}(\ln|x|)\;\bigg{(}\int_{B_{|x|}}{u^{*}_{n}(y)}^{2}\;dy\bigg{)}\;dx.$

Therefore, by the definition of $\eta_{n},\eta$, we have

(3.5)
$\begin{split}V(u^{*}_{n})-V(u^{*})&=\frac{1}{\pi}\int_{\mathbb{R}^{2}}(\ln|x|)|u^{*}_{n}(x)|^{2}\big{(}\eta_{n}(x)-\eta(x)\big{)}\;dx\\\
&\quad+\frac{1}{\pi}\int_{\mathbb{R}^{2}}(\ln|x|)\eta(x)\big{(}|u^{*}_{n}(x)|^{2}-|u^{*}(x)|^{2}\big{)}\;dx\\\
\end{split}$

As $n\to\infty$ the first integral in (3.5) will tend to zero. In order to
analyze the second integral we again decompose $\ln|x|$ according to (2.3).
Then

$\frac{1}{\pi}\int_{\mathbb{R}^{2}}\ln(1+|x|)\eta(x)\big{(}|u^{*}_{n}(x)|^{2}-|u^{*}(x)|^{2}\big{)}\;dx$

will remain nonnegative since the continuous functional
$\phi\to\int_{\mathbb{R}^{2}}\ln(1+|x|)\eta(x)|\phi(x)|^{2}\;dx$ is positive
while

$-\frac{1}{\pi}\int_{\mathbb{R}^{2}}\ln(1+\frac{1}{|x|})\eta(x)\big{(}|u^{*}_{n}(x)|^{2}-|u^{*}(x)|^{2}\big{)}\;dx$

converges to zero since by Hölder’s inequality and inequality (3.4) we have
the estimate

$\begin{split}&\frac{1}{\pi}\int_{\mathbb{R}^{2}}\ln(1+\frac{1}{|x|})\eta(x)\big{(}|u^{*}_{n}(x)|^{2}-|u^{*}(x)|^{2}\big{)}\;dx\\\
&\leq\frac{2}{\pi}||\eta||_{\infty}||\nabla(u^{*}_{n}+u^{*})||_{2}^{1/2}||u^{*}_{n}+u^{*}||_{2}^{1/2}||\nabla(u^{*}_{n}-u^{*})||_{2}^{1/2}||u^{*}_{n}-u^{*}||_{2}^{1/2}\\\
&\leq C||u^{*}_{n}-u^{*}||_{2}^{1/2}\\\ \end{split}$

for a positive constant $C$. Hence

$V(u^{*})\leq\underset{n\longrightarrow\infty}{\lim\inf}\;V(u^{*}_{n})$

proving the theorem. ∎

###  3.2. Uniqueness of ground states

Let $u_{\lambda}$ denote the solution of the minimization problem (5.3) found
in theorem .

######  Theorem 3.3.

For any $\lambda>0$ the solution $u_{\lambda}$ of the minimization problem
(5.3) is unique in the following sense: If $v_{\lambda}\in X$ is such that
$e_{0}(\lambda)=E(v_{\lambda})$ and $N(v_{\lambda})=\lambda$, then
$v_{\lambda}\in\\{e^{i\theta}u^{*}(x-x_{0}),\theta\in\mathbb{R},x_{0}\in\mathbb{R}^{2}\\}$.

###### Proof.

Since by the strict rearrangement inequality $E(u)>E(u^{*})$ for all
$u\notin\\{e^{i\theta}u^{*}(x-x_{0}),\theta\in\mathbb{R},x_{0}\in\mathbb{R}^{2}\\}$
established in the proof of theorem 3.2 it is sufficient to prove uniqueness
in the class of spherically symmetric (decreasing) nonnegative functions. To
do so we consider the Euler-Lagrange equation for $u_{\lambda}$. By theorem
3.2 for any $\lambda>0$ there is a Lagrange multiplier $\omega$ such that, at
least in a weak sense

(3.6) $-\Delta
u_{\lambda}(x)+\frac{\gamma}{2\pi}\bigg{(}\int_{\mathbb{R}^{2}}\ln(|x-y|)\;|u_{\lambda}(y)|^{2}\;dy\bigg{)}\;u_{\lambda}(x)=\omega
u_{\lambda}(x).$

We define the quantities

$I_{\lambda}:=\frac{1}{2\pi}\int_{\mathbb{R}^{2}}\ln(|y|)\;|u_{\lambda}(y)|^{2}\;dy,\quad
E_{\lambda}:=\omega-\gamma I_{\lambda}$

and the potential

$W_{\lambda}(x):=\frac{1}{2\pi}\bigg{(}\int_{\mathbb{R}^{2}}\ln(|x-y|)\;|u_{\lambda}(y)|^{2}\;dy\bigg{)}-I_{\lambda}.$

Then $W_{\lambda}(x)\geq 0$ for all $x\in\mathbb{R}^{2}$, $W_{\lambda}(0)=0$
and $\Delta W_{\lambda}=|u_{\lambda}|^{2}$. The function $u_{\lambda}$ is the
ground state of the Schrödinger operator $-\Delta+\gamma W_{\lambda}$ with
eigenvalue $E_{\lambda}$ and therefore $E_{\lambda}>0$. The rescaled functions

(3.7)
$u(|x|):=\frac{\sqrt{\gamma}}{E_{\lambda}}\;u_{\lambda}(x/\sqrt{E_{\lambda}}),\quad
W(|x|):=\frac{{\gamma}}{E_{\lambda}}\;W_{\lambda}(x/\sqrt{E_{\lambda}})$

then $u,W$ satisfy the universal equations

(3.8) $\Delta u=(W-1)u,\quad\Delta W=|u|^{2}.$

In [7] it was shown that (3.9) admits a unique spherically symmetric solution
$(u,W)$ such that $u$ is a positive decreasing function vanishing at infinity.
See also theorem 5.2 of the present paper where we consider a more general
system. ∎

###  3.3. Dependence on parameters and admissible frequencies

We determine explicitly $e_{0}(\lambda)$ as a function of $\lambda$ and the
range of admissible frequencies $\omega$ for ground states of the form (1.4).
We consider the unique spherically symmetric solution $(u,W)$, $u>0$ and $u$
vanishing at infinity , of the universal system (3.9). Let

(3.9) $N:=\int_{\mathbb{R}^{2}}|u(|x|)|^{2}\;dx,\quad
I:=\frac{1}{2\pi}\int_{\mathbb{R}^{2}}\ln(|x|)\;|u(|x|)|^{2}\;dx.$

These quantities are finite since, for any $p>0$, $u$ decays faster than
$\exp(-px)$ at infinity and in particular $u\in X$. The corresponding
numerical values are easily computed from the solution of and we give them in
the appendix. By the scaling (3.7) we obtain the following identity for the
parameters $\gamma,\lambda$ and the Lagrange multiplier $\omega$:

(3.10)
$\omega=\frac{\gamma\lambda}{N}\big{(}1+I-\frac{N}{4\pi}\ln\frac{\gamma\lambda}{N}\big{)}.$

Multiplying the variational equation (5.4) by $u_{\lambda}$ and integrating we
obtain

$2E(u_{\lambda})-T(u_{\lambda})=T(u_{\lambda})+\gamma
V(u_{\lambda})=\omega\lambda.$

Applying the virial relation (2.7) and using (3.10) we finally have

(3.11)
$e_{0}(\lambda)=\frac{\gamma\lambda^{2}}{16\pi}\Big{(}1+8\pi\frac{1+I}{N}-2\ln\big{(}\frac{\gamma\lambda}{N}\big{)}\Big{)}$

with $N,I$ given in (3.9). Note that (3.11) can be also obtained by
integrating $\omega=\omega(\lambda)$ in (3.10) with respect to $\lambda$
since, at least formally,

$\frac{d\,e_{0}(\lambda)}{d\lambda}=\omega.$

Taking $\phi$ in (2.9) as the unique ground state solution the relation (3.10)
between $\lambda,\omega$ and $\gamma$ simplifies to

(3.12)
$\omega=-\frac{\gamma\lambda}{4\pi}\ln\frac{\gamma\lambda}{\Lambda_{0}}$

with $\Lambda_{0}=N(\phi)$ as defined in (2.10). The energy is then given by

(3.13)
$e_{0}(\lambda)=\frac{\gamma\lambda^{2}}{16\pi}\big{(}1-2\ln\frac{\gamma\lambda}{\Lambda_{0}}\big{)}.$

The relations (3.10), respectively (3.12), yield the following result on
admissible frequencies:

######  Theorem 3.4.

(1) For any $\omega\leq 0$ the Schrödinger-Newton system (1.1) admits a unique
spherically ground state of the form (1.4). In particular, for the solution
$\phi$ of the stationary equation (2.9) the energy $e_{0}(\lambda)$ attains
its maximum.

(2) For any $0<\omega<\omega^{*}$ with

$\omega^{*}=\frac{N}{4\pi
e}\exp\big{(}4\pi\;\frac{1+I}{N}\big{)}=\frac{\Lambda_{0}}{4\pi e}$

the Schrödinger-Newton system (1.1) admits two spherically ground states of
the form (1.4) with different particle numbers.

(3) For $\omega=\omega^{*}$ there is a unique ground state solution of the
form (1.4).

##  4\. A logarithmic Hardy-Littlewood-Sobolev inequality

In [13], E. Carlen and M. Loss proved the following logarithmic Hardy-
Littlewood-Sobolev inequality:

(4.1)
$-\int_{\mathbb{R}^{n}}\int_{\mathbb{R}^{n}}\ln(|x-y|)\,f(x)f(y)\;dxdy\leq\frac{1}{n}\int_{\mathbb{R}^{n}}f(x)\ln
f(x)\;dx+C_{0}$

for all nonnegative real-valued $f$ with

$\int_{\mathbb{R}^{n}}f(x)\;dx=1,\int_{\mathbb{R}^{n}}f(x)\ln(1+|x|)\;dx,\int_{\mathbb{R}^{n}}f(x)\ln
f(x)\;dx<\infty$

and sharp constant $C_{0}$. We focus on the case $n=2$. Replacing $f$ by
$|u|^{2}$ with $N(u)=\lambda$ this reads as

(4.2) $-2\pi
V(u)\leq\frac{\lambda}{2}\int_{\mathbb{R}^{n}}|u(x)|^{2}\ln|u(x)|^{2}\;dx+C_{0}\lambda^{2}-\frac{\lambda^{2}}{2}\ln\lambda.$

Applying the logarithmic Sobolev inequality ( [14], see also [11], theorem
8.14)

(4.3)
$\int_{\mathbb{R}^{2}}|u(x)|^{2}\ln|u(x)|^{2}\;dx\leq\lambda\ln(T(u))-\lambda(1+\ln\pi)$

to the integral on the r.h.s. of (4.2) and using $2C_{0}=1+\ln\pi$ when $n=2$
we get

$-V(u)\leq\frac{\lambda^{2}}{4\pi}\ln\frac{T(u)}{\lambda}.$

However, this inequality is not sharp. Here we give the sharp version:

######  Theorem 4.1.

For any $u\in X$ the following inequality holds:

(4.4) $-V(u)\leq\frac{\lambda^{2}}{4\pi}\ln\frac{8\pi
T(u)}{N\lambda}-\big{(}\frac{I+1}{N}-\frac{1}{8\pi}\big{)}\lambda^{2}$

where $\lambda=\int_{\mathbb{R}^{2}}|u|^{2}\;dx$ and the constants $I$ and $N$
are given in (3.9).

###### Proof.

By theorem 3.2 we have for any $u\in X$ with
$\lambda=\int_{\mathbb{R}^{2}}|u|^{2}\;dx$ and for any coupling constant
$\gamma>0$ the sharp inequality

$T(u)+\frac{\gamma}{2}V(u)\geq e_{0}(\lambda)$

with $e_{0}(\lambda)$ given in (3.11). We optimize with respect to $\gamma$.
The optimal $\gamma$ is given by

$\frac{1}{4\pi}\ln\frac{\gamma\lambda}{N}=-\lambda^{-2}V(u)+\frac{I+1}{N}-\frac{1}{8\pi}$

which yields the desired inequality. ∎

##  5\. Purely angular excitations

In this section we prove the existence of non radial solutions of the
Schrödinger-Newton system (1.1) which in polar coordinates $(r,\theta)$ are of
the form

$\psi(t,r,\theta)=\phi_{\omega}(r)e^{im\theta-i\omega
t},\quad\phi_{\omega}(r)\geq 0$

for positive integers $m$. These are eigenfunctions of the angular momentum
operator $L=-i\partial_{\theta}$. We define the function space $X_{m}^{(s)}$
of spherically symmetric functions $u:\mathbb{R}^{2}\to\mathbb{C}$ by

(5.1)
$X_{m}^{(s)}=\\{u=u(|x|):\int_{\mathbb{R}^{2}}|\partial_{r}u|^{2}+\frac{m^{2}}{r^{2}}||u|^{2}+|u|^{2}+\ln(1+r)\,|u|^{2}\;dx<\infty\\}.$

We define the energy functional for purely angular excitations by

(5.2) $E_{m}(u)=T_{m}(u)+\frac{\gamma}{2}V(u),\quad
T_{m}(u)=\int_{\mathbb{R}^{2}}|\partial_{r}u|^{2}+\frac{m^{2}}{r^{2}}||u|^{2}\;dx,$

We consider the minimization problem

(5.3) $e_{m}(\lambda)=\inf\\{E_{m}(u),u\in X_{m}^{(s)},N(u)=\lambda\\}.$

The following theorem holds:

######  Theorem 5.1.

For any $\lambda>0$ there is a nonnegative $u_{\lambda}\in X_{m}^{(s)}$ such
that $e_{m}(\lambda)=E_{m}(u_{\lambda})$ and $N(u_{\lambda})=\lambda$.

###### Proof.

Consider any minimizing sequence $(u_{n})_{n}$ for $e_{m}(\lambda)$. Since
$E_{m}(u)$ is decreasing under the replacement $u\mapsto|u|$ we may suppose
that $u_{n}\geq 0$. Now we mimic the proof of theorem 3.2 starting from
Newton’s theorem (3.3) replacing the energy $E$ by $E_{m}$ since only the
spherical symmetry of the functions are required in this part of the proof. ∎

The minimizing $u_{\lambda}$ satisfies the Euler-Lagrange equation

(5.4)
$-u_{\lambda}^{\prime\prime}-\frac{1}{r}u^{\prime}_{\lambda}+\frac{m^{2}}{r^{2}}u_{\lambda}+\frac{\gamma}{2\pi}\bigg{(}\int_{\mathbb{R}^{2}}\ln(|x-y|)\;|u_{\lambda}(|y|)|^{2}\;dy\bigg{)}\;u_{\lambda}=\omega
u_{\lambda}.$

for $r\geq 0$ and some multiplier $\omega$. By the same rescaling as in the
proof of theorem 3.3 we obtain a system of universal equations given by

(5.5)
$u^{\prime\prime}+\frac{1}{r}u^{\prime}-\frac{m^{2}}{r^{2}}u=(W-1)u,\quad
W^{\prime\prime}+\frac{1}{r}W^{\prime}=|u|^{2}$

such that $W(0)=W^{\prime}(0)=0$ and $u\geq 0$. We note that
$f(r):=r^{-m}u(r)$ then satisfies $f(0)>0$, $f^{\prime}(0)=0$ and the
differential equation

$f^{\prime\prime}+\frac{2m+1}{r}f^{\prime}=(W-1)f$

from which we easily deduce $f>0$ and $f^{\prime}<0$ using the facts $f\geq 0$
and $f\to 0$ as $r\to\infty$. Also note that for any $p>0$, $u$ decays faster
than $\exp(-pr)$ at infinity. Uniqueness of the solution $u_{\lambda}\in
X_{m}^{(s)}$ follows again from the uniqueness of the solutions of the
universal system (5.5) which we prove in the following theorem:

######  Theorem 5.2.

For any $m\geq 0$ there is a unique solution $(u,W)$ of (5.5) such that $u>0$
for $r>0$ and $u\to 0$ as $r\to\infty$.

###### Proof.

Suppose there are two distinct solutions $(u_{1},W_{1})$, $(u_{2},W_{2})$
having the required properties. We may suppose $u_{2}(r)>u_{1}(r)$ for
$r\in]0,\bar{r}[$. We consider the Wronskian

$w(r)=u_{2}^{\prime}(r)u_{1}(r)-u_{1}^{\prime}(r)u_{2}(r).$

Note that $w(0)=0$ and $rw(r)\to 0$ as $r\to\infty$. It satisfies the
differential equation

$(rw)^{\prime}=r(W_{2}-W_{1})u_{1}u_{2}.$

Suppose $u_{2}(r)>u_{1}(r)$ for all $r>0$. Then $W_{2}(r)>W_{1}(r)$ for all
$r>0$ since $r(W_{2}-W_{1})^{\prime}=\int_{0}^{r}(u_{2}^{2}-u_{1}^{2})s\;ds>0$
and hence $(rw)^{\prime}>0$ for all $r>0$ which is impossible. Hence there
exists $\bar{r}>0$ such that $\delta(r)=u_{2}(r)-u_{1}(r)>0$ for
$r\in[0,\bar{r}[$, $\delta(\bar{r})=0$ and $\delta^{\prime}(\bar{r})<0$.
However, then $w(\bar{r})=\delta^{\prime}(\bar{r})u_{1}(\bar{r})<0$, but
$(rw)^{\prime}(r)>0$ for all $r<\bar{r}$, that is $w(r)>0$ for all $r<\bar{r}$
which is again impossible. ∎

##  Appendix A Numerical values

Let $(u,W)$ be the unique spherically symmetric solution of the universal
equations (3.9) such that $W(0)=W^{\prime}(0)=0$, $u>0$ vanishing at infinity,
i.e.

(A.1) $(ru^{\prime})^{\prime}=r(W-1)u,\quad(rW^{\prime})^{\prime}=r|u|^{2}.$

By integrating the second equation we get

(A.2)
$N=2\pi\int_{0}^{\infty}u^{2}(s)s\;ds=2\pi\underset{r\longrightarrow\infty}{\lim}rW^{\prime}(r).$

Multiplying the second equation by $\ln r$ and using

$(rW^{\prime})^{\prime}\ln r=(rW^{\prime}\ln r-W)^{\prime}$

we obtain after integration

(A.3) $I=\int_{0}^{\infty}u^{2}(s)s\ln
s\;ds=\underset{r\longrightarrow\infty}{\lim}rW^{\prime}(r)\ln r-W(r).$

Solving (A.1) numerically we then find

(A.4) $N=2\pi\cdot 1.64145=10.3135,\quad I=0.2276$

and therefore

(A.5) $\Lambda_{0}=46.03.$

With these numerical values the logarithmic Hardy-Littlewood-Sobolev
inequality (4.4) reads as follows:

(A.6)
$-V(u)\leq\frac{\lambda^{2}}{4\pi}\ln\frac{T(u)}{\lambda}-0.0084\lambda^{2}.$

## References

  * [1] Pekar, S.I., Untersuchungen über die Elektronentheorie der Kristalle, §6, p. 29 -34, Akademie Verlag Berlin, 1954.
  * [2] Lieb, E.H., Existence and uniqueness of the minimizing solution of Choquard’s nonlinear equation, Studies in Applied Mathematics 57, 93-105 (1977).
  * [3] Penrose, R., On gravity’s role in quantum state reduction, Gen. Rel. Grav. 28, 581-600 (1996).
  * [4] Harrison, R., Moroz, I. Tod, K.P., A numerical study of the Schrödinger-Newton equation, Nonlinearity 16, 101-122 (2003).
  * [5] Lions, P.L., The Choquard equation and related questions, Nonlinear Analysis T.M.A. 4, 1063-1073 (1980).
  * [6] Lions, P.L., Solutions complexes d’équations elliptiques semi-linéaires dans $\mathbb{R}^{n}$, C.R. Acad. Sc. Paris t.302, Série 1 ,no. 19, 673-676 (1986).
  * [7] Choquard, Ph., Stubbe, J., Vuffray, M., Stationary solutions of the Schrödinger-Newton model - An ODE approach, Adv. Diff. Eq. to appear (2008).
  * [8] Reed, M. and Simon, B., Methods of modern mathematical physics, Vol. IV, Analysis of operators, Academic Press 1978.
  * [9] Lieb, E.H., Sharp constants in Hardy-Littlewood-Sobolev and related inequalities , Ann. Math. 118, 349 - 374 (1983).
  * [10] Weinstein, M.I., Nonlinear Schrödinger equations and sharp interpolation estimates , Comm. Math. Phys. 87, 567 - 576 (1983).
  * [11] Lieb, E.H. and Loss, M., Analysis, 2nd edition, Graduate Studies in Mathematics, vol. 14, AMS 2001.
  * [12] Choquard, Ph., Stubbe, J. The one-dimensional Schrödinger-Newton equations, Lett. Math. Phys. 81, 177-184 (2007).
  * [13] Carlen, E., Loss, M. Competing symmetries, the logarithmic HLS inequality and Onofri’s inequality on $S^{n}$, Geom. Funct. Anal. 2, 90-103 (1992).
  * [14] Weissler, F., Logarithmic Sobolev Inequalities for the Heat-Diffusion Semigroup, Trans. AMS 237, 255-269 (1978).


# Graph-Free Knowledge Distillation for Graph Neural Networks111Code:
https://github.com/Xiang-Deng-DL/GFKD

Xiang Deng  Zhongfei Zhang State University of New York at Binghamton
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Knowledge distillation (KD) transfers knowledge from a teacher network to a
student by enforcing the student to mimic the outputs of the pretrained
teacher on training data. However, data samples are not always accessible in
many cases due to large data sizes, privacy, or confidentiality. Many efforts
have been made on addressing this problem for convolutional neural networks
(CNNs) whose inputs lie in a grid domain within a continuous space such as
images and videos, but largely overlook graph neural networks (GNNs) that
handle non-grid data with different topology structures within a discrete
space. The inherent differences between their inputs make these CNN-based
approaches not applicable to GNNs. In this paper, we propose to our best
knowledge the first dedicated approach to distilling knowledge from a GNN
without graph data. The proposed graph-free KD (GFKD) learns graph topology
structures for knowledge transfer by modeling them with multivariate Bernoulli
distribution. We then introduce a gradient estimator to optimize this
framework. Essentially, the gradients w.r.t. graph structures are obtained by
only using GNN forward-propagation without back-propagation, which means that
GFKD is compatible with modern GNN libraries such as DGL and Geometric.
Moreover, we provide the strategies for handling different types of prior
knowledge in the graph data or the GNNs. Extensive experiments demonstrate
that GFKD achieves the state-of-the-art performance for distilling knowledge
from GNNs without training data.

##  1 Introduction

Knowledge Distillation (KD) Hinton et al. (2015) aims to transfer useful
knowledge from a teacher network to a student. The effectiveness of KD to
boost the student performance has been demonstrated across a wide range of
applications in artificial intelligence Romero et al. (2015). As the knowledge
in a teacher concentrates on a narrow manifold instead of the full input
space, KD has a strong assumption that either the training dataset or some
representative samples are available. The requirement for observable data
highly limits its applications on data-unavailable cases. For example, a deep
model (e.g., ResNet-152) that is pretrained on a large-scale dataset of
billions of data samples is released online. One may wish to distill the
knowledge from this powerful model into a compact and fast model for the
deployment on resource-limited devices, which requires the access to the
training dataset. However, the dataset is not publicly available as it is not
only large but also difficult to store and transfer. In reality, it is not
rare that a corporation or a leading research group shares their pretrained
models while they do not release the training data due to large sizes,
privacy, confidentiality, or security in medical or industrial domains.

Figure 1: CNNs with grid data and GNNs with graph data

A simple yet effective way to address the issue is to generate fake data for
knowledge transfer by optimizing the inputs to the pretrained teacher to
maximize the class-conditional probability Mordvintsev et al. . This method
Yin et al. (2020) has shown its success on convolutional neural networks
(CNNs) where the inputs are grid data within a continuous space such as images
and videos as the gradients w.r.t. the inputs exist. However, many real data
such as proteins and chemical molecules lie in non-grid domains within a
discrete space and thus call for graph neural networks (GNNs) Kipf and Welling
(2017) that explicitly deal with the topological structures of these graph
data. As shown in Figure 1, different from CNNs handling grid data such as
images, GNNs deal with graph data that contain both node features and
topological structures within a discrete space. Current GNN models learn the
node-level or graph-level representations by aggregating node features based
on local topology structures. The output of a GNN is not differentiable w.r.t.
the topological structures of the input graphs, which makes the CNN-based
approaches not applicable to GNNs.

In this paper, we study how to distill knowledge from a pretrained GNN without
observable graphs and develop to the best of our knowledge the first data-free
knowledge distillation approach (i.e., GFKD) tailored for GNNs. The workflow
of GFKD is shown in Figure 2. GFKD first learns the fake graphs that the
knowledge in the teacher GNN is more likely to concentrate on, and then uses
these fake graphs to transfer knowledge to the student. To achieve this goal,
we propose a structure learning strategy by modeling the topology of a graph
with a multivariate Bernoulli distribution and then introduce a gradient
estimator to optimize it.

Our main contributions are summarized as follows:

  * •

We introduce a novel framework, i.e., GFKD, for distilling knowledge from GNNs
without observable graph data. To the best of our knowledge, this is the first
dedicated data-free KD approach tailored for GNNs. We also provide the
strategies (or regularizers) for handling different priors about the graph
data, including how to deal with one-hot features and degree features.

  * •

We develop a novel strategy for learning graph structures from a pretrained
GNN by using multivariate Bernoulli distribution and introduce a gradient
estimator to optimize it, which paves the way for extracting knowledge from a
pretrained GNN without observable graphs. Note that the current GNN libraries
do not support computing gradients w.r.t. the input graph structures. GFKD
avoids this issue as the structure gradients in GFKD are obtained by only
using GNN forward propagation without backward propagation. GFKD is thus
supported by these libraries.

  * •

We evaluate GFKD on six benchmark datasets in different domains with two
different GNN architectures under different settings and demonstrate that GFKD
achieves the best performance across different datasets.

##  2 Related Work

###  2.1 Graph Neural Networks

GNNs emerge as a hot topic in recent years for their potential in numerous
applications. Bruna et al. Bruna et al. (2013) first generalize CNNs to
signals defined on the non-grid domain. Defferrard et al. Defferrard et al.
(2016) further improve the idea by using Chebyshev polynomials. Kipf and
Welling Kipf and Welling (2017) propose to build GNNs by stacking multiple
first-order Chebyshev polynomial filters. Xu et al. Xu et al. (2019), on the
other hand, propose a simple neural architecture, graph isomorphism networks
(GINs), which generalizes the Weisfeiler-Lehman (WL) test and hence achieves a
powerful discriminative ability. The other GNN architectures including but not
limited to Fey et al. (2018); Li et al. (2015); Hamilton et al. (2017);
Veličković et al. (2017) deal with the features and the topological structures
using different strategies.

###  2.2 Knowledge Distillation

Knowledge distillation aims at transferring knowledge from a teacher model to
a student. Hinton et al. Hinton et al. (2015) propose KD that penalizes the
softened logit differences between a teacher and a student. FitNet Romero et
al. (2015) and AT Zagoruyko and Komodakis (2017) further use the feature
alignment to assist knowledge transfer. Yang et al. Yang et al. (2020) develop
a local structure preserving module for distilling knowledge from GNNs. Other
distillation approaches Tian et al. (2020) use different criteria to align
feature representations. All these approaches require the training dataset for
knowledge transfer, which cannot handle the case where the training data are
not available.

To address this issue, many efforts have been made on distilling knowledge
without training images from a CNN through generating fake images. Lopes et
al. Lopes et al. (2017) make use of meta data instead of real images to
distill knowledge. Yoo et al. Yoo et al. (2019) use a generator and a decoder
to learn fake images for knowledge transfer. Micaelli and Storkey Micaelli and
Storkey (2019) and Chen et al Chen et al. (2019) use generative adversarial
networks (GANs) Goodfellow et al. (2014) to generate fake images. Nayak et al.
Nayak et al. (2019) propose to generate images by modelling the softmax space.
DeepInversion Yin et al. (2020) generates fake images by inversing a CNN and
using the statistics in batch normalization Ioffe and Szegedy (2015).

All these data-free KD approaches are designed for CNNs with images as inputs,
and highly overlook GNNs dealing with non-grid data within a discrete space.
Directly applying these approaches to GNNs is not applicable as graph data
contain both features and topological structures. Thus, it is necessary and
appealing to develop a data-free KD approach tailored for GNNs.

Figure 2: Framework of GFKD

##  3 Framework

In this section, we first provide a brief overview on GNNs. We then present
GFKD and the strategies for dealing with different types of prior knowledge
about the graph data. At the end, we introduce the optimization solution to
GFKD.

###  3.1 Graph Neural Networks

Different from CNNs handling grid data, GNNs can take non-gird data as inputs.
A non-grid data can be represented as a graph $G=\\{V,E\\}$ and a set of
features $h$, where $V$ and $E$ denote the nodes and the edges, respectively.
Modern GNNs typically adopt a neighborhood aggregation strategy by using the
features and graph structures to learn discriminative node or graph
representations. Suppose that $h_{i}\in h$ is the feature of node $v_{i}$. The
operation in layer $k$ of a GNN is:

$h_{i}^{k}=g\left(f\left(h_{i}^{k-1}\right),\\{f(h_{j}^{k-1})|v_{j}\in\mathcal{N}_{i}\\}\right)$
(1)

where $g(.)$ denotes the aggregation function; $f(.)$ is the feature
transformation function; $\mathcal{N}_{i}$ denotes the neighbors of node
$v_{i}$. It is observed that the topological structure of a graph plays a
vital role in learning representations. To transfer the knowledge from a
teacher GNN to a student, it is necessary to know which structures the
knowledge concentrates on.

###  3.2 Graph-Free Knowledge Distillation

Suppose that a teacher GNN $T(.)$ with parameters $W$ is trained over dataset
(X, Y) by minimizing the regular cross-entropy loss, where X and Y are the
graph data and the labels, respectively:

$\mathcal{L}_{CE}=\mathcal{C}(Y,T_{W}(H,A))$ (2)

where $\mathcal{C}(.)$ denotes the loss function such as cross-entropy or mean
square error; $H$ denotes the node features of graphs X; $A$ represents the
graph structure information of X which can be represented as adjacency
matrices consisting of 0s and 1s.

From the Bayesian perspective, learning $W$ by minimizing (2) can be
considered as maximizing the logarithm of class-conditional probability
$\rm\log$ $p(Y|H,A,W)$, i.e., $\mathop{\mathrm{argmin}}_{W}\mathcal{L}_{CE}$ =
$\mathop{\mathrm{argmax}}_{W}$ $\rm\log$ $p(Y|H,A,W)$. Thus, the knowledge is
more likely to concentrate on the graphs that make the teacher output a high
class-conditional probability. When training data are not available but $W$ is
known, one may optimize the inputs by maximizing the class-conditional
probability (i.e., $\mathop{\mathrm{argmax}}_{H,A}$ $\rm log$ $p(Y|H,A,W)$ or
$\mathop{\mathrm{argmin}}_{H,A}\mathcal{L}_{CE}$) to generate fake samples for
knowledge transfer. This inversion technique Yin et al. (2020) has shown its
success on CNNs where the inputs are grid-data images within a continuous
space without topological structures $A$. Unfortunately, this is not
applicable to GNNs as $\mathcal{L}_{CE}$ is not differentiable w.r.t. the
graph structure $A$ (but differentiable w.r.t. features $H$) as seen from (1).

####  3.2.1 Learning Graph Topology with Stochastic Structures

To address the above issue, we propose to model the topological structure of a
graph with a multivariate Bernoulli distribution, thus obtaining a stochastic
structure. Specifically, suppose that $a\in A$ is the structure (i.e.,
adjacency matrix) of a graph. Each element in $a$ follows a Bernoulli
distribution:

$P_{\theta_{ij}}(a_{ij})=a_{ij}\phi(\theta_{ij})+(1-a_{ij})\phi(-\theta_{ij})$
(3)

where $\phi(\theta_{ij})$ is the sigmoid function
$\frac{e^{\theta_{ij}}}{1+e^{\theta_{ij}}}$; $\theta_{ij}$ is a learnable
parameter; $a_{ij}\in\\{0,1\\}$ is the element in the $i$th row and the $j$th
column in $a$ where $a_{ij}=1$ means that node $v_{i}$ is a neighbor of
$v_{j}$ and vice versa.

For directed graphs, $n^{2}$ parameters are used to model the distribution of
$a$:

$P_{\theta}(a)=\prod_{i=1}^{n}\prod_{j=1}^{n}P_{\theta_{ij}}(a_{ij})$ (4)

Note that for undirected graphs, the number of parameters is reduced to
$\frac{n(n+1)}{2}$ as their adjacency matrices are symmetric.

For a batch of graphs, their structures $A$ are independent of each other and
thus follow the joint distribution: $\prod_{i=1}^{m}P_{\theta_{i}}(a)$ where
$m$ is the number of graphs and $\theta_{i}$ is the structure parameter for
graph $i$ . Instead of directly minimizing (2) that is not differentiable
w.r.t. $A$, we generate fake graph data by minimizing the following
expectation222Assume that we have some priors about the magnitude of the
number of graph data nodes.:

$\mathcal{L}_{H,\Theta}=\mathop{\mathbb{E}}_{A\sim
P_{\Theta}(A)}\left[\mathcal{C}\left(Y,T_{W}\left(H,A\right)\right)+\lambda*\mathcal{R}\right]$
(5)

where $P_{\Theta}(A)$ = $\prod_{i=1}^{m}P_{\theta_{i}}(a)$; $H$ represents the
feature parameters; $Y$ is a set of randomly sampled labels; $\mathcal{R}$
denotes the regularizers for different priors that we have about the target
task data; $\lambda$ is a balancing weight. Thus, minimizing (5) w.r.t. $H$
and $\Theta$ can generate the graph data that maximize the GNN output class
probability. We omit $W$ in $T_{W}$ in the following as it is known for a
pretrained teacher GNN.

####  3.2.2 Regularizers for Prior Knowledge

$\mathcal{R}$ in (5) deals with different types of prior knowledge about the
graph data for the target task. We provide the strategies for handling common
priors.

##### Priors in Graph Neural Networks:

Similar to the case in CNNs, many GNNs also benefit from batch normalization
(BN). BN contains statistical information about the data as it accumulates the
moving average of the means and the variances of the features during training.
Similar to the case in CNNs Yin et al. (2020), it is reasonable to force the
fake graph data to have similar feature means and variances to those in the
GNNs accumulated from the real graph data:

$\mathcal{R}_{bn}=(u_{A,\Theta}-u_{T})^{2}+(v_{A,\Theta}-v_{T})^{2}$ (6)

where $u_{A,\Theta}$ and $v_{A,\Theta}$ represent the means and variances of
the features of the generated graphs, respectively; $u_{T}$ and $v_{T}$ are
the means and variances in the teacher GNN, respectively.

##### Priors about Target-Task Graph Data:

Besides the priors embedded in the GNNs, one may have some prior knowledge
about the target task data:

##### (1) One-hot features:

The graph data for many tasks have one-hot features, such as the
classification task on MUTAG. In this case, directly minimizing (5) cannot
lead to one-hot features. To address this issue, we first reparameterize $H$
in (5) with the softmax function $\sigma(\omega)$ where $\omega$ are learnable
parameters and then minimize the entropy of $\sigma(\omega)$:

$\mathcal{L}_{\omega,\Theta}=\mathop{\mathbb{E}}_{A\sim
P_{\Theta}(A)}\left[\mathcal{C}\left(Y,T\left(\sigma(\omega),A\right)\right)+\lambda*Ent(\sigma(\omega))\right]$
(7)

where $Ent(.)$ denotes the entropy and $\sigma(\omega)$ can be seen as the
instantiations of $H$.

##### (2) Degrees as features:

Some graph data use the degrees of the nodes as features. In this case, the
features can be derived from the adjacency matrix $A$. It is thus not
necessary to explicitly learn features $H$ and objective (5) is reduced to:

$\mathcal{L}_{\Theta}=\mathop{\mathbb{E}}_{A\sim
P_{\Theta}(A)}\left[\mathcal{C}\left(Y,T\left(A\right)\right)+\lambda*\mathcal{R}\right]$
(8)

We have discussed some common graph priors while there may be other priors for
different graphs. Fortunately, objective (5) is readily extended to different
graph data.

####  3.2.3 Optimization

To minimize objective (5), we need to compute the gradients regarding $H$ and
$\Theta$. As the gradients of $\mathcal{C}(.)$ w.r.t. $H$ exist, we can easily
estimate them by sampling from $P_{\Theta}(A)$:

$\begin{split}\nabla_{H}\mathcal{L}_{H,\Theta}=\nabla_{H}\mathop{\mathbb{E}}_{A\sim
P_{\Theta}(A)}\left[\mathcal{C}\left(Y,T\left(H,A\right)\right)+\lambda*\mathcal{R}\right]\\\
=\mathop{\mathbb{E}}_{A\sim
P_{\Theta}(A)}\nabla_{H}\left[\mathcal{C}\left(Y,T\left(H,A\right)\right)+\lambda*\mathcal{R}\right]\\\
=\frac{1}{N}\sum_{i=1}^{N}\nabla_{H}\left[\mathcal{C}\left(Y,T\left(H,A^{i}\right)\right)+\lambda*\mathcal{R}\right]\end{split}$
(9)

where $A^{i}\sim P_{\Theta}(A)$ are independent and identically distributed
(iid).

The difficulty lies in computing the gradients w.r.t. $\Theta$ that exist in
the distribution $P_{\Theta}(A)$. We introduce a gradient estimator Yin and
Zhou (2019); Williams (1992) to optimize $\Theta$, which is based on the
reparametrization trick and REINFORCE.

We omit $\lambda*\mathcal{R}$ in the objective (5) in the following for
simplicity. Bernoulli random variables $A$ in (5) can be reparameterized by
two exponential random variables:

$\begin{split}\mathcal{L}_{H,\Theta}=\mathop{\mathbb{E}}_{A\sim
P_{\Theta}(A)}\mathcal{C}\left(Y,T\left(H,A\right)\right)\\\
=\mathop{\mathbb{E}}_{B,M\sim\prod_{i=1}^{n^{2}}Ep(1)}\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[B\odot
e^{-\frac{\Theta}{2}}<M\odot e^{\frac{\Theta}{2}}]}\right)\right)\end{split}$
(10)

where $Ep(.)$ represents the Exponential distribution; $\odot$ denotes the
element-wise product; $\mathbf{1}_{[.]}$ is the indicator function which
equals to one if the argument is true and zero otherwise.

Note that as $B$ follows $\prod_{i=1}^{n^{2}}Ep(1)$, $Q$ = $B\odot
e^{-\frac{\Theta}{2}}$ follows
$\prod_{i=1}^{n^{2}}Ep(e^{\frac{\theta_{i}}{2}})$. Similarly, $S$ = $M\odot
e^{\frac{\Theta}{2}}$ follows $\prod_{i=1}^{n^{2}}$
$Ep(e^{-\frac{\theta_{i}}{2}})$. (10) can be further reparameterized as:

$\begin{split}\mathcal{L}_{H,\Theta}=\mathop{\mathbb{E}}_{Q\sim\prod_{i=1}^{n^{2}}Ep(e^{\frac{\theta_{i}}{2}}),S\sim\prod_{i=1}^{n^{2}}Ep(e^{-\frac{\theta_{i}}{2}})}\\\
\left[\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[Q<S]}\right)\right)\right]\end{split}$
(11)

Next we show how to obtain the gradients w.r.t $\Theta$. Applying REINFORCE to
(11) leads to:

$\begin{split}\nabla_{\Theta}\mathcal{L}_{H,\Theta}=\mathop{\mathbb{E}}_{Q\sim\prod_{i=1}^{n^{2}}Ep(e^{\frac{\theta_{i}}{2}}),S\sim\prod_{i=1}^{n^{2}}Ep(e^{-\frac{\theta_{i}}{2}})}\\\
\left[\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[Q<S]}\right)\right)\nabla_{\Theta}log\left(P_{\Theta}\left(Q\right)P_{\Theta}\left(S\right)\right)\right]\\\
=\mathop{\mathbb{E}}_{Q\sim\prod_{i=1}^{n^{2}}Ep(e^{\frac{\theta_{i}}{2}}),S\sim\prod_{i=1}^{n^{2}}Ep(e^{-\frac{\theta_{i}}{2}})}\\\
\left[\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[Q<S]}\right)\right)\frac{1}{2}(S\odot
e^{-\frac{\Theta}{2}}-Q\odot e^{\frac{\Theta}{2}})\right]\end{split}$ (12)

As $Q$ = $B\odot e^{-\frac{\Theta}{2}}$ and $S$ = $M\odot
e^{\frac{\Theta}{2}}$, (12) is equivalent to:

$\begin{split}\nabla_{\Theta}\mathcal{L}_{H,\Theta}=\mathop{\mathbb{E}}_{B,M\sim\prod_{i=1}^{n^{2}}Ep(1)}\\\
\left[\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{\left[B\odot
e^{\frac{-\Theta}{2}}<M\odot
e^{\frac{\Theta}{2}}\right]}\right)\right)\frac{1}{2}(M-B)\right]\end{split}$
(13)

$B$ and $M$ can be further reparameterized as:

$B=K\odot U,M=K\odot(1-U)$ (14)

where $U$ and $K$ follow $\prod_{i=1}^{n^{2}}Un(0,1)$ and
$\prod_{i=1}^{n^{2}}$ $Gamma(2,1)$, respectively, and $Un()$ and $Gamma()$
denote the uniform distribution and the gamma distribution, respectively. (13)
can be further reparameterized as:

$\begin{split}\nabla_{\Theta}\mathcal{L}_{H,\Theta}=\mathop{\mathbb{E}}_{U\sim\prod_{i=1}^{n^{2}}Un(0,1),K\sim\prod_{i=1}^{n^{2}}Gamma(2,1)}\\\
\left[\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[U<\phi(\Theta)]}\right)\right)\frac{1}{2}(K-2K\odot
U)\right]\end{split}$ (15)

By applying Rao Blackwellization to (15),
$\nabla_{\Theta}\mathcal{L}_{H,\Theta}$ is equal to:

$\begin{split}\mathop{\mathbb{E}}_{U\sim\prod_{i=1}^{n^{2}}Un(0,1)}\left[\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[U<\phi(\Theta)]}\right)\right)(1-2U)\right]\end{split}$
(16)

$U\sim\prod_{i=1}^{n^{2}}Un(0,1)$ implies
$(1-U)\sim\prod_{i=1}^{n^{2}}Un(0,1)$. Thus, $U$ in (16) can be replaced with
$1-U$ to obtain a new unbiased gradient estimator. Taking the average of the
new estimator and (16) can further reduce the sampling variance Yin and Zhou
(2019):

$\begin{split}\nabla_{\Theta}\mathcal{L}_{H,\Theta}=\mathop{\mathbb{E}}_{U\sim\prod_{i=1}^{n^{2}}Un(0,1)}[(\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[U>\phi(-\Theta)]}\right)\right)\\\
-\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[U<\phi(\Theta)]}\right)\right))(U-0.5)]\end{split}$
(17)

By simply sampling from $Un(0,1)$, we can obtain the gradients w.r.t. $\Theta$

$\begin{split}\nabla_{\Theta}\mathcal{L}_{H,\Theta}=\frac{1}{N}\sum_{i=1}^{N}[(\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[U^{i}>\phi(-\Theta)]}\right)\right)\\\
-\mathcal{C}\left(Y,T\left(H,\mathbf{1}_{[U^{i}<\phi(\Theta)]}\right)\right))(U^{i}-0.5)]\end{split}$
(18)

where $U^{i}\sim\prod_{i=1}^{n^{2}}Un(0,1)$ are iid. More derivation details
are given in the Appendix. We set $N$ to 1 in this paper as the variance is
small. As shown in (18), this unbiased gradient estimator allows us to obtain
the gradients w.r.t. the structure parameters $\Theta$ with only GNN forward
propagation and thus it is efficient and supported by the current GNN
libraries.

Datasets |  | MUTAG | PTC | PROTEINS
---|---|---|---|---
Teacher |  | GCN-5-64 | GIN-5-64 | GCN-5-64 | GCN-5-64 | GIN-5-64 | GCN-5-64 | GCN-5-64 | GIN-5-64 | GCN-5-64
Student |  | GCN-3-32 | GIN-3-32 | GIN-3-32 | GCN-3-32 | GIN-3-32 | GIN-3-32 | GCN-3-32 | GIN-3-32 | GIN-3-32
Teacher | 100% training data | 89.5 | 92.9 | 89.5 | 63.5 | 61.5 | 63.5 | 76.9 | 76.0 | 76.9
KD | 100% training data | 84.6$\pm$2.3 | 87.7$\pm$2.1 | 84.4$\pm$1.7 | 60.2$\pm$2.1 | 60.5$\pm$1.9 | 60.6$\pm$2.6 | 76.0$\pm$1.0 | 76.7$\pm$0.6 | 76.3$\pm$0.8
RandG | 0 training data | 39.1$\pm$6.8 | 58.7$\pm$4.2 | 38.8$\pm$5.8 | 43.2$\pm$4.4 | 53.2$\pm$5.8 | 42.9$\pm$8.0 | 56.6$\pm$4.8 | 33.3$\pm$7.4 | 43.4$\pm$9.5
DeepInvG | 0 training data | 58.6$\pm$5.1 | 59.6$\pm$2.9 | 35.4$\pm$2.2 | 52.9$\pm$7.8 | 45.7$\pm$4.1 | 43.9$\pm$9.2 | 65.4$\pm$2.3 | 52.7$\pm$5.7 | 47.4$\pm$9.9
GFKD | 0 training data | 70.8$\pm$2.1 | 73.2$\pm$4.2 | 70.2$\pm$2.0 | 57.4$\pm$2.2 | 57.5$\pm$2.4 | 54.1$\pm$6.4 | 74.7$\pm$1.5 | 60.4$\pm$1.0 | 65.5$\pm$3.1

Table 1: Test accuracies (%) on MUTAG, PTC, and PROTEINS

Datasets |  | IMDB-B | COLLAB | REDDIT-B
---|---|---|---|---
Teacher |  | GCN-5-64 | GIN-5-64 | GCN-5-64 | GCN-5-64 | GIN-5-64 | GCN-5-64 | GCN-5-64 | GIN-5-64 | GCN-5-64
Student |  | GCN-3-32 | GIN-3-32 | GIN-3-32 | GCN-3-32 | GIN-3-32 | GIN-3-32 | GCN-3-32 | GIN-3-32 | GIN-3-32
Teacher | 100% training data | 74.3 | 75.3 | 74.3 | 81.7 | 82.3 | 81.7 | 92.8 | 91.7 | 92.8
KD | 100% training data | 76.4$\pm$0.8 | 75.4$\pm$0.9 | 75.6$\pm$1.3 | 80.9$\pm$0.6 | 81.9$\pm$0.5 | 81.6$\pm$0.4 | 85.7$\pm$0.6 | 88.4$\pm$2.5 | 87.2$\pm$3.4
RandG/DeepInvG | 0 training data | 58.5$\pm$3.7 | 58.7$\pm$4.2 | 55.4$\pm$3.4 | 34.8$\pm$9.0 | 28.4$\pm$7.3 | 27.2$\pm$6.3 | 50.1$\pm$1.0 | 49.9$\pm$0.8 | 48.9$\pm$2.1
GFKD | 0 training data | 69.2$\pm$1.1 | 67.8$\pm$1.8 | 67.1$\pm$2.3 | 67.3$\pm$1.4 | 65.4$\pm$2.7 | 66.8$\pm$1.8 | 66.5$\pm$3.7 | 63.8$\pm$4.5 | 63.1$\pm$5.7

Table 2: Test accuracies (%) on IMDB-B, COLLAB, and REDDIT-B | #Graphs | #Classes | Avg#Graph Size
---|---|---|---
MUTAG | 188 | 2 | 17.93
PTC | 344 | 2 | 14.29
PROTEINS | 1,113 | 2 | 39.06
IMDB-B | 1,000 | 2 | 19.77
COLLAB | 5,000 | 3 | 74.49
REDDIT-B | 2,000 | 2 | 429.62
Table 3: Summary of datasets.

####  3.2.4 Knowledge Transfer with Generated Fake Graphs

As shown in Figure 2, we first update feature parameters $H$ and the
stochastic structure parameters $\Theta$ by minimizing (5) with gradients (9)
and (18), respectively. We then simply obtain fake graphs by using $H$ as the
node features and sampling from $P_{\Theta}(A)$ to generate topological
structures. The teacher GNN outputs a high probability on these graphs and
thus the knowledge is more likely to concentrate on these graphs. We then
transfer knowledge from the teacher to the student by using these fake graph
data $x$ with the KL-divergence loss:

$\tau^{2}KL\left(\sigma\left(\frac{T\left(x\right)}{\tau}\right),\sigma\left(\frac{S\left(x\right)}{\tau}\right)\right)$
(19)

where $\sigma$ is the softmax function; $\tau$ is a temperature to generate
soft labels and $KL$ represents KL-divergence; $S$ is the student GNN. To the
end, we achieve knowledge distillation without any observable graphs.

##  4 Experiments

In this section, we report extensive experiments for evaluating GFKD. Note
that our goal is not to generate real graphs but to transfer as much knowledge
as possible from a pretrained teacher GNN to a student without using any
training data.

###  4.1 Experimental Settings

We adopt six graph classification benchmark datasets Xu et al. (2019)
including three bioinformatics graph datasets, i.e., MUTAG, PTC, and PROTEINS,
and three social network graph datasets, i.e., IMDB-B, COLLAB, and REDDIT-B.
The statistics of these datasets are summarized in Table 3. On each dataset,
70% data are used for pretraining the teachers and the remaining 30% are used
as the test data.

We use two well known GNN architectures, i.e, GCN Kipf and Welling (2017) and
GIN Xu et al. (2019). We distill knowledge in two different settings, i..e,
the teacher and the student share the same architecture or use different
architectures. We use the form of (architecture-layer number-feature
dimensions) to denote a GNN. For example, GIN-5-64 denotes a GNN with 5 GIN
layers and 64 feature dimensions.

As there is no existing approach applicable to GNNs for distilling knowledge
without observable graph data, we design two baselines for references:

  * •

Random Graphs (RandG): RandG generates fake graphs by randomly drawing from a
uniform distribution as node features and topological structures, and then use
these graphs to transfer knowledge.

  * •

DeepInvG Yin et al. (2020): As the original DeepInversion Yin et al. (2020)
cannot learn the structures of graph data, here DeepInvG first randomly
generates graph structures and then uses DeepInversion to learn node features
with objective $C(Y,T(H,A))+\mathcal{R}_{bn}$.

For generating fake graphs, we do 2500 iterations. The learning rates for
structure and feature parameters are set to 1.0 (5.0 on PROTEINS, COLLAB, and
REDDIT-B) and 0.01, respectively, and are divided by 10 every 1000 iterations.
For KD, all the GNNs are trained for 400 epochs with Adam and the learning
rate is linearly decreased from 1.0 to 0. $\tau$ is set to 2. More details are
given in the Appendix.

Figure 3: Ablation studies regarding regularizers

Figure 4: Graph visualization on IMDB-B. Note that there is no correspondence
between the graphs in the two rows.

Figure 5: Ablation studies regarding the number of fake graphs

(a) RandG

(b) GFKD

(c) Teacher

Figure 6: Feature visualization on IMDB-B

###  4.2 Experiments on Bioinformatics Graph Data

Table 1 reports the comparison results on three bioinformatics graph datasets.
It is observed that without using any training data, GFKD transfers much more
knowledge than those of the baselines across different datasets and
architectures, which demonstrates the effectiveness of GFKD. As expected, the
overall performance of RandG is worse than those of the other methods as the
knowledge in the teacher does not concentrate on random graphs. GFKD and
DeepInvG both learn node features for fake graphs, but differ in that GFKD
learns graph structures while DeepInvG randomly generates structures. As shown
in Table 1, GFKD improves the accuracy over DeepInvG substantially. For
example, the accuracy improvement of GFKD is 12.2% over DeepInvG with teacher
GCN-5-64 and student GCN-3-32 on MUTAG. This demonstrates the effectiveness of
GFKD for learning graph structures. We also notice that in the teacher-student
pair with different architectures, i.e, GCN-5-64 and GIN-3-32, GFKD also beats
the baselines significantly on all the three datasets, which demonstrates that
GFKD is applicable to the case where the teacher and the student have
different architectures.

###  4.3 Experiments on Social Network Graph Data

To investigate the generalization of GFKD in different domains, we further
conduct experiments on three social network graph datasets. Table 2 reports
the comparison results. Note that on the three datasets, the node features are
the degrees of the nodes (or a constant) which are derived from the graph
structures. Thus, DeepInvG is reduced to RandG. It is observed that GFKD also
outperforms the baselines substantially on all the three social network
datasets, which demonstrates the generalization and usefulness of GFKD for
different types of graph data. The superiority of GFKD is attributed to its
ability to learn the topology structures of graph data.

###  4.4 Ablation Studies

####  4.4.1 Ablation Studies regarding the Number of Fake Graphs

Theoretically, GFKD can generate infinite fake graphs for knowledge transfer.
However, the quality and the diversity are limited by the pretrained teacher.
We study how the performance of GFKD varies with the number of fake samples.
We denote the ratio of the number of fake graphs to the number of training
samples used by the teacher by $r$.

Figure 6 presents the effects of the number of fake graphs, where GCN-5-64 and
GCN-3-32 are adopted as the teacher and the student, respectively. It is not
surprising that the performances of GFKD first increase and then stabilize.
The reason for the performance stabilization is that the diversity of the
generated graphs is constrained by the pretrained teacher.

####  4.4.2 Ablation Studies regarding the Regularizers

We have introduced two regularizers dealing with BN and one-hot features,
respectively. We evaluate their effects on the performances of GFKD by using
GCN-5-64 and GCN-3-32 as the teacher and the student, respectively. We adopt
MUTAG and PTC datasets as their features are one-hot.

The comparison results are presented in Figure 4, where $\rm GFKD_{w/obn}$,
$\rm GFKD_{w/ooh}$, and $\rm GFKD_{w/obnoh}$ denote GFKD without the BN
regularizer, without the one-hot regularizer, and without neither of the two
regularizers, respectively. We observe that the performance decreases
significantly without either of these two regularizers, which demonstrates the
effectiveness and usefulness of these two regularizers. Meanwhile, this also
indicates that more prior knowledge about the graph data leads to better
performances.

###  4.5 Visualization

Although the goal of GFKD is not to generate real graph data, we present some
fake graphs learned by GFKD in Figure 4, where the fake graphs are learned on
IMDB-B from pretrained teacher GCN-5-64. It is observed that the fake graphs
and the real graphs share some visual similarities.

To further investigate whether GFKD can learn discriminative features from
these fake graphs, we use t-SNE Maaten and Hinton (2008) to visualize the
features learned by different methods. We adopt GCN-5-64 and GCN-3-32 as the
teacher and the student in this experiment, respectively.

Figure 6 presents the visualization of the features learned by RandG, GFKD,
and the teacher. It is observed that the feature representations learned by
RandG are mixed for different classes, which indicates that using randomly
generated graphs cannot learn discriminative features. In contrast, the
features learned by GFKD are well separated for different classes and are as
discriminative as those learned by the teacher. This demonstrates that the
fake samples learned by GFKD are beneficial to representation learning and the
knowledge concentrates on these fake graphs.

##  5 Conclusion

In this paper, we study a novel problem on how to distill knowledge from a GNN
without observable graph data and introduce GFKD as a solution, which is to
our best knowledge the first work along this line. To learn where the
knowledge in the teacher concentrates on, we propose to model the graph
structures with multivariate Bernoulli distribution and then introduce a
gradient estimator to optimize it. Essentially, the structure gradients can be
obtained by only using GNN forward propagation. Extensive experiments on six
benchmark datasets demonstrate the superiority of GFKD for extracting
knowledge from GNNs without observable graphs.

## References

  * Bruna et al. [2013] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun.  Spectral networks and locally connected networks on graphs.  arXiv preprint arXiv:1312.6203, 2013.
  * Chen et al. [2019] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian.  Data-free learning of student networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3514–3522, 2019.
  * Defferrard et al. [2016] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering.  In Advances in neural information processing systems, pages 3844–3852, 2016.
  * Fey et al. [2018] Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller.  Splinecnn: Fast geometric deep learning with continuous b-spline kernels.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 869–877, 2018.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * Hamilton et al. [2017] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In Advances in neural information processing systems, pages 1024–1034, 2017.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * Kipf and Welling [2017] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  International Conference on Learning Representations, 2017.
  * Li et al. [2015] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel.  Gated graph sequence neural networks.  arXiv preprint arXiv:1511.05493, 2015.
  * Lopes et al. [2017] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner.  Data-free knowledge distillation for deep neural networks.  arXiv preprint arXiv:1710.07535, 2017.
  * Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(Nov):2579–2605, 2008.
  * Micaelli and Storkey [2019] Paul Micaelli and Amos J Storkey.  Zero-shot knowledge transfer via adversarial belief matching.  In Advances in Neural Information Processing Systems, pages 9551–9561, 2019.
  * [14] Alexander Mordvintsev, Christopher Olah, and Mike Tyka.  Inceptionism: Going deeper into neural networks.  https://ai.googleblog.com/2015/06/ inceptionism-going-deeper-into-neural.html, last accessed on 12.20.2020.
  * Nayak et al. [2019] Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, R Venkatesh Babu, and Anirban Chakraborty.  Zero-shot knowledge distillation in deep networks.  arXiv preprint arXiv:1905.08114, 2019.
  * Romero et al. [2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  In International Conference on Learning Representations, 2015.
  * Tian et al. [2020] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation.  In International Conference on Learning Representations, 2020.
  * Veličković et al. [2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  arXiv preprint arXiv:1710.10903, 2017.
  * Williams [1992] Ronald J Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine learning, 8(3-4):229–256, 1992.
  * Xu et al. [2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  International Conference on Learning Representations, 2019.
  * Yang et al. [2020] Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang.  Distilling knowledge from graph convolutional networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7074–7083, 2020.
  * Yin and Zhou [2019] Mingzhang Yin and Mingyuan Zhou.  Arm: Augment-reinforce-merge gradient for stochastic binary networks.  In International Conference on Learning Representations, 2019.
  * Yin et al. [2020] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz.  Dreaming to distill: Data-free knowledge transfer via deepinversion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715–8724, 2020.
  * Yoo et al. [2019] Jaemin Yoo, Minyong Cho, Taebum Kim, and U Kang.  Knowledge extraction with no observable data.  In Advances in Neural Information Processing Systems, pages 2705–2714, 2019.
  * Zagoruyko and Komodakis [2017] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  In International Conference on Learning Representations, 2017.