# Generalization of Spivey’s recurrence relation

Taekyun Kim  Department of Mathematics, Kwangwoon University, Seoul 139-701,
Republic of Korea<EMAIL_ADDRESS>and  Dae San Kim  Department of Mathematics,
Sogang University, Seoul 121-742, Republic of Korea<EMAIL_ADDRESS>
###### Abstract.

In 2008, Spivey found a recurrence relation for the Bell numbers $\phi_{n}$.
We consider the probabilistic $r$-Bell polynomials associated with $Y$,
$\phi_{n,r}^{Y}(x)$, which are a probabilistic extension of the $r$-Bell
polynomials. Here $Y$ is a random variable whose moment generating function
exists in some neighborhood of the origin and $\phi_{n}=\phi_{n,0}^{1}(1)$.
The aim of this paper is to generalize the relation for the Bell numbers to
that for the probabilistic $r$-Bell polynomials associated with $Y$.

###### Key words and phrases:

probabilistic $r$-Stirling numbers of the second kind; probabilistic $r$-Bell
polynomials

###### 2010 Mathematics Subject Classification:

11B73; 11B83

##  1\. Introduction

The Stirling number ${n\brace k}$ counts the number of ways to partition a set
of $n$ objects into $k$ nonempty subsets. The Bell polynomials are defined by
$\phi_{n}(x)=\sum_{k=0}^{n}{n\brace k}x^{k}$, with the Bell numbers given by
$\phi_{n}=\phi_{n}(1)$. For $r\in\mathbb{N}$, the $r$-Stirling number
${n+r\brace k+r}_{r}$ enumerates the number of ways to partition a set of
$n+r$ objects into $k+r$ nonempty subsets such that the first $r$ elements are
in distinct subsets. The $r$-Bell polynomials are defined by
$\phi_{n,r}(x)=\sum_{k=0}^{n}{n+r\brace k+r}_{r}x^{k}$, with the $r$-Bell
numbers given by $\phi_{n,r}=\phi_{n,r}(1)$. Assume that $Y$ is a random
variable whose moment generating function exists in some neighbourhood of the
origin. Let $(Y_{j})_{j\geq 1}$ be a sequence of mutually independent copies
of the random variable $Y$, and let $S_{k}=Y_{1}+Y_{2}+\cdots+Y_{k},\ (k\geq
1)$, with $S_{0}=0$. Then we define the probabilistic $r$-Stirling numbers of
the second kind associated with $Y$, ${n+r\brace k+r}_{r,Y}$, as a
probabilistic extension of the $r$-Stirling numbers of the second kind
${n+r\brace k+r}_{r}$. The probabilistic $r$-Bell polynomials associated with
$Y$ are defined by $\phi_{n,r}^{Y}(x)=\sum_{k=0}^{n}{n+r\brace
k+r}_{r,Y}x^{k}$, with the probabilistic $r$-Bell numbers associated with $Y$
given by $\phi_{n,r}^{Y}=\phi_{n,r}^{Y}(1)$.

Spivey found the following recurrence relation on the Bell numbers:

$\phi_{n+k}=\sum_{l=0}^{k}\sum_{j=0}^{n}j^{k-l}\binom{k}{l}{n\brace
j}\phi_{l}.$

The aim of this paper is to generalize the relation for the Bell numbers to
that for the probabilistic $r$-Bell polynomials associated with $Y$. Indeed,
we obtain

(1)
$\phi_{j+n,r}^{Y}(y)=\sum_{l=0}^{n}\binom{n}{l}r^{n-l}\sum_{k=0}^{l}\sum_{m=0}^{j}\frac{y^{k}}{k!}\binom{j}{m}\sum_{l_{1}+\cdots+l_{k}=l}\binom{l}{l_{1},\dots,l_{k}}E\Big{[}S_{k}^{j-m}\prod_{i=1}^{k}Y_{i}^{l_{i}}\Big{]}\phi_{m,r}^{Y}(y),$

where $l_{1},l_{2},\dots l_{k}$ are positive integers.
By using the simple fact (16), we show (1), first for
$\phi_{n}^{Y}=\phi_{n,0}^{Y}(1)$, second for
$\phi_{n}^{Y}(y)=\phi_{n,0}^{Y}(y)$, and finally for $\phi_{n,r}^{Y}(y)$. We
note here that $\phi_{n}=\phi_{n,0}^{1}(1)$. For the rest of this section, we
recall the facts that are needed throughout this paper.

For $n\geq 0$, the Stirling numbers of the second kind are defined by

(2) $x^{n}=\sum_{k=0}^{n}{n\brace k}(x)_{k},\quad(\mathrm{see}\ [1-26]),$

where $(x)_{0}=1,\ (x)_{k}=x(x-1)\cdots(x-k+1),\ (k\geq 1)$.

The Bell numbers are given by

(3) $\phi_{n}=\sum_{k=0}^{n}{n\brace k},\quad(n\geq 0),\quad(\mathrm{see}\
[15,21]).$

The Bell polynomials are defined by

(4) $e^{x(e^{t}-1)}=\sum_{n=0}^{\infty}\phi_{n}(x)\frac{t^{n}}{n!}.$

From (4), we note that

(5) $\phi_{n}(x)=\sum_{k=0}^{n}{n\brace k}x^{k},\quad(n\geq
0),\quad(\mathrm{see}\ [7,8,10,15,21]).$

Note that, for $x=1$, $\phi_{n}=\phi_{n}(1)$ are the Bell numbers.

For $r\in\mathbb{N}$ and $n\geq 0$, the $r$-Stirling numbers of the second
kind are given by

(6) $(x+r)^{n}=\sum_{k=0}^{n}{n+r\brace k+r}_{r}(x)_{k},\quad(\mathrm{see}\
[20]).$

From (6), we note that

(7) $\displaystyle\sum_{l=0}^{n}{n+r\brace l+r}_{r}(x)_{l}$
$\displaystyle=(x+r)^{n}=\sum_{k=0}^{n}\binom{n}{k}r^{n-k}x^{k}$
$\displaystyle=\sum_{k=0}^{n}\binom{n}{k}r^{n-k}\sum_{l=0}^{k}{k\brace
l}(x)_{l}$ $\displaystyle=\sum_{l=0}^{n}\sum_{k=l}^{n}\binom{n}{k}{k\brace
l}r^{n-k}(x)_{l}.$

Thus, by comparing the coefficients on both sides of (7), we get

(8) ${n+r\brace l+r}_{r}=\sum_{k=l}^{n}\binom{n}{k}{k\brace
l}r^{n-k},\quad(n\geq l\geq 0).$

It is known that the Bell numbers satisfy the recurrence relation

(9) $\phi_{n+1}=\sum_{k=0}^{n}\binom{n}{k}\phi_{k},\quad(n\geq
0),\quad(\mathrm{see}\ [14,18]).$

In 2008, Spivey discovered the following remarkable relation

(10) $\phi_{n+k}=\sum_{l=0}^{k}\sum_{j=0}^{n}j^{k-l}\binom{k}{l}{n\brace
j}\phi_{l},\quad(\mathrm{see}\ [24]).$

The $r$-Bell numbers are defined by

(11) $\phi_{n,r}=\sum_{k=0}^{n}{n+r\brace k+r}_{r},\quad(n\geq
0),\quad(\mathrm{see}\ [13-20]).$

The $r$-Bell polynomials are given by

(12)
$e^{rt}e^{x(e^{t}-1)}=\sum_{n=0}^{\infty}\phi_{n,r}(x)\frac{t^{n}}{n!},\quad(\mathrm{see}\
[13-20]).$

Thus, by (12), we get

(13) $\phi_{n,r}(x)=\sum_{k=0}^{n}{n+r\brace k+r}_{r}x^{k},\quad(n\geq
0),\quad(\mathrm{see}\ [6,8,16,17]).$

Note that, for $x=1$, $\phi_{n,r}=\phi_{n,r}(1)$ are the $r$-Bell numbers.

For any integer $k\geq 0$, the partial Bell polynomials are given by

(14)
$\frac{1}{k!}\bigg{(}\sum_{m=1}^{\infty}x_{m}\frac{t^{n}}{n!}\bigg{)}^{k}=\sum_{n=k}^{\infty}B_{n,k}\big{(}x_{1},x_{2},\dots,x_{n-k+1}\big{)}\frac{t^{n}}{n!},\quad(\mathrm{see}\
[10,11,14]),$

where

(15) $\displaystyle B_{n,k}(x_{1},x_{2},\dots,x_{n-k+1})$
$\displaystyle=\sum_{\begin{subarray}{c}l_{1}+\cdots+l_{n-k+1}\\\
l_{1}+2l_{2}+\cdots+(n-k+1)l_{n-k+1}=n\end{subarray}}\frac{n!}{l_{1}!l_{2}!\cdots
l_{n-k+1}!}\bigg{(}\frac{x_{1}}{1!}\bigg{)}^{l_{1}}\bigg{(}\frac{x_{2}}{2!}\bigg{)}^{l_{2}}\cdots\bigg{(}\frac{x_{n-k+1}}{(n-k+1)!}\bigg{)}^{l_{n-k+1}}.$

We will use the following simple fact

(16) $\displaystyle f(x+t)$
$\displaystyle=\sum_{n=0}^{\infty}\frac{f^{(n)}(x)}{n!}(x+t-x)^{n}=\sum_{n=0}^{\infty}\frac{f^{(n)}(x)}{n!}t^{n}$
$\displaystyle=\sum_{n=0}^{\infty}\frac{t^{n}D_{x}^{n}}{n!}f(x)=e^{tD_{x}}f(x),$

where $D_{x}=\frac{d}{dx}$.

Assume that $Y$ is a random variable such that the moment generating function
of $Y$,

$E\big{[}e^{tY}\big{]}=\sum_{n=0}^{\infty}E[Y^{n}]\frac{t^{n}}{n!},\quad(|t|<r),$

exists for some $r>0$ (see [3,14,22]). Here $E$ denotes the mathematical
expectation. Let $(Y_{j})_{j\geq 1}$ be a sequence of mutually independent
copies of the random variable $Y$, and let $S_{k}=Y_{1}+Y_{2}+\cdots+Y_{k},\
(k\geq 1)$, with $S_{0}=0$.

Then we recall the probabilistic Stirling numbers of the second kind
associated with $Y$, ${n\brace k}_{Y}$, which are defined in terms of the
$n$-th moments of $S_{l},\ (l=0,1,2,\dots,k)$:

(17) ${n\brace
k}_{Y}=\frac{1}{k!}\sum_{l=0}^{k}\binom{k}{l}(-1)^{k-l}E\big{[}S_{l}^{n}\big{]},\quad(0\leq
k\leq n),\quad(\mathrm{see}\ [3,15]).$

The probabilistic Bell polynomials associated with $Y$ are defined by

(18) $\phi_{n}^{Y}(x)=\sum_{k=0}^{n}{n\brace k}_{Y}x^{k},\quad(n\geq
0),\quad(\mathrm{see}\ [14,23]).$

From [14], we note that the probabilistic Bell polynomials associated with $Y$
satisfy the recurrence relation

(19)
$\phi_{n+1}^{Y}(x)=x\sum_{k=0}^{n}\binom{n}{k}E\big{[}Y^{k+1}\big{]}\phi_{n-k}^{Y}(x).$

##  2\. Generalization of Spivey’s recurrence relation

Let $(Y_{j})_{j\geq 1}$ be a sequence of mutually independent copies of the
random variable $Y$, and let

$S_{0}=0,\quad S_{k}=Y_{1}+Y_{2}+\cdots+Y_{k},\quad(k\in\mathbb{N}).$

For $r\in\mathbb{N}$, we define the probabilistic $r$-Stirling numbers of the
second kind associated with $Y$ by

(20)
$\frac{1}{k!}\Big{(}E[e^{tY}]-1\Big{)}^{k}e^{rt}=\sum_{n=k}^{\infty}{n+r\brace
k+r}_{r,Y}\frac{t^{n}}{n!},\quad(k\geq 0).$

When $Y=1$, ${n+r\brace k+r}_{r,Y}={n+r\brace k+r}_{r},\ (n\geq k\geq 0)$.

From (20), we note that

(21) $\displaystyle\sum_{n=k}^{\infty}{n+r\brace k+r}_{r,Y}\frac{t^{n}}{n!}$
$\displaystyle=\frac{1}{k!}\Big{(}E\big{[}e^{Yt}\big{]}-1\Big{)}^{k}e^{rt}$
$\displaystyle=\frac{1}{k!}\sum_{j=0}^{k}\binom{k}{j}(-1)^{k-j}E\big{[}e^{(Y_{1}+\cdots+Y_{j})t}\big{]}e^{rt}$
$\displaystyle=\frac{1}{k!}\sum_{j=0}^{k}\binom{k}{j}(-1)^{k-j}E\big{[}e^{(S_{j}+r)t}\big{]}$
$\displaystyle=\sum_{n=0}^{\infty}\bigg{(}\frac{1}{k!}\sum_{j=0}^{k}\binom{k}{j}(-1)^{k-j}E\big{[}(S_{j}+r)^{n}\big{]}\bigg{)}\frac{t^{n}}{n!}.$

Therefore, by comparing the coefficients on both sides of (21), we obtain the
following theorem.

######  Theorem 2.1.

For $n\geq k\geq 0$, we have

${n+r\brace
k+r}_{r,Y}=\frac{1}{k!}\sum_{j=0}^{k}\binom{k}{j}(-1)^{k-j}E\big{[}(S_{j}+r)^{n}\big{]}.$

In view of (13), we define the probabilistic $r$-Bell polynomials associated
with $Y$ by

(22) $\phi_{n,r}^{Y}(x)=\sum_{k=0}^{n}{n+r\brace k+r}_{r,Y}x^{k},\quad(n\geq
0).$

When $Y=1$, $\phi_{n,r}^{Y}(x)=\phi_{n,r}(x)$. For $x=1$,
$\phi_{n,r}^{Y}=\phi_{n,r}^{Y}(1)$ are called the probabilistic $r$-Bell
numbers associated with $Y$.

From (22), we have

(23) $\displaystyle e^{x(E[e^{Yt}]-1)}e^{rt}$
$\displaystyle=\sum_{k=0}^{\infty}\frac{x^{k}}{k!}\Big{(}E[e^{Yt}]-1\Big{)}^{k}e^{rt}$
$\displaystyle=\sum_{k=0}^{\infty}x^{k}\sum_{n=k}^{\infty}{n+r\brace
k+r}_{r,Y}\frac{t^{n}}{n!}=\sum_{n=0}^{\infty}\sum_{k=0}^{n}x^{k}{n+r\brace
k+r}_{r,Y}\frac{t^{n}}{n!}$
$\displaystyle=\sum_{n=0}^{\infty}\phi_{n,r}^{Y}(x)\frac{t^{n}}{n!}.$

Therefore, by (23), we obtain the following theorem.

######  Theorem 2.2.

The generating function of the probabilistic $r$-Bell polynomials associated
with $Y$ is given by

(24)
$e^{x(E[e^{Yt}]-1)}e^{rt}=\sum_{n=0}^{\infty}\phi_{n,r}^{Y}(x)\frac{t^{n}}{n!}.$

From (24), we have

(25)
$\displaystyle\sum_{n=0}^{\infty}\phi_{n,r}^{Y}(x)\frac{t^{n}}{n!}=e^{x(E[e^{Yt}]-1)}e^{rt}=\sum_{k=0}^{\infty}\frac{1}{k!}\bigg{(}\sum_{j=1}^{\infty}xE[Y^{j}]\frac{t^{j}}{j!}\bigg{)}^{k}e^{rt}$
$\displaystyle=\sum_{l=0}^{\infty}\sum_{k=0}^{l}B_{l,k}\Big{(}xE[Y],xE[Y^{2}],\dots,xE[Y^{l-k+1}]\Big{)}\frac{t^{l}}{l!}\sum_{m=0}^{\infty}r^{m}\frac{t^{m}}{m!}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{l=0}^{n}\binom{n}{l}\sum_{k=0}^{l}B_{l,k}\Big{(}xE[Y],xE[Y^{2}],\dots,xE[Y^{l-k+1}]\Big{)}r^{n-l}\frac{t^{n}}{n!}.$

Therefore, by comparing the coefficients on both sides of (25), we obtain the
following theorem.

######  Theorem 2.3.

For $n\geq 0$, we have

$\phi_{n,r}^{Y}(x)=\sum_{l=0}^{n}\binom{n}{l}\sum_{k=0}^{l}B_{l,k}\Big{(}xE[Y],xE[Y^{2}],\dots,xE[Y^{l-k+1}]\Big{)}r^{n-l}.$

By (24), we get

(26) $\displaystyle\sum_{n=0}^{\infty}\phi_{n+1,r}^{Y}(x)\frac{t^{n}}{n!}$
$\displaystyle=\frac{d}{dt}\sum_{n=0}^{\infty}\phi_{n,r}^{Y}(x)\frac{t^{n}}{n!}=\frac{d}{dt}\bigg{(}e^{x(E[e^{tY}]-1)}e^{rt}\bigg{)}$
$\displaystyle=xE[Ye^{Yt}]e^{x(E[e^{Yt}]-1)}e^{rt}+re^{x(E[e^{tY}]-1)}e^{rt}$
$\displaystyle=x\sum_{k=0}^{\infty}E[Y^{k+1}]\frac{t^{k}}{k!}\sum_{m=0}^{\infty}\phi_{m,r}^{Y}(x)\frac{t^{m}}{m!}+r\sum_{n=0}^{\infty}\phi_{n,r}^{Y}(x)\frac{t^{n}}{n!}$
$\displaystyle=\sum_{n=0}^{\infty}\bigg{(}x\sum_{k=0}^{n}\binom{n}{k}E[Y^{k+1}]\phi_{n-k,r}^{Y}(x)+r\phi_{n,r}^{Y}(x)\bigg{)}\frac{t^{n}}{n!}.$

Therefore, by comparing the coefficients on both sides of (26), we obtain the
following theorem.

######  Theorem 2.4.

For $n\geq 0$, we have

$\phi_{n+1,r}^{Y}(x)=x\sum_{k=0}^{n}\binom{n}{k}E[Y^{k+1}]\phi_{n-k,r}^{Y}(x)+r\phi_{n,r}^{Y}(x).$

From (18), we note that

(27) $e^{E[e^{Yx}]-1}=\sum_{n=0}^{\infty}\phi_{n}^{Y}\frac{x^{n}}{n!}.$

By using (16), we have

(28) $\displaystyle e^{tD_{x}}e^{E[e^{Yx}]-1}$
$\displaystyle=e^{E[e^{Y(x+t)}]-1}$
$\displaystyle=e^{E[e^{Yx}(e^{Yt}-1)+e^{Yx}]-1}$
$\displaystyle=e^{E[e^{Yx}(e^{Yt}-1)]}e^{E[e^{Yx}]-1}.$

Now, we observe that

(29) $\displaystyle
e^{E[e^{Yx}(e^{Yt}-1)]}=\sum_{k=0}^{\infty}\frac{1}{k!}\Big{(}E\big{[}e^{Yx}(e^{Yt}-1)\big{]}\Big{)}^{k}$
$\displaystyle=\sum_{k=0}^{\infty}\frac{1}{k!}E\Big{[}e^{(Y_{1}+\cdots+Y_{k})x}(e^{Y_{1}t}-1)(e^{Y_{2}t}-1)\cdots(e^{Y_{k}t}-1)\Big{]}$
$\displaystyle=\sum_{k=0}^{\infty}\frac{1}{k!}\sum_{n=k}^{\infty}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}E\big{[}e^{S_{k}x}Y_{1}^{l_{1}}Y_{2}^{l_{2}}\cdots
Y_{k}^{l_{k}}\big{]}\frac{t^{n}}{n!}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{k=0}^{n}\frac{1}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}E\bigg{[}e^{S_{k}x}\prod_{i=1}^{k}Y_{i}^{l_{i}}\bigg{]}\frac{t^{n}}{n!},$

where $l_{1},l_{2},\dots l_{k}$ are positive integers. From (27), (28) and
(29), we note that

(30) $\displaystyle
e^{tD_{x}}e^{E[e^{Yx}]-1}=e^{E[e^{Yx}(e^{Yt}-1)]}e^{E[e^{Yx}]-1}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{k=0}^{n}\frac{1}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}\frac{t^{n}}{n!}\sum_{j=0}^{\infty}E\bigg{[}S_{k}^{j}\prod_{i=1}^{k}Y_{i}^{l_{i}}\bigg{]}\frac{x^{j}}{j!}\sum_{m=0}^{\infty}\phi_{m}^{Y}\frac{x^{m}}{m!}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{l=0}^{\infty}\sum_{k=0}^{n}\sum_{m=0}^{l}\binom{l}{m}\frac{1}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}E\bigg{[}S_{k}^{l-m}\prod_{i=1}^{k}Y_{i}^{l_{i}}\bigg{]}\phi_{m}^{Y}\frac{t^{n}}{n!}\frac{x^{l}}{l!},$

where $l_{1},l_{2},\dots l_{k}$ are positive integers. On the other hand, by
(27), we get

(31) $\displaystyle
e^{tD_{x}}e^{E[e^{Yx}]-1}=e^{tD_{x}}\sum_{l=0}^{\infty}\phi_{l}^{Y}\frac{x^{l}}{l!}=\sum_{n=0}^{\infty}\frac{t^{n}}{n!}D_{x}^{n}\sum_{l=0}^{\infty}\phi_{l}^{Y}\frac{x^{l}}{l!}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{l=0}^{\infty}\phi_{l+n}^{Y}\frac{t^{n}}{n!}\frac{x^{l}}{l!}.$

Therefore, by (30) and (31), we obtain the following theorem.

######  Theorem 2.5.

For $n,l\geq 0$, we have

$\phi_{l+n}^{Y}=\sum_{k=0}^{n}\sum_{m=0}^{l}\binom{l}{m}\frac{1}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}E\bigg{[}S_{k}^{l-m}\prod_{i=1}^{k}Y_{i}^{l_{i}}\bigg{]}\phi_{m}^{Y},$

where $l_{1},l_{2},\dots l_{k}$ are positive integers.

We note here that this reduces to the Spivey’s relation (10) if $Y=1$.

$\displaystyle\phi_{l+n}$
$\displaystyle=\sum_{k=0}^{n}\sum_{m=0}^{l}\binom{l}{m}\frac{1}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}k^{l-m}\phi_{m}$
$\displaystyle=\sum_{k=0}^{n}\sum_{m=0}^{l}\binom{l}{m}{n\brace
k}k^{l-m}\phi_{m}.$

We extend Theorem 2.5 to the probabilistic Bell polynomials associated with
$Y$, which are given by (see (18), Theorem 2.2)

(32) $e^{x(E[e^{Yt}]-1)}=\sum_{n=0}^{\infty}\phi_{n}^{Y}(x)\frac{t^{n}}{n!}.$

Observe that

(33) $\displaystyle e^{tD_{x}}e^{y(E[e^{Yx}]-1)}$
$\displaystyle=e^{y(E[e^{Y(x+t)}]-1)}$
$\displaystyle=e^{y(E[e^{Yx}(e^{Yt}-1)+E[e^{Yx}]-1)}$
$\displaystyle=e^{yE[e^{Yx}(e^{Yt}-1)]}e^{y(E[e^{Yx}]-1)}.$

Now, we note that

(34) $\displaystyle
e^{yE[e^{Yx}(e^{Yt}-1)]}=\sum_{k=0}^{\infty}\frac{y^{k}}{k!}\Big{(}E\big{[}e^{Yx}(e^{Yt}-1)\big{]}\Big{)}^{k}$
$\displaystyle=\sum_{k=0}^{\infty}\frac{y^{k}}{k!}E\Big{[}e^{(Y_{1}+Y_{2}+\cdots+Y_{k})x}(e^{Y_{1}t}-1)(e^{Y_{2}t}-1)\cdots(e^{Y_{k}t}-1)\Big{]}$
$\displaystyle=\sum_{k=0}^{\infty}\frac{y^{k}}{k!}E\Big{[}e^{S_{k}x}(e^{Y_{1}t}-1)(e^{Y_{2}t}-1)\cdots(e^{Y_{k}t}-1)\Big{]}$
$\displaystyle=\sum_{k=0}^{\infty}\frac{y^{k}}{k!}\sum_{n=k}^{\infty}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}E\Big{[}e^{S_{k}x}Y_{1}^{l_{1}}Y_{2}^{l_{2}}\cdots
Y_{k}^{l_{k}}\Big{]}\frac{t^{n}}{n!}$
$\displaystyle=\sum_{n=0}^{\infty}\bigg{(}\sum_{k=0}^{n}\frac{y^{k}}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}E\bigg{[}e^{S_{k}x}\prod_{i=1}^{k}Y_{i}^{l_{i}}\bigg{]}\frac{t^{n}}{n!},$

where $l_{1},l_{2},\dots l_{k}$ are positive integers. From (33) and (34), we
have

(35) $\displaystyle
e^{tD_{x}}e^{y(E[e^{Yx}]-1)}=e^{yE[e^{Yx}(e^{Yt}-1)]}e^{y(E[e^{Yx}]-1)}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{k=0}^{n}\frac{y^{k}}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}\frac{t^{n}}{n!}\sum_{j=0}^{\infty}E\bigg{[}S_{k}^{j}\prod_{i=1}^{k}Y_{i}^{l_{i}}\bigg{]}\frac{x^{j}}{j!}\sum_{m=0}^{\infty}\phi_{m}^{Y}(y)\frac{x^{m}}{m!}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{l=0}^{\infty}\sum_{k=0}^{n}\sum_{m=0}^{l}\frac{y^{k}}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}\binom{l}{m}E\bigg{[}S_{k}^{l-m}\prod_{i=1}^{k}Y_{i}^{l_{i}}\bigg{]}\phi_{m}^{Y}(y)\frac{t^{n}}{n!}\frac{x^{l}}{l!},$

where $l_{1},l_{2},\dots l_{k}$ are positive integers. On the other hand, by
(32), we get

(36) $\displaystyle e^{tD_{x}}e^{y(E[e^{Yx}]-1)}$
$\displaystyle=\sum_{n=0}^{\infty}\frac{D_{x}^{n}}{n!}t^{n}\sum_{l=0}^{\infty}\phi_{l}^{Y}(y)\frac{x^{l}}{l!}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{l=0}^{\infty}\phi_{l+n}^{Y}(y)\frac{t^{n}}{n!}\frac{x^{l}}{l!}.$

Therefore, by (35) and (36), we obtain the following theorem.

######  Theorem 2.6.

For $n,l\geq 0$, we have

$\phi_{l+y}^{Y}(y)=\sum_{k=0}^{n}\sum_{m=0}^{l}\binom{l}{m}\frac{y^{k}}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}E\bigg{[}S_{k}^{l-m}\prod_{i=1}^{k}Y_{i}^{l_{i}}\bigg{]}\phi_{m}^{Y}(y),$

where $l_{1},l_{2},\dots l_{k}$ are positive integers.

When $Y=1$, we note that

(37) $\displaystyle\phi_{l+n}(y)$
$\displaystyle=\sum_{k=0}^{n}\sum_{m=0}^{l}\binom{l}{m}y^{k}\frac{1}{k!}\sum_{l_{1}+\cdots+l_{k}=n}\binom{n}{l_{1},\dots,l_{k}}k^{l-m}\phi_{m}(y)$
$\displaystyle=\sum_{k=0}^{n}\sum_{m=0}^{l}\binom{l}{m}y^{k}{n\brace
k}k^{l-m}\phi_{m}(y),$

where $l,n$ are nonnegative integers. Note that (37) reduces to the Spivey’s
relation (10) by letting $y=1$.

From (16) and (24), we note that

(38) $\displaystyle
e^{tD_{x}}\Big{(}e^{y(E[e^{Yx}]-1)}e^{rx}\Big{)}=e^{y(E[e^{Y(x+t)}]-1)}e^{r(x+t)}$
$\displaystyle=e^{y(E[e^{Yx}(e^{Yt}-1)])}e^{rt}e^{y(E[e^{Yx}]-1)}e^{rx}$
$\displaystyle=\sum_{k=0}^{\infty}\frac{y^{k}}{k!}E\Big{[}e^{(Y_{1}+Y_{2}+\cdots+Y_{k})x}(e^{Y_{1}t}-1)\cdots(e^{Y_{k}t}-1)\Big{]}e^{rt}e^{y(E[e^{Yx}]-1)}e^{rx}$
$\displaystyle=\sum_{k=0}^{\infty}\frac{y^{k}}{k!}\sum_{l=k}^{\infty}\sum_{l_{1}+\cdots+l_{k}=l}\binom{l}{l_{1},\dots,l_{k}}E\Big{[}e^{S_{k}x}\prod_{i=1}^{k}Y_{i}^{l_{i}}\Big{]}\frac{t^{l}}{l!}e^{rt}e^{y(E[e^{Yx}]-1)}e^{rx}$
$\displaystyle=\sum_{l=0}^{\infty}\sum_{k=0}^{l}\frac{y^{k}}{k!}\sum_{l_{1}+\cdots+l_{k}=l}\binom{l}{l_{1},\dots,l_{k}}\frac{t^{l}}{l!}e^{rt}E\Big{[}e^{S_{k}x}\prod_{i=1}^{k}Y_{i}^{l_{i}}\Big{]}e^{y(E[e^{Yx}]-1)}e^{rx}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{l=0}^{n}\binom{n}{l}r^{n-l}\sum_{k=0}^{l}\frac{y^{k}}{k!}\sum_{l_{1}+\cdots+l_{k}=l}\binom{l}{l_{1},\dots,l_{k}}\frac{t^{n}}{n!}\sum_{j=0}^{\infty}E\Big{[}S_{k}^{j}\prod_{i=1}^{k}Y_{i}^{l_{i}}\Big{]}\frac{x^{j}}{j!}\sum_{m=0}^{\infty}\phi_{m,r}^{Y}(y)\frac{x^{m}}{m!}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{j=0}^{\infty}\sum_{m=0}^{j}\sum_{l=0}^{n}\binom{n}{l}r^{n-l}\binom{j}{m}\sum_{k=0}^{l}\frac{y^{k}}{k!}\sum_{l_{1}+\cdots+l_{k}=l}\binom{l}{l_{1},\dots,l_{k}}E\Big{[}S_{k}^{j-m}\prod_{i=1}^{k}Y_{i}^{l_{i}}\Big{]}\phi_{m,r}^{Y}(y)\frac{t^{n}}{n!}\frac{x^{j}}{j!},$

where $l_{1},l_{2},\dots l_{k}$ are positive integers. On the other hand, by
(24), we get

(39) $\displaystyle e^{tD_{x}}\Big{(}e^{y(E[e^{Yx}]-1)}e^{rx}\Big{)}$
$\displaystyle=\sum_{n=0}^{\infty}\frac{t^{n}}{n!}D_{x}^{n}\sum_{j=0}^{\infty}\phi_{j,r}^{Y}(y)\frac{x^{j}}{j!}$
$\displaystyle=\sum_{n=0}^{\infty}\sum_{j=0}^{\infty}\phi_{j+n,r}^{Y}(y)\frac{t^{n}}{n!}\frac{x^{j}}{j!}.$

Therefore, by (38) and (39), we obtain the following theorem.

######  Theorem 2.7.

For $n,j\geq 0$, we have

$\phi_{j+n,r}^{Y}(y)=\sum_{l=0}^{n}\binom{n}{l}r^{n-l}\sum_{k=0}^{l}\sum_{m=0}^{j}\frac{y^{k}}{k!}\binom{j}{m}\sum_{l_{1}+\cdots+l_{k}=l}\binom{l}{l_{1},\dots,l_{k}}E\Big{[}S_{k}^{j-m}\prod_{i=1}^{k}Y_{i}^{l_{i}}\Big{]}\phi_{m,r}^{Y}(y),$

where $l_{1},l_{2},\dots l_{k}$ are positive integers.

When $Y=1$, we have

$\displaystyle\phi_{j+n,r}(y)$
$\displaystyle=\sum_{l=0}^{n}\sum_{m=0}^{j}\binom{n}{l}\binom{j}{m}r^{n-l}\sum_{k=0}^{l}\frac{y^{k}}{k!}\sum_{l_{1}+\cdots+l_{k}=l}\binom{n}{l_{1},\dots,l_{k}}k^{j-m}\phi_{m,r}(y)$
$\displaystyle=\sum_{l=0}^{n}\sum_{m=0}^{j}\sum_{k=0}^{l}\binom{n}{l}\binom{j}{m}r^{n-l}\phi_{m,r}(y)y^{k}k^{j-m}{l\brace
k},$

where $j,n$ are nonnegative integers.

In particular, for $n=1$, we have

$\phi_{j+1,r}^{Y}(y)=y\sum_{k=0}^{j}\binom{j}{k}E\big{[}Y^{k+1}\big{]}\phi_{j-k,r}^{Y}(y)+r\phi_{j,r}^{Y}(y).$

##  3\. Conclusion

Let $Y$ be a random variable such that the moment generating function of $Y$
exists in a neighborhood of the origin. We introduced probabilistic extensions
of the $r$-Stirling numbers of the second and the $r$-Bell polynomials, namely
the probabilistic $r$-Stirling numbers of the second associated with $Y$,
${n+r\brace k+r}_{r,Y}$ and the probabilistic $r$-Bell polynomials associated
with $Y$, $\phi_{n,r}^{Y}(y)$. Here the latter is a natural polynomial
extension of the former. Spivey found a recurrence relation for the Bell
numbers $\phi_{n}$. In this paper, we generalized the recurrence relation for
$\phi_{n}=\phi_{n,0}^{1}(1)$ to that for $\phi_{n,r}^{Y}(y)$ by utilizing the
simple fact (16).

In more detail, we obtained for ${n+r\brace k+r}_{r,Y}$ an explicit expression
in terms of the $n$th moments of $S_{j}+r,\,\,(0\leq j\leq k)$ in Theorem 2.1.
We derived for $\phi_{n,r}^{Y}(x)$ the generating function, a finite sum
expression involving the incomplete Bell polynomials and a recurrence
relation, respectively in Theorem 2.2, Theorem 2.3 and Theorem 2.4. Then the
relation for the Bell numbers in (10) was generalized to first that for
$\phi_{n}^{Y}=\phi_{n,0}^{Y}(1)$, second that for
$\phi_{n}^{Y}(y)=\phi_{n,0}^{Y}(y)$, and finally that for $\phi_{n,r}^{Y}(y)$,
respectively in Theorem 2.5, Theorem 2.6 and Theorem 2.7.

As one of our future projects, we would like to continue to study
probabilistic extensions of many special polynomials and numbers and to find
their applications to physics, science and engineering as well as to
mathematics.

## References

  * [1] Abbas, M.; Bouroubi, S. _On new identities for Bell’s polynomials,_ Discrete Math. 293 (2005), no. 1-3, 5-10.
  * [2] Abramowitz M.; Stegun, I.A. _Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables,_ Dover, New York, 1992.
  * [3] Adell, J. A. _Probabilistic Stirling numbers of the second kind and applications,_ J. Theoret. Probab. 35 (2022), no. 1, 636-652.
  * [4] Belbachir, H.; Bousbaa, I. E. _Translated Whitney and $r$-Whitney numbers: a combinatorial approach,_ J. Integer Seq. 16 (2013), no. 8, Article 13.8.6, 7 pp.
  * [5] Boubellouta, K.; Boussayoud, A.; Araci, S.; Kerada, M. _Some theorems on generating functions and their applications,_ Adv. Stud. Contemp. Math. (Kyungshang) 30 (2020), no. 3, 307-324.
  * [6] Broder, A. Z. _The $r$-Stirling numbers,_ Discrete Math. 49 (1984), no. 3, 241-259.
  * [7] Carlitz, L. _Some remarks on the Bell numbers,_ Fibonacci Quart. 18 (1980), no. 1, 66-73.
  * [8] Carlitz, L. _Weighted Stirling numbers of the first and second kind II,_ Fibonacci Quart. 18 (1980), no. 3, 242-257.
  * [9] Carlitz, L. _Weighted Stirling numbers of the first and second kind I,_ Fibonacci Quart. 18 (1980), no. 2, 147-162.
  * [10] Comtet, L. _Advanced combinatorics. The art of finite and infinite expansions,_ Revised and enlarged edition. D. Reidel Publishing Co., Dordrecht, 1974.
  * [11] Graham, R. L.; Knuth, D. E.; Patashnik, O. _Concrete mathematics. A foundation for computer science,_ Second edition. Addison-Wesley Publishing Company, Reading, MA, 1994.
  * [12] Kilar, N.; Simsek, Y. _Combinatorial sums involving Fubini type numbers and other special numbers and polynomials: approach trigonometric functions and $p$-adic integrals,_ Adv. Stud. Contemp. Math. (Kyungshang) 31 (2021), no. 1, 75-87.
  * [13] Kim, D. S.; Kim, T. K. _Normal ordering associated with $\lambda$-Whitney numbers of the first kind in $\lambda$-shift algebra,_ Russ. J. Math. Phys. 30 (2023), no. 3, 310-319.
  * [14] Kim, T.; Kim, D. S. _Probabilistic degenerate Bell polynomials associated with random variables,_ Russ. J. Math. Phys. 30 (2023), no. 4, 528-542.
  * [15] Kim, T.; Kim, D. S.; Kwon, J. _Probabilistic degenerate Stirling polynomials of the second kind and their applications,_ Math. Comput. Model. Dyn. Syst. 30 (2024), no. 1, 16-30.
  * [16] Kim, T.; Kim, D. S. _Some identities on degenerate $r$-Stirling numbers via boson operators,_ Russ. J. Math. Phys. 29 (2022), no. 4, 508-517.
  * [17] Kim, T.; Kim, D. S. _Degenerate Whitney numbers of first and second kind of Dowling lattices,_ Russ. J. Math. Phys. 29 (2022), no. 3, 358-377.
  * [18] Kim, T.; Kim, D. S. _Degenerate $r$-Whitney numbers and degenerate $r$-Dowling polynomials via boson operators,_ Adv. in Appl. Math. 140 (2022), Paper No. 102394, 21 pp.
  * [19] Kim, T.; Kim, D. S. _Combinatorial identities involving degenerate harmonic and hyperharmonic numbers,_ Adv. Appl. Math. 148 (2023), Paper No. 102535, 15 pp.
  * [20] Kim, T.; Kim, D. S.; Lee, H.; Park, J.-W. _A note on degenerate $r$-Stirling numbers,_ J. Inequal. Appl. 2020, 2020:225.
  * [21] Roman, S. _The umbral calculus,_ Pure and Applied Mathematics, 111. Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York, 1984.
  * [22] Ross, S. M. _Introduction to probability models,_ Thirteenth edition. Academic Press, London, 2024.
  * [23] Soni, R.; Vellaisamy, P.; Pathak, A. K. _A probabilistic generalization of the Bell polynomials,_ J. Anal. (2023). https://doi.org/10.1007/s41478-023-00642-y
  * [24] Spivey, M. Z. _A generalized recurrence for Bell numbers,_ J. Integer Seq. 11 (2008), no. 2, Article 08.2.5, 3 pp.
  * [25] Ta, B. Q. _Probabilistic approach to Appell polynomials,_ Expo. Math. 33 (2015), no. 3, 269-294,
  * [26] Teicher, H. _An inequality on Poisson probabilities,_ Ann. Math. Statist. 26 (1955), 147-149.


0 Research

# Scene Graph Conditioning in Latent Diffusion

Frank Fundel
Universität Ulm<EMAIL_ADDRESS>
###### Abstract

Diffusion models excel in image generation but lack detailed semantic control
using text prompts. Additional techniques have been developed to address this
limitation. However, conditioning diffusion models solely on text-based
descriptions is challenging due to ambiguity and lack of structure. In
contrast, scene graphs offer a more precise representation of image content,
making them superior for fine-grained control and accurate synthesis in image
generation models. The amount of image and scene-graph data is sparse, which
makes fine-tuning large diffusion models challenging. We propose multiple
approaches to tackle this problem using ControlNet and Gated Self-Attention.
We were able to show that using out proposed methods it is possible to
generate images from scene graphs with much higher quality, outperforming
previous methods. Our source code is publicly available on
https://github.com/FrankFundel/SGCond

##  1 Introduction

Diffusion models [34, 10] have emerged as powerful image generation models, by
gradually removing noise at each timestep. Recent advancements, such as Latent
Diffusion [33], have improved the efficiency and computational cost of
diffusion models. Latent Diffusion applies diffusion processes on a compressed
latent space, achieved through a VQVAE-based autoencoder. Diffusion models
exhibit great stability during training, avoid mode collapse, and generate
images of exceptional quality. However, they are computationally expensive for
sampling new images and lack a low-dimensional latent space compared to
traditional VAEs, limiting conditional control. Nonetheless, diffusion models
have proven to be highly effective in image generation, surpassing the
performance of GANs in terms of image quality and stability [5].

Various techniques have been proposed to add more control to diffusion models.
ControlNet [46] focuses on fine-tuning a copy of the model, while preserving
the original knowledge using zero-convolutions. T2I-Adapter [23] offers a
lightweight model for fine-tuning only a smaller adapter model while leaving
the diffusion model unchanged. GLIGEN [20] introduces grounded text-to-image
diffusion, combining text and grounding embeddings using Gated Self-Attention
to enhance contextual understanding. These techniques showcase the
advancements in controlling diffusion models.

While conditioning on text for image generation models has its limitations,
scene graphs offer a more structured and unambiguous representation of image
content. Text-based descriptions can be long, loosely structured, and subject
to semantic ambiguity. In contrast, scene graphs provide a concise and precise
encoding of object relationships and spatial arrangements within an image.
These advantages make scene graphs a superior choice for conditioning image
generation models when accurate synthesis and fine-grained control over visual
content are crucial.

To address these challenges, different methods for utilizing scene graphs in
image generation have been proposed. Some techniques rely on predicting scene
layouts, which are coarse representations of the intended scenes [16, 37].
Other methods condition on crops containing single objects corresponding to
nodes in a scene graph [21]. Recently, works have been proposed which involve
learning embeddings through masked contrastive pre-training to align image and
graph features [44]. However, images that are conditioned on scene graphs are
not yet comparable to images that are conditioned on other modalities such as
bounding boxes, poses or depth maps.

### Contribution

Our contribution can be summarized with the following points:

  * •

We propose multiple approaches for effectively fine-tuning diffusion models
using scene graphs

  * •

We evaluated the most promising methods to some extend 5.3

##  2 Background

###  2.1 Image Generation

Recently, image generation has acquired significant attention due to the
remarkable advancements and capabilities demonstrated by deep learning models.
These models have the potential to generate highly realistic and diverse
images, often indistinguishable from those captured by cameras or created by
humans [5, 45].

####  2.1.1 Autoencoder

Autoencoders are great at compressing and reconstructing images, while
preserving their most prominent features. By encoding an image using a
Convolutional Neural Network (CNN) into a compressed latent space
representation and then decoding this representation back into an image, a
simple and easy to train method for image generation is at hand - one can
simply manipulate the latent vector and thus, induce the desired changes. But
autoencoders also bring an interesting problem when it comes to image
generation. Autoencoders map an input image directly into a single point in
the latent space which results in a discrete distribution of points [26]. This
makes interpolation between different latent representations difficult.
Additionally, the distribution of points in the latent space is uncertain
which results in poor sampling of novel images.

####  2.1.2 GAN

Generative Adversarial Networks (GANs) [9] solve the issue of non-uniform
latent space by generating images directly from random uniform noise in the
latent space. To train the generator network another network, the
discriminator network, is trained simultaneously to decide if an image is from
a set of example images or if it is generated. As the discriminator becomes
better at deciding if an image is real or fake, the generator becomes better
at generating new images. Because of the adversarial structure of the model,
GANs are both very effective and can generate images almost indistinguishable
from real images, but also difficult to train. Training is highly sensitive to
hyperparameter selection and sometimes the model is unable to converge
properly. Additionally, the generator can also learn to map every latent
vector to the same few output images that are able to fool the discriminator.
This phenomena is called mode collapse is one of the hardest problems GANs
suffer from Ying et al. [45].

####  2.1.3 VAE

Another approach that solves the non-uniform distribution problem of
autoencoders are Variational Autoencoders (VAEs) [17]. Instead of directly
encoding the input data into a single point in the latent space, the encoder
network predicts the parameters $\sigma$ and $\mu$ of a probability
distribution function. To reconstruct the image, a latent vector needs to be
sampled from the distribution. If one took the predicted distribution
parameters for reconstruction, the latent space would again be non-uniform.
The uniformity comes from the sampling process. Rather than uniformly sampling
from the predicted distribution function directly (which is not
differentiable), a random variable $\epsilon$ is sampled from a uniform
distribution e.g. standard Gaussian. A differentiable transformation using
this random variable can then be defined (Eq. 1 compared to the original
autoencoder defined by Eq. 2). To encourage the latent space distribution to
be close to the prior distribution, Kullback-Leibler divergence is
incorporated in the loss function [26].

$z=\mu+\exp(\log(\sigma^{2})*\epsilon),\epsilon\sim\mathcal{N}(0,1)$ (1)
$z=\sigma(Wx+b)$ (2)

This provides more certainty about the latent space distribution and prevents
the formation of imprecise distributions that leads to inaccurate results
[26]. VAEs are easier to train then GANs and are not susceptible to mode
collapse, but commonly generate images with less quality. There are multiple
causes for this issue [3]. One cause is the use of Mean Squared Error (MSE)
loss, because MSE is minimized by averaging all plausible outputs, which
causes blurring. GANs mitigate this problem by using a discriminator network
that only distinguishes real from fake, which is a more high-level goal [15].
The other cause is that VAEs estimate the data distribution to be a normal
Gaussian, where instead the real distribution could much more intricate.

This issue can be solved in VAEs by discretizing the latent space and applying
the MSE not only on the pixel level but also on the latent level. Because the
latent space is discrete, more fine-grained details are preserved [31]. Vector
Quantized VAEs (VQVAEs) [38] are designed for that task. Input images are
encoded to closely match one of the learnable codebook vectors and the index
of the closest (in terms of euclidean distance i.e. MSE) one is selected as
the discrete representation. Due to continuous relaxation [38], all codebook
vectors are updated per image, thus a single codebook vector can be influenced
by multiple similar images, allowing interpolation on a smooth latent space.
Therefore, VQVAEs achieve an optimal combination of a discrete latent space,
without the assumption of normal distributed data, while maintaining a smooth
latent space. There are also applications of vector quantization in GANs
(VQGAN[7]), improving their performance by taming transformers. Previously
transformers have shown better semantic quality because of the lack of
inductive bias towards local interactions of CNNs, but worse resolution due to
the quadratic complexity.

####  2.1.4 Diffusion

Latest image generation models rely on diffusion processes have shown great
capabilities. Diffusion in machine learning was introduced by Sohl-Dickstein
et al. in 2015 [34] as a generative model inspired by non-equilibrium
statistical physics using Markov chains to gradually convert one distribution
into another. This is done by adding a small amount of Gaussian noise to a
sample at each timestep, resembling a Markov chain. In a Markov chain, each
probability at each timestep only depends on the probability of the previous
timestep and can be represented as a transition matrix [28]. This process of
perturbing the input $x_{0}\sim q(x_{0})$ with noise over $T$ timesteps (Eq.
3) is called the forward process [10]. The Markov chain can be learned and
reversed one step at a time by estimating the transpose transition matrix.

$\displaystyle q(x_{t}\mid x_{t-1})$
$\displaystyle=\mathcal{N}(x_{t};\mu_{t}=\sqrt{1-\beta_{t}}x_{t-1},\Sigma_{t}=\beta_{t}\mathbf{I})$
(3a) $\displaystyle q(x_{1:T}\mid x_{0})$
$\displaystyle=\prod_{t=1}^{T}q(x_{t}\mid x_{t-1})$ (3b) $\displaystyle
q(x_{t}\mid x_{0})$
$\displaystyle=\mathcal{N}(x_{t};\sqrt{\bar{a}_{t}}x_{t-1},(1-\bar{a}_{t})\mathbf{I})$
(3c)
$\displaystyle=\sqrt{\bar{a}_{t}}x_{0}+\epsilon\sqrt{1-\bar{a}_{t}},\epsilon\sim\mathcal{N}(0,\mathbf{I})$
(3d)

The first equation Eq. 3a is the multivariate normal distribution, where
$\mathbf{I}$ is the identity, indicating that each dimension has the same
standard deviation $\beta_{t}$. The forward process is mathematically defined
by the second equation Eq. 3b as a product of all perturbations. To speed up
this process, the product of Gaussians can also be defined as a single one
with $\bar{a}_{t}=\prod_{s=1}^{T}(1-\beta_{t})$, see Eq. 3c.

The reverse distribution $q(x_{t-1}\mid x_{t})$ is unknown and statistical
estimates require computations involving the data distribution. However, a
parameterized distribution $p_{\theta}(x_{t-1}\mid x_{t})$ can be estimated by
a neural network i.e. predicting $\mu_{\theta}(x_{t},t)$ and
$\Sigma_{\theta}(x_{t},t)$. Ho et al. [10] claim that it is intractable to
estimate the distribution without additionally conditioning on the original
sample $x_{0}$, that is $q(x_{t-1}\mid x_{t},x_{0})$. The formula is re-
parameterized to
$\mu_{\theta}(x_{t},t)=\mu_{\theta}(x_{t}(x_{0},\epsilon),t)$. The final
formula for $\mu_{\theta}$ shows that predicting the mean of the distribution
is the same as predicting the noise $\epsilon$ at timestep $t$, thus the model
is defined by $\epsilon_{\theta}(x_{t},t)$. The function $q(x_{t}\mid x_{0})$
for noising a sample is consequently also re-parameterized using $\epsilon$,
see Eq. 3d.

Hence, to train the model, each sample is produced by randomly drawing a data
sample $x_{0}$, a timestep $t$, and noise $\epsilon$, which together give rise
to a noised sample $x_{t}$. The objective becomes the MSE between the true
noise and the predicted noise [10]. Specifically, after noising a sample, a
U-Net is utilized to predict the noise, which can then be subtracted from the
noisy sample. Because the amount of noise depends on the timestep, the model
needs to know the current timestep. This is implemented using position
embedding, introduced by Vaswani et al. [39]. Different variance schedules
$\beta_{1},...,\beta_{T}$ can be used e.g. linear, quadratic or cosine. To
generate new samples, Gaussian noise $x_{T}$ and repeatedly predicting
$x{t-1}$ from $x_{t}$. That is, the noise $\epsilon$ is predicted by the model
and subtracted from the sample to obtain an estimate of $x_{0}$, then some
noise is added back to the sample to get $x_{t-1}$. This is done iteratively
until $t=0$, generating better and better approximations of $x_{0}$. Song et
al. [35] later proposed the Denoising Diffusion Implicit Model (DDIM), which
is a diffusion model variant, that generalizes it to a non-Markovian process,
while maintaining the same training procedure. This makes sampling much
faster, while introducing a parameter $\eta$ to control the determinism of the
model.

Training diffusion models is very computationally expensive because of the
gradient computations on the high-dimensional pixel space of RGB images. And
also during inference diffusion models are quite demanding in terms of compute
resources, due to the sequential reverse process. Consequently, the training
of such models is currently limited to a select minority within the field and
furthermore, leaves a substantial carbon footprint [33]. To address these
issues, Rombach et al. introduce Latent Diffusion [33], where the two
diffusion processes are applied not on the pixel space but on a compressed
latent space. This is achieved by, first, training an autoencoder to provide a
lower-dimensional latent space, and second, training a diffusion model on the
compressed samples. Good compression is not necessary, because the only
objective of the latent space is more efficient diffusion, thus training the
autoencoder is much simpler. In particular, the authors utilize a variation of
a VQGAN [7] as the autoencoder and, but with the quantization layer in the
decoder instead of the encoder. The goal of the VQGAN is to learn a latent
space that is visually equivalent from the image space, while also providing a
notable decrease in computational complexity. The U-Net for the noise
prediction is a time-conditional U-Net, mainly comprised of 2D-convolutional
layers. Sampling is done using the DDIM method. Their method achieve state-of-
the-art results while requiring substantially less computational resources.

Diffusion models have shown to generate images with extremely high quality,
even beating GANs [5]. They are more stable during training and do not suffer
from mode collapse. But sampling new images is quite expensive in terms of
time and compute. Additionally, diffusion models lack a low-dimensional latent
space, whereas standard VAEs typically have access to a low-dimensional latent
space, which allows conditional control of the generation [25] but limits
their overall performance [33].

Summarizing, GANs provide fast sampling and high quality samples, whereas VAEs
on the other hand provide a much higher diversity (mode coverage) as well as
fast sampling speed. Diffusion models sampling speed is much slower, but come
with high image quality and diversity.

###  2.2 Image-Text Embeddings

Aligning images with text is the process of accurately matching textual
descriptions with their corresponding images. This alignment allows for a
coherent and meaningful connection between the visual content of an image and
its textual information. CLIP [29] is the most important work in this area.
CLIP is a large transformer model trained from scratch on 400M image-text
pairs scraped from the internet. The model consists of an image encoder (ViT
[6]) and a text encoder (Transformer [40]). For efficiency, instead of
predicting each exact word, the goal is to predict which text as whole belongs
to an image. For that, the cosine similarity over a minibatch of correct
image-text pairs is maximized during training, while minimizing the cosine
similarity of the incorrect pairs. This is also known as contrastive learning,
where a model learns without predicting labels or text, but instead learns a
representation in which similar samples are close together, while dissimilar
ones are far apart. CLIP provides zero-shot capabilities that are even
competitive to supervised models.

###  2.3 Conditional Image Generation

By incorporating additional information into the image generation process,
such as text, or other images, conditional image generation models enable some
control over the generated output. Pix2Pix [15], proposed in 2016, introduced
a conditional image-to-image translation framework based on GANs, with a U-Net
as the generator and a specific loss function to learn from paired images.
This model demonstrated impressive results in various tasks, such as
generating realistic street scenes from semantic labels, transforming day-time
images into night-time equivalents, or converting sketches into photorealistic
images. CycleGAN [47], introduced in 2017, revolutionized conditional image
generation by enabling the transformation of images from one domain to another
without the need for paired training data. It employed a cycle consistency
loss, which ensured that the transformation from one domain to another and
back again would reconstruct the original image. One of the earliest text
conditioned image generation networks was introduced in 2016 by Reed et al.
[32] by concatenating the embedded text to the input noise of the generator
and to the features of the discriminator of a GAN.

Conditioning mechanisms in diffusion models use various forms of conditions,
such as labels, classifiers, texts, images, semantic maps, and graphs. There
are four main types of conditioning mechanisms: concatenation, gradient-based,
cross-attention, and adaptive layer normalization (adaLN). Concatenation
involves combining conditioning information with the input noise or
intermediate denoised targets during the sampling process. For example, text
embeddings [33] and semantic feature maps [41] can be used as guidance using
this technique. Gradient-based mechanism incorporates gradients from a
classifier into the diffusion sampling process [5]. This allows for
controllable generation, such as guiding the diffusion process towards a
specific class label. Cross-attention [39] is effective for training
attention-based multi-modal models. In diffusion models, it is usually applied
to intermediate layers of the denoising U-Net [33]. Adaptive layer
normalization (adaLN) replaces standard layer norm layers in transformer-based
diffusion models with adaptive layer normalization. Instead of directly
learning scale and shift parameters, adaLN regresses them from the sum of the
time and condition embeddings [27], thus shifting the distribution. A variety
of conditional diffusion models have already been proposed.

####  2.3.1 Conditional Diffusion and Guided Diffusion

Diffusion models try to predict the noise $\epsilon$ at timestep $t$, which is
also equivalent to predicting the score function $\nabla_{x}\log p_{t}(x)$
[36]. This process can be conditioned on an arbitrary class $y$, resulting in
the conditional score function $\nabla_{x}\log p_{t}(x\mid y)$. Traditional
class-conditional diffusion models try to model the conditional distribution
$p(x\mid y)$ during training, but performance can be improved when
reformulating the score function to $\nabla_{x}\log p_{t}(x)+\nabla_{x}\log
p_{t}(y\mid x)$ by using Bayes’ rule. Now it can be seen that $\nabla_{x}\log
p_{t}(y\mid x)$ is, in fact, what a classifier does. Dhariwal et al. [5]
showed that the conditional distribution can be approximated with the
unconditional distribution with its mean shifted by $\Sigma\nabla_{x}\log
p_{t}(y\mid x)$. The formula for the shifted mean is given by

$\hat{\mu}_{\theta}\left(x_{t}\mid y\right)=\mu_{\theta}\left(x_{t}\mid
y\right)+s\cdot\Sigma_{\theta}\left(x_{t}\mid y\right)\nabla_{x_{t}}\log
p_{\phi}\left(y\mid x_{t}\right)$ (4)

, where $\theta$ and $\phi$ are the parameters of two different models, and
$s$ is the guidance scale. Now, instead of training a conditional diffusion
model $p(x\mid y)$, a traditional pre-trained diffusion model can be guided
using the gradients of a classifier trained on noisy images.

####  2.3.2 Classifier-Free Guidance

Training a classifier on extremely noisy images is quite difficult and adds
another level of complexity to the image generation process. Additionally,
during sampling, the use of a classifier on perturbed images can bee
interpreted as adversarial attacks and thus limits the performance. In
classifier-free guidance [11], a conditional diffusion model is utilized,
denoted as $p_{\theta}(x_{t}\mid y)$. However, instead of using a specific
label $y$, a null label $\varnothing$ is introduced with a fixed probability
during training (usually 10% of the time). The resulting model can serve as
either a conditional model or an unconditional model, depending on the
presence or absence of a conditioning signal. During sampling, the two cases
$p_{\theta}(x_{t}|y)$ and $p_{\theta}(x_{t}\mid\varnothing)$ can be used to
guide the denoising process by:

$\hat{p}_{\theta}\left(x_{t}\mid
y\right)=p_{\theta}\left(x_{t}\mid\emptyset\right)+s\cdot\left(p_{\theta}\left(x_{t}\mid
y\right)-p_{\theta}\left(x_{t}\mid\emptyset\right)\right)$ (5)

By employing this method, the model does not depend on an additional
classifier, but rather leverages its own knowledge. Finally, conditioning on
free text using class-conditioned diffusion is difficult, as it is not trivial
to build a classifier for that task.

####  2.3.3 Conditioning on Text

Text is a powerful medium for conveying rich and detailed information. By
utilizing text as a modality for image generation, one can leverage the
descriptive nature of language to convey specific attributes, spatial
relationships, and contextual information that can enhance the fidelity and
accuracy of the generated images. Moreover, text offers a flexible and
accessible means of input for users, as it can be easily generated,
manipulated, and understood.

Text-condition in state-of-the-art models is often done using CLIP. Perhaps,
the biggest breakthrough in text-to-image was made by Ramesh et al. with
unCLIP [30] (also known as DALLE-2), introducing a two-stage model that
generates CLIP image embeddings based on text captions and decodes high-
quality images conditioned on those embeddings using diffusion. Thereby,
enabling language-guided image manipulations in a computationally efficient
manner and pushing the boundaries of text-to-image generation.

Prior to unCLIP, the authors of GLIDE [24] compared two variants of text-
conditioned diffusion: CLIP guidance and classifier-free guidance, where the
latter performed better. For the first method, instead of using gradients from
a classifier to shift the mean at each step in the sampling process, the
gradients of the dot product of the image $x_{t}$ and caption $c$ encodings
w.r.t. the image are used (Eq. 6). Similar to classifier guidance, CLIP has to
be trained on noisy images and the diffusion model itself does not have to be
trained specifically.

$\hat{\mu}_{\theta}\left(x_{t}\mid c\right)=\mu_{\theta}\left(x_{t}\mid
c\right)+s\cdot\Sigma_{\theta}\left(x_{t}\mid
c\right)\nabla_{x_{t}}\left(f\left(x_{t}\right)\cdot g(c)\right)$ (6)

For the second method, the authors utilize a Transformer [39] model to create
text embeddings, which are then used as conditioning vectors for the diffusion
model. Additionally, the feature vectors of the last layer of the Transformer
are concatenated on to each attention layer of the denoising U-Net.

unCLIP employs the same architecture as GLIDE, but adding four extra CLIP
tokens to the text embeddings of GLIDE and adding CLIP embeddings to the
existing timestep embedding. The authors implement classifier-free guidance to
enhance the sampling quality. unCLIP uses a prior network to map the CLIP text
embeddings to the corresponding image embeddings. Although CLIP aims to
minimize text and image embeddings for ranking purposes, its training
objective does not explicitly focus on aligning modalities in the latent
space. Instead, the model is trained to ensure that matching examples, where
the text and image correspond to each other, are closer together in the
embedding space compared to non-matching examples. This ranking objective
allows CLIP to associate relevant text and image pairs effectively, enabling
it to perform tasks such as image classification or zero-shot image generation
based on textual prompts, see Sec. 2.2. However, the specific spatial
alignment of the modalities in the latent space is not the primary goal of
CLIP’s training process (also known as modality gap [22]). The authors of
unCLIP state that it is indeed possible to generate images directly on the
CLIP text embeddings or the text tokens itself (e.g. GLIDE [24]), but with
worse results. unCLIP enables three different kinds of image manipulations.
First, one can interpolate between two different images by interpolating
between the CLIP embeddings. Second, by subtracting CLIP embeddings, one can
remove aspects of an image. Third, one can generate variations of an image by
choosing $\eta>0$ (where $\eta=0$ makes the model deterministic).

Latent Diffusion [33] also incorporates classifier-free guidance, but uses a
domain-specific (e.g. CLIP for text) encoder to obtain $y$ and guides the
sampling using cross-attention in the denoising U-Net.

####  2.3.4 Conditioning on other Modalities

Following the successful development of text-to-image diffusion models,
researchers have explored methods to incorporate additional modalities such as
semantic maps, depth maps, and other images. They can be encoded and used as
guidance or used as an initial image and guided with text. In Stable Diffusion
(e.g. diffusers [14]), when used as an initial image, noise is added to the
input. The strength parameter, in essence, determines the extent of
transformation applied to the reference image, with higher strength values
leading to greater noise addition. However, when conditioning the model on an
initial image, depth map, or other modalities, it becomes necessary to fine-
tune the model accordingly for the specific modality.

While fine-tuning diffusion models can be beneficial for incorporating
specific modalities or addressing certain tasks, there are also some
disadvantages associated with this approach. First, the amount of data in
task-specific domains is often fairly limited, which can lead to overfitting
and less generalization when fine-tuning large models. And second, fine-tuning
large diffusion models requires a substantial amount of computational
resources, thereby restricting the potential for personalization [46].

###  2.4 Scene Graphs

Scene graphs are a special kind of graph used in computer vision and provide a
powerful structural representation of an image. They depict scenes as directed
graphs, with objects as nodes and relationships between objects as edges [16].
These graphs can be predicted from images and utilized for tasks such as image
retrieval or image captioning [21]. Scene graphs provide a structured and
versatile means of describing multiple objects and their intricate
relationships within an image. Moreover, they enable generative models to
reason over both objects and their relations when generating images.

##  3 Related Work

###  3.1 Efficient Fine-Tuning

Fine-tuning large diffusion models comes with several challenges, mainly data
and compute resource limitations. Thus, using a smaller network to influence a
large one can be beneficial. NovelAI [1] introduced a method on how to build
and train such a HyperNetwork for Stable Diffusion. Their method consists of a
small number of linear layers that are connected to the cross-attention module
of the denoising U-Net, altering the key and values from the text guidance.
Throughout the training process, only the HyperNetwork is updated, while the
diffusion model remains unchanged. Due to the compact size of the
HyperNetwork, training is fast and requires minimal compute resources.

Another method, LoRA [12], works by training two smaller matrices
$A\in\mathbb{R}^{r\times k}$ and $B\in\mathbb{R}^{d\times r}$ (rank
decomposition matrices), which dot-product $\delta W=BA$ has the same size as
the cross-attention weights $W_{0}\in\mathbb{R}^{d\times k}$ of the denoising
U-Net. Their size can be controlled by $r<<\min(d,k)$. The resulting matrix
$\delta W$ is then added to the initial weight matrix $W_{0}$. During training
the diffusion model remains frozen and only the smaller matrices $A$ and $B$
are updated.

ControlNet (Feb 2023) is a fine-tuning method that aims to preserve the
original knowledge of the larger model, while preventing overfitting when the
fine-tuning dataset is small. ControlNet can be used as a robust method to
fine-tuning large diffusion models. This is achieved by freezing the original
layers of the denoising U-Net and creating trainable copies of them. Zero-
convolution is applied to the additional condition, which is then added to the
input for the trainable copy. Afterwards, zero-convolution is applied again to
the output of the trainable copy, which is then added to the output of the
frozen layer. Zero-convolution, hereby stands for a standard $1\times 1$
convolution layer, where the weights are initialized with zeros. This turns
the ControlNet into the identity function in the beginning of the training,
thus preserving the functionality of each block.

T2I-Adapter (Mar 2023) [23] proposes a light-weight model consisting of four
feature extraction blocks and three downsample blocks, where a feature
extraction block consists of one convolution and two residual blocks. The size
of the features extracted in each feature extraction block align with the
output size of each layer in the denoising U-Net, allowing them to be added.
The diffusion model is not updated during training, only the adapter is.
T2I-Adapter also allows multiple conditions, by adding the features of
multiple adapters.

GLIGEN (Apr 2023) [20] introduces grounded text-to-image diffusion by
combining text embeddings $c$ with certain grounding embeddings $e$. These
grounding embeddings can be generated from a variety of input modalities such
as depth maps, pose estimations, semantic maps etc. but also from a
combination of text and corresponding bounding boxes. To incorporate the new
conditional information, Gated Self-Attention layers are placed between each
Cross-Attention layer. In Gated Self-Attention, the visual tokens from the
diffusion process and the conditioning tokens are combined and Self-Attention
is applied. Afterwards, only the visual tokens are selected and gated by
multiplying it with a learned parameter (initialized with 0). During training,
all other layers are fixed and only the new gated cross-attention layers are
updated.

DiffFit (May 2023) [43] drastically reduces trainable parameters while fine-
tuning large diffusion models by freezing all parameters except for bias,
Layer-Normalization and class embedding. Additionally a trainable scaling-term
$\gamma$ is inserted (initialized to 1) and multiplied to each encoder block.
This method can also be combined with other fine-tuning models such as
ControlNet, freezing the trainable copy.

A graphical summarization of these techniques can be seen in Fig. 4.

###  3.2 Image Generation from Scene Graphs

Graphs offer a powerful means of representing various types of data, including
citations, social and economic networks, biochemical and knowledge graphs, as
they explicitly describe relationships and dependencies. Thus, several methods
have been developed to learn from such graphs, known as Graph Neural Networks
(GNN) [42].

Using text as conditioning in image generation models has certain drawbacks
when compared to other approaches. Image descriptions may be long and loosely
structured and the meaning depends strongly on syntax and language.
Furthermore, text inherently carries ambiguity, as multiple sentences can
express the same concept [37]. Thus, in certain cases where image descriptions
tend to be complex, scene graphs provide a better approach for conditioning
image generation models. Additionally, text embedded by CLIP, which is the
standard in current diffusion models, has some weaknesses on more systematic
or complex tasks e.g. counting objects or distances between objects [29].
Moreover, CLIP is generally not good at capturing structural information e.g.
CLIP scores are almost the same for ”The dog is on the motorcycle” and ”The
motorcycle is on the dog” for an image corresponding to the first description
[13]. GLIGEN [20] uses bounding boxes to ground the text embeddings for more
precise control of location of objects. However, A significant challenge in
utilizing scene graphs for image generation lies in the complexity of aligning
visual and graph features [44]. Most research on scene graphs relies on the
Visual Genome dataset, which contains human-annotated scene graphs [16].

The first image generation model that utilizes scene graphs, Sg2Im [16], uses
a Graph Convolutional Network (GCN) to obtain object features and scene
layouts, which are a course image representations of the intended scenes. The
scene layouts are then up-scaled and refined to generate realistic images. The
generation process is trained adversarially, optimizing GAN losses. PasteGAN
[21] conditions a GAN on image crops that contain single objects corresponding
to certain nodes in a scene graph. Later, Sortino et al. [37] used a similar
approach to predict scene layouts from scene graphs and condition a VQVAE on
those embeddings, thus avoiding the instability and mode collapse problems of
GANs. Recently, Yang et al. [44] proposed a method to condition a latent
diffusion model on scene graphs, by learning embeddings using masked
contrastive pre-training to align image and graph embeddings. This is done by
masking out certain parts of an image and reconstructing it using scene graph
encodings. This encoder can then be utilized to embed scene graphs and feed
the embeddings into a latent diffusion model using cross-attention. On April
2023, Farshad et al. [8] (LMU CompVis, the latent diffusion group) presented a
method which utilizes CLIP embeddings of nodes in a scene graph to predict
scene layouts using a GCN. These scene layouts are then used as guidance in a
latent diffusion model.

##  4 Method

To condition a latent diffusion model on scene graphs, we make use of two
state-of-the-art conditioning techniques, ControlNet [46] and Gated Self-
Attention [20]. Two approaches that depend on the technique can be defined:
1. predict scene layouts from scene graphs  2. directly predict scene
embeddings from scene graphs.  Scene layouts or scene embeddings can then be
used as conditioning. We use these techniques instead of fine-tuning a
diffusion model like all previous works, because datasets of scene graphs are
quite limited in size e.g. the largest scene graph dataset Visual Genome [19]
only consists of 108,077 data points, i.e. $5\times 10^{4}$ times smaller than
LAION5B.

###  4.1 Graph Convolutional Network

Graph Convolutional Networks (GCNs) [18] are a type of neural network designed
to work with graph-structured data. In contrast to traditional convolutional
neural networks (CNNs) that process grid-like data e.g., images, GCNs can
handle complex relationships present in graphs. The key idea behind GCNs is to
generalize convolutional operations to graphs. In a graph, nodes represent
entities, and edges represent connections between them. GCNs leverage this
structural information to perform convolution-like operations. They aggregate
and combine information from neighboring nodes to compute new node
representations. This aggregation involves taking a weighted sum of the
feature vectors of neighboring nodes and combining them with the node’s own
features. By stacking multiple graph convolutional layers, GCNs can capture
deeper structural information.

We first embed each individual node and edge description using CLIP to obtain
text embeddings $c$ and then apply multiple layers of graph convolution to
compute graph features $f_{g}$. Because the meaning of each node is encoded
separately using CLIP and defined explicitly in the graph, the resulting
features are more robust and not limited by CLIP’s lack of structural
semantics.

###  4.2 Via Scene Layouts

We use the graph encoder explained above to obtain graph features $f_{g}$, by
jointly pretraining a small CNN decoder to predict course scene layouts
$f_{s}$, similar to Johnson et al. [16] but using BERT [4] and CLIP [29]
embeddings. For ControlNet, the dimensions of a scene layout has to be the
same as $z_{T}$ i.e. $64\times 64\times 128$. Thus, we use the decoder only up
until an intermediate layer and fine-tune it together with the graph encoder
while training using ControlNet. In this approach, the text prompt, that is
encoded with CLIP, is set to the default prompt ”an image” during training. A
default prompt is used prevent the CLIP text conditioning to influence the
image generation process. An illustration of both approaches can be seen in
Fig. 1.

Figure 1: Architectures of both approaches that utilize scene layouts as an
intermediate step for image generation.

The combination of GCN and CNN to transform graphs to an image representation
is necessary in this approach, because the denoising U-Net that is fine-tuned
using ControlNet expects image-like representations such as depth maps,
semantic maps, edges or poses. During this transformation, spatial information
like location and shape can be estimated from the object and relation nodes of
the scene graph respectively.

###  4.3 Via Graph Embeddings

The resulting graph features $f_{g}$ from the scene layout pretraining can
also be directly used in Gated Self-Attention, without the need of encoding
scene layouts. Controlling spatial arrangements using with ControlNet is rigid
and therefore the structure of the input image is crucial. Gated Self-
Attention allows for a more loosely control of spatial arrangements e.g. using
bounding boxes, key points or other conditional signals. By creating scene
layouts in the pre-training phase, information about specific words such as
actions can get lost. This can be prevented, by fusing the text embeddings $c$
with the corresponding graph features $f_{g}$ using
$h=MLP(concatenate(c,f_{g}))$ [20]. Here, the CLIP encoder for the graph
embedding can also be replaced by BERT [4], similar to [13]. The default
prompt is used and the graph encoder is trainable during fine-tuning, for
architecture see Fig. 2.

Possibly even better embeddings can be obtained by contrastively learning
similarities between images and scene graphs, similar to CLIP but for graphs
instead of text. Standard CLIP does not include negatives for structural
relationships. Within a batch there are no counter-examples for captions such
as ”the large bus and the green grass” $\rightarrow$ ”the green bus and the
large grass”. To do this a scene graph can be modified by switching attributes
that are connected to different objects.

The concept was initially introduced in Structure-CLIP[13], where scene graphs
were utilized to mine hard negatives. Instead of fine-tuning CLIP on graphs,
we leverage the frozen graph knowledge transformer from Structure-CLIP to
obtain meaningful graph embeddings - see Fig. 2.

Figure 2: Architectures of both approaches that utilize graph embeddings for
image generation.

###  4.4 Structure-CLIP

Since CLIP, as a contrastive learning method, does not provide negative
samples for structural information, it performs poorly on tasks that require
detailed semantics. Structure-CLIP [13] proposed a composition-aware hard
negative mining strategy, which involves manipulating scene graphs in a
meaningful way. That is, a sampling function that swaps nodes in a scene
graph, to alter the caption in a way such that the sentence stays similar but
its semantics change. Scene graphs are used because not all possible swaps of
nodes change the semantics or form a correct sentence. Additionally, the
authors propose a ”knowledge-enhanced structured framework” to further capture
structured knowledge from triples formed from the scene graph. A triple is a
pair of nodes and their relation e.g. (Cows, Sit in, Hay). Each word in a
triple is embedded using BERT [4] and each triple is then fed through a
modified Transformer encoder [39] with

$\displaystyle Q=HW_{Q},K$ $\displaystyle=HW_{K},V=HW_{V},$ $\displaystyle
Attention(H)$ $\displaystyle=softmax\left(\frac{QK^{\top}\odot
M}{\sqrt{d_{K}}}+(1-M)*\delta\right)V,$

where $H$ is the sequence of embedded triples, $\odot$ is element-wise
multiplication, $M_{ij}=0$ and $\delta<<0$, so the output is near $0$ at the
beginning. Relying only on structured knowledge may lead to a loss of the
original semantics, to this end the triple embedding is added to the CLIP text
embedding. Incorporating the triple encoding transformer significantly
improved the performance, compared to just using their negative mining
strategy on its on.

##  5 Experiments

###  5.1 Dataset

For the fine-tuning dataset we choose Visual Genome [19], which is comprised
of 108,077 scene graph and image pairs, with additional annotations such as
bounding boxes and object attributes. For pretraining, we use COCO-Stuff [2],
which contains 45,000 images with corresponding bounding boxes and
segmentation masks, where we synthetically generate scene graphs, following
Johnson et al. [16].

###  5.2 Settings

As a base model we always use Stable Diffusion 2.1
111https://huggingface.co/stabilityai/stable-diffusion-2-1. As a baseline we
create text description from the scene graph and feed them into the base model
as CLIP embeddings. We also fine-tune the base model on Structure-CLIP
embeddings using LoRA [12]. For a second approach, we pretrain the scene
layout model and use it to fine-tune the model using ControlNet. We then use
the graph embeddings directly to fine-tune the model using Gated Self-
Attention. We also compare different text embeddings, namely BERT [4] and CLIP
[29]. Finally, we make use of the graph transformer from Huang et al. [13] to
create the embeddings for the Gated Self-Attention model. We use FID-score for
measuring the image sampling quality. The default prompt for all models except
for the first two is ”an image”. To summarize, these are the experiments we
conduct:

  1. (i)

Text (CLIP) $\rightarrow$ Diffusion

  2. (ii)

Text (Structure-CLIP) $\rightarrow$ Diffusion (LoRA)

  3. (iii)

Scene Layouts (BERT) $\rightarrow$ ControlNet

  4. (iv)

Scene Layouts (CLIP) $\rightarrow$ ControlNet

  5. (v)

Embedding (BERT) + Text $\rightarrow$ Gated Self-Attention

  6. (vi)

Embedding (CLIP) + Text $\rightarrow$ Gated Self-Attention

  7. (vii)

Embedding (Structure-CLIP) + Text $\rightarrow$ Gated Self-Attention

###  5.3 Results

The values of each model are either taken from the paper directly or from Yang
et al. [44]. Each model is evaluated without using the ground truth masks.
Some examples can be seen in Fig. 5 and Fig. 6. The results on COCO Stuff [2]
and Visual Genome [19] can be seen in Fig. 3.

Method | Inception Score $\uparrow$ | FID $\downarrow$
---|---|---
| COCO | VG | COCO | VG
Real Image (256x256) | 30.7 | 27.3 | - | -
Sg2Im [16] | 8.2 | 7.9 | 99.1 | 90.5
PasteGAN [21] | 12.3 | 8.1 | 79.1 | 66.5
SGTransformer [37] | 13.7 | 12.8 | 52.3 | 60.3
SGDiff [44] | 17.8 | 16.4 | 36.2 | 26.0
SceneGenie [8] | 22.2 | 20.3 | 63.3 | 42.2
Ours (iii) 5.3 | 32.1 | - | 279.3 5.3 | -
Ours (v) 5.3 | 32.2 | - | 266.4 5.3 | -
Figure 3: Results of our experiment compared to several other methods.
22footnotetext: Because of time and resource constraints, we were unable to
train or fine-tune any model. Additionally, SGTransformer, SGDiff and
Structure-CLIP are not published yet. Therefore, we use the pre-trained scene
layout network from sg2im to perform two experiments:  1. Scene Layouts
(bounding boxes, sg2im) $\rightarrow$ GLIGEN  2. Scene Layouts (semantic map,
sg2im) $\rightarrow$ ControlNet  with 10 scene graphs from COCO for each
experiment. 33footnotetext: FID requires around 50k images according to its
authors. Since we only provide 10 images, these numbers are probably not
meaningful.

##  6 Discussion

The proposed approaches for fine-tuning diffusion models using scene graphs
have demonstrated promising results and significant advancements in image
generation. By incorporating scene graphs as a more structured and precise
representation of image content, we have achieved improved control while
maintaining high image quality. However, there are several important points to
consider and discuss. Because of time and resource constraints, we were not
able to train any model. Thus, the overall image quality and control can be
significantly improved. Scene layouts, including bounding boxes produced by
sg2im [16] are not optimal and very course, which is the current bottleneck of
our system. The pre-trained models used in our experiments were not fine-tuned
to the input, which results in missing image attributes such as objects or
relations or overall bad image quality.

##  7 Conclusion

In conclusion, diffusion models have emerged as powerful image generation
models, but conditioning on text for image generation has limitations due to
the ambiguity and length of textual descriptions. However, scene graphs offer
a structured and unambiguous representation of image content as they provide a
precise encoding of object relationships and spatial arrangements, making them
a superior choice for image generation when fine-grained control and accurate
synthesis are crucial. Different methods have been proposed for utilizing
scene graphs in image generation, including predicting scene layouts,
conditioning on object crops corresponding to nodes in a scene graph, and
learning embeddings through masked contrastive pre-training. However, images
conditioned on scene graphs are not yet comparable to images conditioned on
other modalities like bounding boxes, poses, or depth maps.

In our work, we have contributed multiple approaches for effectively fine-
tuning diffusion models using scene graphs. We have evaluated these methods to
some extent and provided the code for further research and exploration in this
area. Our contributions aim to enhance the capabilities of diffusion models
and facilitate more precise and controlled image generation. Overall, the
advancements in diffusion models and the utilization of scene graphs offer
exciting prospects for the field of image generation. Further research and
development in this area can lead to even more impressive results and
advancements in generating images with fine-grained control and accuracy.

###### Acknowledgements.

I want to thank Ahi’s Pizza who makes the best pizza in Ulm and whoever
invented matcha.

## References

  * [1] Novelai improvements on stable diffusion — by novelai — medium.  https://blog.novelai.net/novelai-improvements-on-stable-diffusion-e10d38db82ac.  (Accessed on 05/25/2023).
  * [2] S. Anwar, M. Tahir, C. Li, A. Mian, F. S. Khan, and A. W. Muzaffar.  Image colorization: A survey and dataset, 2022.
  * [3] L. Cai, H. Gao, and S. Ji.  Multi-stage variational auto-encoders for coarse-to-fine image generation, 2017.
  * [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [5] P. Dhariwal and A. Nichol.  Diffusion models beat gans on image synthesis, 2021.
  * [6] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * [7] P. Esser, R. Rombach, and B. Ommer.  Taming transformers for high-resolution image synthesis, 2021.
  * [8] A. Farshad, Y. Yeganeh, Y. Chi, C. Shen, B. Ommer, and N. Navab.  Scenegenie: Scene graph guided diffusion models for image synthesis, 2023\.
  * [9] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks, 2014.
  * [10] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models, 2020.
  * [11] J. Ho and T. Salimans.  Classifier-free diffusion guidance, 2022.
  * [12] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen.  Lora: Low-rank adaptation of large language models, 2021.
  * [13] Y. Huang, J. Tang, Z. Chen, R. Zhang, X. Zhang, W. Chen, Z. Zhao, T. Lv, Z. Hu, and W. Zhang.  Structure-clip: Enhance multi-modal language representations with structure knowledge, 2023.
  * [14] Huggingface.  diffusers.  https://github.com/huggingface/diffusers/.  (Accessed on 05/25/2023).
  * [15] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.  Image-to-image translation with conditional adversarial networks, 2018\.
  * [16] J. Johnson, A. Gupta, and L. Fei-Fei.  Image generation from scene graphs, 2018.
  * [17] D. P. Kingma and M. Welling.  Auto-encoding variational bayes, 2022.
  * [18] T. N. Kipf and M. Welling.  Semi-supervised classification with graph convolutional networks, 2017\.
  * [19] L. Li, Z. Gan, Y. Cheng, and J. Liu.  Relation-aware graph attention network for visual question answering, 2019\.
  * [20] Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee.  Gligen: Open-set grounded text-to-image generation, 2023.
  * [21] Y. Li, T. Ma, Y. Bai, N. Duan, S. Wei, and X. Wang.  Pastegan: A semi-parametric method to generate image from scene graph, 2019.
  * [22] W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Zou.  Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, 2022.
  * [23] C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, Y. Shan, and X. Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models, 2023.
  * [24] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
  * [25] K. Pandey, A. Mukherjee, P. Rai, and A. Kumar.  Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents, 2022.
  * [26] K. Panguluri and K. Kamarajugadda.  Image generation using variational autoencoders.  IJITEE (International Journal of Information Technology and Electrical Engineering), 9, 03 2020.
  * [27] W. Peebles and S. Xie.  Scalable diffusion models with transformers, 2023.
  * [28] M. A. Pinsky and S. Karlin.  3 - markov chains: Introduction.  In M. A. Pinsky and S. Karlin, editors, An Introduction to Stochastic Modeling (Fourth Edition), pages 79–163. Academic Press, Boston, fourth edition edition, 2011.
  * [29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * [30] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents, 2022\.
  * [31] A. Razavi, A. van den Oord, and O. Vinyals.  Generating diverse high-fidelity images with vq-vae-2, 2019.
  * [32] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee.  Generative adversarial text to image synthesis, 2016.
  * [33] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models, 2022.
  * [34] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
  * [35] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models, 2022.
  * [36] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole.  Score-based generative modeling through stochastic differential equations, 2021.
  * [37] R. Sortino, S. Palazzo, and C. Spampinato.  Transformer-based image generation from scene graphs, 2023.
  * [38] A. van den Oord, O. Vinyals, and K. Kavukcuoglu.  Neural discrete representation learning, 2018.
  * [39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need, 2017.
  * [40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need, 2017.
  * [41] W. Wang, J. Bao, W. Zhou, D. Chen, D. Chen, L. Yuan, and H. Li.  Semantic image synthesis via diffusion models, 2022.
  * [42] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu.  A comprehensive survey on graph neural networks.  IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24, jan 2021.
  * [43] E. Xie, L. Yao, H. Shi, Z. Liu, D. Zhou, Z. Liu, J. Li, and Z. Li.  Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning, 2023.
  * [44] L. Yang, Z. Huang, Y. Song, S. Hong, G. Li, W. Zhang, B. Cui, B. Ghanem, and M.-H. Yang.  Diffusion-based scene graph to image generation with masked contrastive pre-training, 2022.
  * [45] H. Ying, H. Wang, T. Shao, Y. Yang, and K. Zhou.  Unsupervised image generation with infinite generative adversarial networks, 2021.
  * [46] L. Zhang and M. Agrawala.  Adding conditional control to text-to-image diffusion models, 2023.
  * [47] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.

Figure 4: Architectural overview of the mentioned ”efficient fine-tuning”
(top-row) and ”conditional fine-tuning” (bottom row) techniques. Figure 5:
Examples of different models from the same scene graphs. Figure 6: Examples of
different models from the same scene graphs.